A good natural follow-up question to the first post is what concrete examples do you have in mind? The usual disclaimers apply, pretty much all covered by this nice paper that Wei Yang Tham pointed to, but: this doesn’t mean that research to-date hasn’t been valuable (it has!), doesn’t mean we should stop doing theory, or stop doing ‘basic’ research without a specific problem in mind, or stop doing work estimating causal quantities. Just that the pendulum seems to me too far in one direction right now in econ (and consequently in science of science), and I think it would be productive for the field and the world if the pendulum swung a bit more toward implementable solutions.
I’ll approach this from two angles: first, by way of pointing to other instances outside of the field that have done this well from time to time, and second, by diving into some sample problems from science of science and breaking down more specifically what I think should be done.
Market Design / Auction Design
I’ll start with a close cousin of economics, which is market design and auction design. Here we’re thinking in the tradition of Roth, Milgrom, etc. The moment you read one of these papers, it’s apparent they’re different in kind from most other econ. In particular, they are proposing solutions to existing problems, like kidney exchange, hospital residency matching, and FCC spectrum auctions. Crucially, you see this work getting picked up rather directly in practice, with economists like Al Roth and John McMillan directly employed in designing these things.
Modern work carries on this tradition – here’s a beautiful paper from 2015 by Eric Budish examining the game theoretic flaws in continuous time order book markets, and critically for our purposes, proposing and testing a specific solution in the form of discrete time frequent batched auctions. Now, at least as far as 2017 people were still calling Eric a communist for all this, but eventually someone at NASDAQ or another exchange will go ahead and realize he and his coauthors are right and they’ll implement it, or something very like it (if they haven’t already).
There are a few items of note here. First, I’m sure there’s some, but the absence of anyone caring about estimating a causal quantity is notable to me. Second, notice how in all the major cases, we get a readily implementable solution to real world problems. They’re not without flaws, it’s not like theory went out the window or anything, and for the bigger cases you still really want the actual PhDs designing the systems in practice, but they really went out and just did things.
Operations Research
Let’s step one branch further on the family tree, to my dear friends in OR. A relevant historical note is that OR was born out of WWII, and was originally all about solving practical problems with military applications. From there we go onto industrial applications, but I think it’s not a coincidence that from the start this field was solidly aimed at creating things for solving specific, real-world problems.
So basically, anywhere you look in OR we have useful stuff. Say I’m opening an ice cream store, and I want to know how much ice cream to buy from my supplier – hey that’s a Newsvendor problem, we have algorithms for that. We can treat it simply, with an extremely pared down model you could solve with a closed-form solution in your sleep, or we can relax assumptions and make it NP-hard with things like stochastic demand and lead-times. Let’s say my ice cream shop got bigger, and now I’ve got multiple registers and lines are all out of control, how should I set these up to make sure my customers can ice cream faster? Hey now, we’ve got some queuing theory for you! Now I’m really in business, I’ve got an ice cream franchise on my hands, with 5 locations! I can move ice cream from store to store, but of course that costs me – how should I distribute ice cream from the manufacturer among my stores? That’s an inventory placement problem. How should my stores be distributed around the city? Network design. How can I load up my trucks most efficiently? Bin packing.
Now I’m not saying OR has everything solved. Far from it. But imagine you were at the very start, and you just said “How do I optimize my ice cream store?” that seems like a huge impossible problem. But OR folks didn’t approach it by saying, we need microdata from every ice cream store in the U.S., and then we’ll be able to tell you in 5 years if the variable “number of registers” has a positive causal relationship with profitability – they modeled the problem, agreed where the thing was separable, and then went about inventing new techniques for solving progressively harder versions of that problem optimizing for a profit function. And no, not every ice cream store today uses this stuff, but it is widely available, and you absolutely can hire some OR folks to help you optimize your logistics.
Machine Learning
Probably everyone is tired of hearing about this but briefly indulge me. Much to the chagrin of statistics departments everywhere, ML has taken off despite being statistics with a jaunty hat on top. I’m not going to pick out examples because we all already know this stuff – it’s probably so widely applicable as to be a GPT. For our purposes though, I’m going to pick out two particular aspects of the ML adoption ramp-up which I think are salient for us to consider. First, competition on benchmark datasets. Imagenet is the canonical example at this point, but we’ve got plenty of others. You see new papers come out these days, everyone is saying they’re state of the art against other solutions. So, barring things like informational frictions, if I’ve got a computer vision problem, all I really need to do is find an implementation of something from NeurIPS in the past 3 years and bam, I’ve got a working algorithm that does something useful. And this leads to the other salient point – the tooling got really good and really accessible. Stata this is not. There’s a reason almost all of this stuff is in Python, and it isn’t because it’s the fastest language. Now on this last front to be fair to econ I know there are some people doing valiant work, but we should keep this sort of simplicity of deployment (import new_paper_neural_net, model = new_paper_neural_net(x,y)) in mind as a desirable target, not a niche nice-to-have.
Moneyball (Sabermetrics)
I’ll be honest in saying I don’t know very much about this stuff, but suffice to say that I find it embarrassing for scientists, writ large, to have been beaten to the punch on optimizing their work by sports institutions so legendarily resistant to change that Brad Pitt got to play the hero who saves the day (sort of) with data. To pick two examples I do know, the shift in basketball shooting to favor 3-point shots (and subsequent predictable equilibrium with recently increasing midrange shots, as those get easier once defenses prioritize the perimeter), and pitch framing for catchers in baseball, which apparently is now completely ubiquitous. And this is an interesting case because I don’t think this kind of research even produced, let’s say, software tools that are in any way comparable to like an LP or a neural net. But you still have catchers adopting new practices based on data en masse, and I think it’s largely because the research was aimed at saying “Hey, this action is superior to doing not_action” rather than establishing something to like the high bar of causality, and because there’s a pretty nice apparatus in place for communicating these results out more widely. So anyway. I’m sure other people who know this better could make a more thorough and convincing case, but this likewise seems relevant for us, and I have to think there are lessons to be drawn from how critical the Sloan Conference is in all this.
Conference Design, Grant Funding Design, and Lab Management
So now let’s turn to the topics at hand, and see if we can sketch out some fruitful directions for ‘economic engineering’ research in the science of science context, using lessons from the above.
First, let’s consider conference design. The very first useful thing to be done here I think is for all the theoreticians to get together and decide what we want to optimize for, normatively. Could be a cascade optimization, who knows: information diffusion -> peer feedback -> network formation -> good vibes. Then I think it would be awfully useful if some of the modelers would tell us where the most important, separable subproblems in conference design are. Just off the top of my head, I’ll call out how to assign/conduct reviews of submitted works.
This bit is going to pop out as the framing and solution to an optimization problem, so one could imagine we end up with a benchmark dataset (e.g., Imagenet) or canonical model of the problem (e.g., Newsvendor), and then a line of research proposing new algorithms for solving the same (e.g., Computer Vision research), and then folks packaging them up very tightly in Python for use by all the thousands of people who run conferences.
Other bits pop up naturally – what cadence should you have a conference? How big should a conference be? How do you order the sessions? How do you conduct the feedback? Are discussants actually a useful feature? Should we interlace connectivity events (like mixers) with feedback sessions?
Next let’s talk about grant funding design. I think there’s already some interesting writing on this, and I commend Pierre Azoulay for banging his head against the wall with NIH/NSF, but the mode I’d advise against is trying to tackle this either with some natural experiments or with A/B tests. Why? Well at root this can be reframed as a dynamic control problem, with interesting characteristics like the arrival of grants, time cost of writing them, the uncertain distribution of quality, and so on. But let’s say Pierre broke through to the NIH/NSF and they ran an A/B test… would it be enough? How many A/B tests do they have to run until we’re happy about some estimation of a causal quantity? I think instead we need to jump directly to the ‘control’ aspect of the problem, and propose ways to directly optimize the grant funding policy (with, e.g., an LP, or MIP, or Bandits, or RL, take your pick). And then subsequently package that up in Python and filter it out to the many, many granting organizations out there who will never have the power necessary to run statistically valid RCTs. This is one good example I had in mind when saying we should be doing fewer A/B tests and more online optimization.
Finally, consider lab management, which somehow appears basically unstudied. This is where the Moneyball comparison comes in most nicely. A lot of what’s going on here, at the manager level and at the individual level, is the basically arbitrary set up and execution of workflows and behaviors. Not unlike watching a baseball team practice in the early 20th century. It’s all pretty haphazard. For this, I don’t think you particularly need to invent some nice optimizers or Python software, but nor do I think you want to be spending time estimating causal relationships. Instead, I think you want to be doing like the folks at MIT have done, look at the dominant behaviors right now, and propose alternate behaviors. Maybe this is something like Pentland proposing coffee breaks. Maybe it’s something like Agile for researchers. Maybe you realize we need tracking data like stadiums have before we can do this kind of research – fine, let’s go write a grant about it! But fundamentally what are we doing here when scientists are the last profession to realize that just winging it is not going to maximize productivity? And it’s too early to tell, but I’m optimistic about ICSSI or something like it functioning like the Sloan conference, fruitfully bringing together academics and practitioners so there’s a tighter pipeline from research to implementation.
Anyway that’s a lot to think about for now! I’ll make efforts to get even more specific and hone in on one of these problems to start describing the math in more detail, but we’ll save that for a later date…
macroinvention 