I mean not really, but…
Here’s what I worry about. Pierre Azoulay is one of the smartest people I’ve ever met, and he can’t convince the NIH to do RCTs so we can discover better ways of allocating research funding. But more than that, let’s say Pierre was successful, the NIH saw the light, and decided to run a new RCT with its funding model every single year — would that actually maximize research output more than if we just gave the NIH some funding algorithms directly?
For example, one year you might play around with reviewer blinds. The next year maybe you swap around reviewers according to homophily with each other. Then homophily with the PIs. Then research similarity with the PIs. Then number of reviewers. Then the scoring system. Then the scoring system again. Etc., etc. Each time, let’s say for the sake of argument we get back results of the RCT within the year, get a nice causal estimate of X and Y, and then update how we do things next year.
And my worry is, isn’t that way, way, way too slow? Why wouldn’t we instead spend all that research effort just coming up with funding control algorithms, handing them off to NIH managers, and letting competition sort out the best algorithm in practice? Shouldn’t we just hand algorithmic control of the relevant action levers to an optimization agent, and not worry so much about perfect identification strategies to isolate the causal effect of X on Y, when we have maybe hundreds of Xs to sort through? Isn’t every single funding decision made under the previous policy by definition worse than the state of the art policy? Why would we wait a year for a robust causal estimate instead of updating our funding policy the second we get back enough data to shift the policy decision?
Anyway these are the things that keep me up at night. If anyone knows some proper papers that have done all this formally (i.e. under what conditions does it make sense to expend limited research effort on precise causal estimates instead of on direct control/optimization algorithms), do let me know.
macroinvention 