Susan Athey on biggish data and machine learning

 Susan Athey is interviewd in JAMA:

How an Economist’s Application of Machine Learning to Target Nudges Applies to Precision Medicine   by Roy Perlis and Virginia Hunt  JAMA. Published online May 16, 2025. doi:10.1001/jama.2025.4497 

"A recent study by economist Susan Athey, PhD, and her colleagues may shed light on how best to target treatments using machine learning. The investigation, published in the Journal of Econometrics, focused on the effectiveness of text and email reminders, or nudges, sent to students about renewing their federal financial aid. The researchers compared causal targeting, which was based on estimates of which treatments would produce the highest effects, and predictive targeting, which was based on both low and high predicted probability of financial aid renewal.

"In the end, the study found hybrid models of the 2 methods proved most effective. However, the result that may be most surprising to Athey was that targeting students at the highest risk of nonrenewal was actually less effective.

...

"Dr Athey:When I first started working on this, I was like, “Oh, there’s going to be a gold mine. I’m going to go back and reanalyze all of these experiments that have already been run, and we’re going to be doing new scientific discoveries every day.” It didn’t quite work out that way. We had some big successes, but there has been a lot of lack of success.

What are the cases where this doesn’t work? Machine learning is using the data to learn about these treatment effects. You have to do a lot of sample splitting. There’s always a cost to using the data to discover the model. You can do it without sample splitting, but then you have to adjust your P values. There’s no free lunch. If you have a very small dataset, you probably know what the most important factors are. You might be better off prespecifying those and just doing your subgroup analysis. If [there are] hundreds of observations, it’s just unlikely. These techniques are too data hungry to work.

Generally, you need thousands of people in the experiment. Then more than that, the statistical power needed to get treatment effect heterogeneity is large. And even treatment effect heterogeneity is easier—trying to get differential targeting is another thing. Imagine you have 3 drugs. It’s hard enough to say that something works relative to nothing. If you’re trying to say that one drug works better than another drug where both work, that’s hard. Usually you need really large, expensive trials to do that.

Then you add on top of that that I want to say, “This drug is better for these people, and this other drug is better for these other people.” You need 10 times as much data as you would for the basic “is there a treatment effect at all?” Now, of course, sometimes there’s a genetic thing: this drug literally doesn’t work or it has this terrible side effect for some people. That will pop out of the data.

For more subtle effects, you do need larger studies. That’s really been the main impediment. And as an economist, it’s like, why are all these things just barely powered? Why are there so many clinical studies with a t-statistic of 2? Of course, people did the power calculations, and they had some data already when they planned the experiments. If you have more data, maybe you add another treatment arm or something else. You don’t actually overpower an experiment. In my own research, I’ve ended up running my own experiments that are designed to get heterogeneity. I’ve also had a lot of luck when there’s very big administrative datasets, and there’s a really good natural experiment. Then you have lots of data. But former clinical trials are selected to not be good because the researcher themself didn’t overpower their own experiment. That’s why this isn’t so useful.

But nonetheless, that’s not to say it’s not out there. Like in any discovery, if it’s going to save lives and money, it’s worth doing. It’s just that there’s not a whole bunch of low-hanging fruit. There’s no dollars lying on the sidewalk."