Black Hole Research

Last year, Steven Stillman showed that regression discontinuity design doesn't work very well in identifying the effect of changes in the drinking age on other outcomes.

Suppose you wanted to know whether changing the legal drinking age, or alcohol purchase age, would affect outcomes among the cohort subject to the change. A lot of folks will reach for a Regression Discontinuity Design to compare outcomes for those who've just reached the minimum drinking age with those just under it (who are otherwise pretty similar other than a small age difference), and then claim that increasing the drinking age would reduce harms by the magnitude of that discontinuity extended over the interval, or that reducing the drinking age would similarly increase harms.

The problem with that approach is that you're identifying not just on reaching the legal age, but also on having a big birthday milestone. And you could also be picking up effects of inexperienced drinkers. Pretty hard to tell whether the RDD is really catching a drinking age effect or a mix of birthday and experience effects. Hard, that is, unless you actually have a country that changed the purchase age. Stillman and Boes showed that despite an effect showing up in RDD with the New Zealand change from 20 to 18, that effect doesn't extend to a longer term difference-in-difference analysis that tracks actual changes for the affected cohort.

I'd summarised last year:

They also make an important point on method, and I thank Steven for having explained this one to me slowly; hopefully I've understood it properly. Now recall that they found no effect of the law change on traffic accidents: changing the alcohol purchase age from 20 to 18 did not increase the accident rate among youths. Nevertheless, results from a regression discontinuity design comparing accident rates among kids just under the alcohol purchase age with those just over the alcohol purchase age shows an increase in accidents on reaching the alcohol purchase age after the law change. Why could there be an effect in RDD but not overall? Either the RDD is picking up the effect of the alcohol purchase age on the cohort of inexperienced drinkers who change their behaviour on reaching 18 and experience worsened outcomes, or it's picking up a particular changed behaviour around the time of the birthday. We would have overestimated the effects on traffic accidents for 18-19 year olds by extrapolating from the discontinuity around the 18th birthday.

But, we only know this because New Zealand actually had the policy change so we can compare actual outcomes with those estimated by the RDD. Now, suppose you're in a country that hasn't reduced its drinking age and you want to estimate what the effect of decreasing the drinking age might be. If you ran an RDD looking at the effect of reaching the age of majority on traffic accidents, took the break around the birthday as being the effect of being able to drink, then extrapolated that effect back across the cohort of younger drinkers who would be eligible to drink under a different minimum legal drinking age, you could pretty easily be overestimating the likely effect of a real change in the alcohol purchase age. The RDD picks up the effect of reaching the legal drinking age rather than the effect that would obtain by lowering the legal drinking age.

So - we should be wary of RDD estimates of the effects of the alcohol purchase age.

Well, unless you have preferences over outcomes and want it to look like there's a big effect. In that case, RDD's great.

Toumbourou, Kypri, Jones and Hicki survey the literature other than Stillman's work and conclude that the drinking age should jump. Lindo and Siminski reply in a letter to the Medical Journal of Australia:

Toumbourou and colleagues argue that the Australian legal age for buying alcohol should be increased. However, they overstate their case by only citing research that supports their position, giving an impression of scientific consensus on several key issues when there is strong contrary evidence.

They note the Boes and Stillman work ignored by Toumbourou et al.

Toumbourou, Kypri, Jones and Hicki's reply? That the Boes and Stillman paper can be ignored because it's not yet published.

Second case? A piece in the American Journal of Preventative Medicine, again using RDD to claim that Quebec should hike its drinking age, with zero reference to the Boes and Stillman critique.

If only econ journals published as quickly as do some other fields' journals, or, better, that some other fields took the time to get expert referees in to provide comment on technique. Hopefully this will change when the Boes and Stillman work finally comes out. I'm not all that optimistic, but hopeful.