Minimum prices and mortality risk

The latest Stockwell piece on alcohol minimum pricing and alcohol-related fatalities seemed a bit fishy.

I was mostly worried about how they ran a panel study that had zero cross-sectional variation in their main regressor of interest and where the main source of time series variation was CPI adjustments to measured prices, but it looked like there were plenty of other holes for truck-driving expeditions.

George Mason statistician Rebecca Goldin goes for a scenic tour, noting another rather serious problem. Standard t-statistics don't really do the job if you're pouring over multiple lags to look for effects. She writes [ht Forbes]:

What’s troubling here is that they break the data down into many quarters and categories, run multiple statistical tests, but don’t adjust for multiple testing. This results in a table spotted with statistically significant results even as basic statistics tells us this method will produce spurious results.

A close look at the table is suggestive that spurious results are indeed at hand. This table looks at 16 quarters following a minimal price increase, and whether there is a correlated increase or decrease in deaths among acute, chronic or wholly attributable alcohol deaths. The authors point to a statistically significant decrease in wholly attributable deaths in the quarter that a price increase was implemented, as well as in the second and third subsequent quarters. (but not in the first quarter, nor the 4th-15th quarters).

But it also shows a significant increase in acute deaths in the third and fifth quarters after a price increase, and then a statistically significant decrease in acute deaths in the 8th quarter after the price increase. Chronic deaths saw statistically significant decreases in the 8th, 9th and 13th quarter after a price increase, but not in the other 13 quarters. This all suggests that the results could be, at least in part, the result of simply running a lot of tests on a lot of data – and without adjusting for multiple tests, randomness can creep in. Though the results do lean toward decreased death, picking out the most extreme of the results (as the media and the authors of this study did) may be misleading. In fact, if the 1,388 wholly alcohol attributable deaths occurred evenly over the quarters, these numbers refer to trends in about 87 deaths each quarter – trends that would be highly sensitive to a small number of deaths.

So while there seems to be an overall trend of decreased death with increased prices, the failure to account for multiple testing means there could be true correlation or there could be just a statistical fluke.

Eventually, somebody's going to take a proper shotgun to the stuff Stockwell's been up to. When Chris Auld was doing their econometrics, I didn't worry about their empirical results. Things seem to have gone a bit adrift since then.