Confusing the Monster-Ometer with the Frog Exaggerator - again

The latest results from the NZ Health Survey are up.

And so is Alcohol Healthwatch's take on those stats. They take it all as reason for tightening control on alcohol.

Go and have a look at the stats for yourself. For each of a pile of indicators, MoH slices up the data by gender, by age, and by ethnicity. It then says whether the difference between the latest stat and last year's stats, or 2014/15's stats, or the 2011/12 stats for the series that go back that far, are statistically significant.

Now one immediate problem is that if you've sliced up the data two dozen ways and you're running comparisons between three pairs of years for each of those slices, you've got a lot of potential comparisons. 24 comparisons per indicator times 3 year-pairs of comparisons, for the indicators where they have data going back over the whole period - which they don't for some because they changed their definition of hazardous drinking along the way. 

When you're slicing the data that way, you need to adjust your statistical tests for the problem of multiple comparisons. Why? Jellybeans. I'm pretty sure that MoH hasn't done that and that they're just running naive tests for each one.*  

So keep that in mind when noting changes that pop up as significant. Some of those will really just be noise. 

While the mean prevalence figures cited in each group are not age-standardised, the p-values are. 

Here is the full list of all statistically significant changes across the indicators. They always compare 2018/19 to 2014/15 and to 2011/12 when those years can be compared; I don't know why they don't run comparisons to the start of the series in 2006/07. The 2006/07 figures are almost always higher than the current ones. For most stuff, there was a big drop between 06/07 and 11/12, then a flattening. 

• Past year drinking. There are 72 potential comparisons. One is down. Twelve are up. And 59 are unchanged. 
• Up from 78.7% last year to 80.3% this year, an increase significant at p=.05. But not significantly different from 2011/12 (lower, but not significantly) and not significantly different from 2014/15 (higher, but not significantly). 
• Down among 55-64 year olds when compared to 2011/12 (but not when compared to 2014/15 or 2017/18)
• Up among 75+ when compared to 2011/2 or 2017/18, but not when compared to 2014/15
• Up among Maori when compared to 2011/12 or 2017/18 but not when compared to 2014/15.
• Up among Maori men when compared to 2017/18, and up among Maori women when compared to 2011/12 (but not the other years for either)
• Up among Pacific when compared to 2017/18, and up among Pacific women in every year comparison
• Up among Asian women when compared to 2017/18, but not when compared to 2014/15 or 2011/12
• Overall this looks like a bit of an increase on last year's stats, but still more than three percentage points down on 2006/07 (to which they don't make comparisons). 
• Hazardous drinkers (AUDIT score 8 or higher, among total population). Here there are only 24 potential comparisons because their data doesn't go back earlier than 2015/16 because of a data redefinition. 
• Across those 24 potential comparisons, there are zero statistically significant changes. 
• Looking at the changes without looking at significance, compared to the start of the period, 8 of the 24 are higher and the rest are lower. Compared to last year, 17 of the 24 are higher and the rest are lower. 
• Hazardous drinkers again, but this time restricted to the set of those who consumed alcohol in the past year. Same drill as last time on the years of data. And, same as last time, zero significant changes out of 24 comparisons.
• Heavy episodic drinking (at least 6 standard drinks), at least monthly, total population. I don't think 3 pints of decent beer is heavy, but I suppose these things are subjective. 24 potential comparisons because of year restrictions. One statistically significant change over last year: a drop among those aged 75+. That's it.
• Same thing again, but restricted to past-year drinkers: identical. A drop among those aged 75+, no other statistically significant changes. 
• Heavy episodic drinking, defined as before, total population, except this time at least weekly. 
• Up among men; Up among those aged 18-24, Up among those aged 15-24, Up among European/other men - all as compared to 2017/18. The remaining 20 comparisons are unchanged. Doesn't look much different for those as compared to 2015/16, but they don't run that comparison. 
• Heavy episodic drinking, at least weekly, among past-year drinkers.
• This time, it's only up among European/other men. None of the remaining 23 comparisons are significant. 

So. We've got 240 comparisons. Of those, 17 show statistically significant increases, 3 show statistically significant decreases, and 220 show no change. I was doing eyeball-counting here, so let me know if you've caught a miscount. 

If they've not adjusted the p-stats for multiple comparisons, we've likely overstated the number of significant results. 

If you want to use a Frog Exaggerator, you can point to some of the significant changes among the 240 potential comparisons and say that they're big. But folks should know you're using a Frog Exaggerator rather than a Monster-Ometer.

 

And while the old and new indicators on hazardous drinking aren't comparable, we might expect that the direction of change in the measures would be comparable. There were big drops in hazardous drinking among youths 15-17 from 2006/7 to 2011/12 on that earlier measure, then a flattish trend after that - and similarly for those aged 18-24. Over the longer period, drops among youths are washed out by increases in older cohorts. Hazardous drinkers, among those 15-17, dropped by over 40% from 2006/7 to 2015/16, and dropped by about a quarter among those aged 18-24. Any recent flatlining should be read in context of prior unmentioned drops in youth hazardous drinking. 

Similarly, consumption of 6+ drinks at least weekly on the old measure showed that 25.6% of 18-24 year olds were in that category in 2006/07, with prevalence dropping to 20.6% in 2011/12 and then bouncing around to land at 15.1% in 2015/16 - a drop of over ten percentage points over the interval. By the new measure, prevalence in 2015/16 was 20.4%, dropped to under 17%, then came back up to 21.1% in 2018/19. So that's an increase on last year, but it will still be a decline on 06/07.

* Update: MoH confirms that they're reporting the unadjusted stats. That's all fine - there are plenty of folks who'd need the naive t-stats. But it does mean that if you're looking across the set of them, you need to remember that you probably need a tighter threshold for the p-values. 

** For those who don't know that excellent Simpsons episode: Professor Frink thought he'd found the Loch Ness monster. The machine was going nuts with beeping. But it was just a frog. Why? He was using the Frog Exaggerator rather than the Monster-Ometer. One might ask why he bothered bringing a Frog Exaggerator out on that scientific expedition in the first place. But good on Frink for letting everyone know he'd accidentally pulled out the Frog Exaggerator. Not everyone does that.