You need a common base to measure changes

Suppose I developed some new measure of how terrible journalistic reporting on health is. And by my new measure, fully 10% of newspaper articles on Stuff demonstrate a combination of complete innumeracy and absolute credulity. Suppose I then compared it to some prior, entirely different, measure of press innumeracy which listed only 6% of articles as being really bad in 2006.

Suppose I then sent the Sunday Star Times a press release about it and claimed that my new measure showed just how much worse things had gotten in the last few years. Think they'd print it?

Via ed.co.nz, Seamus, and Thomas Lumley, here's a Fairfax piece* the shocking rise in alcoholism:

One in 10 New Zealanders could now be considered "alcoholic" according to new diagnostic criteria - but the majority of those with a drinking problem are unlikely to recognise it because the issue is so common.

The new estimate of 400,000 "alcoholics" in New Zealand - around 10 per cent of our 4.4 million population - was tallied up by Professor Doug Sellman from the National Addiction Centre at the University of Otago.

It is significantly higher than the Ministry of Health's 2006 estimate which says 3 to 6 per cent of the population has an alcohol issue.

Sellman's figures are based on the new diagnostic criteria for "alcohol use disorder" recently published in the fifth edition of the Diagnostic and Statistical Manual (DSM) of the American Psychiatric Association.

First off, alcohol use disorder isn't alcoholism. Lumley hit this one over the weekend, pointing to the statement from the President of the American Society for Addiction Medicine on the DSM-V:

DSM-5 has “Alcohol Use Disorder,” which comes in mild, moderate and severe flavors, suggesting the inadequate pyramid approach. There are 11 possible symptoms of the “use disorder,” of which two are necessary to achieve a mild specifier, four for moderate and six for severe. “Alcohol use disorder is defined by a cluster of behavioral and physical symptoms,” the authors of DSM-5 state. I have no problem with that except that some may confuse “alcohol use disorder” with addictive disease or with alcoholism

The article can serve as example.

But the bigger problem is that you just cannot cannot cannot identify changes in a measure if the base measurement has changed unless you do a lot of work to put the two measures on common footing. Suppose we changed how we measured inflation. We do that from time to time. When that happens, economists and statisticians have to do a lot of work building a linked index where you recalculate the old inflation observations on the new measurement so that you can have meaningful comparisons over time.

Suppose that the government put out a press release saying "Hey, by our new measure, wait times for surgery at hospital are half as long as they were under the old Labour government!". Surely surely the reporter would think to ask whether the change were due to the redefinition of the measure or whether it were due to anything real.

Anyway, by the new measure, as reported in the SST article, I count as having mild alcohol use disorder. I answered two of their 11 questions in the affirmative. Because I can spend a lot of time browsing at The Beer Cellar or over at Whisky Galore, I say yes to #3: "Spending a lot of time getting, using, or recovering from alcohol." It's all because of the "getting" part, but I'm being honest. It's an "or" question. I also say yes to #4, "Cravings and urges for alcohol". Reading articles like this give me strong cravings for strong drink. I answer "no" to all of the others, but that doesn't matter.

So, how did Sellman reckon 400,000? Surely it was something more than the guesstimate here presented.

"We know there are about 800,000 heavy drinkers in NZ [based on Ministry of Health figures] and it could very well be higher," Sellman said.

"A majority of heavy drinkers already are likely to meet one of them [the 11 disagnostic criteria], the acquired tolerance criterion, so that means they only need one more criterion to get there.

"So I'm suggesting that perhaps about a half of heavy drinkers are likely to have at least one more of the diagnostic criteria such as a recurrent problem associated with heavy drinking, which accounts for the 400,000 figure."

I hope it was the journalist inferring a trend based on this stuff rather than Sellman suggesting one. Yikes.

* I think this was initially in the Sunday Star Times, but it was also in The Press.