Tuesday 29 October 2013

You need a common base to measure changes

Suppose I developed some new measure of how terrible journalistic reporting on health is. And by my new measure, fully 10% of newspaper articles on Stuff demonstrate a combination of complete innumeracy and absolute credulity. Suppose I then compared it to some prior, entirely different, measure of press innumeracy which listed only 6% of articles as being really bad in 2006.

Suppose I then sent the Sunday Star Times a press release about it and claimed that my new measure showed just how much worse things had gotten in the last few years. Think they'd print it?

Via ed.co.nz, Seamus, and Thomas Lumley, here's a Fairfax piece* the shocking rise in alcoholism:
One in 10 New Zealanders could now be considered "alcoholic" according to new diagnostic criteria - but the majority of those with a drinking problem are unlikely to recognise it because the issue is so common.

The new estimate of 400,000 "alcoholics" in New Zealand - around 10 per cent of our 4.4 million population - was tallied up by Professor Doug Sellman from the National Addiction Centre at the University of Otago.

It is significantly higher than the Ministry of Health's 2006 estimate which says 3 to 6 per cent of the population has an alcohol issue.

Sellman's figures are based on the new diagnostic criteria for "alcohol use disorder" recently published in the fifth edition of the Diagnostic and Statistical Manual (DSM) of the American Psychiatric Association.
First off, alcohol use disorder isn't alcoholism. Lumley hit this one over the weekend, pointing to the statement from the President of the American Society for Addiction Medicine on the DSM-V:
DSM-5 has “Alcohol Use Disorder,” which comes in mild, moderate and severe flavors, suggesting the inadequate pyramid approach. There are 11 possible symptoms of the “use disorder,” of which two are necessary to achieve a mild specifier, four for moderate and six for severe. “Alcohol use disorder is defined by a cluster of behavioral and physical symptoms,” the authors of DSM-5 state. I have no problem with that except that some may confuse “alcohol use disorder” with addictive disease or with alcoholism
The article can serve as example.

But the bigger problem is that you just cannot cannot cannot identify changes in a measure if the base measurement has changed unless you do a lot of work to put the two measures on common footing. Suppose we changed how we measured inflation. We do that from time to time. When that happens, economists and statisticians have to do a lot of work building a linked index where you recalculate the old inflation observations on the new measurement so that you can have meaningful comparisons over time.

Suppose that the government put out a press release saying "Hey, by our new measure, wait times for surgery at hospital are half as long as they were under the old Labour government!". Surely surely the reporter would think to ask whether the change were due to the redefinition of the measure or whether it were due to anything real.

Anyway, by the new measure, as reported in the SST article, I count as having mild alcohol use disorder. I answered two of their 11 questions in the affirmative. Because I can spend a lot of time browsing at The Beer Cellar or over at Whisky Galore, I say yes to #3: "Spending a lot of time getting, using, or recovering from alcohol." It's all because of the "getting" part, but I'm being honest. It's an "or" question. I also say yes to #4, "Cravings and urges for alcohol". Reading articles like this give me strong cravings for strong drink. I answer "no" to all of the others, but that doesn't matter.

So, how did Sellman reckon 400,000? Surely it was something more than the guesstimate here presented.
"We know there are about 800,000 heavy drinkers in NZ [based on Ministry of Health figures] and it could very well be higher," Sellman said.
"A majority of heavy drinkers already are likely to meet one of them [the 11 disagnostic criteria], the acquired tolerance criterion, so that means they only need one more criterion to get there.
"So I'm suggesting that perhaps about a half of heavy drinkers are likely to have at least one more of the diagnostic criteria such as a recurrent problem associated with heavy drinking, which accounts for the 400,000 figure."
I hope it was the journalist inferring a trend based on this stuff rather than Sellman suggesting one. Yikes.

* I think this was initially in the Sunday Star Times, but it was also in The Press.

10 comments:

  1. Many years ago, a study came out in Canada documenting the extent of sexual harassment in the workplace. The authors were interviewed on a TV news show I was watching where they started out by explaining how there had never before been a study as good as this one; their questions were more comprehensive, they spoke to more people, and they spoke to types of workers who had never been included in previous studies. They then went on to their conclusion which was that sexual harassment was on the increase in Canada. I waited for the interviewing journalist to ask how they were able to estimate a time-derivative from a one-of-a-kind study, but sadly he never did!

    ReplyDelete
  2. When I was practising as an alcoholic and met Doug Sellman you had to get 7 points out of nine on the DSM4. Not too long ago.
    Six out of ten and they sent you home no worries. Then they closed down Mahu clinic and Doug went into a frenzy.
    Yes looking at beer is a very bad sign Eric, I don't hold out much hope for you

    ReplyDelete
  3. A man scores 10 out of 11 on the DSM-V diagnostic test for alcoholism.
    He has to go to see a psychiatrist, but he is dithering at the door of the quack’s office.
    “Come in, come in” says the psychiatrist, “I am not going to hurt you”

    ' Oh yes you are the man says , you are going to lock me up in ward lulu land ’

    “What makes you think that” says the quack; and the man says
    ‘ well you are broadcasting and anyway I can read your mind’

    Jeepers thinks the Psychiatrist I am going to lock him up, and he can read my mind;
    may be I am broadcasting and God how long have I been like this ..

    ReplyDelete
  4. Seamus, anything a Canadian can do we can do better.
    Crazy statistics. In the early 1980’s the NZ Woman’s Weekly ran an article about sexual molestation and ran a survey based on voluntary responses. Guess what it turned out that 1 man in every four was a sexual predator.
    The idea caught on and soon became hysteria.
    They raised money to combat this evil amongst men, and the NZ Mental Health
    Foundation then led by J… W… joined in the belief. We didn’t have Eric Crampton then to point out the stupidity of a magazine survey.
    It became de rigueur to paint slogans on wall ‘all men are rapists’ ‘castrate men’
    and so on. We took to wearing crotch guards it was very dangerous out there, and all because of a survey designed to support belief systems. And it ended with the conviction of Peter Ellis.

    ReplyDelete
  5. It's easy, actually. The statistic showing things are worse is always the right one.

    ReplyDelete
  6. So bottom line, books are cheaper from overseas. Makes me wonder where the difference is. Wholesale cost to NZ is more than to overseas? Bulk delivery cost to NZ more expensive than individually posted items from overseas? NZ website/warehousing more expensive than overseas website/warehousing? NZ retailers profiteering? Other? I would be interested to know. But as you say they should recognise the inefficiency, shut themselves down, use the capital elsewhere and save the consumer some money. As for GST, Customs could setup a website to allow consumers to provide evidence of cost and pre-pay GST. Any item that hasn't had GST paid on it gets sold off or binned by Customs. Too harsh?

    ReplyDelete
  7. I have a hard time seeing how Customs could implement your proposal. Maybe it would work, but I'd bet against it at any odds better than 10% in favour of its working out. The only way I could see it working is if the shipper stuck some ID tag on the shipment that would let it be tied to the customer's GST payment, and international shippers wouldn't be arsed to do it.


    I expect that books are expensive here mostly because of fixed costs in small countries. Book Depository can spread the fixed costs of warehousing and cataloguing over millions of customers around the world. Watch their realtime purchase stats sometime. Here, the per-book part of the warehousing and distribution costs are going to be really high before you even get to the retailer.

    ReplyDelete
  8. A common baseline of socially, unarguable, relevant data over time is very difficult to achieve. The saying " Figures can lie and Liars can figire" still holds. Increasingly, corporate funded 'truth' and statistics are taking over the media. Follow the money and all will be revealed.

    ReplyDelete
  9. Yes linking the product and a GST payment would be the crucial part. Luckily shippers and the receiver already know a value to use as an ID tag that wouldn't require any extra work, I know it as 'name and address'. :-) But I realise that multiple packages, multiple items per package, different spellings of name and address would cause complications. I don't expect it to happen either.

    ReplyDelete
  10. Checkout the Librarist - http://www.librarist.com/nz/book/9781847088765/


    It calculates the pricing including postage and delivery timeframes for you (all converted into your local currency of course) - it also tries to find related editions (bottom of the page) so you can consider buying different years/hardcover/paperback if cheaper.

    ReplyDelete