An upcoming engagement

Coming up next week at The Christchurch Press:

Eric Crampton, Senior Lecturer in Economics at the University of Canterbury, will present an educational workshop for editorial staff on understanding "Basic stats, inference, and the social sciences" . In particular, he will pick apart press releases to query how accurate their assumptions are.

Eric's challenge to us is: "The Press has played pretty close to the researchers' press releases, especially when those press releases make claims that aren't really substantiated by the underlying studies. There are a few simple(ish) things to watch out for and a few easy questions that reporters can ask that can really help them to sort out whether the press releases match what the researchers have found. I know that journalists work under pretty tight time pressure and that it's pretty unlikely that they'd be reading the original source papers. However, we've got to be able to do better than quotes from the press releases followed up by reaction quotes from lobby groups in the area."

This is a great opportunity for us to learn skills that will provide better, more accurate stories for our readers.

I'm giving a three hour session, two sections. Here's my draft outline. Have you any better suggestions of examples I should cover? I don't think I can cover more material - I have about an hour and a quarter for each half, with a break in the middle. I'd far sooner take a sledgehammer to two big problems than throw pebbles at a bunch of smaller ones.

Causation and inference

• The basics of correlation
• Ways that correlation isn't causation:
  
  • Reverse causation
  • Spurious correlation: an underlying omitted causal variable
• How can you tell if the relationship is causal?
  
  • Methods that get closer to showing causality (don't worry - intuitive explanations only, no real statistics - just enough to know what the terms mean if somebody uses them):
    
    • Panel fixed-effects studies
    • Instrumental variable studies
    • Regression discontinuity design
    • Natural experiment
  • If the press release doesn't say anything about how causality was determined, ask the researcher before going out for the reaction quotes! Look how Reuters did it on Sally Casswell's latest piece.
  
  
  
  • If all there is is correlation, think of all the ways that correlation isn't causation. Is there something big and obvious lurking around that could be driving the results?
    
    • Pokies and crime rates: could results here be due to pokie machines being situated in bars? Other neighbourhood characteristics? Let's look at the press release and find the clues that they really haven't established causality.
    • Marriage and mental health: if married people are less likely to show onset of new mental illnesses, is that because marriage prevents mental illness? Or, is it because folks who have had a prior mental illness are more likely to develop new ones and are less likely to get married?
    • Does watching TV really kill you? Or, does having a more sedentary lifestyle correlate both with TV watching and increased mortality?
    • Substance abuse and crime: what does "alcohol-related" or "drug-related" mean in the police stats? About as much as "Oxygen-related": the offender had consumed the substance prior to the offence.
• Workshop time: I propose some correlations: you tell me whether they seem plausible or if something else could be driving things.

Social costs and social benefits
Folks wanting to have a pet project subsidized, or a pet hate regulated or taxed, will commission social benefit and social cost studies to help push their case. They're usually terrible. How?

• Obvious dodginess: ridiculous crazy assumptions. Example: PriceWaterhouse Coopers on Adult and Continuing Education. Is it really plausible that anyone taking a night course in Indian Cooking enjoys a fifty percent reduction in his likelihood of committing any crime?
• Social versus private costs and benefits: why the difference matters
• Are the stated benefits gross or net? Do they confuse stocks and flows? Do they account for the deadweight costs of the taxes needed to fund the project? Do they rely on and abuse multiplier effects?
• Also be careful that the study is counting everything it ought to count. If a study claims to show health benefits of $(big number) if we ban, say, tricycles (kids have accidents on them), have they weighed the cost to kids of not being able to ride around on tricycles? In other words, are they only assessing the benefits on one side while ignoring the costs of implementing the policy - not just the costs to the government, but also the costs falling on those subject to the regulation or policy? This is especially prevalent in public health research that counts as zero all the fun people have doing somewhat risky things.
• How to guard against being abused:
  • Ask about the biggest contributors to the costs and benefits. Do they seem plausible? Are they benefits that mostly accrue to the public at large, or to the person undertaking the activity?
  • Have a sense of scale: what does a $4.8 billion dollar cost or a $6.3 billion dollar benefit really mean in per capita terms? Does it seem plausible? Remember that there are only about 4 million people in the country. Any number around $4 billion means about $1000 per person for every man, woman and child in the country. If the benefits are that high, how is it that we're not already doing it? Your first reaction on seeing a really huge number shouldn't be "Wow, this is a really big problem!" It should be "That number smells bad."
  • Have there been prior estimates of the same thing? Are the current numbers very different? Why? Example: tobacco.
    
  • Treasury's handbook on cost-benefit analysis is an excellent resource. Their section on "Tips and Traps" ought be essential reading for journalists writing stories based on cost-benefit analyses.
  
  
  
• In a perfect world, we'd have only disinterested researchers. In the current world, we have commissioned consultants selling products, academics whose universities' PR offices are trying to sell stories to papers to get the university's name out, and bureaucracies that are trying to build public support for their preferred policies. Guard against all of them!
• Workshop time: I'll give press releases for example cost or benefit studies, you tell me what's wrong with them.

Anything I'm obviously missing?