Reporters do not understand causality

The latest in a continuing series. Today's edition: the effects of video lottery terminals, or "pokies". A couple of public health researchers from Massey presented work at an anti-gambling conference arguing that increased presence of pokie machines causes crime. Here's the Press on it; the story was also picked up in the Dom Post and Southland Times.

The more pokies in your neighbourhood, the higher the crime rate, new research has found.

Problem gamblers stealing to feed their addiction could explain the link to higher crime levels, police and gambling experts say.

The nationwide, year-long study by two Massey University academics used census data to prove the link between crime levels and the location and number of gaming machines in a community.

Co-author Martin Wall said he was surprised the impact of problem gambling on the wider community showed up in national crime statistics. "What was surprising was that there was a statistically significant link between the density of machines and crime rates," he said.

"People lose a lot of money on pokie machines and do not really control it. This affects families and communities.

"We have proven that this effect is big enough to show up in crime rates. That is quite a surprise because the number of problem gamblers in New Zealand is quite low. The effect must be quite big if you can detect it in statistics."

And that is, of course, the first thing that ought to give you pause. Is it more plausible that the small number of problem gamblers are responsible for a whole lot of crime, or that something's funky in the statistical method?

Christchurch problem gambling expert Peter Jamieson said there was a clear link between crime and pokie addiction.

"This research backs up what we see every day," he said.

"The slang for pokie machines is that they are a gambler’s heroin. Crime and gambling go together very well. We get many people sent to us by the courts."

Jamieson, co-ordinator for the Salvation Army’s Oasis Centre for Problem Gambling, said many people were referred for gambling addiction counselling by the justice system.

"Ripping off the bosses – that is the biggie," he said. "We also get people who have committed petty stuff like selling things stolen from work or stealing courier packages . . . They need to get some money because they are desperate to gamble."

Christchurch has 114 venues for gaming machines and a total of 1767 pokies.

Christchurch policing development manager Inspector John Price said police often dealt with the "fallout" of problem gambling.

"There is definite causation between burglaries and gambling," he said. "People commit crimes to fuel their gambling. We just pick up the pieces in the end, unfortunately."

Wall said the next stage of research will look at ways to control gambling harm.

Nowhere did the study address causality. They have regressions with crime and other outcome variables on the left hand side, a pokie machine measure on the right hand side and controls for three confounds: population, urban or rural status, and the area's measure on the New Zealand Deprivation Index. That isn't enough to show a causal relationship; it's just correlation. Imagine that pokie machines have no causal relationship with crime. What might generate the correlation?

Well, pokie machines tend to be placed in bars, either downtown or in suburban commercial areas. If those venues are somehow different than other places of similar population and deprivation, and tend to be associated with higher crime levels, then the correlation could simply mean that pokie machines tend to be placed in spots that tend to have higher crime rates.

So if pokie machines aren't randomly sprinkled across the country but rather are put where their owners expect to make the most money - basically, near the kinds of folks whose impulse control issues would similarly dispose them to criminal activity, then we can't say anything about causation using this method.

So, how do you deal with the problem? Instrumental variables. Find something that's correlated with pokie machine presence that's not correlated with the crime rate. Now that's not easy. But that's what you'd have to start doing to be able to start talking about causality. Another method would be regression discontinuity design. There are licensing boards that decide whether a venue's allowed to have pokie machines. Compare places that just barely passed muster to be given licenses with those that just barely didn't. I have no clue whether the licencing trust board deliberations are public record so I don't know whether that's feasible. Third, natural experiment. Are there places that had something weird happen? Every now and again, Internal Affairs orders that some venues shut down their pokie machines because they're not meeting the terms of their trust deeds. If any of them are shut down for a long enough period, you could check whether the exogenous change in status had any effects on the crime rate.

All of those above approaches would be reasonably hard. Here's one that would be reasonably simple. Run the regressions exactly as they are now, but on data from prior to the legalization of pokie machines. As best I can tell, pokies started being allowed around 1990 and rolled out over time. This one would actually be a fun project for a future honours student project: are pokie machines so evil that their effects run backwards through time like tachyons? Look to see whether places that get pokie machines a year later have higher crime rates than places that don't get pokie machines a year later.

I've also a few worries about their measure of crime. I've talked a few times with the stats folks at NZ Police trying to figure out how to get decent disaggregated crime data. The answer I usually got was that it can't be done. So it's not surprising that the authors here used crime reports by police station, then applied average rates for each police station to its constituent meshblocks, then mapped that back up to Census Area Unit averages. But there are other problems, or at least from my recollections of chats with the police about their statistics. Within a city, some police stations will take lead on different offenses regardless of where they happened. So all fraud cases might be handled downtown with petty crime handled by branch stations. Maybe they've fixed things up since, but when I'd asked a couple of years ago they said there was no way of mapping those crimes back to the originating station.

But let's assume that that's been fixed. We still have the very big problem that crimes will be measured where they happen, not based on the residence of the offender. So if you burglarize a place or commit fraud against your employer, it's the victim's address that enters the crime stats. Again, where are pokie machines? Mostly in bars downtown and in bars in commercial areas. Is the measured strong relationship between pokie availability and fraud then reflecting that problem gamblers might defraud their employers, or that the objects of fraud might be close to the kinds of commercial areas where pokie machines are placed? Do we see a relationship with robbery because of the pokie machines, or because pokie machines tend to be near places that get robbed?

I'd said:

There's one simple question that ought to be required for any journalist interviewing any social scientist on any issue like this: "How did you address causality?" It's a simple question. If the researcher says "Oh, we used a panel design to get within-subject effects" or "Oh, we instrumented by using...", that's a great start! If not, all you've got is correlation.

Kiwi journalists just don't ask the question.