Black Hole Research

Last year, Steven Stillman showed that regression discontinuity design doesn't work very well in identifying the effect of changes in the drinking age on other outcomes.

Suppose you wanted to know whether changing the legal drinking age, or alcohol purchase age, would affect outcomes among the cohort subject to the change. A lot of folks will reach for a Regression Discontinuity Design to compare outcomes for those who've just reached the minimum drinking age with those just under it (who are otherwise pretty similar other than a small age difference), and then claim that increasing the drinking age would reduce harms by the magnitude of that discontinuity extended over the interval, or that reducing the drinking age would similarly increase harms.

The problem with that approach is that you're identifying not just on reaching the legal age, but also on having a big birthday milestone. And you could also be picking up effects of inexperienced drinkers. Pretty hard to tell whether the RDD is really catching a drinking age effect or a mix of birthday and experience effects. Hard, that is, unless you actually have a country that changed the purchase age. Stillman and Boes showed that despite an effect showing up in RDD with the New Zealand change from 20 to 18, that effect doesn't extend to a longer term difference-in-difference analysis that tracks actual changes for the affected cohort.

I'd summarised last year:

They also make an important point on method, and I thank Steven for having explained this one to me slowly; hopefully I've understood it properly. Now recall that they found no effect of the law change on traffic accidents: changing the alcohol purchase age from 20 to 18 did not increase the accident rate among youths. Nevertheless, results from a regression discontinuity design comparing accident rates among kids just under the alcohol purchase age with those just over the alcohol purchase age shows an increase in accidents on reaching the alcohol purchase age after the law change. Why could there be an effect in RDD but not overall? Either the RDD is picking up the effect of the alcohol purchase age on the cohort of inexperienced drinkers who change their behaviour on reaching 18 and experience worsened outcomes, or it's picking up a particular changed behaviour around the time of the birthday. We would have overestimated the effects on traffic accidents for 18-19 year olds by extrapolating from the discontinuity around the 18th birthday.

But, we only know this because New Zealand actually had the policy change so we can compare actual outcomes with those estimated by the RDD. Now, suppose you're in a country that hasn't reduced its drinking age and you want to estimate what the effect of decreasing the drinking age might be. If you ran an RDD looking at the effect of reaching the age of majority on traffic accidents, took the break around the birthday as being the effect of being able to drink, then extrapolated that effect back across the cohort of younger drinkers who would be eligible to drink under a different minimum legal drinking age, you could pretty easily be overestimating the likely effect of a real change in the alcohol purchase age. The RDD picks up the effect of reaching the legal drinking age rather than the effect that would obtain by lowering the legal drinking age.

So - we should be wary of RDD estimates of the effects of the alcohol purchase age.

Well, unless you have preferences over outcomes and want it to look like there's a big effect. In that case, RDD's great.

Toumbourou, Kypri, Jones and Hicki survey the literature other than Stillman's work and conclude that the drinking age should jump. Lindo and Siminski reply in a letter to the Medical Journal of Australia:

Toumbourou and colleagues argue that the Australian legal age for buying alcohol should be increased. However, they overstate their case by only citing research that supports their position, giving an impression of scientific consensus on several key issues when there is strong contrary evidence.

They note the Boes and Stillman work ignored by Toumbourou et al.

Toumbourou, Kypri, Jones and Hicki's reply? That the Boes and Stillman paper can be ignored because it's not yet published.

Second case? A piece in the American Journal of Preventative Medicine, again using RDD to claim that Quebec should hike its drinking age, with zero reference to the Boes and Stillman critique.

If only econ journals published as quickly as do some other fields' journals, or, better, that some other fields took the time to get expert referees in to provide comment on technique. Hopefully this will change when the Boes and Stillman work finally comes out. I'm not all that optimistic, but hopeful.

Do Costs Matter (Revisited)

Crikey's Bernard Keane has been questioning the taxpayer-subsidised anti-alcohol campaign in Oz. And, usefully, he's drawn a response from Sandra Jones. Let's have a bit of a look.

In 19 November's column, Keane looks at the new health paternalists:

There’s little new in all this. Last year, a coalition of preventive health groups demanded the Victorian government move to curb the number of liquor licences. Between trying to reduce the number of alcohol retailers and using taxation to increase the price, the preventive health industry has moved precisely nowhere in over 250 years since the Gin Craze of 18th century England. In what has been called the first drug scare in history, mid-18th century British governments, spurred by an outraged middle class, used exactly the same tactics to attack the prevalence of gin consumption among poorer English people who, it was felt, drank too much and didn’t work hard enough.

It’s fascinating how little the justification for such crackdowns has changed. Attempts to regulate and tax gin out of the reach of poorer people were justified by not merely by moral righteousness but on economic grounds: gin was damaging the capacity of English women to produce children, and consumption of gin caused poverty and idleness in an economy struggling to compete with its European rivals.

The economic justification is no longer couched in such melodramatic terms. Instead, it relies on QALYs — Quality Adjusted Life Years, and AWE-based calculations of lost productivity. But the motivation remains the same: social élites anxious to impose control on what they disapprove of. The big difference now is that nearly all of this is taxpayer-funded: we are paying these élites to rationalise banning, taxing and using surveillance over what they disapprove of.

Michael Thorn and Sandra Jones reply shortly thereafter. The bit that struck me there was their reference to the new Australian report by Marsden Jacob and Associates for FARE. Thorn and Jones write that one reason for increased alcohol regulation is:

Rational thinking: This is not a moral case. There are social, health and economic arguments that fully justify acting to reduce the more than $10 billion a year cost to government. These are tangible alcohol-related police, justice and health care costs that far exceed the $6 billion of alcohol tax collected each year.

They cite the FARE report on this one. It's interesting for two reasons. Back in September, when I was critical of the Collins & Lapsley figure on alcohol's social cost in Oz, Sandra Jones seemed to suggest that measured social costs really didn't matter [my critique was here]:

I think Dr Crampton is missing the point. I for one will not vote for (or against) government policy because the costs of death and suffering have been estimated at $5 billion, any more than I would if they were estimated at $10 billion or $100 billion.
...
Perhaps Dr Crampton – and the alcohol industry – could spend a little less time arguing the exact calculation of the financial costs of alcohol and think about the real costs. I am sure that every parent who has lost a teenager from alcohol misuse would estimate that single cost at much more that $15 billion.

So the numbers don't matter, except when they support her.

And, except that the FARE report didn't find anywhere near a $10b cost to the government, or "alcohol-related police, justice and health care costs" in excess of $6b.

From Table 6 of their report:

Sources of harms to others	Total cost of harms 2009/10 $m
Child protection system	694
Effects of drinking of household/family member or friend
- out of pocket	437
- lost time	964
- loss of quality of life	7,703
Theft, burglary	141
Counselling, advice & treatment	113
Property damage	1,673
Loss of life	1,326
Labour costs on others	828
Hospital costs to others	159
Policing & Justice system	1022
TOTAL	15,061

Ok. Police, hospital, counselling, and child protection sum to just under $2 billion. If we add in theft and burglary, and property damage, we can get to $3802 - but most of those costs fall on private third parties, not on government.

There is absolutely no way we can get a $10b cost to the government out of the FARE report. Why? Just look at the table! $9 billion of the $15 billion cited are intangible costs of lost life and lost quality of life. Neither of those are a cost to the government.

And, unless I'm reading it wrong, the $7.7b on lost quality of life is an updating of the figure I'd critiqued last year - the one where  they compared monetized average quality of life differences among those who report knowing a harmful drinker and those who do not, with no adjustment for that the different cohorts might have other relevant differences.

I've not gone through the new FARE / Marsden Jacob report in great depth as yet, though as semester's now finished, that may change. Their numbers on crime are pretty close to what I'd found; on others, we've some differences.

Keane responds to Jones and Thorn, pointing out some of the stats showing there to be no particular crisis in Australian alcohol consumption. Things aren't that different on this side of the ditch.

Do costs matter?

Sandra Jones says that optimal alcohol policy is invariant to measured social costs:

I think Dr Crampton is missing the point. I for one will not vote for (or against) government policy because the costs of death and suffering have been estimated at $5 billion, any more than I would if they were estimated at $10 billion or $100 billion.

As an Australian, I care that alcohol is second only to tobacco as a preventable cause of drug-related death and hospitalisation, with an estimated 3,000 deaths per annum; I care that our emergency rooms are clogged with people suffering from the immediate effects of their, or others, misuse of alcohol; and I care that every year almost 600 Australians aged 65-74 die from injury and disease, and 6,500 are hospitalised, as a result of drinking above the NHMRC guidelines[3].

I also care that 43% of Australians perceived physical assault in a public place to be a problem in their neighbourhood[4]; that alcohol is a significant contributing factor in domestic violence[5]; and that alcohol abuse is an important risk factor for child abuse, maltreatment and neglect[6].

As a parent, I care that 13% of all deaths among Australians aged 14-17 years – that is one per week – are a direct result of alcohol consumption; and that alcohol is responsible for the hospitalisation of 60 teenagers each week.

Perhaps Dr Crampton – and the alcohol industry – could spend a little less time arguing the exact calculation of the financial costs of alcohol and think about the real costs.  I am sure that every parent who has lost a teenager from alcohol misuse would estimate that single cost at much more that $15 billion.

Let’s stop pretending we don’t have a problem with alcohol.

In one important sense, I agree with Sandra. As we wrote in the introduction to our paper:

We argue that the standard economic case for intervention, which relies on identifying instances in which the marginal social costs of consumption exceed marginal private costs, is not established by Cost of Illness studies. First, these studies summarily ignore the differences between marginal and total costs. Even were it the case that total external cost exceeded total collected taxes, feasible interventions including excise tax increases can remain undesirable where marginal external costs are less than the excise rate.

You can have worlds where the measured social costs are orders of magnitude higher than collected excise revenues but no intervention passes cost-benefit analysis because harms are relatively inelastic to intervention; you can have other worlds where the measured social costs are orders of magnitude lower than the collected excise revenues but particular interventions pass cost-benefit analysis because particular categories of harmful activity are relatively elastic to intervention.

The only reason that I started caring about the measure of the total social cost of alcohol use is that bad estimates of it build demand for intervention on what look like sciency economic grounds when the sciency veneer is actually pretty thin. When BERL came out with its big shiny number, I just shrugged - another dodgy number in a world of dodgy numbers. Until Sir Geoffrey started citing it as motivating his call for much tighter regulations around alcohol and BERL's analysts refused to inform him that their figure was not fit for that purpose.

I'm sure that Sandra Jones' opposition to alcohol is completely invariant to the measure of the social cost. But I don't believe that's true of many voters who are deceived by cost of illness studies that present costs drinkers impose upon themselves as 'social costs', which then are interpreted in public debate as costs to the taxpayer.

It's worth perhaps also having a bit of perspective on where Sandra's coming from on alcohol, and here I'll part company with her rather more substantially.

Here's Hoek and Jones in the Journal of Social Marketing last year. First, some definitions:

The dominant approach focuses on individual-level behaviour change, or what is now described as downstream social marketing. This approach regards behaviour change as voluntary and relies on offers that create greater value than continuation of the risk behaviour. By contrast, "upstream" social marketing focuses on policy and regulation, with the aim of altering environments so these support and promote behaviour change. "Upstream" initiatives have much in common with work public health researchers undertake, suggesting that many benefits could be gained from stronger social marketing and public health alliances.

So upstream social marketing is regulation. Ok. So why do we need that rather than efforts that focus on individual responsibility? Let's check the section of the paper called "The Myth of Individual Responsibility" for some answers.

By promoting individual behaviour change, downstream social marketing campaigns risk reinforcing the impression that consumers themselves are responsible for their own choices, irrespective of the powerful stimuli that promote and reinforce these. Furthermore, individually oriented actions deflect attention from the companies manufacturing, distributing and marketing products known to cause harmful social and health problems and imply these cannot (and perhaps should not) be held responsible for their products' effects. Efforts to relocate responsibility back to these corporations thus require environmental changes that both recognise and ameliorate the influence marketing stimuli exert. 

Is this just tobacco and maybe alcohol?

For example, McDonald's and Coca Cola have undertaken extensive sponsorship of sporting events, from major international gatherings such as the Olympic Games and FIFA World Cup right through to smaller local activities, such as support of junior football codes. These activities indicate their endorsement of sporting activity and exercise, which is clearly part of a healthy lifestyle for all individuals. However, it also supports their claim that no foods are "unhealthy", that a sensible diet has room for "treat" foods, just as it should also include "everyday" foods, and that the way to enable inclusion of these treat foods is to undertake sufficient exercise. Individuals, therefore, need to manage their diets so these represent an appropriate balance of foods.

This reasoning has a powerful intuitive logic and few would deny individuals have an important role in selecting the foods they consume. However, this argument overlooks entirely the role food companies have in shaping individuals' food environments.

 And individual choice?

...far from limiting individuals' freedoms, regulation shaping consumers' choice contexts is actually the means of liberating those environments, thus enabling them to make free choices. Only by balancing environments so the voices of public health are not overwhelmed by the cacophony of commerce can individuals begin to make choices that are in their long-term interests. Without regulation to create these more even choice contexts, consumers faced with unremitting commercial stimuli are, understandably, likely to be swayed by these rather than longer-term social or health interests.

The Ministry of Freedom could maybe be in charge of making sure the choice context is just right.

In a paper section entitled "Tobacco control - daring to dream", they note how regulatory policies denormalizing tobacco use made downstream social anti-smoking campaigns more effective; this is meant to serve as example for the effectiveness of this partnered approach for other industries.

They conclude:

If high-risk behaviours such as smoking, excessive alcohol consumption and consumption of high fat, salt and sugar foods are to change, social marketers need to recognise, then manage, the environmental determinants of risk behaviour. This implies that they explore upstream measures and work alongside public health researchers to create a context where downstream interventions may flourish. [25] Jochelson (2006) summarised this approach when she argued that: "Legislation brings about change that individuals on their own cannot, and sets new standards for the public good". Because upstream measures affect the environments that shape and support behaviour, they should be regarded not as constraints diminishing voluntary behaviour but instead as the pre-requisites enabling full and free choices.

It's like the No Logo movement has bred with the public health movement. Count Chocula should be worried; his days may be numbered.

It's at least mildly amusing that Jones, who views people as massively subject to advertising in their roles as shoppers, seems to dismiss that voters' preferences might be corrupted by shonky cost of illness studies.

At least I have a framework for why I worry a lot more about the latter than the former: individuals bear the costs of getting things wrong in their shopping decisions and so pay a bit more attention there than they do when weighing up political choices where each vote has only a trivial chance of changing anything.