Sugar taxes, yet again.

Yesterday, just after 3, Stuff's Senior Health Reporter Rachel Thomas gave me a call. A new WHO-funded meta-study on sugar taxes was out, and she wanted comment before 4.

I received the study at 3.10 and started in. 

At 3.52, I sent this through:

Cool. This is the background version. Will send quotable version shortly. 

A serious crack at this one would require checking the studies they included against the studies that John Gibson and Bonggeum Kim evaluated in a 2019 piece in the Journal of  Development Economics.

Also attached in case you don’t have access. 

Gibson and Kim found that 80% of the studies they surveyed suffered from a severe methodological error because they’re based on household expenditure share data, not consumption data.

What does that mean?

Most of the time, researchers don’t have a clue how much of anything you actually purchased. They have survey data on household expenditure shares. “How much did you spend on {long list of things} in the past week?”. They then divide total spending in a category, like soda, by the average price of thing, to get a ballpark on quantity. 

And that all sounds great until you remember that there are two margins along which people can respond to a soda tax. Say I spend $50/month on Coke that costs $1/can before a soda tax comes in. The tax is $0.25 per can. If you observe that, in the spending data, I now spend $40 per month, what should you conclude? Have I reduced my consumption to 32 cans per month in response to the price increase? Maybe! But I might have shifted over to the store brand soda that cost $0.50 per can before the tax, and costs $0.75 per can after the tax. If I did, I might be consuming 53 cans per month ($40/$0.75), not 32. 

You can’t tell if all you have is household spend-share data rather than actual unit numbers. And 80% of the studies that Gibson and Kim looked at didn’t deal with the very important problem of making inferences from data when people can be adjusting not only on a quantity margin, but also on a quality margin. I might not even have shifted from Coke to home-brand. I might have shifted from cans to 2 litre bottles that cost far less per drink. But the researcher won’t have that data and has to make inferences. 

Anyway, will send through a quotable version. But that’s the underlying problem with a ton of these studies. And they don’t mention it in their selection criteria for inclusion. They just rely on an external letter grade on quality that it’s really unclear considered the problem either, or to what extent. 

I followed it up shortly thereafter with potentially more useful quotes:

“Metastudies always have difficulty in weighing the quality of the different underlying studies. A metastudy takes a complicated average of effects across studies. If the bulk of the underlying studies have substantial errors in method, then the average effect across those studies will suffer the same flaw.

Many studies looking at the effects of prices on consumption, including studies looking at sugar taxes, use survey data on household spending on different kinds of goods. The surveys do not have information on quantity, just spending. Some researchers then estimate the effect of the tax by dividing the amount a household spends on soda by the average price of soda, before and after the tax. But that method can err severely if, in response to a soda tax or a sugar tax, households shift from higher-cost name-brand drinks like Coke to lower-cost store brands, or from higher-cost cans to lower-cost 2-litre bottles. If a household shifts from spending $100 per month on Coke costing $1 per can before a soda tax, to spending $90 per month on HomeBrand costing $0.75 per can including the new sugar tax, a lot of studies would falsely conclude that their consumption had declined.

When Waikato economist Professor John Gibson surveyed the literature in the Journal of Economic Development in 2019, he found that over 80% of published studies using this kind of data include this kind of error. It is important that a rigorous metastudy account for this kind of problem. That this metastudy does not seem to mention the problem in its discussion of which studies were included in its evaluation, and which were excluded, seems a substantial problem likely to lead to overestimation of the effects of sugar taxes on consumption. But it would take further work looking directly at the studies evaluated to tell how big a problem it will be for this metastudy.”

Then I dug a bit more on the references. Can't do anything comprehensive on a short turnaround. Can I see studies cited in the metastudy that I just remember the names from? If I remember the name, it's either because the thing was actually a decent one, or because it was not. 

First one I check: Colchero et al 2015. Why was the name familiar? John Gibson had pointed to it as having had this specific problem. So I sent this through:

Colchero et al 2015, cited in the meta study as one of the ones they’re using to infer elasticities, has the problem – Gibson cites it specifically as having this problem (using budget share data and the standard unit value method that doesn’t deal well with adjustments on the quality margin) 

I can’t check all of them in the time available. But I can see immediately that it’s a problem in the first one I looked at – but I checked that one because I remembered the name being familiar. 

I had suspected that the metastudy's exclusion criteria did not exclude studies that used expenditure share data without accounting for adjustments on the quality margin. Finding one that had not meant it had not. 

So it would be hard to have any confidence in the rest of the metastudy. 

I didn't expect it would all make it into the piece of course. 

The print version of the morning newspaper was a bit galling. None of it was there! 

Just comment from the usual sugar-tax suspects on how the new study is awesome, proof that the government has to put in a sugar tax - the usual kind of thing. 

Quotes that could have been given without even looking at the study. 

The online version found a bit of room for my warnings about the data issue.

Economist Dr Eric Crampton – who has long been against sugar taxes - said a drop in spending didn’t necessarily mean a drop in sales.

He said at least one study included in the meta-analysis based a drop in sales on household spending data alone.

”If a household shifts from spending $100 per month on Coke costing $1 per can before a soda tax, to spending $90 per month on HomeBrand costing $0.75 per can including the new sugar tax, a lot of studies would falsely conclude that their consumption had declined.”

The study was published in the scientific journal JAMA on Thursday.

Life would be so much easier if I were on the other side on this one, or just stopped caring about the data. 

I could give quotes that would make the print version, and I wouldn't even have to have read or understood the study. Just be on the correct side of the issue. 

No need to blitz through a metastudy in half an hour after refreshing my memory on the specifics of the Deaton-method spend-share issue in these studies, check whether the study's exclusion restrictions seem to handle it, find that that just punts to a Cochrane assessment that I won't have time to evaluate, run back to the bibliography to check what studies they did use because if they included even one known-bad one, their exclusion restriction can't have been right, find one, and write it up. 

I'd have just been able to say something lame like "Anyone who cares about children would want a sugar tax now. Everyone who opposes it must be Big Industry. Here is a picture of some rotten teeth. Just look at it!"

So much easier. And probably even without the "you can ignore this guy because he just doesn't like sugar taxes" tag up front. 

Ah well. These boulders don't push themselves up the hills. Somebody's gotta do it. 

Update: The metastudy is here. Knock yourselves out. Nobody cares whether the thing is right or wrong, just whether it advances the cause of getting a sugar tax.