The problem of social cost studies

Matt Burgess presented some of our work on social cost studies at the NZAE meetings. His presentation is embedded below, but it doesn't scale down well to blogger size, so you might do better by just hitting the link.


Prezi seems a pretty slick way of handling presentations; I'll have to remember it in future. Hit the forward triangle to advance the view. A few references as you work through it:

• On healthists vs economists: Public Health and the New Paternalism
• BERL's study of the social costs of harmful alcohol use:
  
  • Our initial report
  • Their reply
  • Our rejoinder.
  • Further explication of our minimal rationality assumption: we only require that, on average across all drinkers in the group, excess costs of "irrational" or mistaken consumption roughly match consumer surplus from the rational consumption.
  • Seamus also raised an important methodological point: there is no useful comparison between aggregate tax take and social costs where social costs include costs imposed by the drinker on himself.
• BERL's drug harm index
• The PWC study on the benefits of adult and continuing education
• On the problems when government funds studies to lobby itself

Matt correctly concluded that current processes don't protect policy against this kind of behaviour. Reputational mechanisms are very weak because while a consultancy's shonky report may disgrace it among academics and other consultants, the target audience - voters - is oblivious and journalists are not equipped to critically assess these produced reports. So even a shonky firm can get a new contract from a government department that wants a big number. Standard peer review doesn't work because the client, often Ministry of Health, doesn't want a sound number: it wants a politically convenient number. So it'll appoint peer reviewers that will provide its desired level of critique. And, the probability of a thorough external fisking is very slim; Matt and I did it, unpaid, because we were really annoyed that BERL was happy to let its number be compared against the aggregate tax take for purposes of policy discussion - we're masochists.

Matt consequently proposed that the NZAE work as a broker for peer review of consultants' reports. For a fee, the NZAE would assign a paid expert referee who would provide a proper independent review. A Ministry could signal that it wanted a credible number by insisting that such independent review be done. This is probably a more useful suggestion than my alternative: the awarding of an annual prize - the Shonky Award - for the worst piece of economic consulting work produced in that year.

A colleague who attended the session reports that another in attendance said BERL / MoH didn't put much weight on our critique because we were paid by industry. I WISH. I would have no problem at all in being funded to do this kind of work, so long as there were no strings on editorial freedom. But the sum total of anything that could possibly be in any way seen as compensation for our work on the BERL report was that I was invited to give a talk at last year's Beervana conference; when Susan and Ira came along, their flights and the hotel room upgrade / extra night were on my dime. I had to pay for my own ticket for the beer tasting, but didn't have to pay to attend the conference dinner (again, common for plenary speakers). In fact, I had a minor argument with the alcohol folks on the upgrade because the hotel seemed to have charged both us and them for it and the Brewer's Guild wanted reimbursement. Being flown to Wellington to present work at a conference hardly counts as compensation: it's kinda standard drill in academia that somebody who wants to hear you present flies you out there and puts you up. Here's the full list of who's been paid what:

• BERL: $135,000 for its initial report
• Our critique: nothing
• Brian Easton: $7,000 for his independent and impartial (but non-tendered) review of the differences between BERL and Crampton/Burgess
• BERL: we have an OIA request in on whether they were funded for their critique of our report; tomorrow's MoH's due date. Update: MoH says they didn't pay.
• Our rejoinder to BERL: nothing
• Our submission to the Law Commission: nothing
• Marsden Jacob and Associates: $60,000 for their untendered shockingly incompetent review.
• Our rejoinder to MJA: nothing

We're chumps. Every time we critique the healthists, the healthists pay themselves more to reply. Perhaps Marsden Jacob will yet be paid to fix their errors. There's absolutely nothing wrong with being paid to do work. But beware the one-sided skepticism.