Really big numbers

There seems to be a pretty thick market in the provision of really implausibly big numbers here in New Zealand.

Last year, Matt Burgess and I took a sledgehammer to a shonky report on the costs of harmful alcohol use.

Now, Matt Nolan points to one that looks about as bad: PriceWaterhouse Cooper's measure of the benefits of Adult Community Education, commissioned by the Adult Community Education industry association. Writes Nolan:

So some part of $171m-$254m can be seen as a positive externality, a pretty indirect one but lets be generous.

Taking the “possibly policy relevant” factors, and accepting the seemingly artificially inflated numbers (they state that it HALVES the chance of these 400,000 odd people being involved in a crime over their lifetime – see page 57), then the average return per dollar invested is about $2. Furthermore, that is an average not a “marginal” amount (contrary to the claims of the report mind you – see page 5). The marginal return (the amount of social return we will get if we increase spending by a small amount) is likely to be well below $1 per dollar.

Given what I feel has been a generous interpretation of the marginal social benefit, the suggestion would be that we should cut back on public funding for adult community education.

There would be zero demand for these kinds of shonky reports if journalists used a tiny bit of critical thinking before parroting back the numbers, but journalists would only do this if the public much cared about accurate numbers. A quick flip through "Appendix A", in which they justify their shonky analysis, makes pretty clear what's going on. A sample:

This [roughly $250 million] is the PV of the savings in the average cost of crime as a result of an individual being involved in full time employment, participating in higher education, improving their self confidence, and being more involved in their community. Research shows that all these factors have an effect on reducing a person’s likelihood of participating in a crime by up to 50%. Depending on the level of improvement in the above indicated factors, this is calculated as the PV of the reduction in the average costs of a crime multiplied by individual’s likelihood of being involved in a crime (where everyone above 15 in NZ is given an equal likelihood of being involved in a crime).

So folks taking adult ed courses are assumed to have a 50% reduction in their chances of committing a crime. PWC cites a 1999 working paper as evidence; a 2004 AER piece by the same author has the crime reduction associated with high school graduation as being less than half that figure (14-26%). This latter study uses a far more cautious identification strategy: changes in minimum age of dropping out of school as instrument for completion rates. And note that the numbers cited are for HIGH SCHOOL GRADUATION, not for taking a night course in Indian cooking.

For some reason, I'm reminded of Susan Feigenbaum and David Levy's work distinguishing the difference between scientific fraud and biased research.

Fortunately, Finance Minister Bill English is no idiot:

Hon Maryan Street: How does the Minister compare the return on the investment of $35 million into private schools with the PricewaterhouseCoopers’ calculation of the return on the investment in adult and community education funding in 2008 of between $54 and $72 for each dollar of funding?

Hon BILL ENGLISH: Well, if we believed PricewaterhouseCoopers’ evaluation, we would spend $10 billion on adult and community education and would have an economy that is twice the size it currently is. That is clearly the kind of nonsense that the previous Government relied on.

Exactly.

Would "useful idiots" be too strong a term for folks who take these reports' findings uncritically?

Education Directions has further critique of the study.