Threatening data

AUT's Rhema Vaithianathan sees the potential for big data in the public service.

The age of Big Data has come to consumers rapidly, reinventing how we shop, socialise, bank and get from A to B. And a Big Data revolution is slowly rolling through the public sector too.

If we do this right, in 20 years or so the government ministries that currently employ thousands, and sprawl across central Wellington should each fit into a small Thorndon villa.

The vast bureaucracies of Wellington serve two main purposes.

First, they monitor. For example, ensuring that schools are open the requisite days, that police are on the beat and other taxpayer-funded services are working as they are supposed to.

Second, they collect information about these services useful for planning and policy advice. This helps the government to decide how best to spend our money. Is the existing service working? Should it get more, less or no funding? Is a new intervention needed, and how much funding should it get?

Fortunately, when it comes to monitoring and informing, Big Data can really deliver. So once our data is up to the task, these jobs won't need to be done the old-fashioned way by armies of civil servants.

It sounds great. But the power implications within government make it all a bit tricky.

It is thoroughly feasible for NGOs to ask the government to run evaluations on their own effectiveness. They can provide the government with the details on the people they're serving and what outcomes they're targeting. The government can then set a control group of people matched to the NGO's clients in IDI, or to do one better and help the NGO randomise treatment and control where the NGO can't afford to help everyone they'd like to help. Then any outcome measure they want can be tracked and evaluated: future interactions with CYF, prison recidivism, workforce attachment, child doctor visits, child immunisations - anything on which there's administrative data. And, even better, the NGO can then tell the government what their cost of outcome delivery has been.

Once that's in place, interesting things happen. Rhema notes the massive potential disemployment in the Wellington bureaucracies. One of the big advantages Ministries have over their Ministers is information. They're the ones who know things, and who can tell the Ministers things when Ministers want to know things - or refrain from providing useful information. There's a whole public choice literature on bureaucracy, agencies' information advantage over Ministers, and equilibria when Ministers can implement costly legislative control devices to help them better monitor agencies' true costs and output. Big Data, done right, can help route around it all that.

And if that can be combined with real information on the real costs of not just delivering services but of providing the outcomes that the Minister wants - that's a game changer.

In our report on Social Impact Bonds, we looked forward to a world in which any NGO or community group could pitch an outcomes project to Treasury, Treasury would tell them the current going price for improving that outcome, and the NGO would then seek investor funding to deliver the service. Whether you see the resulting data on real outcomes and real costs of providing outcomes as a bug or a feature may depend on whether you're part of a monolithic Ministry with a big information monopoly that's under threat.

I hate economic impact numbers

What the heck is "economic impact" even supposed to mean? Seamus keeps wondering if it wouldn't be fun to set an honours project asking "To what relevant economic question is "economic impact" a useful answer, if any?"

My main reason for hating them is that they're so awfully abused by the press. It's hardly the press's fault though - if we as a profession keep producing these figures without adequately explaining (or understanding, in some cases) what they are, they'll keep abusing them.

Today's example, highlighted by Matt at TVHE: a $200m estimated "economic impact" of health disparities between Maori and Pakeha New Zealanders gets turned by the journalist into a cost to taxpayers.

The University press release:

Avoidable deaths of Māori children in New Zealand are costing the country at least 67 lives and around $200 million per year in economic impact, but greater Government spending on primary care and other key interventions could help to resolve the problem, health researchers say.

The journalist version:

The public health physician said 67 Maori children died avoidable deaths every year, costing taxpayers $200 million annually.

Ummm, no. Unfortunately, the Auckland University press release is about as unhelpful as possible in letting any journalist sort this out. Why? Not only do they fail to link the article, they also fail to name the journal: the study was "published in an international public health journal".

If you search through PubMed on the two authors' last names, you can find the article. It's here. But little chance a journalist will have time to do that.

What do you get if you read the article? Here's the results brief at the start:

Preliminary estimates suggest child health inequities between Māori and non-Māori in New Zealand are cost-saving to the health sector. However the societal costs are significant. A conservative “base case” scenario estimate is over $NZ62 million per year, while alternative costing methods yield larger costs of nearly $NZ200 million per annum. The total cost estimate is highly sensitive to the costing method used and Value of Statistical Life applied, as the cost of potentially avoidable deaths of Māori children is the major contributor to this estimate.

So both the press release and the journalist's piece ignore that Vaithianathan and Reid produced a range of estimates with a $200m upper bound rather than a point estimate around $200m. And, the larger number relies on VSL measures. What's a VSL measure? The intangible costs of a premature death. That's only a cost to the taxpayer in the rather indirect sense that the people who are sad when someone dies prematurely may also be taxpayers, and that some of the VSL measure could be viewed (given how NZ's VSL measure is constructed) as partially being due to that premature deaths are tragic for the family. And, worse, the University press release cites the deaths as though they were something not included in the $200m.

There's other weird stuff in there. There's a measure of time out of work for grieving parents that's based around the median wage rate, but if the whole darned story is about the costs of poverty among Maori in terms of premature mortality, it's just a bit odd to use the median wage rate. But I'd hardly expect a journalist to pick that up.

Why oh why can't university press offices pumping published work done at their schools link to the darned paper and write things that won't be misinterpreted by journalists? Are they trying to confuse people? I could understand it if the article were behind a ton of subscription gates, but this one's free access.

Can we just ban economic impact studies? It's almost inevitable that the things get misinterpreted like this. And they then do far more to confuse than to illuminate.

Costs to the taxpayer of $200 million.... How long 'till somebody cites this in Parliament as a cost to the tax system? Any bets?