Threatening data

AUT's Rhema Vaithianathan sees the potential for big data in the public service.

The age of Big Data has come to consumers rapidly, reinventing how we shop, socialise, bank and get from A to B. And a Big Data revolution is slowly rolling through the public sector too.

If we do this right, in 20 years or so the government ministries that currently employ thousands, and sprawl across central Wellington should each fit into a small Thorndon villa.

The vast bureaucracies of Wellington serve two main purposes.

First, they monitor. For example, ensuring that schools are open the requisite days, that police are on the beat and other taxpayer-funded services are working as they are supposed to.

Second, they collect information about these services useful for planning and policy advice. This helps the government to decide how best to spend our money. Is the existing service working? Should it get more, less or no funding? Is a new intervention needed, and how much funding should it get?

Fortunately, when it comes to monitoring and informing, Big Data can really deliver. So once our data is up to the task, these jobs won't need to be done the old-fashioned way by armies of civil servants.

It sounds great. But the power implications within government make it all a bit tricky.

It is thoroughly feasible for NGOs to ask the government to run evaluations on their own effectiveness. They can provide the government with the details on the people they're serving and what outcomes they're targeting. The government can then set a control group of people matched to the NGO's clients in IDI, or to do one better and help the NGO randomise treatment and control where the NGO can't afford to help everyone they'd like to help. Then any outcome measure they want can be tracked and evaluated: future interactions with CYF, prison recidivism, workforce attachment, child doctor visits, child immunisations - anything on which there's administrative data. And, even better, the NGO can then tell the government what their cost of outcome delivery has been.

Once that's in place, interesting things happen. Rhema notes the massive potential disemployment in the Wellington bureaucracies. One of the big advantages Ministries have over their Ministers is information. They're the ones who know things, and who can tell the Ministers things when Ministers want to know things - or refrain from providing useful information. There's a whole public choice literature on bureaucracy, agencies' information advantage over Ministers, and equilibria when Ministers can implement costly legislative control devices to help them better monitor agencies' true costs and output. Big Data, done right, can help route around it all that.

And if that can be combined with real information on the real costs of not just delivering services but of providing the outcomes that the Minister wants - that's a game changer.

In our report on Social Impact Bonds, we looked forward to a world in which any NGO or community group could pitch an outcomes project to Treasury, Treasury would tell them the current going price for improving that outcome, and the NGO would then seek investor funding to deliver the service. Whether you see the resulting data on real outcomes and real costs of providing outcomes as a bug or a feature may depend on whether you're part of a monolithic Ministry with a big information monopoly that's under threat.