Hard SpiritsI feel bad for Stats.
It was during the discussions of measuring spiritual capital that the ghost of Sir John James Cowperthwaite hovered near. The shade whispered in my ear, “When I was Financial Secretary of Hong Kong, I refused to collect economic statistics for London. Why? For fear that I might be forced to do something about them.”
I wish more people at last week’s Indicators Aotearoa New Zealand indicator selection event had been able to hear him.
Statistics New Zealand’s Indicators Aotearoa project aims to build a comprehensive suite of (you guessed it) indicators measuring wellbeing. Where SNZ’s 2014 Progress Indicators Tool, now discontinued, included 16 measures ranging from adult educational attainment to distribution of selected native species, the new framework aims to include about 100.
The project is linked to Treasury’s Living Standards Framework and to the Government’s desire for more measures of wellbeing. Treasury has been developing its own suite of wellbeing indicators as part of the Living Standards Framework; whose indicators will reign supreme remains a bit up in the air. If Treasury’s framework and indicators wind up satisfying the Government’s wellbeing needs, then Statistics New Zealand’s indicators project – or at least those indicators that do not make it into Treasury’s framework – might yet follow its predecessor into the bin of discontinued data series.
So the bureaucratic stakes are high, as far as these things go.
Statistics New Zealand took a reasonable approach in trying to figure out which measures to include: they asked New Zealanders what wellbeing means to them in a series of workshops around the country, along with an online campaign. Then data experts tried to figure out which of those might possibly be meaningfully quantified among those that are not already measured.
That led to a long list of indicators to be pared down at last week’s indicator selection event, where I was visited by the ghost of Sir Cowperthwaite.
Inputs, outcomes and A3 posters
There’s a management truism that what gets measured gets managed, and what doesn’t get measured doesn’t get managed. Choosing the right measures then matters, especially if these kinds of indicators get targeted by a Government desperate to be seen to be improving our wellbeing.
The big headline measures should reflect final outcomes (like healthy life expectancy) rather than intermediate outcomes (like cancer rates) or inputs (like the proportion of people using sunscreen appropriately).
If there are lots of intermediate outcomes that are all captured by the final outcome, promoting intermediate outcomes into being final outcomes is a very bad idea. It invites paying more attention to those selected measures rather than to other ones that Government might more usefully target. In the example above, if an intervention targeting diabetes did more to improve healthy life expectancy than an intervention targeting cancer, the latter might nevertheless be preferred if cancer is a headline statistic and diabetes is not.
Turning inputs into headline statistics is at least as bad, for similar reasons. Inputs might be detrimental to some measures of wellbeing, but useful for others. An index of health-related behaviours, counting exercise, diet, and various consumption choices would be a poor one to include in headline statistics on health. To the extent that those behaviours improve health, they are already captured in the healthy life expectancy statistic. But including them as a headline measure risks targeting them for improvement in ways that may hurt other measures of wellbeing.
And, finally, the outcome measures chosen should be things that the Government might plausibly have any business doing something about. We might wish to be careful about what gets measured lest we be managed into improving our wellbeing in ways we might not welcome.
To take an obvious example, I have yet to see a survey of wellbeing or happiness that, if it included satisfaction with one’s sex life as a measure, failed to find that it mattered considerably. If any participant in Statistics New Zealand’s workshops had been honest and reported that sexual fulfilment mattered greatly in their wellbeing, their input did not make it into the workshop. Really, if the number proved to be a bit softer than we might have wanted, what on earth should policy seek to do about it? All options seem atrocious.
Unfortunately, the process of outcome selection at the Statistics New Zealand event was less fruitful than it should have been.
About 150 people attended at the Michael Fowler Centre.
Two hours of introductory remarks included a welcome from the Government statistician, a hand-clapping game, an invitation to check our privilege with the help of a bingo-sheet of privilege indicators like not being red-haired, and an extensive discussion from a PhD candidate who warned (among other things) about consulting with Ngāi Tahu on issues Māori because they are too corporate.
The two hours of introductory remarks did not include any discussion of the difference between final outcomes, intermediate outcomes, and inputs. Nor did it point out that current measures of inputs and intermediate outcomes would continue to be collected regardless of whether they became headline statistics.
After being encouraged to reflect on my privileges in being able-bodied, not hard of hearing, having a loud speaking voice, and not being red-headed, I joined a group of about 50 people who all had to stand for an hour around an A3 poster to discuss the proposed health indicators – before moving on to stand and talk around other A3 posters. Those unable to stand for extended periods were denied effective participation, as were those even of normal hearing if they were not very close to some of the very soft-spoken participants. Microphones and chairs may not have gone amiss, but I am probably in too privileged a position to have standing to comment on that. Our facilitator did his best but had a difficult task.
Because nobody had explained the difference between inputs and outcomes, or that failure to select particular measures did not mean they would cease to be measured, conversations had to keep coming back to those basic points. Every worthy-sounding input measure had its proponent for inclusion in the mix.
And many of the indicators seemed, well, problematic. Some outcomes were listed as “contributing unambiguously to progress” when they seemed manifestly contestable. For example, the response to the Initiative’s report last year lauding the merits of diversity and immigration suggested not everyone agrees diversity is a good thing. But appreciation of diversity was included as a potential outcome variable deemed to contribute unambiguously to progress. It seems a political argument to win rather than an outcome variable to technocratically maximise. If it were unambiguously associated with progress, we would have received less hate mail insisting on the demerits of diversity and that New Zealand needs fewer immigrants from culturally dissimilar places.
A dangerous thing to attempt to measure
Most contentious in the health discussion was the inclusion of spiritual health as a desirable outcome. And it is there that the spirit of Sir John James Cowperthwaite visited me – as I am a very spiritual person in my own utterly non-spiritual way.
Sir John is generally credited with Hong Kong’s transformation from a desolate place at the end of the Second World War into about the richest and most economically free city in the world. He credited some of his success to his refusal to provide Whitehall with the statistics they might attempt to use in applying Atlee-style management to the Hong Kong economy. His policy of positive non-interventionism certainly outperformed the UK’s more scientific-looking approach built on a mountain of economic statistics.
His spirit warned me that spiritual health is a dangerous thing to even attempt to measure as a headline wellbeing outcome, even if it is deeply meaningful to many communities. There is the obvious problem that atheists like me would not have a clue how to begin answering a survey question on whether we are spiritually fulfilled as the area, to those who share my views, is meaningless. But there is the far worse problem that what gets measured may be managed. If a headline outcome measure of spiritual fulfilment were half of what people expected, or double(!), what on earth should the Government ever do about it? The measure invites management. And any attempt at management would be worse than the measured problem.
I do not envy Statistics New Zealand the task ahead of them. The workshop will not have been as useful as it could have been in selecting an appropriate set of indicators. Will they weigh more heavily what makes sense as outcome measures, or what participants claimed to want? Either way, they will be stuck with trying to measure the things – while being unsure whether much of the project will be superseded by Treasury’s work. All this while battling with what seems an ever-worsening problem in getting the Census out.
I wish Statistics New Zealand good spiritual health in the months ahead, as they will need it.
They took a fair bit of stick about having us fill out a Privilege Bingo form. I think that we wound up doing the privilege bingo exercise by accident. They had time to fill when a video screwed up, and rather than fill the time with something useful (see the column), the team scheduled to give that presentation scrambled for something tangentially related to their having asked lots of different communities for input into their data exercise.
They were utterly utterly oblivious to that they were stepping in very firmly on one side of the culture wars, perhaps because those involved were so ensconced in one side of it that they couldn't anticipate that it was problematic. If it hadn't been a late fill-in, maybe somebody with some nous would have headed it off. But it wasn't completely last minute. They had had time to have photocopied Privilege Bingo sheets on everyone's seats along with the day's agenda.
And today, the Census problems alluded to in my penultimate paragraph hit the front page of the Dom and Radio NZ.
I wonder about the usefulness of the community consultation exercise. It is very very easy for those things to make it very hard for people with potentially dissenting views to say anything. Remember the kinds of preference falsification exercises that Timur Kuran talked about, and information cascades? I wonder if anybody at Stats reads this kind of stuff.
If you're in a room where a couple people have just made very clear that the inclusion of spiritual wellbeing is more than just important to them, that they'd take it as a slight to their entire way of being and sense of self if anyone challenged that measure's inclusion (like, not explicitly, but in the way they make the argument), how many would be happy to step in and challenge it? At the poster session, we could put stickers on particular measures. Before discussion, there were about 10 red stickers on spiritual health, suggesting ten people wanted it demoted, and 2 green stickers, suggesting two people disagreed with the red stickers. I spoke against the inclusion of the measure, then a couple of people spoke about how sacred the measure was to them, and not a single other person who had put up a red sticker was willing to say a word.
If there's just been strong discussion of the importance of kaitiakitanga in land management, would you be the one willing to say "You know, my family settled our farm a hundred years ago and has worked and loved that land for four generations. Shouldn't we count too?" Like, the two can work together, but it is hard to get to that kind of discussion.
A friend gave an analogy that seems to fit. Suppose you've got a juggler who's really really good at juggling five balls. The business-as-usual stuff at Census is the five balls - there is a pile of it. In a Census year, they have to juggle six or seven balls. And they do a great job of that too, because they can manage six or seven for a short period. But throw another ball in the mix and everything starts falling down because eight is way harder than five. I wonder whether the Indicators project isn't that deadly eighth ball.
No comments:
Post a Comment