From my column in last week's NBR:
A government with access to the world’s best microdata, and with a strong stated commitment to evaluation, simply failed to commission or undertake any real assessment of its experiment with partnership schools.On the plus side, I suppose, National has become great cheerleaders for the schools - after having set up a model that guaranteed they would be killed by an incoming Labour government.
Whether or not the schools actually improved outcomes for the students they taught, an experiment without measurement and evaluation is a failure.
Not rocket science
It is not as though this kind of evaluation work is either rocket science or unprecedented. America’s thousands of charter schools spawned a small academic industry of rigorous assessment. By my rough count, there are about 80 published academic studies of charter schools for every 100 American charter schools – and that counts only those studies published in economics books and journals.
...
Evaluation should have been built into the initiative from day one.
Admission to charter schools should have incorporated a lottery element so that outcomes could have been more fairly compared.
But even absent lotteries, New Zealand’s rich administrative data provides ample opportunities for better evaluation. For every student who switched to a partnership school, an analyst might have found a statistically similar student who stayed with the initial public school – or who switched to a different public school. Outcomes like performance on externally invigilated NCEA standards could have been compared across the groups.
Or, even more simply, outcomes could have been compared for partnership school students and public school students after adjusting for all of the family background characteristics we know matter in educational performance.
But none of that was in the evaluation’s terms of reference. So you might not be surprised that there were no meetings of the ministry’s senior leadership team at which the report was discussed. There was nothing really to learn from the report.
David Seymour was the MP responsible for partnership schools under the previous government. When I asked why there had been no rigorous evaluation of the partnership schools, he was livid. He told me he had desperately wanted proper evaluation, and cited some good evaluation models, but said those efforts had been blocked. He had been told it was not necessary because a better evaluation framework was coming for all schools and so no partnership-specific evaluation was needed. But that framework never came.
The government conducted an educational experiment without learning. We do not know whether students at the different partnership schools did better, worse, or about the same as comparable students in traditional schools.
That is even worse than running an experiment and finding out that all of the schools failed. If they failed, and we knew that, we would at least know that. We might start teasing out what parts succeeded, and which ideas really did not pan out. But we do not even know that. Neither can we celebrate any successes among the schools that did improve outcomes for their students.
Some of that work can still be done. Data still sits with Statistics New Zealand that could help us tell which schools did well under a model that will have ended by the time any analyst looks at it.
A government that loved the model and didn't care about evidence would have just started with a huge broad roll-out, so that any future government killing it would face a lot more angry parents.
A government that was properly interested in the evidence would have set the model early in its administration and would have run evaluation with it all along the way, watching things like school drop-out rates, NCEA subjects chosen and achievement in them, and what happened to students after school - comparing like with like.
And a government that just kinda hated them but was forced into doing something by ACT would do basically what National did - set them up without any proper evaluation so it would be easy for the next government to roll them back.
We've an ungated version of the NBR piece here now.
You are assuming that the reason for charter schools being set up was to improve education. Since that was never the real reason there was no need to evaluate them. If there was evaluation they may well have been shown to be failing, and that would never do.
ReplyDeleteNational brought them in because ACT wanted them. ACT wanted them to break up public schooling as it currently stands (and trashing the school unions along the way as a bonus). National also found them useful for burnishing their credentials with Maori.
Rolling them out big time was never going to work, because other than Maori no-one much was going to enrol in tiny new schools trying untried techniques. ACT, dogmatic on this, never got their heads round public schools are not unpopular. If they'd pushed for old-fashioned single-sex traditional discipline schools the public would have bought in big time. (And the Ministry would have gone mental. They hate the fact that few parents like modern "progressive" education.)
Improved education would be a useful by-product only of charter schools.
I'd love to see a real evaluation too, but finding someone unbiased in this area is a difficult task.