Learning nothing in education

What a waste. New Zealand set up a great little policy experiment in the partnership schools. And now that model's ending without really having been evaluated.

From my column in last week's NBR:

A government with access to the world’s best microdata, and with a strong stated commitment to evaluation, simply failed to commission or undertake any real assessment of its experiment with partnership schools.

Whether or not the schools actually improved outcomes for the students they taught, an experiment without measurement and evaluation is a failure.

Not rocket science

It is not as though this kind of evaluation work is either rocket science or unprecedented. America’s thousands of charter schools spawned a small academic industry of rigorous assessment. By my rough count, there are about 80 published academic studies of charter schools for every 100 American charter schools – and that counts only those studies published in economics books and journals.

...

Evaluation should have been built into the initiative from day one.

Admission to charter schools should have incorporated a lottery element so that outcomes could have been more fairly compared.

But even absent lotteries, New Zealand’s rich administrative data provides ample opportunities for better evaluation. For every student who switched to a partnership school, an analyst might have found a statistically similar student who stayed with the initial public school – or who switched to a different public school. Outcomes like performance on externally invigilated NCEA standards could have been compared across the groups.

Or, even more simply, outcomes could have been compared for partnership school students and public school students after adjusting for all of the family background characteristics we know matter in educational performance.

But none of that was in the evaluation’s terms of reference. So you might not be surprised that there were no meetings of the ministry’s senior leadership team at which the report was discussed. There was nothing really to learn from the report.

David Seymour was the MP responsible for partnership schools under the previous government. When I asked why there had been no rigorous evaluation of the partnership schools, he was livid. He told me he had desperately wanted proper evaluation, and cited some good evaluation models, but said those efforts had been blocked. He had been told it was not necessary because a better evaluation framework was coming for all schools and so no partnership-specific evaluation was needed. But that framework never came.

The government conducted an educational experiment without learning. We do not know whether students at the different partnership schools did better, worse, or about the same as comparable students in traditional schools.

That is even worse than running an experiment and finding out that all of the schools failed. If they failed, and we knew that, we would at least know that. We might start teasing out what parts succeeded, and which ideas really did not pan out. But we do not even know that. Neither can we celebrate any successes among the schools that did improve outcomes for their students.

Some of that work can still be done. Data still sits with Statistics New Zealand that could help us tell which schools did well under a model that will have ended by the time any analyst looks at it.

On the plus side, I suppose, National has become great cheerleaders for the schools - after having set up a model that guaranteed they would be killed by an incoming Labour government.

A government that loved the model and didn't care about evidence would have just started with a huge broad roll-out, so that any future government killing it would face a lot more angry parents.

A government that was properly interested in the evidence would have set the model early in its administration and would have run evaluation with it all along the way, watching things like school drop-out rates, NCEA subjects chosen and achievement in them, and what happened to students after school - comparing like with like.

And a government that just kinda hated them but was forced into doing something by ACT would do basically what National did - set them up without any proper evaluation so it would be easy for the next government to roll them back.

We've an ungated version of the NBR piece here now.

Teacher-run schools

A couple of weeks ago, Rose Patterson wondered why the teacher unions don't start their own charter school as demonstration project: to show what can be done when they call the shots:

Recently, the Washington Post wrote an article titled “to improve schools, let teachers run them”, about 70 U.S. schools that are completely teacher run, where kids are engaged and achieving.

Here is the rub. They are charter schools.

Teacher unions see charter schools as a threat to their existence, but they could also provide some opportunities to improve the status of teachers in the public eye. Imagine, for example, a professional arm of the PPTA setting up a fund to sponsor a group of teachers to start New Zealand’s first teacher-led charter school.

Not all the teacher-led charters have been successes though:

A decade later, the union is closing the school. Capital New York has the details:

 [T]he U.F.T. charter has consistently been one of the lowest performing schools—charter or otherwise—in the city and has received stern warnings from its authorizer, the SUNY Charter School Institute, about its viability.

Last year, SUNY issued a report on the U.F.T. Charter School in which it documented instability in leadership, low test scores particularly in middle school grades, lack of resources and disciplinary issues. 

The school has been an embarrassment for the union from the get-go, starting with an unfortunate 2005 incident in which its principal ordered two boys to clean up another student's feces off the bathroom floor, which, of course, made the tabloids. Since then the school has been plagued by principal turnover, textbook and material shortages, and fiscal problems. There have been 10 reported incidents of corporal punishment.

This hardly damns the model, so long as the system can expeditiously identify failures and either fix or close them. But it does not speak well of the operator.

Benefits of competition

A few years ago, Jerry Hausman showed that Wal-Mart does a lot to benefit even consumers who don't shop there. When a Wal-Mart opens, competitor local supermarkets cut their prices to keep customers. And poor customers reap most of the benefits.

Figlio and Hart, in the latest AEJ: Applied Economics, show a similar effect with school vouchers. An ungated version is here.

Suppose your worry about school vouchers is that low social capital parents' stick with a local underperforming school while kids whose parents have better social capital all flee with their vouchers to the better private schools. And suppose further that you care way more about the potential losses to the former than about the gains for the latter. You might then oppose voucher systems.

Figlio and Hart show that public schools facing competitive pressure from private schools under a new voucher system provided stronger student score improvements. All that concern about kids left behind as the private schools cream off the best voucher kids? Not much of an issue if the public schools facing the competitive pressures perform better as consequence. They find the biggest positive effects in public schools facing strong financial incentives to retain low-income students.

Their identification strategy's pretty decent, exploiting the timing of the voucher roll-outs across the state. But do go have a look to see if they've accounted for your particular objections. And then update your priors.

Charter Schools

Coming up as part of Canterbury University's "What If" lecture series, a lecture from Mike Feinberg on KIPP and the American experience with charter schools. Here's Feinberg at Forbes.com on KIPP at Houston.

From the University's announcement:

What If charter schools could make a difference?

Date: Wednesday 26 September 2012
Time: 6:30 p.m. to 8:00 p.m.Location: University of Canterbury
Contact: For further information regarding this event, please contact Shandley Wenborn by sending email toshandley.wenborn@canterbury.ac.nz or by calling 03 364 2346
Audience: The general public

Presented by Dr Mike Feinberg

Partnership Schools/Kura Hourua – known internationally as charter schools – are one of the Government’s initiatives aimed at addressing student underachievement. As a relatively late adopter of the concept, New Zealand has the advantage of being able to learn from the successes and failures of models in other parts of the world.

Join us for a discussion on what KIPP, a US network of 125 high-performing charter schools, has been able to deliver in terms of educational achievement and new ways for parents and communities to be involved in their children’s education.

Dr Feinberg co-founded KIPP, is currently Executive Vice-Chair of KIPP Houston, and serves on the board of KIPP Foundation. He has won numerous awards for educational excellence and public service/social entrepreneurship.

I will be seeking leave to attend. Evening engagements often require side-payments. But this one's likely to be worth it. I'll hope to see Christchurch-based readers there.

NBER roundup [updated]

The late-night NZ-time twitter feed occasionally brings American morning delights. Tonight, it's the new NBER working papers. In the queue for when I'm back on campus and can read the NBER subscription papers:*

• Callison and Kaestner find tobacco consumption less price sensitive than previously thought; they reckon it would take a 100% tax increase to get a 5% drop in consumption. This would be estimated around American tax levels, which are rather below NZ ones; I'd need to back that out into price elasticity estimates to translate it into effects of NZ tax increases. From the abstract:
  

  ...we focus on recent, large tax changes, which provide the best opportunity to empirically observe a response in cigarette consumption, and employ a novel paired difference-in-differences technique to estimate the association between tax increases and cigarette consumption. Estimates indicate that, for adults, the association between cigarette taxes and either smoking participation or smoking intensity is negative, small and not usually statistically significant. Our evidence suggests that increases in cigarette taxes are associated with small decreases in cigarette consumption and that it will take sizable tax increases, on the order of 100%, to decrease adult smoking by as much as 5%.

  If that's right, Turia's tax increases are more regressive than we'd expected.
  
  Update: Oh wow. Read this bit from the full paper:

  Using this method, we found that for adult smokers ages 18 to 74, a 10% tax increase is associated with between a 0.3% to a 0.6% decrease in smoking participation and a 0.3% to a 0.4% decrease in smoking intensity. More surprisingly, given past research suggesting that youth smoking is more sensitive to taxes and prices, we find very little difference by age in the association between cigarette taxes and cigarette consumption. A 10% increase in state cigarette tax is associated with: between a 0.3% to a 0.7% decrease in smoking participation for those ages 18 to 34; between a 0.2% to a 0.4% decrease in smoking participation for those ages 35 to 54; and between a 0.3% to a 0.6% decrease in smoking participation for those ages 55 to 74. Similarly a 10% increase in state cigarette tax is associated with: between a 0.3% and a 0.5% decrease in smoking intensity for those ages 18 to 34; a 0.3% decrease in smoking intensity for those ages 35 to 54; and between a 0.3% and a 0.4% decrease in smoking intensity for those ages 55 to 74. Finally, standard errors of estimates are of a magnitude that rule out cigarette tax elasticities with respect to smoking participation (intensity) among adults greater (more negative) than -0.12 (-0.13).
  ...
  It is notable that estimates in Table 3 provide no evidence to support the hypothesis that smoking behavior is more responsive to taxes (prices) among younger persons than older persons.

  There may be effects in encouraging kids younger than 18 to avoid starting smoking; the analysis here is restricted to adults. But if this is right, it means that whatever benefits come from Turia's Tax will be over a very long time horizon while the costs on low decile households through reduced net-of-smoking disposable income will be very large for a rather long time. I probably ought to pull this up to being its own post. There are rather a few careful controls in here that need more discussion.


• Reyes gives more evidence that banning leaded gasoline, and other restrictions on environmental lead, was a very good idea. From the abstract:

  The paper finds that elevated levels of blood lead in early childhood adversely impact standardized test performance, even when controlling for community and school characteristics. The results imply that public health policy that reduced childhood lead levels in the 1990s was responsible for modest but statistically significant improvements in test performance in the 2000s, lowering the share of children scoring unsatisfactory on standardized tests by 1 to 2 percentage points. Public health policy targeting lead thus has clear potential to improve academic performance, with particular promise for children in low income communities.

  Reyes previously estimated that reductions in environmental lead can account for a 56% reduction in violent crime in the 1990s.
  
  Update: The paper gives some nice benchmarking of the effects of lead reduction: the improvement in test scores that came of the reduction in the proportion of low income kids with high blood lead concentrations would be comparable to the improvement in test scores you'd expect if per capita incomes improved by 15% in low income communities - a rather substantial effect.


• Hastings et al provide more evidence that kids winning lotteries allowing them to attend the school of their choice enjoy better outcomes. 


• Fergusson, Robinson, Torvik and Vargas set up a model testing an Orwellian idea: that leaders whose power is augmented by warmaking have little incentive to let the war end. They test against Colombian data. From the abstract:

  We find that after the three largest victories against the FARC rebel group, the government reduced its efforts to eliminate the group and did so differentially in politically salient municipalities. Our results therefore support the notion that such politicians need enemies to maintain their political advantage and act so as to keep the enemy alive.

  War is the health of the state...


• And, finally, Lacetera et al on compensation for marrow and organ donation. The abstract:

  In an attempt to alleviate the shortfall in organs and bone marrow available for transplants, many U.S. states passed legislation providing leave to organ and bone marrow donors and/or tax benefits for live and deceased organ and bone marrow donations and to employers of donors. We exploit cross-state variation in the timing and passage of such legislation to analyze its impact on organ donations by living and deceased persons, on measures of the quality of the organs transplanted, and on the number of bone marrow donations. We find that these provisions did not have a significant impact on the quantity of organs donated. The leave legislation, however, did have a positive impact on bone marrow donations. We also find some evidence of a positive impact on the quality of organ transplants, measured by post-transplant survival rates. Our results suggest that these types of legislation work for moderately invasive procedures such as bone marrow donation, but may be too low for organ donation, which is riskier and more burdensome to the donor.

  Becker and Elias reckoned it would take about $15k in compensation to encourage kidney donation. If the tax benefits added up to less than that, it would have been surprising if there had been large effects on live donation rates. The data appendices (free access; the article is gated) shows no state provided more than $10k as tax deduction. Note that a $10k tax deduction isn't $10k in hand: it's $10k that you get to remove from your taxable income total. So it's only worth $10k times your marginal tax rate - in other words, very unlikely to motivate donation from the cohorts more likely otherwise there to be price sensitive.
  
  Update: A few neat bits on seeing the paper rather than just the abstract:
  • Where some worry that paying for organs worsens quality, the authors found instead weak evidence of quality improvement.
  • The authors seem to have reached the same conclusion: the payment levels via tax deductions are likely below the reservation price for live kidney donation.

* I've only caught the abstracts of these thus far; if there are grievous errors in method that aren't obvious from the abstracts, my apologies.

NZBR Blog

I miss Roger Kerr's blog.

The New Zealand Business Roundtable has resumed blogging, though, at Policy Matters.

Roger Partridge points to some of the NZBR's resources on school choice and to a 2006 report by Caroline Hoxby for the New Zealand Education Forum.

Great to see the NZBR back to blogging; I've added Policy Matters to my RSS feed.

School quality and crime

David Deming argues in the Quarterly Journal of Economics (HT: @Crime_Economist) that high risk kids experienced lower likelihood of later criminal activity if they won random lotteries allowing them to attend their first choice public school.

How does it work? High-risk youths spread out over more schools rather than being concentrated in a few bad schools. If peer effects work with thresholds or if there are increasingly negative peer effects as concentrations of high-risk kids increase, this makes a lot of sense.

Lessons for us in New Zealand?

• The lottery design in North Carolina resulted in higher-risk youths being more, not less, likely to flip into competitive schools. This cuts against arguments that school choice and charters necessarily result in pools of worse kids left behind. 
• The gains from moving to charters here might be rather more extensive than just educational outcomes. This is also consistent with evidence from things like the Perry Preschool project where educational outcomes washed out over time but social outcomes, like crime reduction, persisted.

Here's an ungated working paper version.

For and against charters

It being useful to develop the best possible argument against a position you support, here's my best case against New Zealand moving towards the use of charter schools. I'm not convinced by this case. Or, at least I don't think I am. Let's see if I've convinced myself by the end of the argument.

- Begin Advocatus Diaboli -

First, we have to recognize that the New Zealand school system is, on the whole, pretty good as compared to the American school system. Not always and everywhere, but on average. That isn't to say that we can't have improvements. But we have to keep in mind that a lot of the gains found in charter schools in the U.S. are measured against a counterfactual of abysmal urban public schools that compare poorly with some of Dante's circles of hell. Here's a fine description of one such nightmare case. That the enthusiastic young teacher there was able to flip over to a charter school, along with some of the students, produces real gains, both for the students who flip, and for the average quality of those engaged in teaching. The worst US public schools drive the best teachers out of teaching; the Charters give them an option where those who'd rock the boat against horrible school principals and stultifying union rules can flourish.

But are any New Zealand public schools really that bad? If the worst New Zealand public schools don't come within cooie of those American schools, we'd be overestimating the potential gains here from adopting charters. Sure, we can find a few isolated cases where the school principal condoned or covered up horrible bullying. But the worst here just doesn't match the worst there (though I invite readers to show me otherwise).

Further, it's really hard to pin down the gains from American charter schools. Sure, those quasi-experimental studies I cited yesterday are nice. But there's a meta-selection issue: the only schools that use lottery for entry are the really successful charter schools that are oversubscribed. You can identify the gains to the students who make it from the public schools into those schools by that method, but you shouldn't extrapolate from those gains to the average potential gains from widespread use of charters. Then, you have to go to some of the estimates of the overall effects of charter schools, and the empirics there are, frankly, a mess. Selection issues are all over the place. Where do new charters open? Where the worst public schools are as that's where parents want to get their kids out. Which kids flip? Those whose parents care more. Taking average cohort performance across public and charter schools doesn't work if charters disproportionately pick up the better kids from the worst areas, but it's not immediately clear whether that helps or hurts charters' performance stats. And so we find all kinds of studies showing little effect in the aggregate, or strong heterogeneity in results across different charters.

Finally, many of the gains from charters are already available in New Zealand in the form of Integrated Schools - state-funded schools established for a special purpose or with a special mission, but that are restricted to cover the national curriculum. Those aren't impossible to open. What's then gained by allowing entry to those who would perhaps eschew the national curriculum in favour of weird stuff? Yes, it's possible to improve on NCEA, and the best charters will. And, sure, there are already religious integrated schools using public funds to provide a religious message. Brian Tamaki's Destiny Church even runs one. But do we really want Tamaki released from requirements to nominally adhere to the national curriculum?

In the best case, charter schools open in neighbourhoods where zoning keeps poor kids stuck in underperforming schools, parents send their kids there, and public schools are forced to lift their game or suffer enrolment drops, funding drops, and downsizing or closure. And so everything's great. But the more likely case, depending entirely on how the regulations are set up, is that those charters pull the better students from the worst schools. Those better students do fare better in the charter schools. But those left behind fare worse as the public schools remain the residual claimant on the worst cases - the disruptive students that the public school system, for better or worse, cannot help but keep integrated into mainstream classrooms where they ruin outcomes for others. Those students become a greater proportion of those left in the public schools, further depressing outcomes for decent students whose parents just don't care enough to send them over to the charter school; cohort effects matter. Those best students would have made it through regardless, but we worsen outcomes for the middling students of indifferent parents.

- End Advocatus Diaboli -

What do I take from this? Upside gains here, relative to those in the States, are likely to be limited. And, as Seamus noted, it can be tough to extrapolate from US studies to here. But particularly bad outcomes seem unlikely unless we make strong complementary assumptions about parental indifference / parental stupidity and about public school inflexibility in response to the competitive threat.

There will be here, as there are in the States, some charter schools that fail. But even the failures let us learn. Roland Fryer dug into the correlates of charter school success (Freakonomics summary here) and found results a bit divergent from usual teachers' union requests. Input-based solutions like smaller class sizes and higher proportions of staff with teaching certification don't do much. Instead, "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement" explains most differences in outcomes across charter schools. Perhaps it's possible to mandate that state-sector schools here simply adopt best practice. But I worry that that's best-case thinking; the government barely seems able to compel its public schools to disclose student achievement statistics. The threat of student exit seems the best long-term discipline for poorly performing schools. And, as Seamus noted, there's a lot of value in the experimentation to see what methods work here. I find it very plausible that charter schools drawing on tikanga will find more appropriate instruction methods for lower decile Maori groups, and that other approaches will be successful in poorer Pacific Island communities. But we won't know until we try.

More on Charter Schools

I have no doubt that John Pagani is wrong about the charter schools proposal. My fear is that Banks and Key are making the same mistake.

Pagani starts off with similar criticisms to Dim-Post linked to by Eric yesterday: Charter schools will cherry pick the best students and then use raw achievement results to trumpet their success; they will charge top-up fees to provide be able to afford better resourced schools, etc. He may be right, he may not be; the devil is in the details that have not been worked out yet. I hope he is wrong, but I will avoid cheering the policy before the details are known. But this is not why I say that Pagani is clearly wrong. It is this comment that misses the point:

The problem is this: If there is something that a charter school can do better than a state school, why don't we make the state school do it?

This is where I fear that Key and Banks are making the same mistake. The point is not that there is clear evidence that there is an alternative way of doing things that are better and that charter schools will deliver on this. Eric has pointed to useful U.S. studies, but we don't know anything in a New Zealand context. We simply don't know what will work, and experimenting with different approaches will give us useful information. So here is my plea to all the parties involved:

1. Please, all read Tim Harford's Adapt.
2. Involve competent social scientists in the design of the policy, with a view to making sure that useful data is available by which different schools and the system as a whole can be judged. This will involve both making sure that there is some random assignment of students to schools, and also making sure that similar data is collected in areas where the system isn't trialed to tease out whether any differences in outcome between charter and state schools is because of an effect on both with no change in the aggregate. 
3. To the Government: make it your stated policy that this is a trial and that you are not committed to the model if it doesn't seem to work;
4. To the Opposition: make it your stated policy that you accept that this is a trial and that, while you have doubts, you are interested in investigating anything that might help our education system, and will keep an open mind while waiting to see outcome data.

Charter Schools

The coalition agreement between ACT and National has provision for trial charter schools in South Auckland and Christchurch:

With respect to education, the parties have, in particular, agreed to implement a system, enabled under either sections 155 (Kura Kaupapa Maori) or 156 (Designated character schools), or another section if appropriate, of the Education Act, whereby school charters can be allocated in areas where educational underachievement is most entrenched. A series of charters would initially be allocated in areas such as South Auckland and Christchurch. Iwi, private and community (including Pacific Island) groups and existing educational providers would compete to operate a local school or start up a new one. Schools would be externally accountable and have a clearly-defined, ambitious mission. Public funding would continue to be on a per-child basis. (Details are included in the attached Annex)

So, what evidence do we have on the performance of charter schools in the States?

First, a note of caution in all the American studies - charter schools operate under regulatory constraints and vehement opposition from American teachers unions, both of which may affect observed results. Further, suppose a charter school opens in a neighbourhood and you find higher test scores among the students who go there compared to those who stay at the public school. Are the results due to differences in school quality, or selection effects? Two kinds of selection effects will be relevant - schools may try to get the best students, and high quality parents may be more likely to flip their kids out of failing schools. DimPost seems to think everything's due to selection; it would have been more helpful had he hit Google Scholar.

There are ways of controlling for selection effects. Here's Josh Angrist and coauthors from 2009, HT: Marginal Revolution. They used a lottery as random assortment mechanism: Boston charter schools were oversubscribed, so a lottery was used to determine which kids got in. So Angrist can compare those who won lotto with those who didn't. Remember, this is Josh-freaking-Angrist. The econometrics here aren't going to be wrong.

Charter schools are publicly funded but operate outside the regulatory framework and collective bargaining agreements characteristic of traditional public schools. In return for this freedom, charter schools are subject to heightened accountability. This paper estimates the impact of charter school attendance on student achievement using data from Boston, where charter schools enroll a growing share of students. We also evaluate an alternative to the charter model, Boston's pilot schools. These schools have some of the independence of charter schools, but operate within the school district, face little risk of closure, and are covered by many of same collective bargaining provisions as traditional public schools. Estimates using student assignment lotteries show large and significant test score gains for charter lottery winners in middle and high school. In contrast, lottery-based estimates for pilot schools are small and mostly insignificant. The large positive lottery-based estimates for charter schools are similar to estimates constructed using statistical controls in the same sample, but larger than those using statistical controls in a wider sample of schools. The latter are still substantial, however. The estimates for pilot schools are smaller and more variable than those for charters, with some significant negative effects.

Upshot: charter schools do better, and because they're outside of collective bargaining and because they face the risk of shutting down if parents don't choose to send their kids there.

Other evidence?

• In another randomized trial, Caroline Hoxby finds substantial gains for students winning entry to charter schools in New York. Here's an ungated version of the paper. Here's the report.
• New Orleans moved heavily towards charter schools after Katrina; outcomes improved.

Here's Caroline Hoxby explaining her work.

Caroline Hoxby: The Promise and Performance of Charter Schools from The Hoover Institution on FORA.tv

I'd not be surprised if charter schools weren't already in National's plans, but it's still a nice win for ACT.

And it's great fun to watch all those who rallied for MMP now whining about post-election coalition deals. You guys should have ticked the box for FPP.

When you go out to do your own literature search, be sure to upweight results from studies that control for selection effects either by this kind of randomized lottery treatment, or by instrumental variable approaches (like this one). Put less weight on studies that just compare charter and regular schools without addressing selection issues.