Tuesday 6 December 2011

Charter Schools

The coalition agreement between ACT and National has provision for trial charter schools in South Auckland and Christchurch:
With respect to education, the parties have, in particular, agreed to implement a system, enabled under either sections 155 (Kura Kaupapa Maori) or 156 (Designated character schools), or another section if appropriate, of the Education Act, whereby school charters can be allocated in areas where educational underachievement is most entrenched. A series of charters would initially be allocated in areas such as South Auckland and Christchurch. Iwi, private and community (including Pacific Island) groups and existing educational providers would compete to operate a local school or start up a new one. Schools would be externally accountable and have a clearly-defined, ambitious mission. Public funding would continue to be on a per-child basis. (Details are included in the attached Annex)
So, what evidence do we have on the performance of charter schools in the States?

First, a note of caution in all the American studies - charter schools operate under regulatory constraints and vehement opposition from American teachers unions, both of which may affect observed results. Further, suppose a charter school opens in a neighbourhood and you find higher test scores among the students who go there compared to those who stay at the public school. Are the results due to differences in school quality, or selection effects? Two kinds of selection effects will be relevant - schools may try to get the best students, and high quality parents may be more likely to flip their kids out of failing schools. DimPost seems to think everything's due to selection; it would have been more helpful had he hit Google Scholar.

There are ways of controlling for selection effects. Here's Josh Angrist and coauthors from 2009, HT: Marginal Revolution. They used a lottery as random assortment mechanism: Boston charter schools were oversubscribed, so a lottery was used to determine which kids got in. So Angrist can compare those who won lotto with those who didn't. Remember, this is Josh-freaking-Angrist. The econometrics here aren't going to be wrong.
Charter schools are publicly funded but operate outside the regulatory framework and collective bargaining agreements characteristic of traditional public schools. In return for this freedom, charter schools are subject to heightened accountability. This paper estimates the impact of charter school attendance on student achievement using data from Boston, where charter schools enroll a growing share of students. We also evaluate an alternative to the charter model, Boston's pilot schools. These schools have some of the independence of charter schools, but operate within the school district, face little risk of closure, and are covered by many of same collective bargaining provisions as traditional public schools. Estimates using student assignment lotteries show large and significant test score gains for charter lottery winners in middle and high school. In contrast, lottery-based estimates for pilot schools are small and mostly insignificant. The large positive lottery-based estimates for charter schools are similar to estimates constructed using statistical controls in the same sample, but larger than those using statistical controls in a wider sample of schools. The latter are still substantial, however. The estimates for pilot schools are smaller and more variable than those for charters, with some significant negative effects.
Upshot: charter schools do better, and because they're outside of collective bargaining and because they face the risk of shutting down if parents don't choose to send their kids there.

Other evidence?
  • In another randomized trial, Caroline Hoxby finds substantial gains for students winning entry to charter schools in New York. Here's an ungated version of the paper. Here's the report.
  • New Orleans moved heavily towards charter schools after Katrina; outcomes improved.
Here's Caroline Hoxby explaining her work.

Caroline Hoxby: The Promise and Performance of Charter Schools from The Hoover Institution on FORA.tv

I'd not be surprised if charter schools weren't already in National's plans, but it's still a nice win for ACT.

And it's great fun to watch all those who rallied for MMP now whining about post-election coalition deals. You guys should have ticked the box for FPP.

When you go out to do your own literature search, be sure to upweight results from studies that control for selection effects either by this kind of randomized lottery treatment, or by instrumental variable approaches (like this one). Put less weight on studies that just compare charter and regular schools without addressing selection issues.

12 comments:

  1. I think DiM may be right in identifying selection bias as the cause for the different results.

    The teachers selected by charter schools (or more likely, teachers who select charter schools as a place to work) are probably better than average.

    And the leaders of charter schools probably select a better than average expenditure plan.

    ReplyDelete
  2. There are ways of controlling for selection effects. Here's Josh Angrist and coauthors from 2009, HT: Marginal Revolution. They used a lottery as random assortment mechanism: Boston charter schools were oversubscribed, so a lottery was used to determine which kids got in. So Angrist can compare those who won lotto with those who didn't.

    There are ways of attempting to control for selection. In this case, the lottery losers presumably had parents who were motivated and keen to get them out of the public school system and who also presumably were profoundly disappointed that their kids were rejected back into a school system they fully expected to do a poor job. The lottery winners' parents presumably were also motivated and keen to get their kids out of the public school system, felt like they were making a fresh new start on the road to educational success and sent their kids off in the full expectation that they were going to do well. If Angrist has a way to control for that, his paper would be very interesting.

    ReplyDelete
  3. @Kimble: I don't think that's evidence against charter schools' efficacy.

    @Psycho: Unclear that that isn't just attenuation bias.

    ReplyDelete
  4. @Eric: mmm, maybe it is evidence of the opposite?

    ReplyDelete
  5. Eric: I haven't read the links, but I didn't find your reactions to the commenters above all that convincing.

    If Kimble's first point is right (teachers at charter schools are of above average quality), this means that positive results shouldn't be expected to scale up, unless you think that the nationwide introduction of charter schools would magically improve average teacher quality.

    If by attenuation bias you mean bias towards zero introduced by random noise in the predictor variable, that doesn't describe Psycho's point at all. Rather, his point (if I understand it correctly) is that treatment is confounded with parent enthusiasm/disappointment and that effects of the latter would incorrectly be ascribed to treatment in these designs. One might also tell the opposite story (lottery winning parents scale down their own efforts to help their children, etc.), but it's certainly a point worth considering.

    Perhaps also of interest, here's a post on such studies' limitations and here's an anecdote about how "random" assignment may not actually be random.

    ReplyDelete
  6. @Lemmus: It scales to the extent that charters are able to draw better talent into teaching. I've seen decent argument that the current system doesn't do well on that front.

    But you're right on Psycho's point - I got that wrong. I'm not sure how big an effect it would be, but definitely doesn't attenuate the measured effects.

    The best point, I think, in the two links you sent is the non-random selection of which schools wind up being the ones running lotteries; that would mean results are limited to those schools able to emulate what that cohort of schools is doing. It doesn't cut against the point that charter schools' success results from more than just selecting the best incoming students, unless your linked anecdote is widespread practice, but it does suggest that adding up across those students is a high upper bound on system gains. Unless other schools are able to emulate the successes of the best schools (which, I think, is part of Hoxby's point)

    ReplyDelete
  7. @Lemmus: Read the last paragraph of this piece especially for how charters can increase average teacher quality system-wide.
    http://www.city-journal.org/html/13_1_how_i_joined.html

    ReplyDelete
  8. Yup, exit options, even if not exercised, must matter a lot. As for the best point in the links, I would guess that it's the point about peer effects. To simplify, if there were only two kinds of students, "motivated" or "not motivated", I would not be surprised about huge effects on a student in question of whether he's in a class with 80% or 20% motivated other students.

    ReplyDelete
  9. I haven't got an opinion on charter schools, but as far as I can tell their desirability remains an empirical question - and so far you're the only person to put up any evidence.

    Until the people who keep posting newspaper articles that don't like the schools actually start showing real evidence of their position, I'm not going to complain about the schools being trialed.

    ReplyDelete
  10. @Matt: There are tons of studies showing little effect, tons of studies showing very heterogeneous effects (some charters are great, others not). But it's damned hard to draw conclusions out of the literature. Why?

    1. A bunch of these studies are hit pieces funded by the teachers' unions and need to be treated with caution;

    2. Parsing out effects is a nightmare. Even the quasi-experiments above aren't perfect; Lemmus lists some of the potential reasons. I worry less about the peer effects one than he does because that's one of the points of Charters - that they can and do kick out students who are disruptive and make sure that they can't wreck other students' experiences.

    ReplyDelete
  11. In my casual, sporadic reading of the charter school lit, it seems like sometimes they are positive and sometimes not, in some places for some pupils. So, the policy question facing us isn't 'are they good?', but 'how do we make them good for these pupils?'
    That is, if we want to make schools good for children, rather than sacrificing them on the altar of [fill in the blank].

    ReplyDelete
  12. Then you want to read Roland Fryer's work on the characteristics of good charter schools.

    ReplyDelete