Tuesday, 14 August 2012

School heterogeneity

New Zealand schools have not been particularly pleased with the government's plans to release data on student performance that would allow for the construction of league tables; they've argued that heterogeneity in incoming student characteristics produce unreliable results.

A new NBER working paper by Ellison and Swanson suggests that, controlling for these kinds of incoming demographic characteristics, US schools are highly heterogeneous in producing top students. They set up production functions for top math students: those scoring very well on the AMC 12 exam. As expected, there are pretty strong demographic correlates of high-scoring schools: zip codes with more highly educated parents and with more Asian-American students produce higher AMC-12 results. But within sets of schools with similar demographic characteristics, there's still strong heterogeneity in production of top students. 
Our results suggest that there is a lot of variation among seemingly similar schools. The most notable feature of this variation is a thick upper tail of schools in which students are many times more likely to reach high achievement levels than are students in the typical school with similar demographics. This thick upper tail is present in all of our analyses of students with high AMC scores, but is not present when we look at where students with high SAT scores are coming from. This contrast suggests that that the thick tail is not because of the self-selection of high-ability students into a particular subset of schools. We suggest that a potential explanation is that almost all schools see it as their responsibility to provide English and math courses that cover material necessary to do well on the SATs, whereas there is much less uniformity in whether schools encourage gifted students to develop more advanced problem solving skills and reach the higher level of mastery of high school mathematics needed to do well on the AMC.
Relative to the literature on the gender gap in mathematics, our comparison of school effects relevant to girls suggests that schools are perhaps even more important for girls: we estimate that the 99th percentile high school in our sample is producing high-scoring girls at more than ten times the rate of an average school with comparable demographics. We also note that there are many low-performing schools that will only very rarely have girls reach the AMC performance levels we have studied. [emphasis added]
Our finding that there appears to be a lot of variation across schools with similar demographics could be seen as hopeful: the number of high-achieving students would increase substantially if low-achieving schools could be brought up to average; and upper-tail schools might have programs that could be emulated to produce even larger improvements. 
If it's impossible to tell which schools are high- or low-achieving given the demographic characteristics of their incoming cohort of students, it's harder to encourage the low-performers to improve. 


  1. Wow, just wow. Mind you, it is astounding how differently schools treat maths, at least for girls. It would be great to have some metrics that are both clear and useful.

  2. I want a league table of regression residuals where the specification gets outcomes conditional on all the demographics.

  3. What gets measured gets done.... And I expect that while the measurement isn't that great at the moment because its being done, the quality of measuring will also improve - can't wait for those residuals!

  4. Eric,
    as you know the Christchurch Press, and all newspapers NZ, prints the results every year, on a performance basis. Shift that performance basis and play games for idiot parents but the same students will perform.
    straight line graph

  5. Most performance will come down to inherent student quality, sure. But the study noted does show substantial differences in school performance even taking that into account.