School heterogeneity

New Zealand schools have not been particularly pleased with the government's plans to release data on student performance that would allow for the construction of league tables; they've argued that heterogeneity in incoming student characteristics produce unreliable results.

A new NBER working paper by Ellison and Swanson suggests that, controlling for these kinds of incoming demographic characteristics, US schools are highly heterogeneous in producing top students. They set up production functions for top math students: those scoring very well on the AMC 12 exam. As expected, there are pretty strong demographic correlates of high-scoring schools: zip codes with more highly educated parents and with more Asian-American students produce higher AMC-12 results. But within sets of schools with similar demographic characteristics, there's still strong heterogeneity in production of top students. 

Our results suggest that there is a lot of variation among seemingly similar schools. The most notable feature of this variation is a thick upper tail of schools in which students are many times more likely to reach high achievement levels than are students in the typical school with similar demographics. This thick upper tail is present in all of our analyses of students with high AMC scores, but is not present when we look at where students with high SAT scores are coming from. This contrast suggests that that the thick tail is not because of the self-selection of high-ability students into a particular subset of schools. We suggest that a potential explanation is that almost all schools see it as their responsibility to provide English and math courses that cover material necessary to do well on the SATs, whereas there is much less uniformity in whether schools encourage gifted students to develop more advanced problem solving skills and reach the higher level of mastery of high school mathematics needed to do well on the AMC.

Relative to the literature on the gender gap in mathematics, our comparison of school effects relevant to girls suggests that schools are perhaps even more important for girls: we estimate that the 99th percentile high school in our sample is producing high-scoring girls at more than ten times the rate of an average school with comparable demographics. We also note that there are many low-performing schools that will only very rarely have girls reach the AMC performance levels we have studied. [emphasis added]

...

Our finding that there appears to be a lot of variation across schools with similar demographics could be seen as hopeful: the number of high-achieving students would increase substantially if low-achieving schools could be brought up to average; and upper-tail schools might have programs that could be emulated to produce even larger improvements. 

If it's impossible to tell which schools are high- or low-achieving given the demographic characteristics of their incoming cohort of students, it's harder to encourage the low-performers to improve.