Low stakes PISA

New Zealand's low PISA rank seems, in part, due to Kiwi students not taking the test very seriously.

Akyol, Krishna and Wang develop a measure checking whether a student is taking the PISA test seriously (leaving questions blank while having time left, for example), and see whether it affects country rankings compared to simulations where children take the test seriously. [HT: Marginal Revolution].

New Zealand's PISA rank is 17th for the year they're checking. In a simulated world in which students in every country took PISA seriously, New Zealand's ranking would rise to 13th. If New Zealand students stayed as they are but all other countries moved to take PISA seriously, we'd fall to 26th in the simulations. And if only New Zealand moved to have all students take PISA seriously, we'd rise to 10th.

The change for New Zealand is substantial. Portugal would see the largest PISA rise in a simulated world where all students took PISA seriously - an increase of five places, from 31st to 26th. New Zealand's four-place rise from 17th to 13th ties with Russia's rise from 39th to 35th as second highest. Germany, Switzerland and Sweden would all rise three places, Japan would rise by two places, and seven countries would rise by one place.

Singapore's top position would be unaffected in any state of the world.

Unfortunately, this is only a single-year snapshot so we can't say whether it explains any of the decline in NZ's PISA rankings. We'd need a similar simulation for earlier PISA years.

This could be a fixed-effect that has always obtained for slacker Kiwi students. In that case, the PISA decline is real and unaffected.

It could be an effect that's not as bad now as it used to be. The paper suggests that, in countries with a lot of high-stakes testing, test fatigue may have students treating PISA less seriously. Kiwi kids up to that age wouldn't have seen a lot of high-stakes testing, but would have in earlier waves of PISA.* If there's been a decline over the period in the frequency of high-stakes testing, that would suggest more test fatigue in the earlier era. According to the paper, that would mean even more unseriousness in the prior era, so our earlier scores higher scores would be adjusted upward even more than our current scores.

Or you could imagine that the lack of high-stakes testing would have kids just not even thinking that it might matter to put in the best quick guesses on the last questions if they're short on time - in which case the decline could be due in part to kids now generally not knowing how to write tests. That would be consistent with observations of cohorts of incoming university students that seemed to get worse at test-writing.

Whatever the case, it does suggest that New Zealand could improve its ranking by rewarding kids for doing well on their PISA tests.


* The authors don't provide their coding of whether countries are counted as high-stakes or low-stakes.