PBRF games

Martin Lally warned about contracting shenanigans meant to game New Zealand's academic research ranking exercise. Recall that while keeping "R" graded staff has no direct effect on a University's funding, the number of R staff falling into the denominator of the Average Quality Score will affect league tables; consequently, Universities faced no downside cost to ensuring that likely Rs are off the books on the census date.

The Tertiary Education Commission seems to have agreed with his critique.

People are keeping Rs off the books:

40 As part of the preparations for the 2012 Quality Evaluation, the TEC has contracted KPMG to undertake an audit of TEOs. The first stage of the audit process has indicated that there appears to be differences in Human Resource practices across TEOs. In particular, differences in the way in which TEOs are applying the staff eligibility criteria in relation to staff whose EPs are not likely to meet the standard for a funded Quality Category. 

41 The extent of these differences indicates that there is potential for these practices to affect the credibility of the published results of the 2012 Quality Evaluation, including the way in which TEOs are ranked (at both the overall organisational level, and by subject areas), and for the reporting of the relative quality of research by panel, subject area, and nominated academic unit generally.

42 In addition, the TEC has received information from TEOs which indicates that the overall number of PBRF-eligible staff whose EPs are unlikely to meet the standards for a funded Quality Category has declined significantly between 2006 and 2012.

Dave Guerin's excellent ED-insider [subscription, sorry; blog version here] notes some of the fun highlighted in the KPMG report: Otago deems a higher proportion of their staff as being PBRF-eligible than does Canterbury, but then manages to exclude them as being "under strict supervision".

TEC's preferred option is to drop Rs from the denominator for AQS calculation. This seems the most obvious tidy solution. They say universities would then have no incentive to game contracts. I wonder whether that's true. Where universities previously had strong incentive to keep Rs off the books, they'll now have some incentive to keep Cs off the books as every C necessarily brings down the average.

Incentives to game on Cs have to be weaker than gaming on Rs because C-ranked researchers still do bring in some incremental funding to the university.

We'd expect then that universities will crank through an optimization balancing the marginal increase in tuition income (foreign and domestic, with different expected elasticities) with a higher AQS against the marginal decrease in research income from dropping a C. We'd also expect that contractual shenanigans with more research-active staff are more expensive than those for less research-active staff, so total shenanigans ought to go down even in the absence of an income trade-off.

It's hard to imagine a fix that will be immune to gaming. The TEC-proposed one will improve things, but there will still be incentives to game.

Meanwhile, in Economics at Canterbury, we're likely to be punished for working harder to get our two teaching specialists research active; their becoming research active will now wind up hurting our AQS.  Our own fault for not responding to incentives as we should have and getting them officially off the books on census date.