Marsden Maths

Motu's released the working paper they'd presented at the NZAE meetings earlier this year. They show that Marsden grants do increase research output. But as for whether the programme is cost effective? Well, that's more fun.

The average researcher on teams receiving Marsden grants made 6 proposals and received 1.2 grants over the period 2000-2012. The Fund allocated just under $68 million in 2013; the standard grant's maximum budget is $300,000 per year while Fast Start grants are limited to $100,000 per year. Panels reject 71-84% of first round proposals. The Motu analysis looks at second round proposals, 41% of which were funded.

Results? Here's the big headline chart from their press release.

So about the biggest effect that they found was that a researcher who received two successful research grants received a 10% increase in publications and a 14% increase in citations. In the regression discontinuity design, comparing project teams whose proposals just made the cut with those that just didn't, they found no effect.

Their abstract's version:

Overall, we find that funding is associated with a 6-15% increase in publications and a 22-26% increase in citation-weighted papers for research teams. For individuals, funding is associated with a 3-5% increase in annual publications, and a 5-8% increase in citation-weighted papers for 5 years after grant; however, the lag structure and persistence of this effect post-grant is difficult to pin down. Surprisingly, we find no systematic evidence that the evaluation of proposals by the Marsden system is predictive of subsequent success. We conclude that the Marsden Fund is modestly successful in increasing scientific performance, but that the selection process does not appear to be effective in discriminating among second-round proposals in terms of their likely success.

An annual 8% increase in the number of citation-weighted papers for five years after a grant might amount to one full paper over the entire time period. A research grant costing upwards of a hundred thousand dollars per year seems a pretty costly way of generating one citation-weighted paper.

It would be ... interesting ... to compare the cost-effectiveness of Marsden with an alternative scenario in which academics who maintained a reasonable publication record received an unconditional grant and the money spent on Marsden-related administration went instead into hiring more academics. There have to be at least 30 full-time equivalent academic-years that go into Marsden grant writing and grant evaluation.