Marsden Fund under the microscope – Expert reaction

New Zealand’s leading ‘blue skies’ research fund boosts Kiwi science, but could be tweaked for greater efficiency, says a new study.

An evaluation conducted by researchers at Motu Economic and Public Policy Research institute has found that Marsden funding increases the scientific output of the funded researchers.

Compared to similar groups that do not receive funding, a team that is given Marsden funding shows a 6-12 percent increase in their academic publications and a 13-30 percent increase in the papers that cite their work.

Before a decision is made to fund a project, it must go through two rounds of evaluation. The new study asserts that “the significant resources devoted to the second round evaluation could be reduced without degrading the quality” of funded research.

“We were very fortunate to have access to all the funded and unfunded proposals, including their evaluation scores. This meant we could control statistically for potential bias driven by the Fund’s efforts to fund projects that are expected to be successful,” said lead researcher Dr Adam Jaffe. “The Marsden Fund and the Royal Society should be commended for making these data available, something very few research funding agencies do.”

The Science Media Centre collected the following expert commentary from New Zealand researchers. Feel free to use these quotes in your reporting.

Prof Juliet Gerrard, Chair of the Marsden Fund Council, said in a statement:

“We welcome the study and the central result that receiving Marsden funding boosts publications, and the impact of those publications.

“The Marsden Fund has a unique dataset of successful and unsuccessful proposals from many years of operation and was happy for this to be used to see whether funding processes can be improved.

“We are not surprised to find that all the proposals in the second round are of very high quality, and have for many years said that we could double the number funded without dropping quality. This study backs our claim and suggests that the first round process is selecting high quality proposals for full review.

“Publication numbers and citations are only one proxy for quality, and Marsden seeks to fund the very best New Zealand talent to do cutting edge research.  We expect a small number of these grants to make an extraordinary difference.

“We believe that the second round peer review by international referees and a panel of experts, is critical to maintaining rigor in the process and is the method most likely to spot proposals that have something really special.

“We continually seek to improve our processes by benchmarking against other funds.  We note that the HRC has trialled a lottery system in their HRC Explorer grants. It would be good to compare this system to the traditional peer review system which they use for their other grants. At the moment, the confidence given by external peer review, particularly in differentiating the ‘truly excellent’ from the ‘very good’ (who might have similar paper output metrics) is critical to the current success of the Marsden Fund.”

Prof Sir Peter Gluckman, the Prime Minister’s Chief Science Advisor,  comments:

“It is important that we study how science systems operate. I have written before about the challenge of peer review systems and whether the traditional model is the best approach. Internationally, a number of new models of peer review are being explored. Regardless of the system that is used, however, it needs to be free of bias and ensure that the science performed is of the highest quality. This study is a welcome start to looking at the New Zealand system, but it also highlights the challenges of such studies and their interpretation.

“It is clear from this report that the Marsden fund is adding real value to our scientific enterprise as judged by traditional academic criteria. Unfortunately the study design does not allow an assessment of the broader potential impacts of basic investigator led science on, for instance public policy, social or environmental health or commercial opportunities. Discovery research of the type the Marsden scheme funds is ultimately at the core of our ability to innovate, to make informed decisions and to advance as a society with the knowledge and knowledge products that flow from it.

“The bibliometric analysis employed in this review of the Marsden did not allow for evaluation of the very different behaviours across science discipline from, for instance, molecular biology to economics. This lack of granularity may have obscured important detail and led to type two errors (that is reaching conclusions of no effect when there is an effect). However the study does confirm that it is difficult to distinguish, in any logical way, the very best of science and inherently this problem must be more difficult with early-stage research. Unfortunately funding constraints mean that the overall success rate of the Marsden fund is below the 15-20% rates that are generally accepted as allowing peer review to reliably distinguish the best science. It is somewhat reassuring that those who were unsuccessful after the triage stage were able to find funding to allow them to perform as well as those who were funded. But it also suggests that the state of funding of discovery science in New Zealand is such that important opportunities are being missed.

“Scientists put a lot of effort into proposing new research ideas to funders and we have seen from elsewhere that systems that do not employ robust peer review of the science face a decline in the quality of science. Robust peer review, which engages international experts, is critical for all expenditure of taxpayer funds on science funded through contestable systems. It provides assurance of good stewardship of public funds by the science community.

“This report is a timely reminder that we need to be honest about scientifically evaluating the science system and be prepared to think innovatively about how we can improve it. Dr Jaffe and his colleagues are to be congratulated on kicking the tyres and asking some important questions.”

Prof Harlene Hayne, Vice-Chancellor, University of Otago and Chair, Universities New Zealand, comments:

“Since 1994, the Marsden Fund has supported high quality research and has become one of this country’s most prestigious sources of funding for university researchers.  The Motu report confirms that the Marsden fund has had a significant impact on the amount of internationally-cited research that is produced in this country-research that enhances the international rankings of our universities helping us to succeed in an increasingly competitive international market.

“Sadly, the report also shows that high quality research proposals are left unfunded simply because the fund just isn’t large enough and clearly demonstrates the return the Government would achieve from increasing its investment in the Marsden Fund.”

Dr Nicola Gaston, President of the New Zealand Association of Scientists and Principal Investigator, MacDiarmid Institute for Advanced Materials and Nanotechnology, comments:

“The Motu study of the effectiveness of the Marsden Fund is a great step towards transparency and the use of evidence in setting policy around science funding in New Zealand. Given the anecdotal evidence in the scientific community, in particularly that supplied by the winners of our most significant awards for scientific achievement, it is absolutely not surprising to see the positive effect that Marsden funding has on the productivity and impact of researchers.

“What is a little surprising, at first glance, is the conclusion that the rankings made by the panels at the second (and final) stage of decisions does not predict the ultimate impact of the work. However, this is not as surprising at it may seem, given that the panels themselves are broad (for example, spanning Physics, Chemistry, and Biochemistry) and they are heavily oversubscribed, meaning that sometimes panelists may be comparing apples and oranges in terms of the research topic.  What’s more, proposals at this stage have already been through a highly competitive first screening.  If the panels were to be given more choice at the second stage, then the value of the decisions they make might be worth more – but this is a hypothesis that would itself need to be tested. It certainly doesn’t change the conclusion of the researchers – with which I agree wholeheartedly – that this study demonstrates that there would be no diminishing returns if we were to increase – perhaps even to double or triple – the size of the Marsden Fund.

“The second point made by the research team, that the resources invested in the second stage review of proposals may be largely viewed as wasted, I think is rather more nuanced.  From the perspective of the funders this is perhaps true, but the reality is that the expected process of evaluation determines the behaviour of the researchers who apply to the fund. For example, one could imagine that if the second round were perceived to be less rigorous than it currently is, the fund would attract a larger number of less credible applications. So I would recommend caution in any response to this particular conclusion.

“Finally, there are a number of additional salient points addressed in the study which are worth comment. Firstly, the suggestion that there are fewer alternatives to the Marsden Fund for the support of fundamental research in New Zealand than in the U.S., for example, does reinforce the their primary conclusion, that the level of funding distributed via the Marsden Fund is simply too low. Secondly, the simulated trajectories for publication and citation counts, under scenarios with and without Marsden funding, demonstrate one very crucial fact: the baseline slope is negative. Funding for research of the kind available through Marsden is not just a nice-to-have; funding for research is an absolute necessity if we are to sustain a vital and productive research community in New Zealand.”