Reputation risks of lotteries for research grants inhibit funders

Awarding research grant funding using lotteries will lead to only “negligible” savings but could expose funders to considerable reputational risks, funding chiefs who have pioneered such systems have admitted.

With several major European research funders expected to announce experiments with grant lotteries next year, there is growing interest in how partial randomisation of grant selection – in which funding applications that pass a certain quality threshold are chosen at random – will work.

One potential benefit of such modified lotteries, which have been trialled in New Zealand, Germany, Austria and Switzerland in recent years, is that they may remove the need for lengthy discussions on the merits of shortlisted entries, resulting in efficiency gains.

But a study by the Research on Research Institute, which interviewed leaders from six funders who had piloted grant selection lotteries, found that “there was little certainty about its money saving properties and participants who had implemented the method felt that the amount of time saved was negligible”.

“It doesn’t save a lot of time,” explained one interviewee who said that it “might shorten panel meetings, because it relieves the reviewers [of the need] to assess very marginal or non-existing differences among top applications”, but this was a “tiny advantage”.

One research leader claimed that setting up randomisation was more expensive because peer reviewers took longer to identify projects to place into the lottery rather than just selecting the very best applications. Reviewers also had the time-consuming task of identifying projects that were “so bad that it would actually damage the reputation of [our organisation] if they would be drawn by lot”, he added.

That reputational risk of funding a terrible project through the novel lottery system was one of the biggest concerns for funders who feared that the chance nature of selection would expose them to allegations of incompetence, the study continues.

One research leader explained his fear that negative newspaper headlines could lead to a perception that “our organisation isn’t capable of doing what it is supposed to do”.

“We don’t want the newspapers to say the organisation is now gambling with the research funding,” said another interviewee about this “nightmare” scenario.

James Wilsdon, professor of research policy at the University of Sheffield, who carried out the study with Helen Buckley Woods, a research associate at Sheffield, said funders worried that lotteries implicitly acknowledged the “huge amount of randomness” that the selection of grants via traditional means often involved.

“The fear is lotteries are ceding the ground of the expert – from which the authority of funders to award money comes – and admitting there is a huge amount of randomness in who gets money,” said Professor Wilsdon, adding that the introduction of lotteries required an “epistemic humility about claims to excellence within the system” that some funders found difficult.

For most funders, the capacity of lotteries to increase fairness in decision-making by removing the possibility for bias, which may also improve the diversity of researchers chosen, was a major positive; while others felt that riskier projects with a potentially higher pay-off were also more likely to be funded.

“I’m convinced it increases fairness,” explained one research leader. “It accommodates the random or the chance element which we know exists in these panel processes.”

jack.grove@timeshighereducation.com