Bland peer review needs a pinch of salt

Research funding agencies face a daunting task when deciding which proposed research project to fund. It takes a great deal of expertise to distinguish between what investor Warren Buffett once called the “three I’s”: innovators, imitators and idiots. The anonymous peer review system that has emerged as the universal and unquestioned tool for assessing research grant applications reliably eliminates the idiots. But, alas, it inadvertently suppresses the innovators, too.

Left with the imitators, who perform solid, sometimes useful incremental research, we are moving in a circle instead of forward, like circus elephants following each other’s tail.

I am not the first person to note that scientific breakthroughs cannot be predicted. Most people will agree that a key ingredient of groundbreaking discoveries is pure curiosity: the purpose-free, passion-driven research of creative “outside-the-box” thinkers, whose prepared minds are likely – as Pasteur put it – to be favoured by chance. A second ingredient, less romantic but not less important, is technical rigour: the adherence to logics of reasoning and the scientific method. Innovators possess these two ingredients; imitators only the second; and idiots neither.

While they appear to be polar opposites, passionate originality and methodological rigour are not mutually exclusive. To spot the innovator, one must identify the grant applications that are the productive mix of the two. But while standard grant review panels sieve out proposals that fail to meet the standard of technical rigour, they are so blunt a tool that, in the process, they also kill the most innovative, visionary projects.

How could it be otherwise, when reviewers are charged with reducing complex, novel research ideas to numerical scores – the averages of which are used by programme managers to determine their funding decisions, as if the reviewers were as infallible as the Pope? This consensus evaluation avoids risks but neutralises the exceptional evaluator with an eye for the groundbreaking proposal that evades the average mind. Only a mediocre proposal will thrive in such an environment.

The poet William Cyples warned in 1864 of the danger to humanity should the “averagarian” bureaucrats prevail, and this limitation of peer review is well recognised by seasoned research programme officials. Yet more and more grant agencies have adopted what the US National Institutes of Health calls “score-driven funding decisions”, surrendering their ability to make decisions based on the holistic judgement of experienced programme managers who can take into account unquantifiables, such as originality and passion. Perhaps the cause is nothing more sinister than a misguided but well-meaning sense of fairness. But the effects are no less disastrous for that.

Nor does reviewer anonymity help. It is observed, of course, because it is thought to suppress bias and maintain objectivity by protecting reviewers from possible vendettas in case of a negative critique and by preventing exchange of favours through unduly positive evaluation. But no society bestows anonymity on any governing body that makes important, direction-setting decisions. Anonymity dilutes magnanimity and accountability. Without magnanimity, the will to resist personal biases diminishes. And without accountability, diligence drops – and, with it, the drive to go the extra mile to identify non-mainstream, true innovation.

The peer review system becomes an echo chamber that nurtures groupthink. The impressionable minds of junior reviewers, still susceptible to the fads of science, will favour proposals close to their mainstream thinking, and the vicious cycle goes on whereby the rich labs get richer. More than 40 per cent of research dollars dispensed by the NIH go to the top-funded 10 per cent of researchers. Diversity is suppressed as research is channelled in one direction.

To add insult to injury, peer reviewers are rarely true peers in the sense of having equal expertise. There is barely any vetting of their intellectual faculty to comprehend the highly technical content of applications. Hyper-specialisation in modern research almost guarantees that reviewers will have less expertise in the domain of a proposal than its often more senior authors. What one does not understand, one does not judge favourably; only the most noble, wise and visionary will possess intellectual empathy for a foreign idea that does not spring from their own mind and may even contradict their own views.

A new course is needed. Funders must reclaim the discretion that they once had for fostering game-changing discoveries to give innovation a chance. If anonymity of peer review and its low quality is the major culprit, then logically there are two solutions: either anonymity must be lifted or the programme managers should treat the review panel as what it is: a group of unvetted, anonymous scientists whose evaluation must be taken with a pinch of salt, considered only as optional “second opinions”. The wisest and most honourable scientists and programme managers, naturally above the fray, must be charged with the task of detecting originality and devotion among applicants, and they should seek to promote open-ended exploration irrespective of obvious utility.

Otherwise, billions of dollars will continue to be wasted as the future is impoverished of scientific breakthroughs.

Sui Huang is a professor at the Institute for Systems Biology in Seattle.