Highly novel research proposals are being systematically turned down because they fall outside evaluators’ paradigms of understanding, a new study suggests.
It indicates that humans are not good at approving truly creative new ideas, a finding that has implications for the economy and culture, as well as academia.
A team from Harvard and Northeastern universities made the discovery by sending 150 ideas for research projects in endocrinology to 142 academic evaluators, who rated 15 of them each. The proposals were ideas for new research, rather than detailed project plans, so as not to be judged on criteria such as budgeting.
Highly novel ideas received worse ratings than those with only moderate novelty, the study found, although those with little novelty also scored poorly.
Christoph Riedl, a co-author and assistant professor for information systems at Northeastern University, explained to Times Higher Education that “humans just have cognitive limits in what they are able to understand”.
“Evaluators systematically misconstrued ideas that were outside their established paradigms,” he said. This “bounded rationality” was the most convincing reason for why assessors failed to appreciate highly novel plans, according to the paper, “Looking across and looking beyond the knowledge frontier: intellectual distance, novelty, and resource allocation in science”, published in Management Science.
Novelty was measured using keywords in each proposal. If two keywords rarely appeared together in the existing literature, an idea was considered novel.
The findings have implications well beyond academia, Professor Riedl said.
“We think that the findings generalise to…pretty much any field where we evaluate how to allocate our money,” including companies looking to develop new products, start-ups looking for investment or publishers reading manuscripts of new books, he argued.
The current economy is “systematically underinvesting in new ideas”, he said.
The paper also discovered that evaluators marked ideas more harshly when they had particular expertise in that area. When a proposal was in an evaluator’s field of specialism, on average it dropped about 30 places in a ranking of the 150 proposals. “That’s huge,” said Professor Riedl. The penalty for high levels of novelty was “comparable in size”, he added.
One way to combat this bias against highly novel ideas was to measure proposals’ novelty – potentially using the team’s keyword method – and then factor this in when deciding which ones to fund, he suggested.
Asked whether the bias against novelty in science might be increasing, Professor Riedl said: “We do know that science and academia has this focus on incremental research within certain paradigms.
“As science funds get more competitive, people might focus on things that are easier to get funded,” he continued.
