Life sciences' REF triumph 'lacks credibility'

The implication of the research excellence framework results that the amount of world-leading research carried out in the UK has doubled in the past six years “lacks credibility”.

This is the claim of a team led by Jonathan Grant, director of the Policy Institute at King’s College London, which used bibliometrics to examine the apparently “remarkable improvement” in UK life science research that saw a doubling in the proportion of work graded 4* (“world-leading”) in the 2014 REF, compared with the 2008 research assessment exercise (rising from 11.8 to 23.9 per cent of the total).

The team found an increase of only 25 per cent in the proportion of submitted papers that were in the top 10 per cent of global papers published in the same field in the same year. Since 9 per cent fewer life science articles were submitted to the REF, the absolute number of papers in the top 10 per cent was just 10 per cent higher.

The results are recorded in a paper, “UK doubles its ‘world-leading’ research in life sciences and medicine in six years: testing the claim?”, published in Plos One on 23 July.

The number of life sciences articles rated 4* in the RAE was the same as the number of submitted papers that were in the top 4.4 per cent of global papers ranked by citation impact. That figure was 7.3 per cent for the REF. This, according to the paper, suggests that the main reason for grade inflation in the REF was that its panels used “a somewhat lower threshold of acceptance for a 4* level”.

By comparison, the difference between thresholds in the physical sciences was less marked. The number of such papers rated 4* in the RAE was the same as the number in the top 6.5 per cent globally, compared with 7.7 per cent in the REF.

Thresholds may have been lowered, the paper suggests, due to the increased reward in 2014 for 4* papers in the formula allocating quality-related research funding. Since the REF results “have huge implications for the relative standing of fields, the research funding of universities, and funding allocated within universities to different research groups” it is “understandable…the REF evaluations are likely to be influenced by these external factors”.

Metrics are, of course, controversial. But the paper argues that, especially at a large scale, they can “provide an external indicator of research quality”.

“For the advancement of science and health, evaluation of research quality requires consistency and validity [and] the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF,” it adds.

Speaking to Times Higher Education, Professor Grant added that grade inflation “makes it harder to distinguish between different degrees of ‘excellence’ within the 4* category”.

This “potentially means the performance-related funding aspects are undermined and the incentive to grow true quality is diluted”, he said.

“Being in the top 4.4 or 7.3 per cent [by citation impact] is clearly very high performance…but to claim that the amount of ‘world-leading’ research has doubled over that period lacks credibility and validity internationally.”

paul.jump@tesglobal.com