‘Validity’ of journal impact factor highlighted by Covid effect

The “validity” of using journal impact factors (JIFs) to evaluate research has been further questioned by an analysis that shows how the “avalanche” of research on Covid-19 could skew them.

According to the paper comparing citing patterns before and during the pandemic, two major general medical journals could double their JIF – a measure of the average citations gained by papers in a particular journal – due to the influence of highly cited Covid papers.

Specialist journals in areas closely related to Covid – such as virology – could also see a massive uptick in their JIF, the study in Scientometrics predicts.

The use of JIFs to evaluate research has come in for growing criticism over the past decade due to the way that highly cited papers can influence them.

They are a particular focus of the 2012 San Francisco Declaration on Research Assessment (Dora), which says they should not be used “to assess an individual scientist’s contributions, or in hiring, promotion or funding decisions”.      

In the paper on the impact of Covid research, Yves Fassin, a professor in innovation and entrepreneurship at Ghent University, details how the highest number of citations for a 2020 article on Covid reached almost 9,000 compared with about 1,500 for a paper on cancer.

“That places that most-cited cancer article around position 1,250 in the all-time ranking of articles on cancer,” the paper says, while the most-cited Covid article would already be ranked at 30.

In one journal, the New England Journal of Medicine (NEJM), only eight articles reached more than 250 citations in the first year after publication between 2015 and 2019, while 27 articles passed this threshold in 2020, all of them papers on Covid. 

The citation figures “are even more impressive” for a specialised publication, the Journal of Medical Virology (JMV), the paper says. Up to 2019, no article in the journal got more than 26 citations in the year after publication. For 2020, 40 papers reached more than 50 citations and one recorded more than 550.

Professor Fassin goes on to show how “two of the already best-ranked journals” in terms of JIF, the NEJM and Lancet, may possibly double their JIF over the next couple of years due to Covid research, “which is an exceptional phenomenon in bibliometrics”.

The JIF of other highly ranked but more multidisciplinary journals, such as Science and Nature, is unlikely to be affected as much, the paper says, due to them having a “larger and wide spread of articles in various research areas”.

Meanwhile, the JMV could see an “explosion” in its JIF that might make it up to 10 times higher and push it into the top 300 of all journals based on JIFs.

“Journal impact factors are heavily influenced by a small number of highly cited papers,” the paper reiterates, and “this is precisely what is going to happen with the avalanche of articles on Covid-19”.

The study says the analysis also “confirms the critique of bibliometric scholars on the use of the journal’s JIF in the evaluation of individual researchers”.

“Several Covid-19 articles in NEJM or in JMV have no citations, and thus no impact, despite the increased JIF of the journal,” it points out. However, the authors of such papers could still stand to gain if JIFs were used to evaluate their research.

simon.baker@timeshighereducation.com