Quarter of citations in top journals ‘wrong or misleading’

One in four references found within leading scientific journals are incorrect or misleading or fail to provide evidence for the claims made by authors, says a new study on academic “quotation errors”.

In a paper published in Proceedings of the Royal Society A, researchers from the US and South Korea analysed some 250 citations taken evenly from five highly prestigious science journals to check whether the references supported the statements made by authors.

While three-quarters of references in the sample taken from Science, Science Advances, Nature, Nature Communications and Proceedings of the National Academy of Sciences were accurate, 25 per cent were deemed not to substantiate the points raised by authors, says the paper, titled “Quotation errors in general science journals”.

About half of faulty citations (12.9 per cent of all citations in the study) were assessed as “impossible to substantiate”, where authors put forward “citations that could not possibly be substantiated by an outside reference”, such as when their own missing methods section was compared to methods sections used by other experiments.

“Sometimes the entire methods section of a paper would be one sentence, explaining how the authors used the same methods used in another paper,” explained Aaron Cumberledge, from the Daegu Gyeongbuk Institute of Science and Technology in South Korea, who co-authored the paper with Neal Smith, from Florida-based AdventHealth University.

“But when you looked closely, it was very difficult to see if they had followed that paper’s methods entirely – there might have been one crucial variant that changed, but it was impossible to tell,” Mr Cumberledge told Times Higher Education.

Some 8.5 per cent of the citations analysed, which were randomly chosen from 500 articles that included a total of 26,344 references, were found to be “unsubstantiated” and 3.6 per cent were judged “partially unsubstantiated” when the error was deemed minor and the general proposition of the quotation was supported by the citation.

In some cases, papers included citations to papers that appeared to contradict their own propositions or failed to provide evidence for them, explained Mr Cumberledge, who teaches science writing at his institution’s College of Transdisciplinary Studies.

Even when the two researchers were able to substantiate citations, the imprecise nature of many references made the checking process hugely time-consuming, particularly as page numbers were generally not included, he added. “It is really difficult to verify things when the only reference is to an old edition of a medical textbook which is 1,000 pages long,” said Mr Cumberledge, who added: “In that case, we managed to substantiate the reference but it was hard work.”

While it was “unfeasible” for editors or reviewers to check all citations for accuracy given the volume of work involved – it took two months for the study’s authors to check 500 citations – Mr Cumberledge called on authors to redouble their efforts to make their citations more precise and verifiable.

By doing so, abusive or unprofessional citation practices, such as spurious citations, in which authors unnecessarily cite the work of colleagues or friends, who, in turn, cite them back, would become much easier to detect, he said.

“I could see some people saying we are being overly fastidious about this, but it would take so little effort to do these citations properly and the science would be so much more solid and evidence-backed.”

jack.grove@timeshighereducation.com