Academics cite work ‘they don’t know particularly well’

Researchers “tend to cite works they are not influenced by and that they do not know particularly well”, according to a new study that raises further questions over whether citations should be used to judge the quality of academic work.

Some universities have turned to citations data – such as a scientist’s h-index, a controversial measure partly based on citations per publication – to make hiring and promotion decisions, according to the research, conducted by academics from Harvard University and the University of Chicago.

Using an online survey across six science and humanities fields, the authors set out to discover whether scholars cite work because of its actual influence on their research – or simply because it “support claims they want to make” and is “familiar to the intended audience”.

The results revealed that more than 60 per cent of citations were said by respondents to have had merely a “minor” or a “very minor influence” over their article.

It also emerged that around 40 per cent of cited articles are known only “slightly well” or “not well” by academics who included them in their papers. Academics were particularly likely to say that they were unfamiliar with the contents of highly cited papers.

In another part of the survey, the researchers found that when it was flagged up that a paper had received many citations, academics rated its quality more highly.

They also found that although researchers do refuse to cite papers that fall below a certain level of quality, “above this threshold, frequency of use is unrelated to quality,” they write.

The findings “severely undermine” the idea that academics cite high-quality work that influenced them, and should spur a “radical reassessment of the role of citations in evaluative contexts”, the authors write.

James Wilsdon, a professor of research policy at the University of Sheffield and an expert on the use of metrics in research evaluation, said that the findings reinforce the “need to handle citation data with care”, and to “always place it in context”.

But he said that he was not “hugely surprised” by the results. “I think most of us recognise that there’s often a performative, strategic or rhetorical dimension to citation – in any given paper, one may cite certain influential or canonical sources, or colleagues, in order to signify a wider appreciation of the field,” he said.

Harriet Barnes, head of policy for higher education and skills at the British Academy, said that researchers were “growing ever more conscious about the limits of citation metrics as proxies for evaluation of quality or impact”.

The results of the research, Why (almost) Everything We Know About Citations is Wrong: Evidence from Authors, were presented at a conference called Science, Technology and Innovation Indicators in Transition, held in Leiden in September. They are based on data from a pilot survey, with the full results still under analysis.

david.matthews@timeshighereducation.com