REF 2021: Did game-playing shift from researchers to outputs?

The dust may not have completely settled on the outcomes of the 2021 Research Excellence Framework, but some key questions are starting to crystallise around how changes made since the 2014 iteration of the REF have influenced the results.

Perhaps chief among these is the extent to which the shift from the selection of individual researchers towards the submission of all research-active staff – albeit with greater flexibility on outputs – has influenced the 2021 data.

Although it seems to be generally accepted that game-playing in the form of selecting only those researchers whom institutions feel would perform well has been largely eliminated – notwithstanding gripes about those individuals who had been moved away from research contracts in recent years – has it merely been replaced by strategising around outputs?

This time around, instead of the 2014 rule that each member of staff entered to the REF needed to have four outputs attributed to them, departments had the flexibility to attribute between one and five outputs to each individual entered, provided the overall volume of outputs was equivalent to an average of 2.5 per full-time equivalent staff member.

It does not take too much to imagine the kind of strategies that REF managers in universities could have employed around this framework: essentially submit five outputs for those seen as “research stars” and spread the rest among everyone else.

In a research world that has become increasingly more collaborative over the past decade, this arguably also allows co-authorship to become a powerful weapon in a department’s submission armoury. Provided that co-authors on the same paper were from different institutions, or subject areas, and made a “substantial” contribution to the research, the same output could conceivably have been entered to the REF several times.

There was even an exception to this in the arts and humanities-leaning Main Panel D, which accepted “the inclusion of the same co-authored output up to two times in a submission” to a unit of assessment by a university.

Rules allowing papers by former members of staff to be submitted – as well as the same output being entered by that academic’s new employer – could also have led to outputs appearing more than once, while major pieces of work such as monographs could – as in 2014 – be “double-weighted”.

The full impact of this on the REF 2021 data is still to emerge, but some interesting observations are already visible in the overall data as well as in commentary included in the main panel reports.

A single output was submitted for 44 per cent of researchers who participated in REF 2021, with this figure varying from about 36 per cent to 37 per cent in subject areas such as history, anthropology, theology and maths to 51 per cent in the units of assessment (UoAs) covering education and art. The maximum of five outputs was, meanwhile, attributed to about 10 per cent of submitted staff in many UoAs.

The significance of this might not become clear until there are more data on what proportion of these researchers with single outputs were co-authors on the same projects.

But some concern about multi-authored work and identifying contributions towards them can be detected. For example, in the Main Panel C overview, the “growing phenomenon of research collaboration and activity” was noted alongside a call for institutions to be given “more detailed guidance” on how to explain the significance of a co-author’s contribution. One particular subpanel also reported that “a major observation was the growth in submitted outputs that were conspicuously multi-authored”.

“It was not always clear what the contribution of the named author had been to the underlying research, resulting in many audit queries being raised, and sometimes inadequately answered,” the report says.

Meanwhile, on double-weighted outputs, the data show that there was a big increase in requests, from 2,850 in 2014 to 8,570 in 2021. The vast majority of the double-weighted outputs came in the social science-leaning Main Panel C (2,424, or 4.5 per cent of outputs) and Main Panel D (6,065 requests, 17 per cent of submitted outputs), as might be expected given the greater prevalence of books in these fields.

It may be that these trends simply reflect what the change in the rules for 2021 was trying to achieve – to reveal excellent research wherever it might lie instead of leaving it in the dark because researchers were not being submitted, as in 2014.

In terms of how prevalent multi-authored papers might be in the final REF 2021 body of work, James Wilsdon, Digital Science professor of research policy at the University of Sheffield, said he did not “inherently see any problem” in outputs being submitted more than once by different institutions in cases where academics had collaborated.

“After all, collaboration is something the system should encourage, not penalise,” said Professor Wilsdon, who is revisiting the question of how metrics could be used in the REF in a review due to report later this year.

“If a paper has dozens of authors…then there may be some case for looking at this more closely – but this is why you have UoA panels with field expertise, so they can judge what is and isn’t reasonable to claim as a contribution,” he added.

Kieron Flanagan, professor of science and technology policy at the University of Manchester, pointed out that there was even an argument for saying that co-authors in the same department should be more readily able to be associated with the same output.

“That is a barrier to collaboration. I write a lot with a colleague here, which means I have to write twice as many papers. And that’s a bit silly,” he said, although he acknowledged that the partial decoupling of outputs from individuals this time around had helped.

Professor Flanagan also said that even if collaborative output had helped to drive the increased share of what was deemed to be 4* research, this was not necessarily a problem because he did buy “to some extent” the argument that REF 2021 had helped to unearth quality that might have been missed through individual selection in 2014.

“My feeling is that these results are more likely to be a better representation of the reality,” he said, although he added that “it will never be a perfect representation of the reality”.

Professor Flanagan also said it was inevitable that universities would have spent a lot of time strategising their submissions, but this would likely have happened whatever the rules.

“Universities will always try to be strategic no matter how the rules are because the stakes are high but also because they can’t help it. There’s lots of managers whose job it is” to ensure good performance on the REF, he said.

simon.baker@timeshighereducation.com