Preprints ‘largely unchanged’ by peer review, even during Covid

Most preprint studies are being fully published without major changes, including those that were quickly shared on servers in the early days of the pandemic, according to two separate studies.

In one study, researchers from the UK, the US and Germany manually compared the abstracts, figures and tables of more than 180 Covid and non-Covid-related preprints from the bioRxiv and medRxiv preprint servers with their peer-reviewed versions in a bid to ascertain how the publication process may have altered them.

Although the Covid-related papers had been taken from the first few months of the pandemic, when there was an explosion of quickly shared research findings, 83 per cent of the studies underwent no major changes to their conclusions by full publication. In the case of non-Covid-related research, the figure was 93 per cent.

According to the researchers, who published their findings in Plos Biology, the majority of changes made to abstracts were “textual alterations that led to a minor change or strengthening or softening of conclusions”, while about a quarter to a third of preprints “underwent no meaningful change” at all before being published.

Even among the 7 per cent of non-Covid articles and 17 per cent of Covid papers deemed to have had “discrete” major changes to abstracts, the majority of these did not “qualitatively change the conclusions of the paper”. The researchers found that just one main conclusion had been contradicted upon full publication of the article.

The study suggested that preprints “were most often passing into the ‘permanent’ literature with only minor changes to their conclusions” and that the “entire publication pipeline is having a minimal”, albeit “beneficial”, effect.

Meanwhile, a separate study by US-based researchers, also published in Plos Biology, used machine learning and textual analytics to compare about 18,000 bioRxiv preprints with their published versions.

The researchers found that most changes appeared “to be associated with typesetting, journal style, and an increasing reliance on additional materials after peer review”; otherwise, the articles tended to be broadly similar.

Casey Greene of the University of Colorado School of Medicine and a co-author of the machine learning study, said taken together, the studies “provide evidence supporting the reliability and use of preprints both during a global pandemic and for general scientific outputs”.

A co-author of the manual comparison study, Jonathon Coates from Queen Mary University of London’s William Harvey Research Institute, added that their paper also could raise questions about use of funds and researcher time in the peer-reviewing and full publication process.

“If the key conclusions are unchanged, but perhaps a little strengthened, is this a justified use of charitable funds and ECR [early career researcher] time?” he asked.

The conclusions were also a reminder that “we should treat all scientific outputs with care and a critical eye”, regardless of publication status. “Peer review often bestows a ‘validation’ of the work, which is an incorrect and dangerous assumption,” he said.

simon.baker@timeshighereducation.com