Journals fail to take action over ‘fatally flawed’ experiments

Biomedical journals are failing to take action when errors that would likely invalidate experiments are spotted, says a study which reveals how publications treat identical concerns differently.

In a paper in Scientometrics, researchers from Australia and France explain how they contacted 13 separate journals to report concerns over 31 published articles on genetics, all of which had relied on a control reagent from the same supplier that had been wrongly identified.

According to the paper’s authors, the inadvertent use of the wrong chemical substance would “almost certainly invalidate any experiment that uses such reagents”, with the faulty results likely to be “incorporated into future studies, potentially leading to failed experiments”.

While the identified error was deemed serious enough for 14 of the 31 papers to be retracted, author corrections were sufficient for seven of the papers while five papers were made subject to expressions of concern. However, for six of the papers, it was stated no action should be taken.

Only six of the responses were issued within six months, with one retraction taking almost three years, says the report by academics from the universities of Sydney and Toulouse, and Grenoble Alpes University.

Noting the “huge variance in responses” to very similar errors, the study’s lead author, Jennifer Byrne, professor of molecular oncology at Sydney, told Times Higher Education that the study’s results illustrated how “even when faced with the same concerns and exactly the same fatal experimental error type, journals still decide to do very different things”.

Those that took action were journals which had a “care factor” that some journals did not, said Professor Byrne, who added that journals are also “clearly influenced by whether authors respond or not, as author corrections can only be issued if authors respond”.

Some journals also appeared not to understand the issues at hand, Professor Byrne continued. In one case, an editor “accepted a clearly incorrect interpretation of the incorrect negative control that we described, essentially saying that this didn’t matter”.

The study proposes that scientific journals adopt an industry-wide reporting template for post-publication notices, which would state the time taken to investigate complaints, the likely impact of any error and what action was taken.

“The system is too slow, partly because at present, journals don’t need to disclose when they receive error notifications,” Professor Byrne said.

“If error notifications were transparently available in real time, journals might be more motivated to act, and to act in a timely manner,” Professor Byrne added, warning that the “current system doesn’t serve the interests of readers who rely upon the published literature”.

jack.grove@timeshighereducation.com