‘Give up freedoms’ to solve reproducibility crisis, says expert

Scientists need to ask “uncomfortable questions” about the value of their own research and potentially give up their line of inquiry to work on larger, group-run projects to help address science’s reproducibility crisis, an influential voice on research integrity has argued.

Daniel Lakens is an experimental psychologist at Eindhoven University of Technology and was involved in the groundbreaking Open Science Collaboration project in 2015, which found that just 40 per cent of 100 leading psychology experiments were replicable.

He told a conference organised by Vrije Universiteit Amsterdam that science had been “busy making incremental changes” to improve reproducibility in scientific research but “had not really fixed the core of why these problems emerge”.

“But the fixes are of such a level that the requirements [needed] are highly likely to make us uncomfortable as scientists,” explained Dr Lakens at the online event on 25 August. He added that these fixes “will change the way we do science and require us to give up certain freedoms” as researchers.

In psychology, it would mean “a lot more work” before researchers could confidently make claims based on their results, he said. Studies would not just need to be replicated but reproduced numerous times in numerous settings to ensure that results replicated in a university lab involving Dutch undergraduates could also be reproduced in non-campus environments, he added.

“It’s a lot more work than [was required] when I entered the field,” admitted Dr Lakens, who said that the time- and resource-intensive nature of this work meant it was impractical for individuals or small teams to carry it out.

Instead, researchers should work in “much larger teams than we are doing now”, even if that required some scholars to sacrifice their own line of enquiry. “If we want to raise the bar, we have to have these uncomfortable discussions,” said Dr Lakens, who recounted frank discussions with researchers who had admitted that “their research did not have any worth at all”.

In one case, a researcher admitted that their research was so highly theoretical it could struggle to have value, revealed Dr Lakens, while another said they had been given grant funding to answer a scientific question that had already been answered.

“We will not get better at fixing crises unless we are willing to talk about the value of our research,” he said.

Dr Lakens highlighted three key causes of the replicability crisis: “low power” results where the collection sample is “so small you cannot make any claims from it”; “publication bias” in which scholars are either dissuaded from publishing when their studies do not produce results with statistical significance, or cannot find an outlet to publish these results; and “p-hacking”, in which data and results are manipulated to create the semblance of a statistically significant result.

In the latter case, p-hacking was only made possible because a researcher considered their work was “so inconsequential no one will notice that you are making false claims”, said Dr Lakens.

“Checking others’ code is a chore, but there is value in it,” said Dr Lakens, who believed the incremental changes happening at present would only lead to “decades of whack-a-mole” as new crises and problems arose.

“If we want to move forward, we need to be happy to give up some freedoms to ensure the resources we have are put to the best uses,” he concluded.

jack.grove@timeshighereducation.com