Tackle ‘reporting bias’ in pedagogy research, paper warns

Learning and teaching journals are “not doing enough” to combat reporting bias in research from the field, according to the author of a new paper on the subject.

Phillip Dawson, associate director of the Centre for Research in Assessment and Digital Learning (Cradle) at Deakin University, also said that research funding organisations must help to reduce the phenomenon, which occurs when researchers selectively report significant positive results and omit non-significant or negative results.

“When higher education research is used to inform practice, positive biases can falsely inflate our confidence that a particular learning and teaching approach is going to achieve the desired outcome,” his paper, “Sharing successes and hiding failures: ‘reporting bias’ in learning and teaching research”, states.

The study found reporting bias during a meta-analysis that compared lectures and active learning in science, technology, engineering and maths (STEM) disciplines. The analysis incorporated two approaches to understanding the prevalence of reporting bias: sourcing authors’ publication habits and examining publication patterns around recently completed doctorates.

“This meta-analysis produced [results] which equated to students in lecture classes being 1.5 times more likely to fail compared to their active learning peers,” the paper in the journal Studies in Higher Education states. After re-examining the data according to the dissemination type of the studies – published studies or dissertations – the “strength of support for active learning differed”.

Published studies were “far more supportive of active learning than dissertations” and there was “statistical evidence of four missing studies…likely to have been conducted” but not published or captured by the researchers. These studies would have “reached the opposite conclusion” to those included. This is “problematic”, Professor Dawson writes, because it “represents bias”.

The analysis suggests that learning and teaching research is particularly susceptible to reporting bias because of a range of factors, including the “positive agendas” of funding bodies.

“Funding bodies want to fund innovation in learning and teaching, which is a positive thing,” he told Times Higher Education. “[Australia’s] Office for Learning and Teaching (OLT), and the [UK’s] Higher Education Academy (HEA) and Jisc, want to fund studies that work.

“We didn’t find evidence of projects funded by [these organisations] publishing negative results. Literature that’s just full of success stories…is not a full picture.”

The paper suggests there are also issues around academic freedom, and teaching and learning researchers may not publish poor teaching practice at their institution for fear of reprisal. Publishing negative results would be “bold” for a professional staff member who lacks academic freedom. By encouraging this, however, universities may be “institutionalising reporting bias”.

Professor Dawson suggested that higher education needs to do more to combat reporting bias because “left unchecked” it could result in “policymakers and educators distrusting supposedly evidence-based improvements to teaching and learning”.

“Journals are not doing enough to address reporting bias, in fact they are doing nothing I am aware of,” he said. “Funding bodies [also] have a role to play. At the end of the day, the OLT and HEA, they want to advance teaching and learning. People within them…believe sharing bad results is a good thing.

“[These groups] need to say: we want the full story. Journals should encourage authors to include all results from their studies, not just the sexy results. [And] universities need to say that your academic freedom extends to both your teaching and research, but also to the nexus between them.”

In response, Mark Jones, HEA’s chief operating officer, said it was in everybody’s interests that research gives a “balanced and accurate picture”, and “we strongly encourage and expect researchers to follow this principle”.

Phil Richards, chief innovation officer at Jisc, added that the organisation “no longer funds research in this space”, but was “very aware” of reporting bias being a potential problem in research of this kind and was compiling an evidence base to help universities understand what works in technology enhanced learning.

“For this to work for the benefit of students paying £9K fees, it is essential that the underlying research is sound and balanced; anything that calls that into question must potentially be a matter of concern,” he said.

“When we funded development and evaluation projects, we were very aware of reporting bias as a potential problem and were always at pains to emphasise to projects that [a] failure to achieve a positive result was an entirely valuable conclusion, and one to be shared.”

john.elmes@tesglobal.com