NSS manipulation claims ‘raise questions’ about data reliability

Allegations that universities have attempted to inappropriately influence responses to the National Student Survey have been revealed – including staff filling in responses – raising questions about the exercise’s planned role in the teaching excellence framework.

The Higher Education Funding Council for England, which is responsible for coordinating the questionnaire across the UK, has investigated claims that institutional staff have tried to get undergraduates to inflate their course ratings on 30 occasions over the past seven years.

The data, released to Times Higher Education under a Freedom of Information request, were described by researchers as likely being a snapshot of a wider problem, the result of intense pressure on staff to help institutions maintain their position in league tables that draw on the NSS.

In most cases, the alleged interference focused on inappropriate influence via email, briefing or other communication: for example, reminding students of the impact that the NSS has on employers’ perceptions of institutions and, therefore, on their job prospects as graduates. But in six instances, staff were suspected of being directly involved in completing NSS responses.

Of the 27 investigations that have concluded, 17 resulted in institutions being asked to take specific actions, and there was insufficient evidence in six others. In four cases, a panel convened by Hefce judged that, on the balance of probabilities, a course’s NSS data were unreliable, and the results were suppressed on sources such as the Unistats website.

There is also evidence that allegations of inappropriate influence are being made more frequently, with all but 12 of the complaints having been lodged since 2014.

Camille Kandiko Howson, senior lecturer in higher education at King’s College London, said that this was concerning.

“I still think that the sector can have confidence in the data, but I think there is a worrying trend,” she said. “If, as time goes on, people become more aware and start to report more activities, it may be that this is a more extensive practice than we have been made aware of previously.”

Concerns for ‘integrity’ of data

Details of the allegations emerged in the wake of the government’s decision to confirm plans to use responses to NSS questions as a key metric in the TEF, meaning that students’ responses could ultimately influence the tuition fees that an institution is allowed to charge.

The allegations of inappropriate influence can be seen alongside concern about the growing phenomenon of “yea-saying” in the NSS, whereby respondents give the same – usually positive – response to every question. In the 2014 survey, 6.1 per cent of those completing the NSS online did this, up from 1 per cent in 2005, and it has been suggested that “overzealous promotion” by universities concerned about their public profile may be to blame.

Dr Kandiko Howson also highlighted the recent vote by the National Union of Students to either sabotage or boycott the survey, in protest against plans for the TEF, and noted that this action, combined with the other issues, “does raise questions about the overall integrity of the data”.

Questions have also been raised about Hefce’s investigative capabilities, with the funding council confirming that its usual response to allegations is to ask institutions in question to conduct an investigation themselves, and to order an independent probe only if it is not satisfied with the provider’s approach.

Dr Kandiko Howson said that the introduction of the fee link via the TEF would likely “require more substantial action" to be taken, "given the importance of the outcomes moving forward”.

A Hefce spokesman argued that the number of cases where data had been suppressed was “very small” compared with the hundreds of thousands of students and hundreds of institutions that take part in the NSS every year, and added that the funding council took its role in protecting the robustness of the NSS “seriously”, issuing guidance to institutions and regularly checking data for evidence of inappropriate influence.

“Students and the wider public can be reassured that we take clear measures to protect the robustness of the NSS,” the spokesman said.

“Given that Hefce receives just a handful of allegations each year and these are related to specific courses, rather than institution-wide, there is no evidence to suggest either that there is a wide-scale problem with inappropriate influence or that Hefce’s process of investigating allegations is not effective or fit for purpose.”

chris.havergal@tesglobal.com

Releasing names ‘not in public interest’

The names of universities accused of exerting inappropriate influence in the National Student Survey have been kept secret.

Times Higher Education argued that it was in the public interest for the identities of the institutions involved – or at least those where the allegations were serious enough for results to be suppressed – to be released.

But Madeleine Atkins, chief executive of the Higher Education Funding Council for England, ruled that the public interest lay with allowing allegations to be investigated and responded to “without being inhibited by the public gaze”.

In a letter to THE, Professor Atkins says that Hefce relies on institutions to investigate allegations of inappropriate influence internally and to report the outcome to the funding council, and that publicity could hamper this process.

She says that all the cases have been related to a specific course, rather than a whole institution, and that as such releasing the names would have “a disproportionate effect” on a university’s reputation.

THE appealed against the ruling, highlighting the increasing significance of NSS results in the reputation and recruitment efforts of institutions, and also their forthcoming role in the teaching excellence framework.

THE also argued that increased openness might encourage more students to report concerns that they have about completion of the NSS.

However, the appeal was rejected. A Hefce spokesman said that the decision was taken “to protect the student interest and the robustness of the NSS”.

“When allegations of inappropriate influence are raised, institutions are requested to undertake an internal investigation and report their findings, which in turn are considered by a Hefce panel,” the spokesman said. “Consequently, the process relies on institutions to be frank and candid in their response so that appropriate action can be taken.

“Hefce’s ability to respond to allegations would be hampered if institutions were named because they would be less forthcoming in the information provided to the panel.”