TEF to limit use of ‘flawed’ NSS scores

Student satisfaction scores will have only a limited impact on whether universities are judged gold, silver or bronze in the teaching excellence framework, the assessment panel’s chair has said.

As the University of Cambridge’s students’ union became the 25th union to boycott the National Student Survey (NSS) over the poll’s influence on the TEF’s ratings, Chris Husbands told an audience at the House of Commons that his team would “not be overweighting the NSS” when calculating which medal an individual university would receive.

Speaking at an event organised by the Higher Education Academy and the Higher Education Policy Institute on 24 January, the vice-chancellor of Sheffield Hallam University said that the TEF “should not place an overemphasis on satisfaction data”.

“I do not think student satisfaction is an accurate proxy for teaching quality,” said Professor Husbands, a former director of the UCL Institute of Education who was named chair of the TEF panel by the Department for Business, Energy and Industrial Strategy in August 2016.

Professor Husbands drew attention to US studies that show that student satisfaction scores are heavily influenced by non-academic factors, including the sex or ethnicity of a lecturer.

“Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” he said of the US research findings.

He added that the TEF panel would “not draw policy from a single data point” and that “all data” are “flawed” in some respect. However, he went on, the challenge was to recognise and understand the flaws, and to learn how the information could be used effectively.

Professor Husbands’ statement may allay the fears of many UK universities whose students have joined a National Union of Students-led movement to boycott the NSS. They hope to sabotage the TEF by rendering the NSS results unusable, thereby making it impossible for policymakers to differentiate tuition fees in future years.

Universities have already been made aware whether they would gain a gold, silver or bronze rating based on existing metrics, which include NSS scores, graduate employment data and student non-completion rates.

Universities must also submit a 15-page narrative statement on the institutional context of teaching. The statements, due by 26 January, will be used by the TEF panel to decide if an institution’s provisional rating based on metrics can be upgraded once its individual circumstances, its students and its local context are taken into account.

Ratings for UK universities will be announced in May 2017, with about 20 per cent of institutions likely to be awarded a bronze rating, 50 to 60 per cent silver and 20 to 30 per cent gold, according to a Department for Education report issued in September 2016.

At the HEA-Hepi event, one vice-chancellor, speaking under Chatham House rules, described the narrative statements as a “creative writing dialogue” with the Department for Education.

Another vice-chancellor wondered how the TEF could use NSS scores as its main measure of teaching excellence when they are widely recognised as poor proxies for teaching quality.

However, Professor Husbands defended the approach of using both metrics and written submissions, saying that the TEF was “underpinned by metrics” but also allowed for more nuanced analysis of teaching.

Evaluations would take into account teaching “based on institutional procedures and how they work to drive student outcomes”, he said.

jack.grove@tesglobal.com