TEF 2017: ‘meaningless’ results ‘devoid of credibility’, says v-c

One of the UK’s most senior vice-chancellors has denounced the teaching excellence framework (TEF) as being “fundamentally flawed” and having “no value or credibility” after his university was awarded bronze.

Sir Christopher Snowden, vice-chancellor of the University of Southampton, made a blistering attack on the TEF as his institution became one of three universities in the Russell Group, alongside the London School of Economics and the University of Liverpool, to be given the lowest rating in the first sector-wide assessment of teaching quality.

Sir Christopher, a former Universities UK president, said the bronze award handed to Southampton – ranked 121^st globally in the latest Times Higher Education World University Rankings – was inexplicable given its strong student outcomes and universally positive feedback from TEF assessors.

“There is no logic in our result at all,” Sir Christopher told THE. “How can you have so many positive comments and exceed many of your benchmarks by a colossal margin and still get a bronze?” he said, adding that the university would appeal the result.

Sir Christopher, who led the University of Surrey for 10 years before arriving at Southampton in 2015, said he was particularly mystified by the bronze award as his university had performed strongly on the student satisfaction, graduate earnings and course completion data used to rate institutions.

“I know we have done significantly better [on metric scores] than many gold-rated institutions with a similar student profile – I can show you the evidence,” he said.

That suggested that the TEF standings were heavily influenced by the information contained in a 15-page submission from universities, said Sir Christopher.

“It tells you the way they treated the statements was completely different,” he said. “I know I am not alone in having deep concerns about its subjective assessment, its lack of transparency, and with different benchmarks for each institution removing any sense of equity and equality of assessment.”

The TEF’s assessment of performance against expected benchmark values based on student profiles, rather than absolute outcomes, made it “incredibly easy for some institutions to exceed their benchmarks”, whereas, for Southampton, “the benchmark was so high we could never achieve a positive ‘flag’,” Sir Christopher added.

For instance, Southampton was required to beat its benchmark target of 4.5 per cent for student dropout rates by two percentage points to achieve a positive flag, said Sir Christopher. It did so by just one percentage point.

“It would mean we need a dropout rate of 2.5 per cent – I'm not sure any university in the country could get that,” he said, saying the target was “ludicrous” and that “the benchmarking is fundamentally flawed”.

With universities held to different standards and judged against expected performance, it was analogous to holding a race between “Usain Bolt and a snail”, said Sir Christopher, adding that “you would only need the snail to go a bit faster [than expected] to win as that is how you are judging performance”.

Sir Christopher added that the “concept of gold, silver and bronze was absolutely meaningless” for students seeking educational excellence, while the TEF’s name was misleading as its metrics were not robust measures of teaching quality.

“I think the Competition and Markets Authority may want to have a word with [universities minister] Jo Johnson,” said Sir Christopher. “The TEF has no credibility and I do not even see how league table compilers will use it as it has no value.”

His comments were echoed by other leading UK universities handed a bronze award. Deborah Johnston, pro-director of teaching and learning at Soas, University of London, said the metrics used by the TEF did “not accurately reflect [its] distinctive mission and student body”.

“There is clearly also a London effect with one in three London institutions achieving bronze, compared to just one in eight outside London,” added Professor Johnston, saying metrics “were not benchmarked for London”, where lower retention rates were often due to costs of living.

Julia Black, interim director of the LSE, said its “students’ high attainment and outstanding performance in highly skilled job markets [are] unfortunately…not captured by the TEF metrics”.

And Sir Christopher’s criticisms were echoed by Sorana Vieru, vice-president (higher education) of the National Union of Students, who called the TEF “another meaningless university ranking system…[which] fail[s] to capture anything about teaching quality”.

jack.grove@timeshighereducation.com