Who even wants the TEF?

I’ll say this for the White Paper on Higher Education, Success as a Knowledge Economy: it’s not as bad as the Green Paper that preceded it.

The Green Paper had me abandoning my Christmas shopping for furious tirades against the errors and illogicality that were scattered among the exhausted clichés and management speak (see here and here). So appalled was I at the shoddy standards evident in the Green Paper that I actually went through all the sources quoted in the first section of the White Paper to contact the authors to ask if they were happy with how their work had been reported.

I'm pleased to say that out of 12 responses I got, 10 were entirely satisfied, and one had just a minor quibble. But what about the twelfth, you ask. What indeed?

When justifying the need for a teaching excellence framework (TEF) last November, Jo Johnson used some extremely dodgy statistical analysis of the National Student Survey to support his case that teaching in some quarters was “lamentable”. I was pleased to see that this reference was expunged from the White Paper.

But that left a moth-eaten hole in the fabric of the argument: if students aren’t dissatisfied, then do we really need a TEF? One could imagine the civil servants rushing around desperate to find a suitably negative statistic. And so they did, citing the 2015 HEPI-HEA Student Academic Experience Survey as showing that “many students are dissatisfied with the provision they receive, with over 60 per cent of students feeling that all or some elements of their course are worse than expected and a third of these attributing this to concerns with teaching quality” (p 8, para 5).

The same report is subsequently cited as showing that: “…applicants are currently poorly-informed about the content and teaching structure of courses, as well as the job prospects they can expect. This can lead to regret: the recent Higher Education Academy (HEA)–Higher Education Policy Institute (HEPI) Student Academic Experience Survey found that over one third of undergraduates in England believe their course represents very poor or poor value for money.”

The trouble is, both of these quotes again use spin and dodgy statistics.

Let’s take the 60 per cent dissatisfaction statistic first. The executive summary of the report stated: “Most students are satisfied with their course, with 87 per cent saying that they are very or fairly satisfied, and only 12 per cent feeling that their course is worse than they expected. However, for those students who feel that their course is worse than expected, or worse in some ways and better than others, the number one reason is not the number of contact hours, the size of classes or any problems with feedback but the lack of effort they themselves put in.”

So how do we get to 60 per cent dissatisfied? This number is arrived at from the finding that 12 per cent said that their experience had been worse than expected, 49 per cent said that it had been better in some ways and worse in others.

So it is literally true that there is dissatisfaction with “some or all elements”, but the presentation of the data is clearly biased to accentuate the negative. One is reminded of Hugh in political comedy The Thick of It saying: “I did not knowingly not tell the truth.”

But it gets worse.

As pointed out on the Wonkhe blog, among “key facts” in a briefing note accompanying the White Paper, the claim was reworded to say “over 60 per cent of students said they feel their course is worse than expected”. The author of the blog referred to this as substantial misrepresentation of the survey.

This is serious, because it appears that in order to make a political point, the government is spreading falsehoods that could cause reputational damage to universities. Moving on to perceptions of “value for money”, there are two reasons for giving these low ratings: you are paying a reasonable amount for something of poor quality; or you are paying an unreasonable amount for something of good quality.

Alex Buckley, one of the authors of the report, replied to my query to say that while the numeric data were presented accurately, crucial context was omitted. This made it crystal clear that it was the money side of the equation that concerned students.

He wrote: “Figure 11 on page 17 of the 2015 HEPI-HEA survey report shows that students from England (paying £9k) and students from Scotland studying in Scotland (paying no fees) have very different perceptions of value for money. And Figure 12 shows that the perceptions of value for money of students from England plummeted at the time of the increase in fees. Half of 2nd year students from England in 2013 thought they were getting good or very good value for money. In 2014, when 2nd years were paying £9k, that figure was a third. (Other global perceptions of quality – satisfaction etc – did not change.) There is something troubling about the Government citing students’ perceptions of value for money as a problem for the sector, when they appear to be substantially determined by government policy, i.e. the level of fees. The survey suggests that an easy way to improve students’ perceptions of the value for money of their degree would be to reduce the level of fees – presumably not the message that the Government is trying to get across.”

So do students want the TEF? All the indicators say no.

Chris Havergal wrote in Times Higher Education about a report by David Greatbatch and Jane Holland in which students in focus groups gave decidedly lukewarm responses to questions about the usefulness of TEF. Insofar as anyone wants information about teaching quality, they want it at the level of courses rather than institutions, but, as an ONS interim review pointed out, the data are mostly too sparse to reliably differentiate among institutions at the subject level.

Meanwhile, the NUS has recommended boycotting the National Student Survey, which forms a key part of the metrics to be used by TEF. This is all rather rum, given that the government claims its reforms will put students at the heart of higher education.

It seems that they have underestimated the intelligence of students, who can see through the weasel words and recognise that the main outcome of all the reforms will be further increases in fees. It’s widely anticipated that fees will rise because of the market competition that the White Paper lauds as a positive stimulus to the sector, and it was clear in the Green Paper that one goal of the reforms was to tie the TEF to a regulatory mechanism that would allow higher fees to be set by those with good TEF scores.

Perhaps less widely appreciated is that the plan is for the new Office for Students to be funded largely by subscriptions paid by higher education providers. They will have to find the money somewhere, and the obvious way to raise the cash will be by raising fees.

So students will be in the heart of the reforms in the sense that having already endured dramatic rises in fees and loss of the maintenance grant, they will now also be picking up the bill for a new regulatory apparatus whose main function is to satisfy a need for information that they do not want.

Dorothy Bishop is professor of developmental neuropsychology at the University of Oxford. This post originally appeared on her blog.