Andreas Schleicher: TEF relies on ‘proxies’ for teaching quality

England’s current plans for a teaching excellence framework rely on “proxies” for teaching quality and there can be “no substitute” for measures of graduate learning outcomes, according to the Organisation for Economic Cooperation and Development’s head of education.

Andreas Schleicher, director for education and skills at the OECD, made the comments while delivering the Higher Education Policy Institute’s annual lecture last night.

He called for the world of higher education to “try harder” on developing an international system to measure learning outcomes, despite opposition from influential high-status institutions in Western nations “that feel they have most to lose”.

The OECD has spent many years developing its Assessment of Higher Education Learning Outcomes (AHELO) project – which would see graduates from different universities around the world tested on their knowledge and skills.

But the project, regarded as the higher education equivalent of the OECD’s Pisa tests for school pupils, has failed to win sufficient backing from OECD member nations to get off the ground, after meeting opposition from leading universities in the West.

Read more: Drop link between TEF and fees, vice-chancellors urge MPs
Read more: Is the TEF the right tool with which to fix HE teaching?

The Westminster government had at one stage appeared supportive of the idea of joining AHELO – which could have seen a full project go ahead – but eventually said that England would not take part.

Mr Schleicher’s lecture – titled “Value added? How do you measure whether universities are delivering for their students?” – appeared to be an attempt to rally support from a UK audience for the principles behind an international project to measure learning outcomes. He also expressed scepticism about the Westminster government’s plans for a TEF.

Mr Schleicher said that on the question of what value universities deliver, “we owe our students an honest answer to this question” as well as “parents who pay a lot of money”  and employers.

“The public, taxpayers, actually want better information on judging the quality of universities,” he also said.

Mr Schleicher argued: “We should become better at developing metrics of learning outcomes that are really direct. It’s very hard to find proxies or shortcuts.”

He said that it was “very hard to improve what you can’t really measure”, adding that at present many judgements about universities are “purely idiosyncratic” as they are “based on rankings that distort institutional behaviour”.

On the Westminster government’s plans in England, he said: “We look at your Green Paper, you try to use proxies: do people complete their studies, all of those kinds of things. But at the end of the day it’s all about learning outcomes.”

He also referred to plans to include National Student Survey results within the TEF, stating: “I’m sceptical this is sufficient as a proxy for learning outcomes…I don’t think you can ever substitute learning outcomes with what people think about learning outcomes.”

Mr Schleicher called for a balance in evaluating graduate learning outcomes between looking at those related to specific discipline studies and more “transversal” skills.

There should be evaluation not just of the “bottom line” in terms of graduate outcomes but also “at the same time” measurement of the “value add question”, he said, looking at outcomes in terms of “where students come from” in terms of their prior achievement or background.

Mr Schleicher concluded his lecture by arguing that “student learning outcomes should be in the critical path of assessing outcomes of higher education. I really don’t believe we can find a shortcut that bypasses that…we can find many proxies, many variables that are correlated. But at the end of the day that [the learning outcome] is what we go to university [for].”

Answering questions from the audience, Mr Schleicher acknowledged that opposition to AHELO had been led by universities perceived to have the strongest reputations – likely to indicate the US and the UK.

He said that “the political economy of this is very, very tough. We can see this from our international work.

“Unsurprisingly, the biggest opponents come from those countries that feel they have most to lose. The biggest opponents within countries come from those institutions that fear they will never do as well as their reputation suggests. These are dynamics that are very natural. But they are very difficult to deal with. Those institutions obviously have a lot of voice. But again that is no reason why we shouldn’t be trying harder.”

Mr Schleicher said that the OECD had planned for AHELO to evaluate institutions, rather than national systems as Pisa does. He suggested that 10 to 15 institutions per country could be sufficient to get the project off the ground.

“That would be a critical mass to start a conversation…It wouldn’t need to be hundreds, it wouldn’t need to be thousands,” he said. “We started Pisa with a group of five countries. At that time, most of the remaining OECD countries were telling us this cannot be done, this should not be done.”

john.morgan@tesglobal.com