Poor students’ ‘cognitive gain’ may play role in TEF

Metrics including “standardised tests” for students as well as measures of the number of disadvantaged students admitted to universities and their “journey travelled” are under consideration for the teaching excellence framework.

Other key issues to be determined in the months of debate ahead on the planned TEF for higher education institutions in England include whether the framework will be a continuous exercise or take place at a point in time that could even be alongside the research excellence framework.

Jo Johnson, the universities and science minister, announced his plan to introduce a TEF, which would include “outcome-focused” metrics, to drive up university teaching standards in a speech last week. It is thought that a timetable for introduction of the TEF has been discussed by government but is yet to be settled.

The government is expected to set up a series of advisory groups to feed into the autumn Green Paper on the TEF announced by Mr Johnson. The Green Paper is likely to set out principles rather than an exact framework, and would be followed by a formal consultation.

The framework itself would then be developed and piloted – but sources expect pilots to happen “sooner rather than later”.

The question of what metrics will go into the TEF remains to be decided, but the government has some clear priorities already.

David Willetts, the former universities and science minister, regarded by some as the source of the idea for a TEF, said that his personal view was that “quite an eclectic mix” of metrics would be needed.

One key area, likely to prove controversial for universities, will be how the TEF assesses the “learning gain” made by their students during their courses.

The Higher Education Funding Council for England has commissioned a number of pilots of projects to measure “learning gain”, the results of which will be considered in planning for the TEF.

Mr Willetts referred to the Collegiate Learning Assessment used by more than 200 higher education institutions in the US.

He said that this involved “fairly basic cognitive tests” administered to students “on a survey basis in their first term and their final term”.

He added: “I don’t think it should be beyond the wit of man to get some kind of measure of cognitive gain like that [in England].”

Hefce states on its website that the learning gain pilots will look at a range of possible methods including “standardised tests”, which would measure “the acquisition of certain skills” and could be “administered to students either as part of their formative or summative assessment for their degree or as an additional exercise alongside the course”. The tests could be “discipline-specific, or focus on generic skills”.

Other methods to assess learning gain to be examined in the pilots include “measuring the progress in students’ achievement by comparing the difference between grades at two points in time”. This could, Hefce states, include “comparing the difference between actual grades at two points in time, using a standardised measure of these grades (such as the grade point average)”.

Mr Johnson called for universities to move to a GPA system in his speech.

The minister also identified four aims for the TEF, including “to recognise those institutions that do the most to welcome students from a range of backgrounds and support their retention and progression to further study or a graduate job”.

There are suggestions that the government may consider including in the TEF a metric on the proportion of disadvantaged students admitted by each institution.

Mr Willetts said: “What we are really after is quite simply a measure of journey travelled. If a university is taking students predominantly from disadvantaged backgrounds with lower A levels and is then getting them into graduate jobs and giving them real cognitive gain, that achievement is not caught by current league tables, which are entirely focused on research and prior attainment.

“What I’m sure Jo is after, and what I believe in, is some measures that capture universities as doing well at that challenge of really accelerating people’s intellectual development.”

Hefce is already piloting new questions on “student engagement” for the National Student Survey, also seen as likely to play a part in the TEF.

john.morgan@tesglobal.com

By the numbers: metrics-based approach to TEF must be carefully considered

The architects of the teaching excellence framework should be wary of uncritically adopting a metrics-driven approach, according to the chair of a major review of the use of quantitative indicators in research.

The independent review of the role of metrics in research assessment and management, commissioned last year by David Willetts when he was minister for universities and science, concludes that although metrics have a place, they need to be used “responsibly” (see this week's feature).

In relation to the research excellence framework, the review endorses the use of metrics as background information for assessors but does not recommend any significantly greater role for them in the next exercise, likely to occur in 2020.

The review’s chair, James Wilsdon, professor of science and democracy at the University of Sussex, said that he had been struck by the fact that in the week before the review reported, the current universities and science minister, Jo Johnson, had set out plans for a TEF based on “a clear set of outcome-focused criteria and metrics…underpinned by an external assessment process undertaken by an independent quality body”.

Professor Wilsdon said that many teaching metrics were “more well established and…less controversial in terms of their inherent construction” than those used in research. But he still worried that the TEF was “being presented unproblematically as if it should include metrics” and called for the debates about the principles underpinning the TEF and the REF to be “joined up”.

He said that the “main warning” of his review was that “every indicator brings with it inbuilt assumptions, biases and politics” that will “push the sector in one way or another” because of universities’ “strategic responses” to them. Hence, although metrics “aren’t inherently a bad idea, we do need to think through carefully which ones are being selected, how they are being weighted and the additional pressures, burdens and positive incentives they will create”.

Professor Wilsdon was also troubled by Mr Johnson’s remarks, in his speech at Universities UK on 1 July, that he had “no intention of replicating the individual and institutional burdens of the REF” and that the TEF would be “proportionate and light touch, not big, bossy and bureaucratic”.

“That is not exactly a glowing endorsement of the REF,” Professor Wilsdon said. But, in his view, the results of the REF largely commanded confidence and were vital to convincing the government that quality-related funding was being well spent.

This was particularly crucial because it was not a “given” that the dual support system for research would survive the next spending review – particularly in light of the Nurse Review’s focus on the research councils.

Paul Jump