Flawed indicators used to compare the academic market

Any market in university fees will be based solely on branding and reputation unless institutions are held to account by valid indicators of teaching quality, including class size, the standard of teaching staff and the time and effort students devote to their studies.

The warning is made by Graham Gibbs, a leading educationalist, as a major report indicates that a government drive to provide better information for student "customers" may be focusing on the wrong measures.

Writing in Times Higher Education this week, Professor Gibbs, former head of the Oxford Learning Institute, says universities have been allowed to "get away with" increasing class sizes, cutting feedback and using "cheap and inexperienced" teachers in their pursuit of higher status. They often choose to invest in research rather than teaching, he adds.

His study for the Higher Education Academy, Dimensions of Quality, published on 30 September, draws together research spanning three decades to investigate the best indicators of a good-quality university education.

It finds that data on university funding, research performance, reputation and student-entry grades - often used by newspaper league tables - are poor indicators of quality.

Information about graduate earnings and employment rates also tell applicants little about the quality of education they can expect to receive, it says. This is because the data are strongly influenced by factors such as institutional reputation, "invalid" league tables, students' entry grades and social class.

Instead, what best predicts a good-quality education are measures of "educational process" including class size, teaching staff, the effort students make and the quantity and quality of feedback they receive.

Professor Gibbs argues that all these indicators can be measured. In the US, for example, the National Survey of Student Engagement asks students how often they discuss grades or assignments with tutors, talk over ideas with faculty members outside class and contribute to class discussions. In the UK, the Higher Education Policy Institute has gathered data on the number of hours students study, revealing wide discrepancies between courses.

But Professor Gibbs says that the Quality Assurance Agency does not ask universities to provide information about these areas, nor are they the focus of the UK's National Student Survey (see box below).

The NSS collects students' views on feedback, but broader judgements on whether teaching is "good" are "open to all kinds of subjective variation in the interpretation of what 'good' means", the report notes.

It concludes that "political demands for 'better information for customers' cannot be met with current data-gathering and analysis methods".

Credibility gap

Bahram Bekhradnia, director of Hepi, said the study showed a clear need for "systematic, national and credible" surveys to provide students with information on the contact they could expect with which staff and the effort required for different degrees.

"If students are to pay more and market mechanisms prevail, then better information will be required to inform the market," he said.

Sir David Watson, principal of Green Templeton College, Oxford, said Professor Gibbs "exposes ruthlessly the potential - and frequently the actual - gap between institutional reputation and the quality of the academic experience of students on courses".

He added: "What he doesn't cover - he was not asked to - is the extent of applicants' and students' understanding of this gulf and the reasons for it. Many students will choose 'reputation' in the cold-eyed understanding of what it will bring: personal networks, social recognition and an employment premium.

"Others will choose to study in places where they will be more comfortable (again for social reasons), but also where they are confident that they will be taught conscientiously, imaginatively and effectively."

Sir David said that "the answer is not to chase the fool's gold of rankings based on these latter criteria. Instead, we should seek to continue to enhance the quality of university teaching across the full range of our institutions."

rebecca.attwood@tsleducation.com