New US college ranking to place students at its heart

Times Higher Education has outlined more details of its planned new US college rankings, which are designed to address dissatisfaction with current rankings in the country by putting student learning at their heart.

The ranking of more than 1,000 colleges and universities in the US, to be published in September 2016, will focus on teaching, including graduate outcomes, and the student experience with a range of new performance metrics announced in a webinar last week.

Phil Baty, editor of the THE’s World University Rankings, explained: “We are trying to do something different.

“We want to move away from measuring selectivity in admissions, we want to move away from measuring just how rich a college is, or how hard it is to get into. We want to get away from creating perverse incentives that distort universities’ activities.

“We want to capture the key mission of a university, which is transforming individuals’ lives – delivering quality teaching and quality education, capturing the value added.”

THE’s director of data and analytics, Duncan Ross, announced that the rankings would use a balanced scorecard approach with a variety of measures derived from the theoretical framework from Biggs, Gibbs et al.

While the original framework suggests a “3p” model (presage, process and product) of measuring teaching excellence, THE has adapted and developed the approach based on local conditions, data availability and the results of a year-long consultation with key sector bodies, academic experts and stakeholders across the US.

The result is THE’s “4c” model which evaluates universities over four core competencies: “capacity” for teaching; “capability” for engaging students; “credibility” in terms of producing good outcomes for students; and “community”.

Each of the four competencies has been chosen to address the types of questions prospective students are concerned with. 

Mr Ross also suggested “candidate metrics” he would use to answer these questions and evaluate the teaching environment across different types of US colleges, including spending power per student, teaching staff per student, the number of programmes available to students, graduation rate, student diversity and academic reputation.

“The challenge here is always going to be that for each of these core areas, can we identify one or more proxies which we can measure effectively and consistently?” he said.

“For these measures and this approach to be effective we need to have a good solid dataset underpinning the work we are doing.”

Some indicators, like student satisfaction, faculty-student interaction and measures of the learning environment and engagement will take data from a Times Higher Education US student survey, which has already been launched.

Current students in taught courses at more than 1,300 colleges across the United States have been targeted for the survey, in a process managed by established market research organisations and supported by university outreach activity.

Other indicators will use IPEDS data, the US college scorecard and the THE Reputation Survey.

A key challenge will be to present the results in an informative, interesting and responsible way, said Mr Ross.

“Were we to present just a list of 1,000 colleges, people might reasonably say we were comparing apples with pears or apples with oranges,” he said.

The solution could be to allow students to sort and explore the dataset, and to generate six different rankings as “sensible comparison groups”. THE has proposed including universities competing internationally – as defined by participation in the Times Higher Education’s World University Rankings – in a “World League”, while colleges with a significant proportion of students from out of state will appear in “National League” set. A further four “Regional Leagues” would band colleges that primarily compete within a sub-national region by broad geographical area in the United States.

In the question and answer session, Mr Ross reassured participants that the US student survey would be conducted largely independently of colleges, following best practice for minimum number of responses required to produce reliable results. He explained that any college outreach activity to encourage student responses would widen the sample, providing more opportunities for analysis and understanding of students’ engagement across the whole of America’s colleges.

Mr Ross did not reveal any specific weightings for the proposed metrics, but said that he thought student engagement indicators and benchmarked measures of “value added” were particularly interesting.