Institutional strategy should never be driven by rankings

University leaders from all over Kazakhstan gathered in Almaty last week for an intensive two-day workshop, titled “Experience of Kazakhstan Universities in the Times Higher Education World University Rankings”.

The title offered something of a misleading premise, as there are of course no Kazakhstani institutions in Times Higher Education’s prestigious top 200 list, nor in the 200-400 bands we also publish each year. But that may be something that could change, given time.

As Times Higher Education has reported, Kazakhstan is undergoing dramatic reform of its higher education system, shaking off its Soviet inheritance to embrace the European degree system. There are moves to attract more international scholars and students to the country, there is investment in infrastructure, and research funding is to be allocated on a more competitive basis, offering more opportunities for emerging talent to shine.

These are all positive developments, and global university rankings are playing a central role in helping to shape Kazakhstan’s reform process, delegates heard at the meeting, hosted by Al-Farabi Kazakh National University, and supported by the Ministry of Science and Education and the British Council.

When compiled rigorously and transparently, rankings can help institutions and nations benchmark performance across a variety of indicators with a wide range of global comparators and competitors.

As the 2009 report from the US Institute for Higher Education Policy, Impact of College Rankings on Institutional Decision Making: Four Country Case Studies, said, rankings can have many positive effects. They can: “prompt change in areas that directly improve student learning experiences”; “encourage institutions to move beyond their internal conversations to participate in broader national and international discussions”; and perhaps most significantly, “foster collaboration, such as research partnerships, student and faculty exchange programmes”.

They can even, the report said, “play an important role in persuading the Government and universities to rethink core national values”.

This is clearly the case in Kazakhstan.

But if there was one key message I sought to leave with the delegates in Almaty, it was that while policy and institutional strategy can and should be informed by global university rankings, they should never be driven solely by league tables. Some rankings systems and some rankings indicators, for example, can be relatively easy to “game”. If proxy indicators, such as staff-to-student ratios or overseas student numbers are given too much weight, it is relatively easy for an institution to make changes that may allow them to creep up some ranking tables but may not necessarily reflect true performance improvement or which may not necessarily be in their best interests. A key reason why Times Higher Education employs the largest range of indicators of all the global ranking systems – 13 in number – is to prevent manipulation and to ensure that progress up the table reflects real improvements across the board.

Similar problems exist with reputational surveys. Delegates in Almaty were concerned that the almost 31,000 academics from 149 countries that have responded, over two annual rounds, to the Academic Reputation Survey used in the Times Higher Education World University Rankings may not be familiar with Kazakh institutions. They offered to provide lists of names and contacts from the region who may be able to fill in the questionnaire. But that misses the point. The annual survey, by Thomson Reuters, used for Times Higher Education’s World University Rankings and World Reputation Rankings is invitation-only.

We use United Nations data to ensure that the survey is distributed around the world to statistically represent the true demographics of global scholarship and the spread of academic disciplines. We accept no nominations by institutions and we do not allow any voluntary sign-up, as other rankings do. This is to ensure that there is no bias and that no manipulation is possible.

One thing is clear: an institution can only perform well in the Academic Reputation Survey if it enjoys a global reputation for excellence, in both teaching and research, among informed experienced academic peers.

This means a university must make sure its academics are publishing cutting-edge research, in journals that are widely regarded and widely read, that they are attending the right conferences, are part of the right research networks, are forging international partnerships, and are nurturing the next generation of international scholars and are producing employable graduates who become ambassadors for their alma mater.

It is not the supply of lists or volunteers to fill in surveys that will help Kazakhstan rise up the tables, it is real improvement in both teaching and research. And the signs here are encouraging.