Judging REF 2014 was ‘inspiring and humbling’

Source: Patrick Welham

The REF is not responsible for the management strategies of individual institutions or particular responses to research assessment

Four years ago I was invited to join the English subpanel for the research excellence framework 2014.

I accepted. Having just run a “mini-REF” in my own department, I wanted to be involved in the real thing, seeing participation in the process of peer review as a vital service to the discipline. Here was an opportunity to immerse myself in the work of colleagues across the sector and to acquaint myself with the workings of other higher education institutions. I see virtue in making comparisons and identifying best practice. Research assessment has been a part of the academic landscape since I was appointed at Goldsmiths, University of London in 1992 as a temporary lecturer. I took part in a pilot for the 1996 research assessment exercise there, and as an early career researcher found that exercise both chastening and useful.

After experiencing the “churn” of restructuring, I saw in the REF something positive. An expert peer review process with panels composed of senior academics with considerable experience across the spectrum of research carried out in my discipline struck me as a sensible way to judge the quality of work being produced. The REF builds on its predecessor, the RAE, and for my subject in ways I found encouraging. As someone who had pioneered creative writing at Goldsmiths and launched the master’s in that subject at the University of Glasgow, I was heartened by the inclusive criteria that the panel established as the framework for assessment. Work in the fullest range of forms and formats would be eligible. Creative, co-authored, collaborative and interdisciplinary work would take its place alongside more familiar types of research.

Yet I was unsurprised by the negative reaction from some quarters. These are not good times for higher education. A climate of change and cuts is not conducive to collegiality or to trust in processes that, sound in themselves, may be used for purposes other than those for which they were intended. The REF is not responsible for the management strategies of individual institutions or particular responses to research assessment.

I feel privileged to have been part of what one fellow panellist characterised as “a well-run exam board”. We adhered strictly to the published criteria, despite suggestions that there would be nods and winks behind the scenes. The experience was inspiring and humbling in equal measure: mind-expanding in the way that reading intensely over a nine-month period the finest work in one’s field is bound to be; ego-shrinking in that I emerged with a more modest idea of my own achievements, and a strong sense of the range and richness of work in my subject area.

Some cast doubt on the capacity of the panel to have read with sufficient care the work submitted. I can vouch for the commitment of the panel on which I served. Outside the REF, I have PDFs for breakfast, so while this was the most demanding period of reading I have undertaken in my career it was familiar territory. Impact was new, but there too attention to detail was unflagging, as was adherence to published criteria. Many criticisms of REF resort to assumptions that would be dispelled by careful scrutiny of the documentation.

According to Derek Sayer, professor of cultural history at Lancaster University, “the REF casts a long shadow over British academic life” (“Protest for a REFormation”, 11 December 2014). REF 2014 – the first research excellence framework, and the first such exercise to assess “impact” – entailed hundreds of academic colleagues carefully calibrating the quality of work and workplace of thousands of other colleagues and, assisted by dozens of expert impact assessors, examining the effects of that work on the world beyond the academy. One scholar on a crusade, or some scholars confusing shadows cast from other obstacles to academic freedom with a process of peer review that’s been with us for nearly 30 years, cannot compete with the comprehensive experience and expertise of panels put together by the Higher Education Funding Council for England. As academics, we are all players, and all referees, but above everything, we are all colleagues.

Quixotic crusades make good headlines. Journalistic conclusions come easy. The hard work of assessment takes time. I do not accept Sayer’s characterisation of the REF: “What began as a ‘light-touch’ periodic audit in 1986 has spawned university bureaucracies that continually monitor and seek to manage individuals’ research within REF priorities and timelines.” I have issues with the new managerialism, with the risks to academic freedom posed by changes in university governance, and the shift from collegial to corporate models. I see long shadows cast by cutbacks and privatisation, but I have complete confidence in the REF, as I had in the RAE.

Metrics might suit me personally – I have quantity of output in spades – but if I envisage the process that would best support work done in my field in all its variety and vibrancy then it would be the exercise in which I have just participated. My experience of REF 2014 was indeed of a well-run exam board, one that took a generous view of the work that colleagues do, the circumstances in which they do it, and the impacts it is having, and arrived at robust judgements.