REFs and referees

Leading academics should be encouraged to watch the 2010 Fifa World Cup, particularly those who will be working so hard over the summer to shape the future of research impact assessment.

The 1,800 members of the Economic and Social Research Council’s new “peer review” college would surely enjoy a game or two. And the expert panels appointed by the Higher Education Funding Council for England to conduct the “impact pilot exercise” as part of the work on the forthcoming research excellence framework also deserve a chance to see things in a different way.

And apart from enjoying the delights of many marvellous games of football, it would be useful if they took a close look at the role of the referee in the games they watch.

Three important questions can be raised about research impact assessment. First, who should make these judgements – should it be research users rather than research providers? Second, should those who judge research impact be independent of vested interests? And third, should these judges be well prepared and highly incentivised?

In South Africa, the first question becomes: should the referees be entirely independent of the footballers whose performance they are assessing? The second becomes: should the referee decide whether it is a goal or not? Third: should the referee be properly paid for preparing well and doing a thorough job?

The football answers are yes, yes and yes. Unfortunately, the research assessment exercise answers are no, no and no. This largely explains why the RAE has been dumped.

Perhaps the architects of the RAE should be given an opportunity to present their approach to Fifa. On our first question, they would suggest that the footballers should referee their own games, as nobody else can understand the rules. The fact that they may want to bias the results in favour of their own team should be ignored.

Second, they would contend that clearly it should be the footballers who decide whether a goal has been scored or not, as an independent figure cannot be trusted with this important responsibility. Rather, groups of players should be asked to give their opinions on whether the ball has crossed the line. This model ensures that the teams with the biggest players can make sure the decisions go their way.

Third, they would argue that referees (read “research users”) should be entirely feeble figures who have no power or responsibility. It should be made clear to them that they are attending the game merely to give the appearance that the process of arriving at a result is fair.

We can safely assume that Fifa would be unimpressed with such a model. More troubling is the fact that the REF could be just as flawed.

Hence the importance of our South African Summer School. At half time, our relaxing scholars should focus sharply on the neglected role of the research users in the assessment of research impact. Are these individuals to be respected referees or undervalued ball boys?

Many will be quick to point out that there are significant differences between a football game and conducting research. They would be right to do so. However, we should recall that the new definition of research adopted by the REF is “a process of investigation leading to new insights effectively shared”.

This is a seismic shift, and one that is still not widely appreciated in the academy. Research that is not “effectively shared” no longer cuts it.

The key point here is that research users do not need to be experts in research methods in order to judge whether or not a research effort has had impact. Similarly, the referee in a World Cup game does not need to know whether a team has cleverly shifted from 4-4-2 to 4-3-2-1 part-way through the second half in order to decide, with the assistance of their support referees on the line, whether a goal has been scored or not.

I have argued before that UK higher education should welcome research impact, for both intellectual and strategic reasons.

However, and notwithstanding the important review work now going on, it seems clear that fresh thinking is needed in relation to the role of research users. Unless big steps are made on this front, the REF process will be vulnerable to the criticism that it has failed to respond to non-campus voices when it comes to research impact assessment – and ministers might just notice this.

The World Cup suggests that research users – like referees – should take impact assessment out of the hands of vested interests. They alone should decide, on the basis of the evidence presented, whether a given research effort has had impact or not. And, of course, British amateurism should be discarded. Research users should be highly trained and highly paid. This is the only way to ensure that they are not intimidated by noisy groups of oversized players.