"The European Union economy doesn't look too bad - if you exclude Greece and perhaps a couple of others from the ." "Spurs are a decent team, if you don't count those games in which goalkeeper Heurelho Gomes gifted points to the opposition." "2011 was an excellent financial year for our unit trust, if you discount the poor performance of one or two of the companies in which we invest."
Yet we don't rate the EU by choosing which economies to include or exclude, nor does the team that wins the Barclays Premier League get to miss out its worst performances before the final ranking. And investors would be living in cloud cuckoo land if they thought they could ignore a unit trust's poorly performing components. We do, however, allow something similar to happen when assessing research in British universities. Why?
The guidance for the research excellence framework makes it clear that while higher education institutions "are strongly encouraged to submit the work of all their excellent researchers", they don't have to. They should develop a "code of practice on the fair and transparent selection of staff for inclusion in REF submissions", but for REF 2014, just as for the research assessment exercises it supplants, universities will be allowed to cherry-pick: "Each HEI must decide which individuals to select for submission, in accordance with its internal code of practice."
What lies behind this policy? What are the arguments for and against? Does it deliver transparency to the taxpayers who ultimately fund our universities?
One argument in support appears to be that we should not heap undue pressure on individuals who carry a high teaching and/or administrative load, and therefore cannot be expected to contribute four high-ranked research outputs. Yet an argument against is that we should value - and for the sake of transparency disclose - the research contributions of all full-time members of staff. Perhaps colleague X is carrying a high load, but then isn't it all the more impressive that s/he manages four papers at all, some of which even fall into the 2* category? (Even 1* is recognised "nationally in terms of originality, significance, and rigour".)
This raises the question of why we do not include people who have fewer than four research outputs. This again seems arbitrary. In my field, psychology, books don't stack up against 4* high-impact journal articles. Yet some scholars might contribute, say, one major monograph and one article, and on this basis be excluded from the REF (although it does allow for the submission of fewer than four outputs under special circumstances, such as illness). We also know that some papers require years of research, so it would be unreasonable to expect someone to submit four of these in one REF. Yet the current rules mean that such a person, if they have no other 4* outputs, will not be returned.
What about full-time staff who don't contribute at all on the research front? If they are carrying an exceptionally heavy teaching or administrative load, it would seem better to admit this fact and move them to a different contract. If they are not carrying such a burden, then this would seem to be the kind of decision that institutions have well-paid senior administrators to deal with. Fudging the issue of how we assess research is not the solution.
Finally, let us consider transparency. Suppose taxpayers want to see where their money is going or how well their local university is doing; perhaps the pressing question is helping their offspring to choose where to study. To keep things simple, let's assume they want to read psychology. From the RAE 2008 website, you can see what percentage of research activity in each university department's submission fell into each quality band (the "quality profile"). You can also see the number of "FTE (full-time equivalent) Category A staff submitted", but not the proportion. So the University of Oxford's 39.10 versus the University of Cambridge's 24 is pretty inscrutable.
If, however, you go up to the level of the university as a whole, you can see the percentage of staff submitted. At this point, the potential pitfalls of the current system become evident. The top three universities appear to have strength and depth (returning between 95 per cent and 97 per cent of their staff); but within the top 10, one (completely legitimately) submitted just 79 per cent. More worryingly, 55 institutions submitted less than 50 per cent of their scholars to RAE 2008.
As an Arsenal fan, I hope Tottenham Hotspurs' keeper Gomes makes a comeback (with a few more howlers). But we must continue to judge Spurs on all their games, with no exceptions. For the sake of transparency, and in fairness to taxpayers, the academy should do the same by including all full-time academic staff in assessments.