Peer pressure

A while ago I was asked by a journal of high standing to referee a paper on a topic of which I had little knowledge. I immediately pointed this out, but the editorial assistant soon afterwards replied that it would be fine as they valued my opinion nevertheless. I was uncomfortable. The journal in question had, during the course of my career, rejected every one of my submissions, now numbering almost a dozen. Their position seemed to be that while they had consistently deemed me unworthy of publication, I was nevertheless worthy of passing judgement on what was good enough for them to publish: outside my own topic too. If I really knew what they wanted, I would no doubt have produced it myself by now. I turned down the assignment.

All academics scratch their heads over the peer-review system from time to time. On this occasion, I had a further worry. If this journal was happy to use a referee who was unqualified in at least two respects, who had been refereeing and rejecting all my submissions these past years? It brought home to me that the flaws of peer review affect us directly, that their caprice can make and break careers, and that academic publishing ought to be better controlled: perhaps even as a matter of public policy.

My experience concerned the appointment of the unqualified reviewer. Experts in a field are usually busy and unavailable to perform reviews. Editors often resort to lesser expertise. Early career academics might be especially tempted to take assignments outside their immediate competence if refereeing for a journal that will look good on their CV. Before one accepts such a task, however, I'd like us all to think about who will be refereeing our own submissions in turn.

The reasons peer review permits unqualified reviewers are no doubt complex. Of particular concern, however, is the fact that reviewers are also largely unaccountable. Most reviews are probably performed within an hour or two, and there appears to be little sanction against a sloppy negative review when editors are always short of space. Nothing is as infuriating as working on a paper for a long time - sometimes years - only to be told by a reviewer who has looked at it for an hour that it's not been thought through. All sorts of spurious counter-arguments can be ventured by a referee and the author has no right of reply. Referees know that they will almost certainly never have to explain or justify their criticisms, so they have a free shot at someone else's work. And needless to say, referees have to undergo no formal training for their role.

There is a big intellectual flaw in the peer-review system. It is inherently conservative. Suppose an editor succeeds in securing an ideal referee, eminent in the field and working on the same subject area. Such a reviewer could well have a vested interest in protecting a particular view or theory. Paradigm-changing work is unlikely to go down well with those who support an existing paradigm. Many papers in my own field work within an existing set of shared assumptions, offering only small additions or footnotes, which produce a dull read.

There is a variant on the paradigm-sustaining form of research publication, which I like to call the cartel. This is a theoretical possibility, although I have sometimes suspected it of happening for real. If an area is relatively small or new, with only a limited number of experts, then they could referee and recommend acceptance of each other's papers, thus boosting the reputation of the area and each other. Egregious cases, which I again take only as theoretical, would be the editorial-board cartel, where a journal publishes a disproportionately high number of papers by its own board, and a cartel among the editors themselves, where they publish each other's work as a mutual back-scratching.

I will add straight away that clearly we cannot assume malpractice just because a journal publishes work by a board member or other editor. Academics gain such positions chiefly because they are excellent scholars who, of course, should not be excluded from publishing just because of their position. But what we lack is the transparency and public accountability that could reassure us that malpractice never occurs. Government research funding to universities is often allocated according to publication record, yet there is little by way of quality assurance: no system of accreditation, for instance. Editorial and board appointments are usually a matter for learned societies and publishers. But we are also entitled to ask whether they are representative for gender, age, race, social background and so on. Is there anything other than self-regulation that stops them being Oxbridge/London old boy clubs?

And if public funding really is to be determined by the decisions of the journals, let us make them transparent. Let us see with each published paper a date of submission, date of acceptance and names of the referees. Let authors know in advance the average decision time. Let us know if asked to revise and resubmit whether it will go back to the same referees or to different ones.

A journal system introduces a degree of inertia into our intellectual output. The supposed benefits of peer review are offered as justification. But if we really want it as an academic community, we must try harder to iron out its obvious imperfections.