Should post-publication peer review be anonymous?

Among the criticisms that are frequently made of peer review is that referees’ anonymity allows laziness, discourtesy and conflicts of interest to flourish. Editors do their best to broker honestly, but grievances were not in short supply when Times Higher Education asked around earlier this year for particularly egregious examples of reviewer comments that people had received (“The worst piece of peer review I’ve ever received”, Features, 6 August).

There is even greater concern about anonymity in the case of so-called post-publication peer review (PPPR), which permits anyone with an internet connection to comment on published papers. In October, the journal Plant Physiology published a scathing editorial stating that contributions to the largely anonymous PPPR forum PubPeer often lacked “courtesy and common sense”. The article, written by the journal’s editor-in-chief Michael Blatt, Regius professor of botany at the University of Glasgow, said: “Given that the majority of comments show the most petty kind of scientific criticism, can there be any doubt that the intent often is to pillory, to do so publicly and without accountability?”

PubPeer’s founders were themselves anonymous until August, when they revealed their identity as part of an application to establish the site as a non-profit organisation. Here, in an exchange of letters with Philip Moriarty, professor of physics at the University of Nottingham, they set out their case for commenter anonymity, while Moriarty counters that Blatt is right to worry about the tone that this lends to discussion.

Meanwhile, Paul Benneworth, a principal researcher at the University of Twente in the Netherlands, argues that anonymity remains crucial to all forms of review – and outs himself as the probable author of the offending referee’s report highlighted by one of the contributors to THE’s article. Publishing may be in an era of increasing openness and transparency, but it is clear that the case for abandoning anonymity in research is far from clear-cut.

Dear Philip,

There is a growing awareness that a large proportion of published scientific research is of low quality and is unreproducible, with the problem being acute in life and social sciences. This has potentially enormous economic and human costs. Using taxpayers’ money to produce and build on unreliable research is wasteful, and researchers’ attempts to extend unreliable research will be unproductive and may damage their careers. Public policy or medical guidelines based upon flawed research may cause further waste and even cost lives.

Science is said to be self-correcting, but within the traditional publishing system, effective correction requires another publication. Such “rebuttals” are often discouraged by journals’ editorial guidelines. Even when they are not, writing and publishing a whole new paper via traditional publishing is often time-consuming, onerous, expensive and excruciatingly slow. Internet technologies can enable instant, widespread dissemination of any information relevant to a publication, and there is a broad consensus that this would offer a huge improvement over the current system. The only real point of dispute is whether anonymous commenting on published scientific work should be banned.

The US Constitution provides strong protection for anonymous speech in order, according to the US Supreme Court, to “protect unpopular individuals from retaliation” (McIntyre v. Ohio Elections Commission, 1995). But it is not an absolute right. How a community regulates anonymity typically hinges upon practical and moral considerations. In the absence of an overriding imperative, the decision to allow anonymous scientific discussion on PubPeer is based upon our careful balancing of the risks and benefits of doing so.

Our experience has been that anonymity greatly facilitates scientific discussion with little if any effect on the quality of comments. The success and growth of PubPeer (we receive approximately 1,000 comments and 500,000 page views a month) can be compared with the activity on other services that forbid anonymity: the absolutely moribund commenting facilities provided by journals and the relatively steady, low-level use of the PubMed Commons system that is offered by the PubMed biomedical archive. In addition, we find that without anonymity, many serious issues are simply never aired – presumably because the competitive and hierarchical environment in research causes most researchers to avoid public criticism of colleagues.

The main arguments against anonymity are that it allows commenters to blacken the names of their competitors with impunity and to hide conflicts of interest. Again, our practical experience has been that these issues rarely arise, for several reasons. First, comments on PubPeer must be based upon verifiable information (usually the data in a publication), which focuses discussion on the science. We moderate comments and provide reporting facilities to enforce this guideline. Second, only cogent scientific arguments will convince readers. Third, authors are notified of comments by email and are encouraged to explain or defend their work; many now do so. (Incidentally, most open questions on PubPeer would be resolved instantly by access to the raw data described in the publication, so we can expect such cases to decline with the trend towards data sharing.)

A weaker argument against anonymity is more subjective: the world would be a better place if everybody signed their comments. We agree. But our experience today shows that a critical mass of comments cannot be achieved without anonymity. Faced with this reality, PubPeer has chosen a commenting policy that is aimed at maximising the dissemination of relevant information and reducing the economic and human waste from unreliable research, at the potential cost of a few ruffled academic feathers.

Yours,
The PubPeer organisers

Dear PubPeer organisers,

I am an enthusiastic supporter of PPPR. A publication should represent the start, not the end, of scientific discussion and debate. I also regularly sing the praises of PubPeer, and your comments regarding open access to raw data also resonate strongly with me.

I would quibble, however, with your claim that there is a “broad consensus” that online PPPR is a welcome change. Very many academics and publishers see it as rather more of a significant threat than an opportunity. An editorial in the American Chemical Society’s Nano journal a couple of years back forcefully made this point. “When plagiarism or data manipulation is suspected, accusations should be reported directly to the journal in which the article is published. It is best not to make an accusation anonymously,” it said.

I disagree entirely that the most appropriate channel for scientific critique is a protracted and opaque letter-to-the-editor approach; my colleagues and I know from bitter experience just how ineffectual that process can be. And we also know just how essential PubPeer can be in highlighting questionable research.

But my views diverge significantly from yours when it comes to the ACS Nano editors’ misgivings – repeated in the recent Plant Physiology editorial – on the central role that anonymity plays in PPPR. I am firmly of the opinion that, too often, online anonymity debases communication and that it is impeding the wider acceptance of PPPR as the natural platform for scientific critique, criticism and debate.

You say that the “only real point of dispute” is whether anonymous commenting should be banned. This, however, is not at all representative of my position. I want to strongly discourage anonymous commenting, to attempt to change the commenting culture so that anonymity is not the norm. But I do not want to ban it. (Just how could one realistically ban it in any case?)

You admit that “the world would be a better place if everybody signed their comments” but you are resigned to accepting anonymous commenting as the norm because “the competitive and hierarchical environment in research causes most researchers to avoid public criticism of colleagues”. This is a shocking indictment of our research system; it’s a timid, rather grubby and damaging mindset that has no place in science. As my colleague Julian Stirling, a research fellow at the National Institute of Standards and Technology in Maryland, once put it during a lengthy online debate: “If you plan to get to the top of your discipline without ever publicly questioning established views, then you have a very strange view of science.”

Yours,
Philip Moriarty

Dear Philip,

Your response indicates that our positions are in fact quite close: you support post-publication peer review and recognise that anonymous commenting is here to stay. Our main disagreement is over the degree to which that anonymity should be encouraged or discouraged. Our decision at PubPeer has been to focus on the content of comments. If that is factual and relevant, then anonymity is acceptable, especially as it has in practice greatly encouraged information-sharing. In our experience, there is little truth in the assertion sometimes made that anonymous comments are of lower quality.

Like many other professional fields, science is competitive and hierarchical. In such systems, it is not surprising that the less established in particular may worry about retaliation for voicing criticism. We agree with you that the fear of retribution implied by the current desire for anonymity is a “shocking indictment of our research system”, but we blame the system, not the commenters.

We are hopeful that services such as PubPeer will slowly tip the balance towards the acceptance of open discussion and criticism. For this to happen, the research community will need to stop turning a blind eye to poor-quality work and become much more supportive of those who raise valid concerns. In the meantime, we think it is unfair of opponents of anonymity to demand that PhD students, postdocs or indeed any other scientists take risks that most established researchers, journals and institutions currently avoid. At PubPeer, we do not consider those who choose to share their expertise anonymously to be “grubby and damaging”.

Yours,
The PubPeer organisers

Dear PubPeer organisers,

Our positions on PPPR are indeed very close, but I remain soundly of the opinion that, too often, anonymity debases debate. You argue that if comments are “factual and relevant” then “anonymity is acceptable”. But acceptable to whom?

In the lengthy (and sometimes rather overheated) PubPeer thread that dissects the Plant Physiology editorial, one of the contributors draws a rather telling comparison between online anonymity and cold-calling. Would you find it acceptable to receive a phone call from an individual who refused to tell you their name, or anything about their background, but criticised your most recent paper at length?

I’ll again stress that this is not a question of the quality of the comments – I will agree that often, but certainly not invariably, anonymous comments are scientifically sound. My key concerns about anonymity are expressed much more eloquently than I could manage by Blatt himself: “Anonymity is intimidating in itself, regardless of how polite a commenter may be; psychologically, it wrong-foots the author from the start, and the price often is an absence of worthwhile discussion.” This asymmetry in communication is key.

You’ve argued in the past that anonymity counters the (greater?) asymmetry that exists between early career researchers who challenge published research and those prestigious professors whose work is being criticised. While I have quite some sympathy for your stance, it is nonetheless the case that anonymity is now the norm in PPPR, regardless of the extent to which career “fallout” might be a possibility. The PubPeer thread on the Blatt editorial is a great example of this – the vast majority of those commenting are anonymous, yet their comments are innocuous in the context of any type of potential damage to their careers. I will always challenge that norm.

Yours,
Philip Moriarty

Source: 

Corbis

In praise of harsh reviews

I read the THE feature highlighting supposedly egregious peer reviews with a growing sense of unease. One anonymous contributor recalled a referee who had criticised his paper for its supposedly poor conceptual awareness, “sub-tabloid” writing style and “abysmal” grammar, likening it to a mid-level undergraduate essay. The feature then stated that the paper had subsequently received plaudits after being published elsewhere, inviting us to conclude that traditional, “blind” peer review has descended into a snake pit of personal abuse, petty jealousies and pet theories, and no longer serves as a dispassionate system of scientific judgement.

An increasingly common view seems to be that if referees were obliged to sign their reviews, this would force them to offer more temperate, considered and constructive criticism, and that science would be all the better for it. As an editor of Regional Studies, Regional Science, a journal that proudly offers early career authors a mentored publication route to short-circuit these issues, I’m well aware of the pitfalls of standard peer review. But I would also argue that the option to write harsh reviews is central to any thriving science system that prizes integrity over system-gaming.

I must here declare an interest, because the comments that the THE contributor received may very well have come from me; I have certainly written a handful of similarly harsh reviews in my time. But I’ve also received two particularly scathing reviews myself that stuck in my memory. The first was on a paper I wrote as a postdoc: it said that my whole premise was based on my excitement with a literature that was probably new only to me. The second was an eight-page line-by-line demolition of a qualitative paper by a reviewer who apparently wanted quantitative tests of statistical significance.

On each occasion, I was stunned at first, but I have to concede that, on reflection, the reviewers were almost certainly right. Both papers were submitted to leading journals that favour a very particular style of article, and my research was not a comfortable fit in either. Without those harsh reviews, the editors might have been tempted to give me a chance to squeeze my findings into their journals’ style. And with the lure of a leading publication, the younger me might have been tempted to “play the game” and say something other than what my findings would strictly have warranted.

Many advocates of open review believe that it should be carried out post-publication. But if that model took off, I predict that a new class of specialist “star reviewers” would emerge. They would acquire the power to make or break papers in the same way that influential bloggers can make or break new products, and authors and editors would respond by mechanically writing and selecting only those papers that they believed would attract the star reviewers’ praise. This is hardly a recipe for fostering the creation of challenging new ideas.

Meanwhile, if open reviewing were carried out pre-publication, as it already is in some journals, reviewers would inevitably be inclined to shy away from negative comments for fear of upsetting authors who are also their potential future referees. Either way, publishing would become a little bit less honest – and it would be science and scientists that lost out.

Paul Benneworth is a principal researcher in the Center for Higher Education Policy Studies at the University of Twente in the Netherlands.