The Problem with Peer Review

Peer Review, Reviewed

Winston Churchill said, “Democracy is the worst form of government, except for all those other forms that have been tried. …”¹² In a nutshell, that summarizes how most of us feel about peer review. We’re not happy with it, but we would be less happy with the alternatives.

Richard Smith, the former editor of the British Medical Journal, identified the following flaws with peer review:¹³

• Lack of agreement among referees: Several studies indicate peer reviewers often disagree on the value of an article;
• Identification of peers: Some people are truly peerless; it can be challenging to identify a referee who truly has expertise in all of the areas represented by the authors;
• Introduction of delays: The peer review process introduces a bottleneck, which delays publication;
• Bias: Referees may be unfairly biased toward or against specific investigators or institutions. In one famous example, Peters and Ceci took 13 articles published by researchers from prominent institutions, changed the institution names, and resubmitted the articles to the same journals. Nine of the 13 articles were rejected as fatally flawed.¹⁴
• Bias against innovation: Truly novel hypotheses often fare poorly in the peer review process, which is more likely to reward incremental advances that reflect current models.

How can we improve the peer review process? One possibility is to consider how the process is blinded. Most peer review is single-blind, meaning the referees know the authors’ identities, but the authors never know who reviewed their work. Double-blind reviews, in which the authors’ identities are hidden from the reviewers, may prevent the referee from being dazzled by an author’s impressive credentials.

Although this seems like a sensible innovation, in practice, it may not improve the quality of peer review. Amy Justice, MD, PhD, et al. conducted a study in which articles submitted to Annals of Emergency Medicine, Annals of Internal Medicine, JAMA, Obstetrics & Gynecology, and Ophthalmology underwent both single-blind and double-blind review. Each article was assigned to two reviewers; one of the two reviewers was not allowed to know the identity of the authors.

Justice et al. found that double-blinding made no difference.¹⁵ Of the 118 articles reviewed, double-blinding had no discernable impact on the assessment of the article. Also, the blinding didn’t work. For one-third of the articles, the reviewers were able to correctly guess the identity of the author; this rate increased when an author was well known and would presumably increase further when the exact subject matter is especially abstruse.

Open peer review is another strategy that has been adopted by several journals over the past few years. Central to this strategy is the use of open reports, in which the referees’ comments are published along with the article, like a Letter to the Editor.

The open peer review strategy may potentially benefit the reader, the reviewer and the reviewed. By seeing how the criticism influenced the final product, the reader gains a deeper understanding of the science behind the paper. Knowing the review will be made public may encourage reviewers to write better reviews. Finally, signed reviews would be published and citable, which could serve as added incentive to serve as a peer reviewer.

That last point is key: Every year, investigators volunteer 68.5 million hours of their time working on peer review for journals. At the end of the day, peer review is fueled by altruism, and the truth is that the system is running out of gas. The Global State of Peer Review report, which surveyed more than 11,000 investigators worldwide, identified reviewer fatigue as a growing problem in the peer review process. In 2013, 1.9 invitations had to be issued to identify one peer reviewer; in 2017, that number increased to 2.4.¹⁶

Little wonder. I know that every article I review takes time away from signing notes, putting together research proposals, teaching, and other activities for which I am paid a salary. One of the great ironies of the peer review system is that the investigators who are most qualified to function as a peer are often the least likely to participate as a peer reviewer, because of other constraints on their time.

Moreover, you get what you pay for. It is difficult for most of us to prioritize a service we are providing gratis. Paying reviewers for their time might go a long way toward improving the quality of peer reviews.