Effects of Editorial Peer Review: A Systematic Review

Context Editorial peer review is widely used to select submissions to journals for publication and is presumed to improve their usefulness. Sufficient research on peer review has been published to consider a synthesis of its effects.

Methods To examine the evidence of the effects of editorial peer-review processes in biomedical journals, we conducted electronic and full-text searches of private and public databases to June 2000 and corresponded with the World Association of Medical Editors, European Association of Science Editors, Council of Science Editors, and researchers in the field to locate comparative studies assessing the effects of any stage of the peer-review process that made some attempt to control for confounding. Nineteen of 135 identified studies fulfilled our criteria. Because of the diversity of study questions, methods, and outcomes, we did not pool results.

Results Nine studies considered the effects of concealing reviewer/author identity. Four studies suggested that concealing reviewer or author identity affected review quality (mostly positively); however, methodological limitations make their findings ambiguous, and other studies' results were either negative or inconclusive. One study suggested that a statistical checklist can improve report quality, but another failed to find an effect of publishing another checklist. One study found no evidence that training referees improves performance and another showed increased interrater reliability; both used open designs, making interpretation difficult. Two studies of how journals communicate with reviewers did not demonstrate any effect on review quality. One study failed to show reviewer bias, but the findings may not be generalizable. One nonrandomized study compared the quality of articles published in peer-reviewed vs other journals. Two studies showed that editorial processes make articles more readable and improve the quality of reporting, but the findings may have limited generalizability to other journals.

Conclusions Editorial peer review, although widely used, is largely untested and its effects are uncertain.

The use of peers to assess the work of fellow scientists goes back at least 200 years.¹ It is usually assumed to raise the quality of the end product and to provide a mechanism for rational, fair, and objective decision making. Despite the fact that peer review has such a long history and is so well established, research into its effects is a recent phenomenon. However, the body of original research on the effects of peer review has been growing, and systematic review and synthesis may now be possible.² This review assesses the effects of processes undertaken as part of editorial peer review of original research studies submitted for paper or electronic publication in biomedical journals.

We used Cochrane methods to carry out our review, which will also be published in the Cochrane Database of Methodology Reviews.³ A detailed description of the search strategy, sources, and terms used is available in online Table A (available in PDF format).^4-6

We considered for inclusion all types of comparative studies in which some attempt to control for confounding had been made. This was an attempt to widen the criteria to collect as much evidence as possible, without collecting studies in which obvious confounding might explain any observed effect. We considered studies assessing the effects of any stage of the peer review process except those solely concerned with technical editing, which was the subject of another review.⁷ For example, we looked for studies of the effects of different ways of assigning submissions to assessors and eliciting opinions, decision-making procedures, and methods of feedback to authors.

Two reviewers (T.J. and P.A.) examined all retrieved citations. Studies for possible inclusion were retrieved in full. The same 2 reviewers then examined studies independently against the selection criteria. The same reviewers extracted information on study design and outcomes. We collected descriptive information on study quality, as reported herein. We identified 135 reports of studies that could possibly fulfill our inclusion criteria. Nineteen of these fulfilled our criteria and we excluded the remaining 116 studies from our review.

We did not pool the results of similar studies into a formal meta-analysis because no 2 studies were alike and all asked a slightly different study question or used different designs or outcome measures. We decided to group studies by the broad issues they addressed.

A summary of the 19 included studies is shown online Table B (available in PDF format ). A more detailed description will be available in our Cochrane review.³

We included 9 studies assessing the effect of blinding/masking or revealing authors' and/or reviewers' identities in some way (eg, exchanging reviews between peer reviewers) on the quality of external reviews.^8-16 All of the studies in this group were randomized controlled trials. Five studies^11-15 reported no apparent effect in quality of reviews, time taken to review, or tone of reviews. The 4 studies^8-10,16 that suggested that concealing reviewer or author identity affected review quality had methodological limitations that make their outcomes ambiguous. The most interesting methodological issue we identified in the design of the studies was the consistent difficulty in ensuring robust blinding procedures. This was probably a reflection of the relatively small world of editorial peer review, especially when applied to specialist areas of knowledge. Overall, most of the studies were inconclusive because blinding was ineffective and confidence intervals were not sufficiently narrow to allow firm conclusions to be made.

We included 2 studies that assessed the effects of submission checklists on outcomes.^17,18 The study by Gardner and Bond ¹⁸ shows some benefit from using a statistical checklist, although this finding may be of limited generalizability given the setting of the study (BMJ) and its small size. The other study found that publication of a checklist for economic studies did not appear to improve submissions.¹⁷

Two studies assessed the effects of the media used by journals to communicate with reviewers. One found that posting to the Internet may benefit authors but does not appear to affect the quality of reviews.¹⁹ The randomized controlled trial by Neuhauser and Koran²⁰ showed that warning reviewers of an impending review by telephone lengthened total turnaround time, although it shortened review time.

Two studies assessed the effects of training reviewers.^21,22 The results were ambiguous in the study by Callaham et al,²¹ showing no obvious effect, in contrast with the results of Strayhorn et al,²² which showed that interrater reliability increased. The open design of both studies makes minimization of biases impossible, and results are therefore difficult to interpret. Generalizability of results beyond the context of the specific interventions tested is questionable.

A randomized controlled trial by Ernst and Resch²³ assessed the presence and effects of reviewer bias. They found no evidence of reviewer bias toward an unconventional treatment. Despite its randomized design, bias may distort the results of this study (the response rate was 61%).

We included 1 study assessing the effects of peer review on study validity,²⁴ which failed to show any difference, although the small size and nonrandomized design of the study again makes generalizability of its results uncertain.

Two studies assessed the effects of peer review on study report quality.^25,26 Both studies showed a beneficial effect, but results may again have limited generalizability because of atypical settings (both journals studied are well resourced and keen on improving quality).

In summary, the quality of randomized studies was superior to that of trials with open design. (Quality refers to the ability of a study to answer its research question.) Overall, the small numbers of reviews and reviewers involved, potentially atypical settings (with a few major journals being the object of many studies), and many methodological weaknesses make both the internal and external validity of these studies difficult to interpret.

Our review was limited to evidence about peer review as practiced by biomedical journals. We discovered remarkably few well-designed studies assessing the effects of this process. Of the studies identified, the majority were focused on specific editorial processes, such as masking of reviewers and authors or use of checklists. Very few examined the broader effects of peer review. We assess the implications of this in a separate study.²⁷ The only study that attempted to address broader issues²⁴ is difficult to interpret because of methodological weaknesses and the likelihood of limited generalizability. The results of our systematic review are consistent with those of an earlier descriptive review.²

Given the widespread use of peer review and its importance, it is surprising that so little is known of its effects. However, the research needed to address these questions would require a well-funded and coordinated effort involving several sectors of the scientific community as well as the cooperation of large numbers of authors and editors, and the methodological issues in conducting proper studies of the subject are daunting.

The 2 principal functions of peer review, filtering out incorrect or inadequate work and improving the accuracy and clarity of published reports, directly reflect the 2 virtues that Francis Bacon in 1605 attributed to doubt: guarding "against errors" and causing issues that "would have been passed by lightly without intervention" to be "attentively and carefully observed." As such, the doubts contributed by peer review are an intrinsic and essential part of science, layered over the critical reviews that take place during the process of doing the work. Future research in biomedical peer review needs to recognize these basic concepts and must consider a number of specific underlying methodological issues. Among these are both the positive contributions of peer review and its abuses, the complexity of biomedical science, and the incremental nature of scientific progress. This is based on rare major discoveries and very frequent minor ones, clarifying patterns of life and disease and society's response to them. Moreover, most studies of biomedical journal peer review have been concerned with manuscripts that were accepted for publication. A number of important questions about peer review can only be answered, however, by studying rejected manuscripts as well as those that are accepted.

Until such research is undertaken, peer review should be regarded as an untested process with uncertain outcomes.