Single-blind vs Double-blind Peer Review in the Setting of Author Prestige

Most medical journals practice single-blind review¹ (authors’ identities known to reviewers), but double-blind review (authors’ identities masked to reviewers) may improve the quality of reviews.² Bias with single-blind review might be greatest in the setting of author or institutional prestige.²

This study was approved by the University of Washington institutional review board and conducted at Clinical Orthopaedics and Related Research (CORR), an orthopedic journal (2015 impact factor, 3.127; acceptance rate, 20%) that allows authors to select single-blind or double-blind peer review. Potential reviewers (based on expertise) were identified from the journal’s database, informed that a study on peer review would occur in the coming year, and allowed to opt out. To avoid influencing behavior, the details and timing of the study were not described and the trial was registered after completion. The protocol appears in the Supplement.

Between June 2014 and August 2015, reviewers were randomized via random number table (1:1 ratio in blocks of 8) to receive single-blind or double-blind versions of an otherwise identical fabricated manuscript, which was putatively written by 2 past presidents of the American Academy of Orthopaedic Surgeons from prominent institutions. The manuscript described a prospective study on a nonclinical topic of broad interest (team training to improve communication and safety in the operating room). Five subtle errors were included, ranging from numerical mistakes to an error in the conclusion, to determine differences in how critically the manuscript was examined.

The primary outcome was recommendation of acceptance or rejection. Grades of “accept” and “reject” were taken at face value and, for grades of “major revision” and “minor revision,” reviewers’ comments were analyzed for language recommending rejection by researchers blinded to group allocation. Secondary outcomes were the number of intentionally placed errors detected and quality scores for the Methods. Post hoc, scores for the other categories usually solicited by CORR were compared.

Based on 80% power and an α of .05, 98 reviewers were needed to detect a 20% difference in acceptance rates. Statistical analysis used χ², Fisher exact, student t, and negative binomial testing, and multivariable logistic regression (SAS [SAS Institute], version 9). Significance was defined as a 2-sided P value of less than .05.

Of 905 potential reviewers, 80 opted out. Of 256 invited to participate, 119 completed the review; this participation rate (46%) is typical for CORR. Participants had completed more prior reviews than nonparticipants (12 vs 7; P < .001), but were similar by sex and nationality. There were no significant differences between the single-blind (n = 62) and double-blind (n = 57) groups on sex (men: 95% vs 88%), nationality (American: 68% vs 74%), prior reviews completed (11 vs 12), or mean time to review completion (13 days for both groups; P > .05 for all comparisons).

Reviewers were more likely to recommend acceptance when the prestigious authors’ names and institutions were visible (single-blind review) than when they were redacted (double-blind review) (87% vs 68%; multivariable relative risk, 1.28 [95% CI, 1.06-1.39], P = .02) (Table 1) and also gave higher ratings for the methods and other categories (Table 2). There was no difference in the number of errors detected.

Although 1 study found double-blind reviews to be of higher quality,³ others detected no differences.^4-6 However, these studies were limited by unsuccessful blinding^3-5 and conducted at journals that routinely use a single form of peer review.^3-6 This study was conducted at a journal that routinely uses both approaches to peer review, reducing the likelihood of a Hawthorne effect. The study also examined peer review in the setting of author and institutional prestige, a situation potentially yielding biased reviews.

The study was conducted at a single orthopaedic journal; generalizability to other journals and other fields of medicine is unknown. With larger numbers, sex and nationality differences between the single-blind and double-blind groups could have been significant. The study used a single test manuscript; results could have differed with different quality manuscripts. Although reviewers could have identified the test manuscript as part of a study, there was no evidence that anyone did, and several steps were taken to prevent this including sending invitations in small batches and not inviting concurrent reviewers from the same institution or reviewers from the high-prestige authors’ institutions.

Corresponding Author: Kanu Okike, MD, MPH, Department of Orthopaedic Surgery, Kaiser Moanalua Medical Center, 3288 Moanalua Rd, Honolulu, HI 96821 (okike@post.harvard.edu).

Author Contributions: Dr Okike had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All Authors.

Acquisition, analysis, or interpretation of data: All Authors.

Drafting of the manuscript: Okike.

Critical revision of the manuscript for important intellectual content: All Authors.

Statistical analysis: Okike.

Administrative, technical, or material support: All Authors.

Study supervision: Kocher, Leopold.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Okike reports educational meeting support from Depuy, Stryker, Synthes and Zimmer, and teaching honoraria from Synthes. Dr Kocher reports stock options from Fixes 4 Kids and Pivot Medical; consultancy fees from OrthoPediatrics, Ossur and Smith & Nephew; research support from Ossur; royalties from Saunders/Mosby-Elsevier; and financial or material support from Saunders/Mosby-Elsevier. Dr Leopold reports serving as the editor in chief of Clinical Orthopaedics and Related Research (CORR). No other disclosures are reported.

Additional Contributions: We thank Brian Robinson, BA (CORR), for his assistance with data collection and the overall conduct of this investigation; compensation was received for these activities as part of his employment by CORR. The authors would also like to thank Daniel Berry, MD (Mayo Clinic), and James Herndon, MD, MBA (Massachusetts General Hospital), for lending their names to the test manuscript, as well as all the reviewers who participated in the study; no compensation was received for any of these activities.

Trial Registration: clinicaltrials.gov Identifier: NCT02739737.