[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
[Skip to Content Landing]
July 13, 1994

Do Readers and Peer Reviewers Agree on Manuscript Quality?

Author Affiliations

From the Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania Medical Center, Philadelphia (Drs Justice and Berlin); Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia (Dr Justice); Editorial Offices, Annals of Internal Medicine, Philadelphia (Drs S. Fletcher and R. Fletcher); and Department of Oncology, The Johns Hopkins University School of Medicine, Baltimore, Md (Dr Goodman). Drs S. Fletcher and R. Fletcher are now with the Department of Ambulatory Care and Prevention, Harvard Medical School and Harvard Community Health Plan, Boston, Mass.

JAMA. 1994;272(2):117-119. doi:10.1001/jama.1994.03520020043011

Objective.  —To study readers' judgments of manuscript quality and the degree to which readers agreed with peer reviewers.

Design.  —Cross-sectional study.

Setting.  Annals of Internal Medicine.

Subjects.  —One hundred thirteen consecutive manuscripts reporting original research and selected for publication. Each of two manuscript versions (one before and one after revision) was judged by two readers, randomly sampled from those who said (based on the title) that they would read the article; one peer reviewer (peer), chosen in the usual way for Annals; and one expert in clinical research methods (expert). Each judge completed an instrument that included a 10-point subjective summary grade of manuscript quality.

Main Outcome Measures.  —Agreement on the 10-point summary grade of manuscript quality between reader-expert, reader-peer, and reader-reader.

Results.  —Readers and peers gave high grades (77% and 73% gave a grade of 5 or better, respectively), while experts were more critical (52% gave a grade of 5 or better; P<.0001). Agreement was relatively high among judge groups (in all cases, >69%) but agreement beyond chance was poor (κ<0.04). One third of readers (33%) thought that the manuscript had little relevance to their work.

Conclusion.  —Readers, like most peer reviewers, are generally satisfied with the quality of manuscripts but would like research articles to be more relevant to their clinical practice.(JAMA. 1994;272:117-119)