[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.159.197.114. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Article
July 13, 1994

Evaluating Peer ReviewsPilot Testing of a Grading Instrument

Author Affiliations

From the Journal of Vascular and Interventional Radiology, Nashville, Tenn (Ms Feurer); Journal of Vascular and Interventional Radiology Editorial Office and the Miami Vascular Institute, Baptist Hospital of Miami, Miami, Fla (Dr Becker and Ms Ramirez); the Mallinckrodt Institute of Radiology, Washington University School of Medicine, St Louis, Mo (Drs Picus, Darcy, and Hicks).

JAMA. 1994;272(2):98-100. doi:10.1001/jama.1994.03520020024006
Abstract

Objective.  —To measure the reliability and preliminary validity of a grading instrument for editors to evaluate the quality of peer reviews.

Design.  —The consecutive sample design included 53 reviews of 23 manuscripts. Reviews were systematically assigned to interrater reliability (n=41; power greater than 0.90 to detect a difference of greater than one point) and preliminary criterion-related validity (n=12) subsamples. Content validity was closely examined.

Setting.  —Nonclinical.

Participants.  —Three graders evaluated reliability. One individual examined content validity and two editors tested preliminary criterion-related validity.

Intervention (Instrument).  —Attributes reflecting two basic dimensions, review content and format, were identified and scored (values are possible points/percent contribution): timeliness, 3/21%; grade sheet, 1/7%; etiquette, 1/7%; sectional narratives, 3/21%; citations, 2/14%; narrative summary, 2/14%; and insights, 2/14%. A scoring guide was provided.

Main Outcome Measures.  —Statistical analyses used to test the interrater reliability of the total score included the intraclass correlation coefficient and analysis of variance with the expectation to uphold the null hypothesis. Kendall's coefficient of concordance was used to test preliminary criterion-related validity.

Results.  —The intraclass correlation coefficient was.84 (P<.001) and a lack of difference between mean scores was demonstrated by analysis of variance (P=.46). Content validity was confirmed and preliminary criterion-related validity was indicated (Kendall's coefficient of concordance=.94, P=.038). Conclusions.—The instrument is reliable. Content validation has been completed, and further criterion-related validation is warranted.(JAMA. 1994;272:98-100)

×