[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Original Investigation
June 2018

Accuracy and Reliability of Eye-Based vs Quadrant-Based Diagnosis of Plus Disease in Retinopathy of Prematurity

Author Affiliations
  • 1Department of Ophthalmology, Casey Eye Institute, Oregon Health and Science University, Portland
  • 2Department of Ophthalmology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
  • 3Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown
  • 4Massachusetts General Hospital and Brigham and Women’s Hospital Center for Clinical Data Science, Boston
  • 5Department of Ophthalmology and Visual Sciences, Illinois Eye and Ear Infirmary, University of Illinois at Chicago
  • 6Graduate School of Dentistry, Kyung Hee University, Seoul, Republic of Korea
  • 7Center for Global Health, College of Medicine, University of Illinois at Chicago
  • 8Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland
JAMA Ophthalmol. 2018;136(6):648-655. doi:10.1001/jamaophthalmol.2018.1195
Key Points

Question  Is plus disease in retinopathy of prematurity more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of individual quadrants?

Findings  In this multicenter cohort study including 141 preterm infants, graders diagnosed 197 eyes by reviewing the entire wide-angle retinal images (eye-based diagnosis; plus vs not plus) and subsequently by reviewing each of the cropped quadrant images (plus vs not plus), which was combined into a quadrant-based diagnosis. Graders had lower intragrader and intergrader agreement and agreement with reference standard diagnosis using quadrant-based diagnosis than eye-based diagnosis.

Meaning  Eye-based diagnosis may have advantages over quadrant-based diagnosis regarding reliability and accuracy.


Importance  Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants.

Objective  To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process.

Design, Setting, and Participants  In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017.

Interventions  Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc.

Main Outcomes and Measures  Intragrader and intergrader reliability (absolute agreement and κ statistic) and accuracy compared with the reference standard diagnosis.

Results  Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen κ range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss κ, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader.

Conclusions and Relevance  Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.