[Skip to Content]
[Skip to Content Landing]
Figure 1.
Workflow of Image Grading With Eye-Based Diagnosis
Workflow of Image Grading With Eye-Based Diagnosis
Figure 2.
Workflow of Image Grading With Quadrant-Based Diagnosis
Workflow of Image Grading With Quadrant-Based Diagnosis
Figure 3.
Representative Case of Eye-Based Diagnosis Showing the Entire Posterior Retinal Image
Representative Case of Eye-Based Diagnosis Showing the Entire Posterior Retinal Image

All 6 graders diagnosed this eye as plus disease based on the entire retinal image. The numerical value indicates the number of graders classifying the image as plus disease (first number), pre-plus disease (middle number), or normal (last number).

Figure 4.
Representative Case of Quadrant-Based Diagnosis Showing the Cropped Quadrant Images
Representative Case of Quadrant-Based Diagnosis Showing the Cropped Quadrant Images

Cropped quadrant images of Figure 3. Quadrant-based diagnosis, which combines independent grading of individual quadrants, was not plus disease by all 6 graders. The numerical value on each image indicates the number of graders classifying that image as plus disease (first number), pre-plus disease (middle number), or normal (last number).

Table.  
Intergrader Reliability for Plus Disease Diagnosis Using Eye-Based and Quadrant-Based Approachesa
Intergrader Reliability for Plus Disease Diagnosis Using Eye-Based and Quadrant-Based Approachesa
1.
Gilbert  C.  Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control.  Early Hum Dev. 2008;84(2):77-82.PubMedGoogle ScholarCrossref
2.
Hartnett  ME.  Pathophysiology and mechanisms of severe retinopathy of prematurity.  Ophthalmology. 2015;122(1):200-210.PubMedGoogle ScholarCrossref
3.
International Committee for the Classification of Retinopathy of Prematurity.  An international classification of retinopathy of prematurity.  Pediatrics. 1984;74(1):127-133.PubMedGoogle Scholar
4.
International Committee for the Classification of Retinopathy of Prematurity.  The International Classification of Retinopathy of Prematurity revisited.  Arch Ophthalmol. 2005;123(7):991-999.PubMedGoogle ScholarCrossref
5.
Cryotherapy for Retinopathy of Prematurity Cooperative Group.  Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results.  Arch Ophthalmol. 1988;106(4):471-479.PubMedGoogle ScholarCrossref
6.
Early Treatment For Retinopathy Of Prematurity Cooperative Group.  Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial.  Arch Ophthalmol. 2003;121(12):1684-1694.PubMedGoogle ScholarCrossref
7.
The STOP-ROP Multicenter Study Group.  Supplemental Therapeutic Oxygen for Prethreshold Retinopathy Of Prematurity (STOP-ROP), a randomized, controlled trial: I: primary outcomes.  Pediatrics. 2000;105(2):295-310.PubMedGoogle ScholarCrossref
8.
Mintz-Hittner  HA, Kennedy  KA, Chuang  AZ; BEAT-ROP Cooperative Group.  Efficacy of intravitreal bevacizumab for stage 3+ retinopathy of prematurity.  N Engl J Med. 2011;364(7):603-615.PubMedGoogle ScholarCrossref
9.
Chiang  MF, Jiang  L, Gelman  R, Du  YE, Flynn  JT.  Interexpert agreement of plus disease diagnosis in retinopathy of prematurity.  Arch Ophthalmol. 2007;125(7):875-880.PubMedGoogle ScholarCrossref
10.
Reynolds  JD, Dobson  V, Quinn  GE,  et al; CRYO-ROP and LIGHT-ROP Cooperative Study Groups.  Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies.  Arch Ophthalmol. 2002;120(11):1470-1476.PubMedGoogle ScholarCrossref
11.
Wallace  DK, Quinn  GE, Freedman  SF, Chiang  MF.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity.  J AAPOS. 2008;12(4):352-356.PubMedGoogle ScholarCrossref
12.
Gschließer  A, Stifter  E, Neumayer  T,  et al.  Inter-expert and intra-expert agreement on the diagnosis and treatment of retinopathy of prematurity.  Am J Ophthalmol. 2015;160(3):553-560.e3.PubMedGoogle ScholarCrossref
13.
Slidsborg  C, Forman  JL, Fielder  AR,  et al.  Experts do not agree when to treat retinopathy of prematurity based on plus disease.  Br J Ophthalmol. 2012;96(4):549-553.PubMedGoogle ScholarCrossref
14.
Hewing  NJ, Kaufman  DR, Chan  RV, Chiang  MF.  Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts.  JAMA Ophthalmol. 2013;131(8):1026-1032.PubMedGoogle ScholarCrossref
15.
Campbell  JP, Kalpathy-Cramer  J, Erdogmus  D,  et al; Imaging and Informatics in Retinopathy of Prematurity Research Consortium.  Plus disease in retinopathy of prematurity: a continuous spectrum of vascular abnormality as a basis of diagnostic variability.  Ophthalmology. 2016;123(11):2338-2344.PubMedGoogle ScholarCrossref
16.
Kalpathy-Cramer  J, Campbell  JP, Erdogmus  D,  et al; Imaging and Informatics in Retinopathy of Prematurity Research Consortium.  Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis.  Ophthalmology. 2016;123(11):2345-2351.PubMedGoogle ScholarCrossref
17.
Campbell  JP, Ataer-Cansizoglu  E, Bolon-Canedo  V,  et al; Imaging and Informatics in ROP (i-ROP) Research Consortium.  Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis.  JAMA Ophthalmol. 2016;134(6):651-657.PubMedGoogle ScholarCrossref
18.
Campbell  JP, Ryan  MC, Lore  E,  et al; Imaging & Informatics in Retinopathy of Prematurity Research Consortium.  Diagnostic discrepancies in retinopathy of prematurity classification.  Ophthalmology. 2016;123(8):1795-1801.PubMedGoogle ScholarCrossref
19.
Keck  KM, Kalpathy-Cramer  J, Ataer-Cansizoglu  E, You  S, Erdogmus  D, Chiang  MF.  Plus disease diagnosis in retinopathy of prematurity: vascular tortuosity as a function of distance from optic disk.  Retina. 2013;33(8):1700-1707.PubMedGoogle ScholarCrossref
20.
Rao  R, Jonsson  NJ, Ventura  C,  et al.  Plus disease in retinopathy of prematurity: diagnostic impact of field of view.  Retina. 2012;32(6):1148-1155.PubMedGoogle ScholarCrossref
21.
Wallace  DK, Freedman  SF, Zhao  Z, Jung  SH.  Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity.  Arch Ophthalmol. 2007;125(11):1523-1530.PubMedGoogle ScholarCrossref
22.
Abbey  AM, Besirli  CG, Musch  DC,  et al.  Evaluation of screening for retinopathy of prematurity by ROPtool or a lay reader.  Ophthalmology. 2016;123(2):385-390.PubMedGoogle ScholarCrossref
23.
Daniel  E, Quinn  GE, Hildebrand  PL,  et al; e-ROP Cooperative Group.  Validated system for centralized grading of retinopathy of prematurity: telemedicine approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study.  JAMA Ophthalmol. 2015;133(6):675-682.PubMedGoogle ScholarCrossref
24.
World Medical Association.  World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects.  JAMA. 2013;310(20):2191-2194. doi:10.1001/jama.2013.281053.Google ScholarCrossref
25.
Ryan  MC, Ostmo  S, Jonas  K,  et al.  Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology.  AMIA Annu Symp Proc. 2014;2014:1902-1910.PubMedGoogle Scholar
26.
StatsToDo. Kappa (Cohen and Fleiss) for ordinal data program. https://www.statstodo.com/CohenFleissKappa_Pgm.php. Accessed September 5, 2017.
27.
Landis  JR, Koch  GG.  The measurement of observer agreement for categorical data.  Biometrics. 1977;33(1):159-174.PubMedGoogle ScholarCrossref
28.
Gschließer  A, Stifter  E, Neumayer  T,  et al.  Effect of patients’ clinical information on the diagnosis of and decision to treat retinopathy of prematurity  [published online October 3, 2017].  Retina. 2017. doi:10.1097/IAE.0000000000001864PubMedGoogle Scholar
29.
Fleck  BW, Williams  C, Juszczak  E,  et al; BOOST II Retinal Image Digital Analysis (RIDA) Group.  An international comparison of retinopathy of prematurity grading performance within the Benefits of Oxygen Saturation Targeting II trials.  Eye (Lond). 2018;32(1):74-80.PubMedGoogle ScholarCrossref
30.
Wittenberg  LA, Jonsson  NJ, Chan  RV, Chiang  MF.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity.  J Pediatr Ophthalmol Strabismus. 2012;49(1):11-19.PubMedGoogle ScholarCrossref
31.
Wallace  DK.  Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis).  Trans Am Ophthalmol Soc. 2007;105:594-615.PubMedGoogle Scholar
32.
Koreen  S, Gelman  R, Martinez-Perez  ME,  et al.  Evaluation of a computer-based system for plus disease diagnosis in retinopathy of prematurity.  Ophthalmology. 2007;114(12):e59-e67.PubMedGoogle ScholarCrossref
33.
Gelman  R, Martinez-Perez  ME, Vanderveen  DK, Moskowitz  A, Fulton  AB.  Diagnosis of plus disease in retinopathy of prematurity using Retinal Image Multiscale Analysis.  Invest Ophthalmol Vis Sci. 2005;46(12):4734-4738.PubMedGoogle ScholarCrossref
34.
Ataer-Cansizoglu  E, Bolon-Canedo  V, Campbell  JP,  et al; i-ROP Research Consortium.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis.  Transl Vis Sci Technol. 2015;4(6):5.PubMedGoogle ScholarCrossref
35.
Cabrera  MT, Freedman  SF, Kiely  AE, Chiang  MF, Wallace  DK.  Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity.  J AAPOS. 2011;15(1):40-44.PubMedGoogle ScholarCrossref
36.
Kalpathy-Cramer  J, Campbell  JP, Kim  S,  et al.  Deep learning for the identification of plus disease in retinopathy of prematurity  [ARVO abstract 5554].  Invest Ophthalmol Vis Sci. 2017;58:5554.Google Scholar
37.
Wilkinson  CP, Ferris  FL  III, Klein  RE,  et al; Global Diabetic Retinopathy Project Group.  Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales.  Ophthalmology. 2003;110(9):1677-1682.PubMedGoogle ScholarCrossref
38.
Quinn  GE, Ying  GS, Daniel  E,  et al; e-ROP Cooperative Group.  Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity.  JAMA Ophthalmol. 2014;132(10):1178-1184.PubMedGoogle ScholarCrossref
39.
Daskivich  LP, Vasquez  C, Martinez  C  Jr, Tseng  CH, Mangione  CM.  Implementation and evaluation of a large-scale teleretinal diabetic retinopathy screening program in the Los Angeles County Department of Health Services.  JAMA Intern Med. 2017;177(5):642-649.PubMedGoogle ScholarCrossref
40.
Vinekar  A, Gilbert  C, Dogra  M,  et al.  The KIDROP model of combining strategies for providing retinopathy of prematurity screening in underserved areas in India using wide-field imaging, tele-medicine, non-physician graders and smart phone reporting.  Indian J Ophthalmol. 2014;62(1):41-49.PubMedGoogle ScholarCrossref
41.
Myung  JS, Paul Chan  RV, Espiritu  MJ,  et al.  Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training.  J AAPOS. 2011;15(6):573-578.PubMedGoogle ScholarCrossref
42.
Patel  SN, Martinez-Castellanos  MA, Berrones-Medina  D,  et al; GEN-ROP; i-ROP Research Consortium.  Assessment of a tele-education system to enhance retinopathy of prematurity training by international ophthalmologists-in-training in Mexico.  Ophthalmology. 2017;124(7):953-961.PubMedGoogle ScholarCrossref
43.
Nagiel  A, Espiritu  MJ, Wong  RK,  et al.  Retinopathy of prematurity residency training.  Ophthalmology. 2012;119(12):2644-5.e1, 2.PubMedGoogle ScholarCrossref
Original Investigation
June 2018

Accuracy and Reliability of Eye-Based vs Quadrant-Based Diagnosis of Plus Disease in Retinopathy of Prematurity

Author Affiliations
  • 1Department of Ophthalmology, Casey Eye Institute, Oregon Health and Science University, Portland
  • 2Department of Ophthalmology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
  • 3Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown
  • 4Massachusetts General Hospital and Brigham and Women’s Hospital Center for Clinical Data Science, Boston
  • 5Department of Ophthalmology and Visual Sciences, Illinois Eye and Ear Infirmary, University of Illinois at Chicago
  • 6Graduate School of Dentistry, Kyung Hee University, Seoul, Republic of Korea
  • 7Center for Global Health, College of Medicine, University of Illinois at Chicago
  • 8Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland
JAMA Ophthalmol. 2018;136(6):648-655. doi:10.1001/jamaophthalmol.2018.1195
Key Points

Question  Is plus disease in retinopathy of prematurity more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of individual quadrants?

Findings  In this multicenter cohort study including 141 preterm infants, graders diagnosed 197 eyes by reviewing the entire wide-angle retinal images (eye-based diagnosis; plus vs not plus) and subsequently by reviewing each of the cropped quadrant images (plus vs not plus), which was combined into a quadrant-based diagnosis. Graders had lower intragrader and intergrader agreement and agreement with reference standard diagnosis using quadrant-based diagnosis than eye-based diagnosis.

Meaning  Eye-based diagnosis may have advantages over quadrant-based diagnosis regarding reliability and accuracy.

Abstract

Importance  Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants.

Objective  To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process.

Design, Setting, and Participants  In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017.

Interventions  Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc.

Main Outcomes and Measures  Intragrader and intergrader reliability (absolute agreement and κ statistic) and accuracy compared with the reference standard diagnosis.

Results  Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen κ range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss κ, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader.

Conclusions and Relevance  Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

Introduction

Retinopathy of prematurity (ROP) is a proliferative retinopathy affecting premature infants and is a major cause of childhood blindness worldwide.1,2 Development of an international classification system of ROP (ICROP) has provided the infrastructure for improving clinical care and supporting multicenter research trials in ROP.3-8 Among the diagnostic parameters, presence of plus disease has been found by the Cryotherapy for ROP5 and the Early Treatment for ROP6 studies to be the most critical finding that indicates potentially blinding disease that requires treatment.

Plus disease is defined as venous dilatation and arteriolar tortuosity within the posterior retinal vessels, which is greater than or equal to that of a standard published photograph selected by expert consensus during the 1980s.3 Beyond comparison with a standard photograph, the Cryotherapy for ROP5 and the Early Treatment for ROP6 trials did not specifically indicate how plus disease should be diagnosed. In 2000, the multicenter Supplemental Therapeutic Oxygen for Prethreshold ROP7 study defined that plus disease should be diagnosed if there was sufficient dilatation and tortuosity in at least 2 quadrants of the retina. In 2005, this definition requiring 2 or more quadrants of vascular abnormality was incorporated into the revised ICROP.4 However, to our knowledge, no specific guidance was provided regarding how to apply this definition toward actual clinical diagnosis.

Despite a standardized definition of plus disease, there is significant diagnostic variability even among experts in the diagnostic outcome as well as in the diagnostic process of plus disease.9-20 This is a major problem because the presence of plus disease is the key indicator for severe, treatment-requiring disease.6 Furthermore, experts often deviate from the published definition of plus disease by incorporating features into their diagnosis that are not part of the published definition.9-15,18-20

Both ophthalmoscopic and image-based examination provide information about all quadrants simultaneously. For that reason, it has never been clear whether plus disease diagnosis should be performed using quadrant-based (ie, diagnose each quadrant individually as plus, pre-plus, or normal and integrate findings into an overall diagnosis)11,21-23 or eye-based (ie, assess overall retinal appearance)12,17 methods. Furthermore, it is not clear which of these approaches is more accurate. The purpose of this study is to directly compare the intragrader reliability, intergrader reliability, and overall accuracy of eye-based vs quadrant-based diagnosis for plus disease.

Methods

This study was approved by the institutional review board at the coordinating center (Oregon Health and Science University, Portland) and at each of 8 study centers (Columbia University, New York, New York; University of Illinois at Chicago; William Beaumont Hospital, Royal Oak, Michigan; Children’s Hospital Los Angeles, Los Angeles, California; Cedars-Sinai Medical Center, Los Angeles, California; University of Miami, Miami, Florida; Weill Cornell Medical Center, New York, New York; and Asociacion para Evitar la Ceguera en Mexico, Mexico City, Mexico). This study was conducted in accordance with the Declaration of Helsinki.24 Written informed consent for the study was obtained from parents of all infants enrolled.

Data Set

As part of the Imaging and Informatics in Retinopathy of Prematurity study, a multicenter cohort study, we developed a database of 197 wide-angle retinal images of the posterior retina from 141 preterm infants, which were taken using a wide-angle fundus camera (RetCam; Natus Medical Incorporated) between July 2011 and December 2016. The mean (SD) gestational age of included infants was 27.0 (2.6) weeks, and 65 (46.1%) were female. Of the 141 infants, 116 (82.3%) were white, 12 (8.5%) were African American, and 13 (9.2%) were other races/ethnicities. At each study center, infants underwent serial clinical examinations by a retinal specialist or pediatric ophthalmologist experienced in ROP. This was done by ophthalmoscopic examination at 8 study centers and a combination of telemedicine and ophthalmoscopy at 1 study center. All clinical examination findings were documented using ICROP criteria. A reference standard diagnosis (RSD) using ICROP criteria was assigned to each of the 197 images, as previously described.17,18,25 In brief, the RSD was established based on the consensus diagnosis that combined the image-based diagnosis (typically from 5 images: posterior, temporal, nasal, superior, and inferior retina) by 3 independent trained graders and the clinical diagnosis as described above. Among the 197 study images, 31 (15.7%) had an RSD of plus disease, 62 (31.5%) had an RSD of pre-plus disease, and 104 (52.8%) had an RSD of normal. Each of the 197 wide-angle retinal images was cropped into 4 quadrant images (superotemporal, inferotemporal, superonasal, and inferonasal) by dividing the posterior retinal images with vertical and horizontal lines bisecting the optic disc using image processing software (ImageJ; National Institutes of Health).

Image Grading

Six trained graders (4 ophthalmologists experienced in ROP and 2 nonphysicians experienced in review of ROP images) independently graded 197 images with eye-based diagnosis using a web-based system developed by the authors. Additional demographic information, such as gestational age, was not provided to graders. An eye-based diagnosis was defined as the diagnosis (plus, pre-plus, or normal) given after reviewing the entire image covering all 4 quadrants (Figure 1).

Subsequently, the 788 quadrant images were provided to the graders 1 at a time in random order (Figure 2). Graders were asked to diagnose the image as plus, pre-plus, or normal. A quadrant-based diagnosis of plus disease diagnosis was assigned when 2 or more quadrants of an image were independently diagnosed as having plus disease. The workflow of image grading is shown in Figure 1 and Figure 2.

Statistical Analysis

Two-level plus disease diagnosis (plus or not plus) was used for all statistical analyses. Intragrader agreement between eye-based and quadrant-based diagnosis and between individual quadrant (superonasal, inferonasal, superotemporal, and inferotemporal quadrant) assessment and eye-based diagnosis was analyzed using absolute agreement and κ statistic for 6 graders. Accuracy of each diagnostic approach and individual quadrant assessment compared with the RSD was also analyzed. Intergrader agreement of 6 graders in eye-based and quadrant-based diagnosis was analyzed using absolute agreement, κ statistic, and intraclass correlation coefficient. Cohen and Fleiss κ statistics were used for assessing agreement between 2 graders and among more than 2 graders, respectively. The symmetry of disagreement between eye-based and quadrant-based diagnosis was evaluated with McNemar and binomial tests, which determined whether the quadrant-based diagnosis undercalled or overcalled plus disease compared with eye-based diagnosis.

Data analysis was performed using SPSS Statistics version 24 (IBM), Microsoft Excel for Mac version 15.33 (Microsoft), R version 3.4.1 (The R Foundation), and the web-based Kappa Program.26 The κ statistic was interpreted using a commonly accepted scale: 0.21 to 0.40 indicated fair agreement, 0.41 to 0.60, moderate agreement, 0.61 to 0.80, substantial agreement, and 0.81 to 1.0, near-perfect agreement.27 All P values were 2-tailed, and significance was set at P < .05.

Results
Intragrader Reliability

eTable 1 in the Supplement shows intragrader agreement between eye-based and quadrant-based diagnosis by 6 graders. Cohen κ ranged from 0.32 to 0.75, showing variable agreement (fair to substantial) between eye-based and quadrant-based plus disease diagnoses. McNemar tests indicated statistically significant discordance for plus disease diagnosis for 5 of 6 graders between eye-based and quadrant-based methods (eTable 1 in the Supplement). Among the 5 graders with discordant diagnoses, 4 showed a discrepancy in the direction of not plus disease for quadrant-based diagnosis (eTable 1 in the Supplement).

Figure 3 and Figure 4 show a representative case of the discrepancy between eye-based and quadrant-based diagnosis. The RSD for this eye was plus disease, and all 6 graders diagnosed the eye as plus disease based on the full retinal image (Figure 3). However, the quadrant-based diagnosis was not plus disease by all 6 graders (ie, all 6 graders diagnosed fewer than 2 quadrants as plus disease when viewed individually) (Figure 4).

Agreement between individual quadrant assessment and eye-based diagnosis or RSD in all 197 images found no remarkable differences between individual quadrants compared with eye-based diagnosis or RSD (eTables 2 and 3 in the Supplement). However, when analyzing 31 images with RSD of plus disease, the inferotemporal quadrant showed higher absolute agreement with eye-based diagnosis than the inferonasal quadrant in 5 of 6 graders (eTable 4 in the Supplement), the inferotemporal quadrant showed higher absolute agreement with RSD than the inferonasal quadrant in 5 of 6 graders, and the superotemporal quadrant showed higher absolute agreement with RSD than the inferonasal quadrant in 4 of 6 graders (eTable 5 in the Supplement).

Intergrader Reliability

Intergrader agreement among the 6 graders is shown in the Table. Cohen κ statistic of pair-wise agreement between all graders indicated that eye-based diagnosis showed substantial to near-perfect agreement. By contrast, quadrant-based diagnosis showed only fair to substantial agreement. Fleiss multigrader κ statistic showed substantial agreement for eye-based diagnosis while quadrant-based diagnosis had only moderate agreement. Moreover, nonoverlapping 95% confidence intervals of Fleiss κs and intraclass correlation coefficients suggest that intergrader agreement for eye-based diagnosis was higher than for quadrant-based diagnosis and individual quadrant diagnosis.

Diagnostic Accuracy

Cohen κ statistic showed that the accuracy of eye-based diagnosis by the 6 graders compared with the RSD was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader (eTable 6 in the Supplement). The 95% confidence intervals of 2 graders do not overlap between eye-based and quadrant-based plus diagnosis (eTable 6 in the Supplement).

Discussion

This study assessed whether grading plus disease individually by quadrant provided the same diagnosis as grading at the whole eye level. There are 3 key findings. First, intragrader agreement between eye-based and quadrant-based diagnosis is limited. There was variability among the graders, and 4 of 6 graders underdiagnosed plus disease when using a quadrant-based approach compared with an eye-based approach. Second, intergrader agreement with quadrant-based diagnosis was lower than with eye-based diagnosis. Third, diagnostic accuracy with quadrant-based diagnosis was lower than with eye-based diagnosis. Taken together, these findings suggest that eye-based diagnosis may have advantages over quadrant-based diagnosis regarding reliability and accuracy.

The first key finding is that intragrader agreement between eye-based and quadrant-based plus diagnosis is imperfect. There are several potential explanations. First, real-world assessment of plus disease grading in individual quadrants may be influenced by the appearance of adjacent quadrants. This could explain the underdiagnosis of plus disease in this study when the masked quadrant-based approach was applied (eg, Figure 3 and Figure 4). In other words, although this study did not formally address this question, it may be that the grading of individual quadrants using telemedicine or ophthalmoscopy is influenced by the presence of information in adjacent quadrants and yields a different quadrant-based diagnosis, thus potentially yielding a different overall diagnosis even for examiners trying to apply a strict quadrant-level approach based on the definition. A 2017 study28 also suggested that that even experts could be biased by factors other than retinal image findings (eg, gestational age). Because both ophthalmoscopic and image-based diagnosis provides information about all quadrants simultaneously, interaction between quadrants may be inevitable in a real clinical setting, and eye-based diagnosis is presumably a more natural method for retinal assessment. Second, when assessing plus disease, graders may have different interpretations regarding the criterion of 2 or more quadrants of arterial dilation and venous tortuosity as shown in the standard published photograph, particularly given that the photograph has differing findings in each quadrant.5,6

The second key finding is that quadrant-based diagnosis showed lower intergrader agreement than eye-based diagnosis. In this study, the agreement of quadrant-based diagnosis between graders was lower than that of eye-based diagnosis. Interobserver disagreement in plus disease has been well documented, including ways in which experts deviate from the published definition of plus disease, such as by incorporating features outside of the posterior pole and nonstandard features, such as venous tortuosity.9-20 Variable adherence to the number of involved quadrants between examiners could be another explanation. Additional specification regarding this definition of plus disease may improve diagnostic consistency in the future.

The third key finding is that accuracy of quadrant-based diagnosis, based on agreement with the RSD, was lower than of eye-based diagnosis. Because ophthalmologists are trained and accustomed to assess retinal information in all quadrants during standard clinical examination, it may not be surprising that experts performed better when given the information from the whole rather than an individual quadrant. Nonetheless, it is important to be aware that the 2 approaches (eye-based vs quadrant-based) do not always give the same diagnosis. Based on these data, it is hard to argue for a strict interpretation of a quadrant-based approach because it is less likely to produce a diagnosis that agrees with the majority of expert observers.

To measure accuracy in this study, we calculated agreement of each grader’s diagnosis with an RSD, as described above.25 We have shown in previous studies17 that an RSD for plus disease developed using these methods agreed with the majority vote of clinical experts in nearly all cases, suggesting very high external validity of our RSD. In other studies,15 we have also demonstrated that the average plus disease severity score from clinical experts was very close to this RSD. Therefore, we believe that agreement with this RSD may be interpreted as diagnostic accuracy, with higher validity than ophthalmoscopic or image-based diagnosis by an individual expert.

There is growing clinical evidence that interobserver disagreement in plus disease diagnosis yields clinically meaningful differences in patient care. In the Cryotherapy for ROP study,5 examiners disagreed on 12% of cases with threshold ROP.10 Since that time, to our knowledge, there has been very little direct comparison of treatment decisions between experts outside of research studies that have consistently shown imperfect agreement.9,11-18 However, 2018 results from the Benefits of Oxygen Saturation Targeting (BOOST)-II trial29 found evidence of variation in treatment recommendations between physicians for the same level of disease based on fundus photographs. This reaffirms the importance of addressing the underlying causes and finding solutions for the wide variation in plus disease diagnosis. Despite the accepted definition of plus disease, there is now evidence that clinicians incorporate information from outside the posterior pole,17 assign diagnostic importance to features that are not in the official definition (eg, venous tortuosity),17 disagree on the level of dilation and tortuosity sufficient for diagnosis of plus disease,15 and may use eye-based or quadrant-based approaches.11,17,21-23 In other words, there is not a single part of the official definition of plus disease that consistently corresponds to expert behavior, which suggests that a more explicit definition may be useful.

An increasingly popular approach toward making plus disease diagnosis more objective and quantitative is computer-based image analysis.30-34 To date, development of these systems has been deductive, starting with the components of the accepted definition of plus disease and evaluating various defined vascular features within this context. For example, one system (ROPTool) uses metrics of arterial tortuosity and venous dilation and combines the values from the 2 most extreme quadrants to detect plus disease.30,35 Findings from the current study suggest that deductive computer-based algorithms that are eye-based rather than quadrant-based may be more successful.16,17 Several computer-based image analysis systems have been developed that use various combinations of quantitative measures to objectively measure plus disease severity.30-32 In addition to their not being widely available and requiring varying amounts of user input to produce a diagnosis, widespread use of these systems has been limited by their variable agreement with clinical plus disease diagnosis.17,22,30,34 Moreover, results from the current study add to the prior literature demonstrating how the clinical diagnosis of plus disease often deviates from the published definition. Therefore, an inductive approach to algorithm development, such as with deep learning, may better produce results that correspond to clinical diagnosis.36 By analyzing the outputs of deep learning–based algorithms, we might gain insights into the inductive diagnostic process used by experts.

More broadly, these study findings have implications for diagnosis of other retinal diseases and for emerging diagnostic modalities, such as telemedicine. In diseases such as diabetic retinopathy, large-scale clinical trials have developed methods for disease classification that are quadrant-based (eg, the 4-2-1 rule for the diagnosis of severe nonproliferative diabetic retinopathy37). Similarly, reading centers have been developed for remote telemedicine grading of diseases, such as diabetic retinopathy and ROP, which may be operated by trained nonphysician readers who interpret images based on specific algorithms.38-40 In these other settings, analysis of agreement between eye-based vs quadrant-based diagnosis may be warranted.

Limitations

There are several limitations of this study. First, only 6 graders were included. This may limit generalizability of our findings. However, we note that this topic has never been studied before, to our knowledge. Second, there are technical limitations on image grading. Most study images included more visible temporal than nasal retina, and the areas displayed for each quadrant were not consistent among different images, which may have affected both eye-based and image-based diagnosis, as previous studies have shown that field of view could affect the plus disease diagnosis.17,20 Standardizing areas of each quadrant by generating multi-image mosaics may overcome this limitation in future studies. In addition, dividing images into quadrants was done by vertical and horizontal lines passing through the center of the optic disc, which may not have completely matched with the anatomical quadrants. This might have limited grader’s ability to access the entire contour of the vasculature, especially when quadrant cutoff divided major vessels. Third, 3 of the 6 graders contributed to establishing the RSD for the analyzed images, which may have created bias. However, the RSD was determined by combining the image-based diagnoses of 3 graders with the actual clinical diagnosis by an independent expert, and this study was performed several years after the initial RSD was established for each image. Thus, we feel it is unlikely that the results of this study were significantly affected by overlap in graders. Fourth, this study did not assess the importance of pre-plus disease in the diagnosis because ICROP provides no guidance on the relevance of the numbers of quadrants of pre-plus disease involved.

Conclusions

We believe these study findings provide guidance for ophthalmologists about the best way to diagnose plus disease and that this has important implications for quality of care, delivery of care, and education. In the future, we feel that more precise definitions of plus disease regarding the criterion requiring 2 or more quadrants of abnormality will provide additional education and diagnostic standardization for practicing ophthalmologists.41-43 Next-generation computer-based image analysis systems as well as imaging modalities such as optical coherence tomography and optical coherence tomographic angiography may build on this work toward developing more objective and reproducible diagnostic methods. These future metrics will require prospective validation to determine when the appropriate intervention to treat ROP should be, but our hope is that in developing these new definitions, we can at least all agree.

Back to top
Article Information

Accepted for Publication: March 13, 2018.

Corresponding Author: Michael F. Chiang, MD, Department of Ophthalmology, Casey Eye Institute, Oregon Health and Science University, 3375 SW Terwilliger Blvd, Portland, OR 97239 (chiangm@ohsu.edu).

Published Online: April 26, 2018. doi:10.1001/jamaophthalmol.2018.1195

Author Contributions: Dr Chiang had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Kim, Campbell, Kalpathy-Cramer, Chan, Chiang.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Kim, Campbell, Chan.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Kim, Choi.

Obtained funding: Kalpathy-Cramer, Chiang.

Administrative, technical, or material support: Kalpathy-Cramer, Ostmo, Jonas, Chiang.

Study supervision: Campbell, Choi, Chan, Chiang.

Group Members: Members of the Imaging and Informatics in Retinopathy of Prematurity Research Consortium include: Oregon Health and Science University, Portland: Michael F. Chiang, MD; Susan Ostmo, MS; Sang Jin Kim, MD, PhD; Kemal Sonmez, PhD; and J. Peter Campbell, MD, MPH; University of Illinois at Chicago: R. V. Paul Chan, MD; and Karyn Jonas, RN; Columbia University, New York, New York: Jason Horowitz, MD; Osode Coki, RN; Cheryl-Ann Eccles, RN; and Leora Sarna, RN; Weill Cornell Medical College, New York, New York: Anton Orlin, MD; Bascom Palmer Eye Institute, Miami, Florida: Audina Berrocal, MD; and Catherin Negron, BA; William Beaumont Hospital, Royal Oak, Michigan: Kimberly Denser, MD; Kristi Cumming, RN; Tammy Osentoski, RN; Tammy Check, RN; and Mary Zajechowski, RN; Children’s Hospital Los Angeles, Los Angeles, California: Thomas Lee, MD; Evan Kruger, BA; and Kathryn McGovern, MPH; Cedars Sinai Hospital, Los Angeles, California: Charles Simmons, MD; Raghu Murthy, MD; and Sharon Galvis, NNP; LA Biomedical Research Institute, Los Angeles, California: Jerome Rotter, MD; Ida Chen, PhD; Xiaohui Li, MD; Kent Taylor, PhD; and Kaye Roll, RN; Massachusetts General Hospital, Boston: Jayashree Kalpathy-Cramer, PhD; Northeastern University, Boston, Massachusetts: Deniz Erdogmus, PhD; and Stratis Ioannidis, PhD; and Asociacion para Evitar la Ceguera en Mexico, Mexico City, Mexico: Maria Ana Martinez-Castellanos, MD; Samantha Salinas-Longoria, MD; Rafael Romero, MD; Andrea Arriola, MD; Francisco Olguin-Manriquez, MD; Miroslava Meraz-Gutierrez, MD; Carlos M. Dulanto-Reinoso, MD; and Cristina Montero-Mendoza, MD.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Kalpathy-Cramer has received personal fees from Infotech Soft. Dr Chan has served as a consultant for Alcon, Allegan, and Bausch and Lomb and serves on the scientific advisory board for Visunex Medical Systems. Dr Chiang has received grants from the National Institutes of Health, National Science Foundation, and Research to Prevent Blindness, serves on the scientific advisory board for Clarity Medical Systems, and has served as a consultant for Novartis. No other disclosures were reported.

Funding/Support: This work is supported by grants R01EY019474, P30EY10572, and P41EB015896 from the National Institutes of Health, grants SCH-1622542, SCH-1622536, and SCH-1622679 from the National Science Foundation, and unrestricted departmental funding from Research to Prevent Blindness.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Gilbert  C.  Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control.  Early Hum Dev. 2008;84(2):77-82.PubMedGoogle ScholarCrossref
2.
Hartnett  ME.  Pathophysiology and mechanisms of severe retinopathy of prematurity.  Ophthalmology. 2015;122(1):200-210.PubMedGoogle ScholarCrossref
3.
International Committee for the Classification of Retinopathy of Prematurity.  An international classification of retinopathy of prematurity.  Pediatrics. 1984;74(1):127-133.PubMedGoogle Scholar
4.
International Committee for the Classification of Retinopathy of Prematurity.  The International Classification of Retinopathy of Prematurity revisited.  Arch Ophthalmol. 2005;123(7):991-999.PubMedGoogle ScholarCrossref
5.
Cryotherapy for Retinopathy of Prematurity Cooperative Group.  Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results.  Arch Ophthalmol. 1988;106(4):471-479.PubMedGoogle ScholarCrossref
6.
Early Treatment For Retinopathy Of Prematurity Cooperative Group.  Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial.  Arch Ophthalmol. 2003;121(12):1684-1694.PubMedGoogle ScholarCrossref
7.
The STOP-ROP Multicenter Study Group.  Supplemental Therapeutic Oxygen for Prethreshold Retinopathy Of Prematurity (STOP-ROP), a randomized, controlled trial: I: primary outcomes.  Pediatrics. 2000;105(2):295-310.PubMedGoogle ScholarCrossref
8.
Mintz-Hittner  HA, Kennedy  KA, Chuang  AZ; BEAT-ROP Cooperative Group.  Efficacy of intravitreal bevacizumab for stage 3+ retinopathy of prematurity.  N Engl J Med. 2011;364(7):603-615.PubMedGoogle ScholarCrossref
9.
Chiang  MF, Jiang  L, Gelman  R, Du  YE, Flynn  JT.  Interexpert agreement of plus disease diagnosis in retinopathy of prematurity.  Arch Ophthalmol. 2007;125(7):875-880.PubMedGoogle ScholarCrossref
10.
Reynolds  JD, Dobson  V, Quinn  GE,  et al; CRYO-ROP and LIGHT-ROP Cooperative Study Groups.  Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies.  Arch Ophthalmol. 2002;120(11):1470-1476.PubMedGoogle ScholarCrossref
11.
Wallace  DK, Quinn  GE, Freedman  SF, Chiang  MF.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity.  J AAPOS. 2008;12(4):352-356.PubMedGoogle ScholarCrossref
12.
Gschließer  A, Stifter  E, Neumayer  T,  et al.  Inter-expert and intra-expert agreement on the diagnosis and treatment of retinopathy of prematurity.  Am J Ophthalmol. 2015;160(3):553-560.e3.PubMedGoogle ScholarCrossref
13.
Slidsborg  C, Forman  JL, Fielder  AR,  et al.  Experts do not agree when to treat retinopathy of prematurity based on plus disease.  Br J Ophthalmol. 2012;96(4):549-553.PubMedGoogle ScholarCrossref
14.
Hewing  NJ, Kaufman  DR, Chan  RV, Chiang  MF.  Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts.  JAMA Ophthalmol. 2013;131(8):1026-1032.PubMedGoogle ScholarCrossref
15.
Campbell  JP, Kalpathy-Cramer  J, Erdogmus  D,  et al; Imaging and Informatics in Retinopathy of Prematurity Research Consortium.  Plus disease in retinopathy of prematurity: a continuous spectrum of vascular abnormality as a basis of diagnostic variability.  Ophthalmology. 2016;123(11):2338-2344.PubMedGoogle ScholarCrossref
16.
Kalpathy-Cramer  J, Campbell  JP, Erdogmus  D,  et al; Imaging and Informatics in Retinopathy of Prematurity Research Consortium.  Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis.  Ophthalmology. 2016;123(11):2345-2351.PubMedGoogle ScholarCrossref
17.
Campbell  JP, Ataer-Cansizoglu  E, Bolon-Canedo  V,  et al; Imaging and Informatics in ROP (i-ROP) Research Consortium.  Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis.  JAMA Ophthalmol. 2016;134(6):651-657.PubMedGoogle ScholarCrossref
18.
Campbell  JP, Ryan  MC, Lore  E,  et al; Imaging & Informatics in Retinopathy of Prematurity Research Consortium.  Diagnostic discrepancies in retinopathy of prematurity classification.  Ophthalmology. 2016;123(8):1795-1801.PubMedGoogle ScholarCrossref
19.
Keck  KM, Kalpathy-Cramer  J, Ataer-Cansizoglu  E, You  S, Erdogmus  D, Chiang  MF.  Plus disease diagnosis in retinopathy of prematurity: vascular tortuosity as a function of distance from optic disk.  Retina. 2013;33(8):1700-1707.PubMedGoogle ScholarCrossref
20.
Rao  R, Jonsson  NJ, Ventura  C,  et al.  Plus disease in retinopathy of prematurity: diagnostic impact of field of view.  Retina. 2012;32(6):1148-1155.PubMedGoogle ScholarCrossref
21.
Wallace  DK, Freedman  SF, Zhao  Z, Jung  SH.  Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity.  Arch Ophthalmol. 2007;125(11):1523-1530.PubMedGoogle ScholarCrossref
22.
Abbey  AM, Besirli  CG, Musch  DC,  et al.  Evaluation of screening for retinopathy of prematurity by ROPtool or a lay reader.  Ophthalmology. 2016;123(2):385-390.PubMedGoogle ScholarCrossref
23.
Daniel  E, Quinn  GE, Hildebrand  PL,  et al; e-ROP Cooperative Group.  Validated system for centralized grading of retinopathy of prematurity: telemedicine approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study.  JAMA Ophthalmol. 2015;133(6):675-682.PubMedGoogle ScholarCrossref
24.
World Medical Association.  World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects.  JAMA. 2013;310(20):2191-2194. doi:10.1001/jama.2013.281053.Google ScholarCrossref
25.
Ryan  MC, Ostmo  S, Jonas  K,  et al.  Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology.  AMIA Annu Symp Proc. 2014;2014:1902-1910.PubMedGoogle Scholar
26.
StatsToDo. Kappa (Cohen and Fleiss) for ordinal data program. https://www.statstodo.com/CohenFleissKappa_Pgm.php. Accessed September 5, 2017.
27.
Landis  JR, Koch  GG.  The measurement of observer agreement for categorical data.  Biometrics. 1977;33(1):159-174.PubMedGoogle ScholarCrossref
28.
Gschließer  A, Stifter  E, Neumayer  T,  et al.  Effect of patients’ clinical information on the diagnosis of and decision to treat retinopathy of prematurity  [published online October 3, 2017].  Retina. 2017. doi:10.1097/IAE.0000000000001864PubMedGoogle Scholar
29.
Fleck  BW, Williams  C, Juszczak  E,  et al; BOOST II Retinal Image Digital Analysis (RIDA) Group.  An international comparison of retinopathy of prematurity grading performance within the Benefits of Oxygen Saturation Targeting II trials.  Eye (Lond). 2018;32(1):74-80.PubMedGoogle ScholarCrossref
30.
Wittenberg  LA, Jonsson  NJ, Chan  RV, Chiang  MF.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity.  J Pediatr Ophthalmol Strabismus. 2012;49(1):11-19.PubMedGoogle ScholarCrossref
31.
Wallace  DK.  Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis).  Trans Am Ophthalmol Soc. 2007;105:594-615.PubMedGoogle Scholar
32.
Koreen  S, Gelman  R, Martinez-Perez  ME,  et al.  Evaluation of a computer-based system for plus disease diagnosis in retinopathy of prematurity.  Ophthalmology. 2007;114(12):e59-e67.PubMedGoogle ScholarCrossref
33.
Gelman  R, Martinez-Perez  ME, Vanderveen  DK, Moskowitz  A, Fulton  AB.  Diagnosis of plus disease in retinopathy of prematurity using Retinal Image Multiscale Analysis.  Invest Ophthalmol Vis Sci. 2005;46(12):4734-4738.PubMedGoogle ScholarCrossref
34.
Ataer-Cansizoglu  E, Bolon-Canedo  V, Campbell  JP,  et al; i-ROP Research Consortium.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis.  Transl Vis Sci Technol. 2015;4(6):5.PubMedGoogle ScholarCrossref
35.
Cabrera  MT, Freedman  SF, Kiely  AE, Chiang  MF, Wallace  DK.  Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity.  J AAPOS. 2011;15(1):40-44.PubMedGoogle ScholarCrossref
36.
Kalpathy-Cramer  J, Campbell  JP, Kim  S,  et al.  Deep learning for the identification of plus disease in retinopathy of prematurity  [ARVO abstract 5554].  Invest Ophthalmol Vis Sci. 2017;58:5554.Google Scholar
37.
Wilkinson  CP, Ferris  FL  III, Klein  RE,  et al; Global Diabetic Retinopathy Project Group.  Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales.  Ophthalmology. 2003;110(9):1677-1682.PubMedGoogle ScholarCrossref
38.
Quinn  GE, Ying  GS, Daniel  E,  et al; e-ROP Cooperative Group.  Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity.  JAMA Ophthalmol. 2014;132(10):1178-1184.PubMedGoogle ScholarCrossref
39.
Daskivich  LP, Vasquez  C, Martinez  C  Jr, Tseng  CH, Mangione  CM.  Implementation and evaluation of a large-scale teleretinal diabetic retinopathy screening program in the Los Angeles County Department of Health Services.  JAMA Intern Med. 2017;177(5):642-649.PubMedGoogle ScholarCrossref
40.
Vinekar  A, Gilbert  C, Dogra  M,  et al.  The KIDROP model of combining strategies for providing retinopathy of prematurity screening in underserved areas in India using wide-field imaging, tele-medicine, non-physician graders and smart phone reporting.  Indian J Ophthalmol. 2014;62(1):41-49.PubMedGoogle ScholarCrossref
41.
Myung  JS, Paul Chan  RV, Espiritu  MJ,  et al.  Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training.  J AAPOS. 2011;15(6):573-578.PubMedGoogle ScholarCrossref
42.
Patel  SN, Martinez-Castellanos  MA, Berrones-Medina  D,  et al; GEN-ROP; i-ROP Research Consortium.  Assessment of a tele-education system to enhance retinopathy of prematurity training by international ophthalmologists-in-training in Mexico.  Ophthalmology. 2017;124(7):953-961.PubMedGoogle ScholarCrossref
43.
Nagiel  A, Espiritu  MJ, Wong  RK,  et al.  Retinopathy of prematurity residency training.  Ophthalmology. 2012;119(12):2644-5.e1, 2.PubMedGoogle ScholarCrossref
×