[Skip to Content]
[Skip to Content Landing]
Figure 1.  Computer-Based Image Analysis
Computer-Based Image Analysis

A, Standard published photograph of plus disease. B, A representative fundus image, where the white dashed line demarcates the approximate field of view of the standard published photograph. C, Results of manual segmentation of retinal vessels. D, Circular crops with the radius of 1 to 6 disc diameters for subsequent analysis.

Figure 2.  Examples of Study Images With Reference Standard Diagnosis of Plus Disease
Examples of Study Images With Reference Standard Diagnosis of Plus Disease

A, Standard photograph. B, Representative wide-angle example of plus disease. C, The same retinal image as in part B, cropped to the field of view of the standard photograph. D, Study image with plus disease, with venous tortuosity out of proportion to arterial tortuosity.

Table 1.  Effect of Field of View and Retinal Vessel Subtype on Accuracy of Plus Disease Diagnosisa
Effect of Field of View and Retinal Vessel Subtype on Accuracy of Plus Disease Diagnosisa
Table 2.  Accuracy of Plus Disease Diagnosis by Clinical Experts and i-ROP Image Analysis System
Accuracy of Plus Disease Diagnosis by Clinical Experts and i-ROP Image Analysis System
Table 3.  Accuracy of Computer-Based Diagnosis of Plus Disease by i-ROP System
Accuracy of Computer-Based Diagnosis of Plus Disease by i-ROP System
1.
International Committee for the Classification of Retinopathy of Prematurity.  The International Classification of Retinopathy of Prematurity revisited.  Arch Ophthalmol. 2005;123(7):991-999. doi:10.1001/archopht.123.7.991.PubMedGoogle Scholar
2.
Cryotherapy for Retinopathy of Prematurity Cooperative Group.  Multicenter trial of cryotherapy for retinopathy of prematurity:preliminary results.  Arch Ophthalmol. 1988;106(4):471-479. doi:10.1001/archopht.1988.01060130517027.PubMedGoogle ScholarCrossref
3.
Good  WV, Hardy  RJ.  The multicenter study of Early Treatment for Retinopathy of Prematurity (ETROP).  Ophthalmology. 2001;108(6):1013-1014.PubMedGoogle ScholarCrossref
4.
Mintz-Hittner  HA, Kennedy  KA, Chuang  AZ; BEAT-ROP Cooperative Group.  Efficacy of intravitreal bevacizumab for stage 3+ retinopathy of prematurity.  N Engl J Med. 2011;364(7):603-615.PubMedGoogle ScholarCrossref
5.
Sommer  A, Taylor  HR, Ravilla  TD,  et al; Council of the American Ophthalmological Society.  Challenges of ophthalmic care in the developing world.  JAMA Ophthalmol. 2014;132(5):640-644. doi:10.1001/jamaophthalmol.2014.84.PubMedGoogle ScholarCrossref
6.
Gilbert  C, Foster  A.  Childhood blindness in the context of VISION 2020—the right to sight.  Bull World Health Organ. 2001;79(3):227-232.PubMedGoogle Scholar
7.
Wallace  DK, Quinn  GE, Freedman  SF, Chiang  MF.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity.  J AAPOS. 2008;12(4):352-356.PubMedGoogle ScholarCrossref
8.
Chiang  MF, Jiang  L, Gelman  R, Du  YE, Flynn  JT.  Interexpert agreement of plus disease diagnosis in retinopathy of prematurity.  Arch Ophthalmol. 2007;125(7):875-880. doi:10.1001/archopht.125.7.875.PubMedGoogle ScholarCrossref
9.
Hewing  NJ, Kaufman  DR, Chan  RVP, Chiang  MF.  Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts.  JAMA Ophthalmol. 2013;131(8):1026-1032. doi:10.1001/jamaophthalmol.2013.135.PubMedGoogle ScholarCrossref
10.
Rao  R, Jonsson  NJ, Ventura  C,  et al.  Plus disease in retinopathy of prematurity: diagnostic impact of field of view.  Retina. 2012;32(6):1148-1155.PubMedGoogle ScholarCrossref
11.
Keck  KM, Kalpathy-Cramer  J, Ataer-Cansizoglu  E, You  S, Erdogmus  D, Chiang  MF.  Plus disease diagnosis in retinopathy of prematurity: vascular tortuosity as a function of distance from optic disk.  Retina. 2013;33(8):1700-1707.PubMedGoogle ScholarCrossref
12.
Myung  JS, Paul Chan  RV, Espiritu  MJ,  et al.  Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training.  J AAPOS. 2011;15(6):573-578.PubMedGoogle ScholarCrossref
13.
Paul Chan  RV, Williams  SL, Yonekawa  Y, Weissgold  DJ, Lee  TC, Chiang  MF.  Accuracy of retinopathy of prematurity diagnosis by retinal fellows.  Retina. 2010;30(6):958-965.PubMedGoogle ScholarCrossref
14.
Wong  RK, Ventura  CV, Espiritu  MJ,  et al.  Training fellows for retinopathy of prematurity care: a Web-based survey.  J AAPOS. 2012;16(2):177-181.PubMedGoogle ScholarCrossref
15.
Wittenberg  LA, Jonsson  NJ, Chan  RV, Chiang  MF.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity.  J Pediatr Ophthalmol Strabismus. 2012;49(1):11-19.PubMedGoogle ScholarCrossref
16.
Chiang  MF, Gelman  R, Martinez-Perez  ME,  et al.  Image analysis for retinopathy of prematurity diagnosis.  J AAPOS. 2009;13(5):438-445.PubMedGoogle ScholarCrossref
17.
Ataer-Cansizoglu  E, Bolon-Canedo  V, Campbell  JP,  et al; i-ROP Research Consortium.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis.  Transl Vis Sci Technol. 2015;4(6):5.PubMedGoogle Scholar
18.
Ryan  MC, Ostmo  S, Jonas  K,  et al.  Development and evaluation of reference standards for image-based telemedicine diagnosis and cinical research studies in ophthalmology.  AMIA Annu Symp Proc. 2014;2014:1902-1910.PubMedGoogle Scholar
19.
Ataer-Cansizoglu  E, You  S, Kalpathy-Cramer  J, Keck  K, Chiang  MF, Erdogmus  D.  Observer and feature analysis on diagnosis of retinopathy of prematurity.  IEEE Int Workshop Mach Learn Signal Process. 2012;1-6.PubMedGoogle Scholar
20.
Gelman  SK, Gelman  R, Callahan  AB,  et al.  Plus disease in retinopathy of prematurity: quantitative analysis of standard published photograph.  Arch Ophthalmol. 2010;128(9):1217-1220. doi:10.1001/archophthalmol.2010.186.PubMedGoogle ScholarCrossref
21.
Wilson  CM, Wong  K, Ng  J, Cocker  KD, Ells  AL, Fielder  AR.  Digital image analysis in retinopathy of prematurity: a comparison of vessel selection methods.  J AAPOS. 2012;16(3):223-228.PubMedGoogle ScholarCrossref
22.
Wallace  DK.  Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis).  Trans Am Ophthalmol Soc. 2007;105:594-615.PubMedGoogle Scholar
23.
Abbey  AM, Besirli  CG, Musch  DC,  et al.  Evaluation of screening for retinopathy of prematurity by ROPtool or a lay reader.  Ophthalmology. 2016;123(2):385-390.PubMedGoogle ScholarCrossref
24.
Chan  RV, Patel  SN, Ryan  MC,  et al.  The Global Education Network for Retinopathy of Prematurity (Gen-Rop): development, implementation, and evaluation of a novel tele-education system (an American Ophthalmological Society thesis).  Trans Am Ophthalmol Soc. 2015;113:T21-T226.PubMedGoogle Scholar
25.
Trese  MT.  What is the real gold standard for ROP screening?  Retina. 2008;28(3)(suppl):S1-S2.PubMedGoogle ScholarCrossref
26.
Reynolds  JD, Dobson  V, Quinn  GE,  et al; CRYO-ROP and LIGHT-ROP Cooperative Study Groups.  Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies.  Arch Ophthalmol. 2002;120(11):1470-1476.PubMedGoogle ScholarCrossref
27.
Cabrera  MT, Freedman  SF, Kiely  AE, Chiang  MF, Wallace  DK.  Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity.  J AAPOS. 2011;15(1):40-44.PubMedGoogle ScholarCrossref
28.
Cabrera  MT, Freedman  SF, Hartnett  ME, Stinnett  SS, Chen  BB, Wallace  DK.  Real-time, computer-assisted quantification of plus disease in retinopathy of prematurity at the bedside.  Ophthalmic Surg Lasers Imaging Retina. 2014;45(6):542-548.PubMedGoogle ScholarCrossref
29.
Chiang  MF, Wang  L, Busuioc  M,  et al.  Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality.  Arch Ophthalmol. 2007;125(11):1531-1538. doi:10.1001/archopht.125.11.1531.PubMedGoogle ScholarCrossref
30.
Quinn  GE, Ying  G-S, Daniel  E,  et al; e-ROP Cooperative Group.  Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity.  JAMA Ophthalmol. 2014;132(10):1178-1184. doi:10.1001/jamaophthalmol.2014.1604.PubMedGoogle ScholarCrossref
31.
Chiang  MF, Boland  MV, Brewer  A,  et al; American Academy of Ophthalmology Medical Information Technology Committee.  Special requirements for electronic health record systems in ophthalmology.  Ophthalmology. 2011;118(8):1681-1687.PubMedGoogle ScholarCrossref
Original Investigation
June 2016

Expert Diagnosis of Plus Disease in Retinopathy of Prematurity From Computer-Based Image Analysis

Author Affiliations
  • 1Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland
  • 2Cognitive Systems Laboratory, Northeastern University, Boston, Massachusetts
  • 3Department of Computer Science, Universidade da Coruña, A Coruña, Spain
  • 4Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown
  • 5Department of Ophthalmology, Weill Cornell Medical College, New York, New York
  • 6Department of Ophthalmology, Ross Eye Institute, State University of New York at Buffalo
  • 7Department of Ophthalmology, Columbia University, New York, New York
  • 8Department of Ophthalmology, Sidra Medical and Research Center, Doha, Qatar
  • 9Retina Consultants, Chicago, Illinois
  • 10Wilmer Institute, Johns Hopkins University School of Medicine, Baltimore, Maryland
  • 11Long Island Vitreoretinal Consultants, Great Neck, New York
  • 12Associated Retinal Consultants, Oakland University, Royal Oak, Michigan
  • 13Asociacion para Evitar la Ceguera en Mexico, Mexico City, Mexico
  • 14Department of Ophthalmology and Visual Sciences, Illinois Eye and Ear Infirmary, University of Illinois at Chicago
  • 15Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland
JAMA Ophthalmol. 2016;134(6):651-657. doi:10.1001/jamaophthalmol.2016.0611
Abstract

Importance  Published definitions of plus disease in retinopathy of prematurity (ROP) reference arterial tortuosity and venous dilation within the posterior pole based on a standard published photograph. One possible explanation for limited interexpert reliability for a diagnosis of plus disease is that experts deviate from the published definitions.

Objective  To identify vascular features used by experts for diagnosis of plus disease through quantitative image analysis.

Design, Setting, and Participants  A computer-based image analysis system (Imaging and Informatics in ROP [i-ROP]) was developed using a set of 77 digital fundus images, and the system was designed to classify images compared with a reference standard diagnosis (RSD). System performance was analyzed as a function of the field of view (circular crops with a radius of 1-6 disc diameters) and vessel subtype (arteries only, veins only, or all vessels). Routine ROP screening was conducted from June 29, 2011, to October 14, 2014, in neonatal intensive care units at 8 academic institutions, with a subset of 73 images independently classified by 11 ROP experts for validation. The RSD was compared with the majority diagnosis of experts.

Main Outcomes and Measures  The primary outcome measure was the percentage of accuracy of the i-ROP system classification of plus disease, with the RSD as a function of the field of view and vessel type. Secondary outcome measures included the accuracy of the 11 experts compared with the RSD.

Results  Accuracy of plus disease diagnosis by the i-ROP computer-based system was highest (95%; 95% CI, 94%-95%) when it incorporated vascular tortuosity from both arteries and veins and with the widest field of view (6–disc diameter radius). Accuracy was 90% or less when using only arterial tortuosity and 85% or less using a 2– to 3–disc diameter view similar to the standard published photograph. Diagnostic accuracy of the i-ROP system (95%) was comparable to that of 11 expert physicians (mean 87%, range 79%-99%).

Conclusions and Relevance  Experts in ROP appear to consider findings from beyond the posterior retina when diagnosing plus disease and consider tortuosity of both arteries and veins, in contrast with published definitions. It is feasible for a computer-based image analysis system to perform comparably with ROP experts, using manually segmented images.

Introduction

Retinopathy of prematurity (ROP) is a vasoproliferative disease affecting premature infants. Since the 1980s, clinical diagnosis has been standardized using the International Classification for Retinopathy of Prematurity (ICROP).1,2 Diagnostic cut points identifying severe ROP requiring treatment have been proposed and tested.3 As a result of multicenter clinical trials, it is known that severe ROP may be successfully treated if it is diagnosed early.3,4 Furthermore, in spite of these advances in diagnosis and treatment, ROP continues to be a leading cause of childhood blindness worldwide.5,6

The Early Treatment for Retinopathy of Prematurity multicenter clinical trial showed that plus disease is the most important ICROP parameter for identifying ROP requiring treatment.3 Plus disease is defined as arterial tortuosity and venous dilation in the posterior pole greater than that of a standard published photograph selected by expert consensus during the 1980s.1 Thus, accurate and consistent diagnosis of plus disease is critical to ensure that infants at risk for blindness receive the appropriate screening and treatment. Since that time, a newer pre-plus category has been defined by the revised ICROP as retinal vascular abnormalities that are insufficient for plus disease but have more arterial tortuosity and venous dilation than normal.1

There are numerous limitations in the definition of plus disease. Studies have found that clinical diagnosis of plus disease is subjective and varies among experts.7,8 The definition of ICROP explicitly states that plus disease refers only to arterial tortuosity and venous dilation within the posterior pole vessels, and the standard published photograph displays only a very narrow-angle retinal view of 2 to 3 disc diameters (DDs) (Figure 1A). However, previous work suggests that experts consider additional retinal features (such as venous tortuosity) and larger retinal fields of view during real-world clinical diagnosis.9-11 Better understanding of the retinal vascular abnormalities that characterize plus disease will lead to improved clinical diagnosis, education,12-14 and methods for automated, computer-based diagnosis.15,16

Our purpose is to identify quantitative retinal vascular features that correlate with diagnosis of plus disease by ROP experts. We have developed a computer-based image analysis system (Imaging and Informatics in ROP [i-ROP]) and have demonstrated that it can accurately identify plus disease and pre-plus disease.17 In this study, we use i-ROP to correlate quantitative vascular features with a reference standard diagnosis (RSD) defined by consensus of image reviews by 3 expert ROP image graders combined with the clinical diagnosis.

Box Section Ref ID

Key Points

  • Question What does computer-based image analysis demonstrate regarding how experts classify plus disease?

  • Findings This study used a computer-based image analysis program to determine which vascular features best correlated with expert diagnosis of plus disease in retinopathy of prematurity. The computer-based image analysis system accurately classified 95% of images only when analyzing with the widest field of view and including tortuosity information from both arteries and veins.

  • Meaning These findings suggest that experts may use information outside of the technical definition of plus disease and that computer-based image analysis systems may perform as well as human graders.

Methods
Description of Data Set

We developed a database of 77 wide-angle retinal images acquired during routine clinical care and established an RSD for each image using previously published methods that combine interpretations and the clinical diagnosis from 3 expert graders (independent, masked gradings from 2 ophthalmologists and 1 ROP study coordinator: M.F.C., R.V.P.C, and S.O.).18 When the majority of the experts independently agreed with the clinical diagnosis (made at bedside using indirect ophthalmoscopy by the examining physician), this consensus became the RSD. If there was disagreement, the image was discussed among the group for consensus to determine the RSD. Among these 77 images, 14 had an RSD of plus disease, 16 had pre-plus disease, and 47 were normal. Each image was manually segmented by one of us (S.N.P.) to identify retinal vessels for computer-based analysis and cropped into circles in a range of sizes based on methods described previously (Figure 1).17 This study was approved by the Institutional Review Board at Oregon Health & Science University, and followed the tenets of the Declaration of Helsinki. Written informed consent was obtained from parents of infants enrolled in this study.

Computer-Based Image Analysis

We developed the i-ROP computer-based image analysis system.17 We used 11 previously described measurements of dilation and tortuosity and designed the i-ROP system to identify the vascular features that best classified normal images, pre-plus images, and plus disease images into the correct categories compared with the RSD.17,19 From each measurement algorithm, we obtained a set of distribution values for each image and analyzed them using a 2-component gaussian mixed model system, which helped us represent the image features extracted from a vasculature structure as a probability distribution (with 2 components, respectively, representing the straight and tortuous segments) rather than a set of values coming from regular statistics, such as mean and median. Finally, we compared the ordered probability distributions for each image and compared them with the RSD to design the system to correctly classify the images as plus, pre-plus, or normal. We examined several algorithms to quantify vascular tortuosity, and found that the most accurate algorithm used an acceleration function, which was a point-based feature, defined as the second derivative of the best-fit line at each point on the vascular tree.17

Statistical Analysis

Agreement of the RSD (plus, pre-plus, or normal) with findings from computer-based image analysis was examined over a range of circular image crop sizes from a 1-DD radius to a 6-DD radius (Figure 1). Agreement of the RSD with findings from computer-based image analysis was then examined when considering arteries only, veins only, and all vessels. To improve the external validity of image analysis, we used a leave-one-out cross-validation process, which involves a repeated process of using a subset of the data to train the system and compare with the remaining data set.17 We then used a jackknife variance estimate to calculate standard deviation and 95% CIs for the performance of the system for each subset of values along with Stata, version 11.0 (StataCorp) for determination of statistical significance using the t test function.

Finally, to determine whether the RSD was adequately representative of the general population of physicians, a subset of 73 of the 77 wide-angle retinal images was separately classified by 11 clinical experts as part of a larger set of 100 images (4 images [all of which were unanimously diagnosed as plus disease by all experts, clinical diagnosis, and the i-ROP system] were excluded from the larger data set for methodological reasons) from June 29, 2011, to October 14, 2014. All 11 experts were experienced in ROP examination and treatment and had all participated as experts in ROP research studies funded by the National Institutes of Health and/or published at least 2 peer-reviewed articles on ROP. To examine the classification performance of the i-ROP computer-based image analysis system, we calculated the tabular confusion matrix for the plus disease classification of the RSD compared with the i-ROP system for each of the 77 images.

Results
Effect of Field of View and Vessel Subtype on Plus Disease Diagnosis

The performance of i-ROP computer-based analysis improved with increased field of view: accuracy compared with the RSD generally improved from a 1-DD radius field of view (accuracy, 64%; 95% CI, 61%-67% when considering all vessels) to a 3-DD radius field of view (approximate field of view of the standard photograph; accuracy, 85%; 95% CI, 83%-86%) to a 6-DD radius field of view (accuracy, 95%; 95% CI, 94%-95%; P < .001 vs 3DD) (Table 1). Findings also demonstrated improved i-ROP system performance when considering all vessels (accuracy, 95%; 95% CI, 94%-95% with a 6-DD radius field of view) rather than only arteries (accuracy, 87%; 95% CI, 86%-88% with a 6-DD radius field of view; P < .001) or only veins (accuracy, 79%; 95% CI, 77%-81% with a 6-DD radius field of view; P < .001). None of the dilation features performed better than 80% accuracy.17 Examples of study images demonstrating these points are shown in Figure 2.

Diagnostic Accuracy of i-ROP Image Analysis System and Outside Experts

Table 2 displays the accuracy of the i-ROP system in diagnosing plus disease compared with the RSD. For an additional comparison, the diagnostic accuracy of 11 clinical experts was also determined using a subset of 73 of 77 images. Among the 73 images, accuracy of the 11 clinical experts ranged from 79%-99% (mean, 87%), and accuracy of the i-ROP image analysis system was 95% (69 of 73 images). When the diagnosis selected by the majority of the 11 experts was compared with the RSD, the accuracy of the clinical experts was 97% (71 of 73 images).

Table 3 provides details about diagnostic performance of the i-ROP system on the full data set of 77 images, including the 4 images that were diagnosed incorrectly compared with the RSD. The i-ROP system performed similarly to the 3 expert image graders who collectively determined the RSD for the 77 images, who individually agreed with the RSD in 72 (94%), 74 (96%), and 71 (92%) images.

Discussion

This study examines the retinal features used by ROP experts while classifying plus disease by correlating their diagnoses with prespecified quantitative parameters from computer-based image analysis. Key results of this study are that ROP experts appear to consider findings from beyond the central retina when diagnosing plus disease and consider tortuosity of both arteries and veins, and it is feasible for a computer-based image analysis system to perform comparably to ROP experts, using manually segmented images.

These findings suggest that physicians incorporate information beyond the ICROP definition of plus disease, which is based only on abnormality within the posterior pole vessels. Although the 2005 revised ICROP classification provided examples of wide-field fundus images of pre-plus disease, there continues to be only a single standard published photograph of plus disease, which displays a very narrow field of view (Figure 1A and Figure 2A).1 In our study, a computer-based machine learning system for identifying plus disease was more accurate when evaluating larger-angle fields of view (Table 1). Similarly, a previous study has demonstrated that the accuracy of experts decreases when fundus images are cropped closer to the field of view of the standard photograph (Figure 2B).10 In combination, this finding suggests that the mid-peripheral and peripheral vessels contain information that physicians use diagnostically to improve their diagnosis, but that information was not included in the original definition of plus disease, which is consistent with previously published literature.9-11 Similarly, in a different study using qualitative research methods to analyze the diagnostic process of experts, we found that some experts specifically cited peripheral retinal vascular features as being useful for the diagnosis of plus disease.9

Our findings also show that diagnosis of plus disease by experts incorporates tortuosity of both arteries and veins rather than only the arteries as suggested in the ICROP definition.1 In fact, a previous study has shown that veins in the standard published photograph (Figure 1A and Figure 2A) actually have greater tortuosity than the arteries.20 This outcome is consistent with findings by Wilson et al,21 who demonstrated that quantitative arterial and venous tortuosity both increased with worsening clinical stage of ROP, and both increased regardless of whether 4 or 8 vessels were analyzed. This finding may be useful from a clinical perspective because distinguishing between retinal arteries and veins in infants with ROP can be difficult for a computer system or even for expert physicians.22 Furthermore, results of this study are consistent with other ROP algorithms and suggest that retinal vascular tortuosity is more correlated with expert diagnosis than venous dilation, although this finding may result in part because quantitative analysis of vascular dilation is dependent on external factors, such as image magnification.23

Because this study finds that expert diagnosis of plus disease is not consistent with published ICROP definitions and there is some interexpert diagnostic variability, it may be that individual experts are weighting particular retinal features differently to arrive at their diagnosis.8,9 This finding has implications for ROP education given that previous work has shown that trainees often perform poorly in clinical diagnosis.10,12-14,24 An important step toward improving clinical diagnosis and education would be through dissemination of wider-angle representative retinal images of ROP. To foster that development, we are creating a website (http://www.i-rop.com) that provides examples of such images. We acknowledge that it may be problematic to change the definition of plus disease ex post facto in terms of extrapolating prognostic information to a population that may be different than the original population, but we believe it would be helpful for all experts to use the same validated parameters. In the original Cryotherapy for Retinopathy of Prematurity study when using only the criterion standard of binocular indirect ophthalmoscopy, experts disagreed on the need for treatment based on a diagnosis of threshold disease in 12% of cases.25,26

These study results confirm that computer-based image analysis systems can perform similarly to physicians and provide objective measurements of the vascular features labeled plus disease and pre-plus disease. In addition, our findings raise the question as to whether our attempts at developing computer-based image analysis systems ought to be constrained by existing clinical definitions (eg, arterial tortuosity and venous dilation) or whether, as in this case, they should model real-world physician improvement and be designed to fit the data. As described above, our study results are consistent with previous work suggesting that distinguishing between arteries and veins is not necessary for computer-based image analysis development and that quantitative measurements of vascular dilation are less accurate than measurements of tortuosity for diagnosis of plus disease diagnosis.16,21 From this analysis, we cannot conclude that experts are ignoring dilation features; for example, it may be that measurements of tortuosity are more continuous as severity of plus disease increases, whereas venous dilation correlates strongly only at one or both ends of the spectrum of plus disease, and therefore tortuosity works better as a single quantitative variable. This theory may partially explain why other image analysis systems, such as ROPTool (FocusROP), have lower receiver operating characteristic curves with dilation measures, or it may be related to the imprecision and technical challenges of measuring vascular dilation on RetCam (Clarity Medical Systems) images.22,23,27

Although it is difficult to directly compare the performance of the i-ROP system with prior systems, such as ROPTool, given the use of manual segmentation, different populations studied, and different end points for comparison, the i-ROP system appears to have at least 2 key advantages to previously reported systems, which have been discussed in detail in a prior publication.17 Most important, given the high interexpert variability for the diagnosis of plus disease, the i-ROP system was trained against a rigorously developed RSD as opposed to a single clinical examiner’s impression. Second, the i-ROP system is the first reported system to accurately incorporate pre-plus disease into the semi-automated classification. Using the i-ROP system, no eyes with plus disease were classified as normal, and no normal eyes were classified as having plus disease (Table 3). In comparison, published results using ROPTool have reported a sensitivity of plus disease detection (against a clinical examiner and without identification of pre-plus disease) between 71% and 86%27,28 and, in a recent publication by Abbey et al,23 a sensitivity of 91% for any vascular abnormality (≥1 on a scale of 0-16).

The advantage of the continuous scale by Abbey et al23 is that it avoids the inherent limitation in all studies involving computer-based diagnosis, which is the identification of a criterion standard.15 In our study, the reference standard was developed by combining the image-based diagnoses of 3 experts in ROP image grading combined with the actual clinical diagnosis. The majority vote of the 11 physicians agreed with the RSD in 71 of 73 (97%) cases, suggesting that the external validity of our RSD is high, and therefore the implications of these findings are likely generalizable to the larger population of ROP physicians. Although the majority consensus of the 11 physicians performed well with the reference standard, there was variability even among experts in this group, which may imply different thresholds for each physician, attention to different vascular features, or imprecision in the classification of the physicians. The clinical relevance of this variability is unknown, and these questions will be avenues of future research.

There are several additional limitations to this study. First, the i-ROP system was trained using manually segmented images, which limits its immediate clinical utility. For the purposes of designing the machine learning software, we wanted to avoid the potential noise caused by automated segmentation algorithms; however, we are working on using the information gleaned from this work to develop a fully automated system. Second, despite the cross-validation procedures, these data will be more convincing when they are validated against other data sets. With only 77 images in the data set, it is possible that different parameters would be found to be more important in other image sets. Third, we did not analyze whether a combination of image features would have worked better than individual features, so we did not compare arterial tortuosity and venous dilation with our current system, which uses only a measurement of tortuosity. It may be that some combination of image features actually performs better than the current i-ROP system. Last, there is variability in the routine use of photographic diagnosis during routine clinical care, and some of the variability of the expert consensus may be owing to unfamiliarity with this technique and not represent true clinical differences.

Conclusions

These findings have important implications for quality of care, education, and delivery of care with emerging technologies, such as computer-based decision support and telemedicine.29-31 Physicians and trainees can use these findings to improve standardization of plus disease in diagnosis of ROP. Using the i-ROP system, it may be possible to develop a fully automated classification system for plus disease that performs as well as experts at 3-level classification.

Back to top
Article Information

Submitted for Publication: November 30, 2015; accepted February 8, 2016.

Corresponding Author: Michael F. Chiang, MD, Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, 3375 SW Terwilliger Blvd, Portland, OR 97239 (chiangm@ohsu.edu).

Published Online: April 14, 2016. doi:10.1001/jamaophthalmol.2016.0611.

Author Contributions: Dr Chiang had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Campbell, Ataer-Cansizoglu, Bolon-Canedo, Patel, Martinez-Castellanos, Jonas, Chan, Chiang.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Campbell, Bolon-Canedo, Patel, Chan, Chiang.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Campbell, Ataer-Cansizoglu, Bozkurt, Erdogmus, Patel, Martinez-Castellanos.

Obtained funding: Erdogmus, Kalpathy-Cramer, Chiang.

Administrative, technical, or material support: Ataer-Cansizoglu, Kalpathy-Cramer, Patel, Reynolds, Shapiro, Repka, Drenser, Ostmo, Jonas, Chiang.

Study supervision: Erdogmus, Kalpathy-Cramer, Ferrone, Martinez-Castellanos, Chan, Chiang.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Chiang reported serving as an unpaid member of the Scientific Advisory Board for Clarity Medical Systems and as a consultant for Novartis. Dr Reynolds reported serving as a consultant for Novartis. No other conflicts were reported.

Funding/Support: This study was supported by grants R01 EY19474, P30 EY010572 (Drs Chiang and Campbell), and R21 EY022387 (Drs Erdogmus, Chiang, and Kalpathy-Cramer) from the National Institutes of Health, unrestricted departmental funding from Research to Prevent Blindness (Drs Campbell, Reynolds, Repka, Chan, and Chiang and Mss Patel, Ostmo, and Jonas), the St Giles Foundation (Dr Chan), and the iNsight Foundation (Dr Chan and Ms Jonas).

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: The Imaging and Informatics in ROP (i-ROP) Research Consortium includes Michael F. Chiang, MD, Susan Ostmo, MS, Kemal Sonmez, PhD, and J. Peter Campbell, MD, MPH (Oregon Health & Science University, Portland, OR); R.V. Paul Chan, MD, and Karyn Jonas, RN (Cornell University, New York, NY); Jason Horowitz, MD, Osode Coki, RN, Cheryl-Ann Eccles, RN, and Leora Sarna, RN (Columbia University, New York, NY); Audina Berrocal, MD, and Catherin Negron, BA (Bascom Palmer Eye Institute, Miami, FL); Kimberly Drenser, MD, Kristi Cumming, RN, Tammy Osentoski, RN, and Tammy Check, RN (William Beaumont Hospital, Royal Oak, MI); Thomas Lee, MD, Evan Kruger, BA, and Kathryn McGovern, MPH (Children’s Hospital Los Angeles, Los Angeles, CA); Charles Simmons, MD, Raghu Murthy, MD, and Sharon Galvis, NNP (Cedars Sinai Hospital, Los Angeles, CA); Jerome Rotter, MD, Ida Chen, PhD, Xiaohui Li, MD, and Kaye Roll, RN (LA Biomedical Research Institute, Los Angeles, CA); Jayashree Kalpathy-Cramer, PhD (Massachusetts General Hospital, Boston, MA); Deniz Erdogmus, PhD (Northeastern University, Boston, MA); and Maria Ana Martinez-Castellanos, MD, Samantha Salinas-Longoria, MD, Rafael Romero, MD, and Andrea Arriola, MD (Asociacion para Evitar la Ceguera en Mexico (APEC), Mexico City, Mexico).

References
1.
International Committee for the Classification of Retinopathy of Prematurity.  The International Classification of Retinopathy of Prematurity revisited.  Arch Ophthalmol. 2005;123(7):991-999. doi:10.1001/archopht.123.7.991.PubMedGoogle Scholar
2.
Cryotherapy for Retinopathy of Prematurity Cooperative Group.  Multicenter trial of cryotherapy for retinopathy of prematurity:preliminary results.  Arch Ophthalmol. 1988;106(4):471-479. doi:10.1001/archopht.1988.01060130517027.PubMedGoogle ScholarCrossref
3.
Good  WV, Hardy  RJ.  The multicenter study of Early Treatment for Retinopathy of Prematurity (ETROP).  Ophthalmology. 2001;108(6):1013-1014.PubMedGoogle ScholarCrossref
4.
Mintz-Hittner  HA, Kennedy  KA, Chuang  AZ; BEAT-ROP Cooperative Group.  Efficacy of intravitreal bevacizumab for stage 3+ retinopathy of prematurity.  N Engl J Med. 2011;364(7):603-615.PubMedGoogle ScholarCrossref
5.
Sommer  A, Taylor  HR, Ravilla  TD,  et al; Council of the American Ophthalmological Society.  Challenges of ophthalmic care in the developing world.  JAMA Ophthalmol. 2014;132(5):640-644. doi:10.1001/jamaophthalmol.2014.84.PubMedGoogle ScholarCrossref
6.
Gilbert  C, Foster  A.  Childhood blindness in the context of VISION 2020—the right to sight.  Bull World Health Organ. 2001;79(3):227-232.PubMedGoogle Scholar
7.
Wallace  DK, Quinn  GE, Freedman  SF, Chiang  MF.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity.  J AAPOS. 2008;12(4):352-356.PubMedGoogle ScholarCrossref
8.
Chiang  MF, Jiang  L, Gelman  R, Du  YE, Flynn  JT.  Interexpert agreement of plus disease diagnosis in retinopathy of prematurity.  Arch Ophthalmol. 2007;125(7):875-880. doi:10.1001/archopht.125.7.875.PubMedGoogle ScholarCrossref
9.
Hewing  NJ, Kaufman  DR, Chan  RVP, Chiang  MF.  Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts.  JAMA Ophthalmol. 2013;131(8):1026-1032. doi:10.1001/jamaophthalmol.2013.135.PubMedGoogle ScholarCrossref
10.
Rao  R, Jonsson  NJ, Ventura  C,  et al.  Plus disease in retinopathy of prematurity: diagnostic impact of field of view.  Retina. 2012;32(6):1148-1155.PubMedGoogle ScholarCrossref
11.
Keck  KM, Kalpathy-Cramer  J, Ataer-Cansizoglu  E, You  S, Erdogmus  D, Chiang  MF.  Plus disease diagnosis in retinopathy of prematurity: vascular tortuosity as a function of distance from optic disk.  Retina. 2013;33(8):1700-1707.PubMedGoogle ScholarCrossref
12.
Myung  JS, Paul Chan  RV, Espiritu  MJ,  et al.  Accuracy of retinopathy of prematurity image-based diagnosis by pediatric ophthalmology fellows: implications for training.  J AAPOS. 2011;15(6):573-578.PubMedGoogle ScholarCrossref
13.
Paul Chan  RV, Williams  SL, Yonekawa  Y, Weissgold  DJ, Lee  TC, Chiang  MF.  Accuracy of retinopathy of prematurity diagnosis by retinal fellows.  Retina. 2010;30(6):958-965.PubMedGoogle ScholarCrossref
14.
Wong  RK, Ventura  CV, Espiritu  MJ,  et al.  Training fellows for retinopathy of prematurity care: a Web-based survey.  J AAPOS. 2012;16(2):177-181.PubMedGoogle ScholarCrossref
15.
Wittenberg  LA, Jonsson  NJ, Chan  RV, Chiang  MF.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity.  J Pediatr Ophthalmol Strabismus. 2012;49(1):11-19.PubMedGoogle ScholarCrossref
16.
Chiang  MF, Gelman  R, Martinez-Perez  ME,  et al.  Image analysis for retinopathy of prematurity diagnosis.  J AAPOS. 2009;13(5):438-445.PubMedGoogle ScholarCrossref
17.
Ataer-Cansizoglu  E, Bolon-Canedo  V, Campbell  JP,  et al; i-ROP Research Consortium.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis.  Transl Vis Sci Technol. 2015;4(6):5.PubMedGoogle Scholar
18.
Ryan  MC, Ostmo  S, Jonas  K,  et al.  Development and evaluation of reference standards for image-based telemedicine diagnosis and cinical research studies in ophthalmology.  AMIA Annu Symp Proc. 2014;2014:1902-1910.PubMedGoogle Scholar
19.
Ataer-Cansizoglu  E, You  S, Kalpathy-Cramer  J, Keck  K, Chiang  MF, Erdogmus  D.  Observer and feature analysis on diagnosis of retinopathy of prematurity.  IEEE Int Workshop Mach Learn Signal Process. 2012;1-6.PubMedGoogle Scholar
20.
Gelman  SK, Gelman  R, Callahan  AB,  et al.  Plus disease in retinopathy of prematurity: quantitative analysis of standard published photograph.  Arch Ophthalmol. 2010;128(9):1217-1220. doi:10.1001/archophthalmol.2010.186.PubMedGoogle ScholarCrossref
21.
Wilson  CM, Wong  K, Ng  J, Cocker  KD, Ells  AL, Fielder  AR.  Digital image analysis in retinopathy of prematurity: a comparison of vessel selection methods.  J AAPOS. 2012;16(3):223-228.PubMedGoogle ScholarCrossref
22.
Wallace  DK.  Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis).  Trans Am Ophthalmol Soc. 2007;105:594-615.PubMedGoogle Scholar
23.
Abbey  AM, Besirli  CG, Musch  DC,  et al.  Evaluation of screening for retinopathy of prematurity by ROPtool or a lay reader.  Ophthalmology. 2016;123(2):385-390.PubMedGoogle ScholarCrossref
24.
Chan  RV, Patel  SN, Ryan  MC,  et al.  The Global Education Network for Retinopathy of Prematurity (Gen-Rop): development, implementation, and evaluation of a novel tele-education system (an American Ophthalmological Society thesis).  Trans Am Ophthalmol Soc. 2015;113:T21-T226.PubMedGoogle Scholar
25.
Trese  MT.  What is the real gold standard for ROP screening?  Retina. 2008;28(3)(suppl):S1-S2.PubMedGoogle ScholarCrossref
26.
Reynolds  JD, Dobson  V, Quinn  GE,  et al; CRYO-ROP and LIGHT-ROP Cooperative Study Groups.  Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies.  Arch Ophthalmol. 2002;120(11):1470-1476.PubMedGoogle ScholarCrossref
27.
Cabrera  MT, Freedman  SF, Kiely  AE, Chiang  MF, Wallace  DK.  Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity.  J AAPOS. 2011;15(1):40-44.PubMedGoogle ScholarCrossref
28.
Cabrera  MT, Freedman  SF, Hartnett  ME, Stinnett  SS, Chen  BB, Wallace  DK.  Real-time, computer-assisted quantification of plus disease in retinopathy of prematurity at the bedside.  Ophthalmic Surg Lasers Imaging Retina. 2014;45(6):542-548.PubMedGoogle ScholarCrossref
29.
Chiang  MF, Wang  L, Busuioc  M,  et al.  Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality.  Arch Ophthalmol. 2007;125(11):1531-1538. doi:10.1001/archopht.125.11.1531.PubMedGoogle ScholarCrossref
30.
Quinn  GE, Ying  G-S, Daniel  E,  et al; e-ROP Cooperative Group.  Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity.  JAMA Ophthalmol. 2014;132(10):1178-1184. doi:10.1001/jamaophthalmol.2014.1604.PubMedGoogle ScholarCrossref
31.
Chiang  MF, Boland  MV, Brewer  A,  et al; American Academy of Ophthalmology Medical Information Technology Committee.  Special requirements for electronic health record systems in ophthalmology.  Ophthalmology. 2011;118(8):1681-1687.PubMedGoogle ScholarCrossref
×