Comparison of Subjective Assessment and Precise Quantitative Assessment of Lesion Distribution in Diabetic Retinopathy

Key Points

Question  Does subjective assessment of diabetic retinopathy (DR) lesion distribution differ from precise quantitative classification of predominantly peripheral lesions?

Findings  This multisite cross-sectional study demonstrated a difference between subjective and quantitative classification of DR lesion distribution for both frequency-based and surface area–based methods.

Meaning  These findings highlight the potential importance of objective quantitative approaches to DR assessment, which may facilitate the development of a more precise DR scoring system.

Importance  Predominantly peripheral disease in eyes with nonproliferative diabetic retinopathy (DR) is suggested as a potential strong risk factor for progression to proliferative disease. However, the reliability and optimal method for the assessment of lesion distribution are still uncertain.

Objective  To compare agreement between subjective assessment and precise quantification of lesion burden in ultrawidefield (UWF) images of eyes with DR.

Design, Setting, and Participants  This multisite cross-sectional study examines UWF pseudocolor images acquired from DR screening clinic patients from December 20, 2014, through August 1, 2014. Of 104 cases, 161 eyes with DR were included. Data analysis was conducted from June 1, 2016, through December 1, 2016 at the Doheny Image Reading Center.

Main Outcomes and Measures  Distribution of DR lesions in eyes was assessed subjectively and quantitatively, and eyes were classified as having predominantly central lesions (PCLs) or predominantly peripheral lesions (PPLs). The frequency and surface area (SA) of each lesion type were quantified. Intergrader and subjective vs quantitative classification were compared for level of agreement. Several methods of determining PPL distribution were also compared.

Results  On subjective frequency-based evaluation by graders, 133 eyes were classified as having PCL, and 28 eyes as having PPL. On exact quantification of lesion SA, 121 eyes were classified as PCL, and 40 eyes as having PPL. On SA-based quantification, 134 eyes were classified as having PCL, and 27 eyes as having PPL. There was a significant difference between qualitative and quantitative classification of DR lesion distribution for both frequency-based (mean difference [SD]: PCL, 6 [2]; PPL, 13 [6]; P < .001) and SA-based (mean difference [SD]: PCL, 6 [1]; PPL, 20 [7]; P < .001) methods. Both intergrader reproducibility and subjective vs quantitative agreement were higher with frequency-based classification.

Conclusions and Relevance  Subjective assessment of PPL DR lesions on UWF images differed in some cases from precise quantitative assessments, particularly when considering the area of lesions. These findings highlight the benefit of objective quantitative approaches to DR assessment, which may facilitate the development of a more precise DR scoring system.

The current gold standard for evaluating and staging diabetic retinopathy (DR) is largely derived from the Early Treatment Diabetic Retinopathy Study (ETDRS) protocol, and it consists of mydriatic stereoscopic 30°color photographs obtained using a defined protocol of 7 standard retinal fields.^1-3 However, this method only captures approximately 30% of the entire retinal surface. With the advent of ultrawidefield (UWF) imaging, 82% of the retinal surface can be visualized, allowing evaluation of large regions of the retina periphery not previously considered for assessing the severity of DR. The ability to capture the peripheral retina has improved the rate of DR detection and assessment of DR severity, particularly in individuals with involvement of the peripheral retina.⁴ Numerous studies have supported the superiority of UWF imaging for detecting peripheral lesions in DR compared with conventional fundus color photography, with growing evidence that many eyes contain more severe disease than was previously visible.^5-8 There is also evidence to support excellent agreement of nonmydriatic UWF retinal images with dilated ETDRS photographs and dilated fundus examination in determining the severity of DR and diabetic macular edema.⁹

With the use of UWF pseudocolor imaging technology, a new phenotype of DR lesion distribution has been described—eyes that contain a larger portion of DR lesions outside the standard 7 ETDRS fields than within the 7 ETDRS fields—designated as predominantly peripheral lesions (PPLs).^10,11 Approximately one-third of hemorrhage and/or microaneurysms, intraretinal microvascular abnormalities, and new vessels elsewhere are located predominantly outside the ETDRS fields. Lesions found in the periphery on UWF images but not ETDRS film photographs suggest a more severe DR level in 10% of eyes.¹⁰ Additionally, data suggest eyes that were subjectively determined to contain PPLs identified on UWF imaging had greater than a 4-fold increased risk for progression to proliferative DR (PDR) when compared with eyes with DR lesion burden predominantly within the 7 ETDRS fields.¹¹ The ability to capture a larger portion of the retina has identified eyes that should be assigned a higher degree of retinopathy severity when comparing UWF images with the standard ETDRS grading scale.^12,13 Other studies have had similar findings, with 15% of eyes receiving higher DR severity grade when using UWF pseudocolor images compared with the 7 ETDRS fields,¹⁴ and almost 12% of eyes containing new vessels detected by UWF images in an area of the retina not covered by the classic 7 ETDRS fields.⁸ Additionally, eyes with PPLs identified on UWF imaging had more than a 3-fold increased risk of 2-step or more DR progression and almost a 5-fold increased risk for progression to PDR.¹¹

This evidence, if confirmed in replication studies, highlights the potential need to image and analyze the peripheral retina to more accurately grade the severity of DR and identify patients who may need more careful monitoring. Moreover, these findings demonstrate the importance of standardizing an accurate and validated method to classify eyes as containing either predominantly central lesions (PCLs) or PPLs on UWF retinal images to accurately identify patients who are more likely to have progression to PDR.

The aim of this study was to compare subjective and precise quantitative classification of DR lesion distribution from nonmydriadic UWF pseudocolor images to determine the reliability of subjective classification of eyes as containing either a PCL or PPL distribution.

This multiple-site, retrospective, cross-sectional study was approved by the institutional review boards of the University of California, Los Angeles, and the Narayana Nethralaya Eye Institute and was conducted in accordance with the Health Insurance Portability and Accountability Act and the Declaration of Helsinki. Written informed consent was obtained from all patients prior to ophthalmoscopic examination and UWF image acquisition.

Ultrawidefield pseudocolor images from patients who underwent DR screening at the Narayana Nethralaya Eye Institute in Bangalore, India, from December 20, 2014, through December 30, 2014, and at the retina clinic at Jules Stein Eye Institute from August 1, 2014, through February 2, 2016, were studied. Images were manually segmented between June 1, 2016, and December 1, 2016. Only eyes with evidence of DR were included in this study.

Exclusion criteria for this study included lack of DR lesions (ie, no visible DR) and poor UWF image quality due to obscured retina caused by misalignment, media opacity, and/or substantial eyelash artifact. Eyes with evidence of retinal venous occlusive disease or other significant pathology other than DR were also excluded. Diabetic retinopathy levels ranged from mild nonproliferative DR to PDR.

Nonmydriatic stereoscopic 200° UWF pseudocolor images were acquired using the Optos Daytona (Optos PLC) in India and the Optos P200Tx in Los Angeles. For each eye, the highest-quality image was selected. Ultrawidefield images were exported and subsequently manually segmented by experienced graders at the Doheny Image Reading Center. Disease boundaries of DR lesions including microaneurysms, hemorrhages, hard exudates, cotton-wool spots, intraretinal microvascular abnormalities, venous beading, venous loops, neovascularization of the disc, new vessels elsewhere, and preretinal hemorrhages were outlined using a grading software program (GRADOR) (Figure 1).

Before frequencies and SA of individual DR lesions were generated, manually segmented images were stereographically projected to a format that unifies the projection of the curved retina onto the imaging plane, accounting for differences in gaze direction, pixel-to-micrometer ratios, and UWF device.

The general method for positioning the ETDRS regions was to use spherical geometry to estimate the outlines of 30° circular regions on the projected Optomap. The transformation from the original to the projected Optomap was determined from optical models of the Optos Daytona and Optos P200Tx systems and a model eye. In the projected Optomap, the angular coordinates of every pixel are known, allowing the positions of ETDRS fields and measurements in real units (millimeters) to be estimated in a systematic and validated fashion.

Custom ETDRS grids along with defined peripheral fields were precisely generated for each image based on the specific location of the fovea and optic nerve head in that image and were superimposed onto the projected Optomap. Peripheral fields 3 to 7 were defined as extensions of the respective ETDRS fields 3 to 7 that were not covered by the ETDRS area.¹⁰ Diabetic retinopathy lesion, frequencies, and SA were quantified for each eye.

Ultrawidefield images were subjectively classified by 2 expert masked graders at Doheny Image Reading Center as having PCL if DR lesion burden appeared to be greater within the 7 ETDRS fields, and PPL if lesion burden appeared greater outside the 7 ETDRS fields. Classifications were repeated twice based on the numbers of lesions and based on the total area of lesions. Assessments were repeated by 1 grader several weeks later, to assess intragrader reproducibility.

For each lesion type, individual lesion frequencies, total SA (millimeters and pixels), and average SA were calculated for each ETDRS field, each peripheral field, and the overall retinal surface. The distance of each individual lesion to the foveal center and optic nerve head was calculated using the center of gravity of each lesion (the average x and y coordinate of the lesion). Lesions located in areas where the ETDRS fields overlap were counted separately for each individual ETDRS and peripheral field, but were only counted once when the ETRDS fields were considered altogether. Diabetic retinopathy lesion distribution in eyes was precisely classified in 2 ways: (1) global method in which eyes were designated as containing PCL if more DR lesions were located within all the combined ETDRS fields compared with all the combined peripheral fields, or PPL if more DR lesions were located within the combined peripheral fields, and (2) single-field method in which eyes were designated as having PPL when at least 1 peripheral field 3 to 7 had more DR lesions than its respective adjacent ETDRS field 3 to 7.

Intergrader, intragrader, and subjective vs quantitative classification were compared for level of agreement, based on both frequency and SA of lesions. Frequency-based classification of DR lesions in combined ETDRS fields and combined peripheral fields was compared with frequency-based classification of DR lesions in individual ETDRS fields and their respective peripheral fields.

The corresponding subjective intergrader and intragrader agreement as well as subjective vs quantitative agreement of PCL and PPL classifications from UWF images were calculated using weighted κ values. The DR lesion counts within each field were compared using paired t test analyses. All analyses were conducted using SPSS statistical software (version 18.0; IBM).

One hundred sixty-one eyes from 104 patients with clinical evidence of DR on imaging were enrolled in this study. Seventy-one patients (104 eyes) were recruited in India, and 33 patients (57 eyes) were enrolled in Los Angeles. The mean (SD) age was 63 (10) mm; 72 patients were men and 32 were women. Seventy-one patients were Indian, 17 white, 6 African American, 4 Asian, 3 Latino, and 3 other.

A total of 11 982 DR lesions were segmented on UWF images, with 9339 total DR lesions (78%) located within the ETDRS fields and 2643 total DR lesions (22%) located outside the ETDRS fields. The distribution of frequencies and SA of each DR lesion type including microaneurysms, hemorrhages, hard exudates, cotton-wool spots, intraretinal microvascular abnormalities, venous beading, venous loops, neovascularization of the disc, new vessels elsewhere, and preretinal hemorrhage were quantified for each ETDRS field, each peripheral field (peripheral fields 3-7), combined ETDRS fields (central retina), combined peripheral fields, and the entire retina captured in the UWF image (Table). Lesion distributions of all 11 982 DR lesions were compared within and outside the ETDRS fields by each lesion type (Figure 2).

On subjective frequency-based evaluation by graders, 133 eyes were classified as having PCL, and 28 eyes as having PPL. On exact quantification of lesion SA, 121 eyes were classified as having PCL, and 40 eyes as having PPL. On SA-based quantification, 134 eyes were classified as having PCL, and 27 eyes as having PPL. There was a significant difference between qualitative and quantitative classification of DR lesion distribution for both frequency-based (mean difference [SD]: PCL, 6 [2]; PPL, 13 [6]; P < .001) and SA-based (mean difference [SD]: PCL, 6 [1]; PPL, 20 [7]; P < .001) methods. The intragrader agreement for frequency-based vs SA-based classification of PCL and PPL distribution was 88% (n = 141) (weighted κ = 0.630) (eTable 1 in the Supplement). The intergrader agreement for subjective classification of PCL and PPL DR lesion distribution between 2 graders based on frequency compared with SA is presented in eTable 2 in the Supplement. There is considerable intergrader agreement of 97% (n = 156) in frequency-based classification of PCL and PPL DR lesion distribution (weighted κ = 0.884; P < .001). Similarly, there was 94% (n = 152) intergrader agreement in SA-based classification of PCL and PPL DR lesion distribution (weighted κ = 0.841; P < .001).

The agreement between subjective and quantitative classification of PCL and PPL DR lesion distribution is presented in eTable 3 in the Supplement. Agreement between subjective and quantitative frequency-based classification of PCL and PPL DR lesion distribution is 96% (n = 155) (weighted κ = 0.858; P < .001). Agreement between subjective and quantitative SA-based classification is 89% (n = 144) (weighted κ = 0.683; P < .001).

The agreement for quantitative frequency-based vs SA-based classification of PCL and PPL DR lesion distribution was 84% (n = 136) (weighted κ = 0.640) (eTable 4 in the Supplement).

When classifying DR lesion distribution using the global method, 140 eyes were classified as having PCL and 22 eyes (13.5%) were classified as having PPL. When classifying DR lesion distribution using the single-field method, 85 eyes were classified as having PCL and 77 (47.5%) were classified as having PPL (Figure 3).

This study compared subjective classification of DR lesion distribution on UWF images with precise quantitative measurements of lesion frequencies and surface areas, and designated DR lesion distribution of each eye as either PCL or PPL based on predominance of lesions within or outside the 7 ETDRS fields. Some eyes that were subjectively rated to have predominantly peripheral disease were found to have predominantly central disease when evaluated in a precise quantitative fashion, and there was disagreement in subjective and quantitative PCL and PPL classification of eyes when eyes were classified based on frequency compared with SA. Perhaps most importantly, we observed a stark difference in classification of DR distribution when using a global lesion approach compared with a single-field approach, with classification skewed toward PPL with the single-field approach.

There was substantial intergrader reproducibility in subjective classification of DR distribution; however, it is important to note that agreement was higher when based on frequency of lesions when compared with SA (94% vs 97%). This difference between SA and lesion number based classification was also evident when comparing subjective with quantitative classification (96% vs 89%).

These findings highlight several key points. First, subjective and quantitative classification of global DR lesion distribution in eyes as either PCL or PPL differed when eyes were classified solely based on lesion frequency compared with lesion SA. The possible explanation for this difference is 2-fold: some eyes contain a greater frequency of small DR lesions within the combined 7 ETDRS fields, designating the DR lesion distribution as PCL by frequency. However, the same eyes may contain fewer large DR lesions that encompass a larger SA of the peripheral retina, designating these eyes as PPL when classifying by SA alone. Additionally, the difference in pixel-to-micrometer ratios between the central retina and the peripheral retina may affect subjective classification of DR lesion distribution. For instance, at the center of the retina the pixels are smaller and toward the periphery they are larger; each pixel in the periphery represents a smaller area, resulting in lesions appearing larger than they truly are. Therefore, accurate subjective assessment of eyes can be difficult because of the differences in pixel-to-micrometer ratios in the central and peripheral retina, leading to overestimation of lesion size in the periphery.

The second key finding was the difference in DR distribution classification when using a global method vs a single-field method. When comparing the global and single-field methods of classification of PCL and PPL, the single-field method skewed classification to PPL, as the threshold for PPL classification was much lower. When using the global method, 13.6% of eyes were classified as having PPL distribution (n = 22), whereas 47.5% of eyes were classified as having PPL distribution (n = 77) when using the single-field method. The description of DR disease distribution as PCL and PPL as initially introduced is based on the single-field method of classification.¹¹ Although this method does not take lesion distribution in the global image into account, Silva et al¹¹ demonstrated a significant trend of increasing progression of nonproliferative DR and PDR onset with increasing number of PPL fields involved.

The discordant classifications of DR lesion distribution when using different methods to designate eyes as having PCL or PPL distribution raises a key question: what is the true definition of predominantly peripheral? Should it be based on lesion number, lesion size, or a combination of both? Should classification of DR lesion distribution be based on a global method wherein the total number of lesions within the combined peripheral and combined ETDRS fields are compared, or should it be based on a single-field method with comparison of the number of DR lesions in any single peripheral field relative to its corresponding ETDRS field? The improvement in the degree of peripheral retinal visualization with UWF images raises another question: in classifying eyes as having PCL or PPL distribution, should we define the center of the retina as all 7 ETDRS fields, ETDRS fields 1 through 3, or an anatomic landmark such as the equator because it is bounded by all the vortex veins?

In this study, precise quantitative parameters were shown to be superior in the assessment of the distribution of DR compared with previous subjective approaches. This highlights the need for evolving from subjective systems developed in an era of film photography toward quantitative systems for disease staging enabled by digital imaging and advances in image processing. Manual segmentation of lesions, as was done for this study, is not practical for large-scale clinical trials or clinical practice. Several Conformité Européenne (CE)–marked systems are available for detection of referral-warranted retinopathy and have demonstrated good performance in large studies,^15-17 with even better performance observed with newer machine learning systems.¹⁸ If DR lesions can be detected automatically by computer algorithms, rather than defining a lesion as central or peripheral based on an arbitrary threshold, the distance of the lesion from a central reference point (eg, fovea or optic disc center) can be considered as a continuous variable.

We demonstrated that using 2 different methods of classification (global vs single field) yielded different classification of DR lesion distribution, with single-field classification favoring PPL distribution. Given the apparent importance of peripheral lesions in DR progression, this study underscores a need to standardize the methods for rating the severity of peripheral disease using UWF images. An objective quantitative approach may be preferred and may translate to a more precise scoring system. Additionally, we must determine a method to adjudicate any discrepancies in classification of eyes that are quantitatively designated as PCL by frequency but PPL by SA, and vice versa.

Our study has a few limitations that should be considered when assessing our results. First, our sample size, while larger than some previous reports on this topic, is relatively small. Second, the distribution of lesions seen in our cohort may not reflect the general population with diabetes. Additionally, the UWF images used in this analysis were collected retrospectively, and thus the study is subject to ascertainment bias. Perhaps most importantly, our study is cross-sectional and lacks longitudinal information on DR progression and clinical outcomes. Although previous subjective single-field approaches may be less precise, they were at least shown to have some predictive value. Thus, while our findings do highlight that there may be variability in defining predominantly peripheral disease, it is uncertain which definition is most informative or indicative of possible disease progression.

Our study also has many strengths, including the use of certified reading center graders, a meticulous approach to segment all DR lesions, and high levels of reproducibility.

We report that subjective determinations of predominantly peripheral disease in patients with DR may differ from objective lesion quantification approaches and may differ based on the grading definitions used. Taken together, our findings highlight the challenges of subjective classification approaches and the need for a high level of precision in the definitions of new phenotypes such as predominantly peripheral DR. Given the apparent importance of peripheral DR lesions (to be further studied in ongoing Diabetic Retinopathy Clinical Research Network [DRCR.net] Protocol AA) as well as new therapeutic trials aimed at reversing the background retinopathy, the development of automated quantitative approaches to precisely stage DR is of great interest and may be within reach with advances in image processing.

Accepted for Publication: January 1, 2018.

Corresponding Author: SriniVas R. Sadda, MD, Doheny Image Reading Center, Doheny Eye Institute, PO Box 86228, Los Angeles, CA 90086 (ssadda@doheny.org).

Published Online: February 22, 2018. doi:10.1001/jamaophthalmol.2018.0070

Author Contributions: Ms Sears and Dr Sadda had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Sears, van Hemert, Sadda.

Acquisition, analysis, or interpretation of data: Sears, Nittala, Jayadev, Verhoek, Fleming, van Hemert, Tsui.

Drafting of the manuscript: Sears.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Sears, Nittala, Verhoek.

Obtained funding: Sears, Sadda.

Administrative, technical, or material support: Sears, Jayadev, Fleming.

Study supervision: Nittala, van Hemert, Sadda.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Drs Verhoek, Fleming, and Hemert are employees of Optos. Dr Sadda receives research support and serves as a consultant for Optos (although the research support did not support this project). Dr Sadda also receives research support from Carl Zeiss Meditec and serves as a consultant for Heidelberg Engineering and Centervue. No other disclosures were reported.

Funding/Support: Ms Sears is supported by a Research to Prevent Blindness Medical Student Eye Research Fellowship from Research to Prevent Blindness.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.