A-D, Underestimated discs; E-H, overestimated discs.
O’Neill EC, Gurria LU, Pandav SS, Kong YXG, Brennan JF, Xie J, Coote MA, Crowston JG. Glaucomatous Optic Neuropathy Evaluation ProjectFactors Associated With Underestimation of Glaucoma Likelihood. JAMA Ophthalmol. 2014;132(5):560-566. doi:10.1001/jamaophthalmol.2014.96
Glaucoma is a significant health problem for which diagnosis remains suboptimal. Optic disc evaluation, which is fundamental to the diagnosis, is a difficult skill to acquire.
To determine the optic disc characteristics that most influence decision making in the assessment of glaucoma likelihood and to ascertain the optic disc features associated with overestimation and underestimation of glaucoma likelihood.
Design, Setting, and Participants
This prospective, observational, Internet-based study with multinational participation included 197 ophthalmic clinicians (37 glaucoma subspecialists, 51 comprehensive ophthalmologists, and 109 ophthalmology trainees) from 22 countries who self-registered for the Glaucomatous Optic Neuropathy Evaluation (GONE) Project from December 1, 2008 through June 30, 2010.
A series of 42 monoscopic optic disc photographs of healthy and glaucomatous eyes were presented to clinicians using the GONE Project Program. Participants were asked to assess each disc according to 9 conventional topographic features and assign a presumptive grade for glaucoma likelihood.
Main Outcomes and Measures
Agreement (κ and weighted κ) among participants for disc signs and glaucoma likelihood and contributions of disc-related factors to overestimation and underestimation of glaucoma likelihood.
Ophthalmology trainees and comprehensive ophthalmologists underestimated glaucoma likelihood in a mean (SD) of 22.1% (1.6%) and 23.8% (1.8%) of discs, respectively. Underestimation of vertical cup-disc ratio and failure to identify retinal nerve fiber layer loss, disc hemorrhage, or rim loss were most likely to lead to underestimation of glaucoma. When all 4 features were inaccurately assessed, underestimation of glaucoma likelihood increased to 43.0%. Ophthalmology trainees and comprehensive ophthalmologists overestimated glaucoma likelihood in a mean (SD) of 13.0% (1.2%) and 8.9% (1.3%) of discs, respectively. Overestimation of glaucoma likelihood was associated with overestimation of retinal nerve fiber layer loss, rim loss, vertical cup-disc ratio, disc hemorrhage, and incorrect assessment of disc tilt and was more likely in large discs.
Conclusions and Relevance
Ophthalmology trainees and comprehensive ophthalmologists underestimated glaucoma likelihood in approximately 1 in 5 disc photographs and were twice as likely to underestimate as overestimate glaucoma likelihood. Underestimating the vertical cup-disc ratio and cup shape and missing retinal nerve fiber layer defects and disc hemorrhage were the key errors that led to underestimation. When all 4 parameters were incorrectly assessed, underestimation increased to almost 1 in 2.
The Glaucomatous Optic Neuropathy Evaluation (GONE) Project was developed to benchmark optic disc examination among eye care professionals and gain further insights into the decision-making processes that clinicians adopt in determining whether an optic disc appearance is glaucomatous or not.1 Data demonstrating the utility of the GONE Project in studying the levels of agreement among a large cohort of ophthalmologists for a selected set of healthy and glaucomatous monoscopic optic disc photographs were previously published. The levels of agreement were significantly higher among glaucoma subspecialists compared with ophthalmology trainees, highlighting the importance of training and experience in evaluation for glaucoma.1 Interobserver agreement in glaucoma likelihood was substantial for our expert panel (κ, 0.63; 95% CI, 0.60-0.67) and was significantly higher than for comprehensive ophthalmologists (κ, 0.51; 95% CI, 0.48-0.55) and ophthalmology trainees (κ, 0.48; 95% CI, 0.44-0.51).1 The GONE Project also evaluates individual clinical signs associated with the optic nerve head and retinal nerve fiber layer (RNFL). This evaluation allowed us to determine which clinical signs were most predictive of a correct or incorrect glaucoma diagnosis, which may point to key features that should be highlighted when teaching optic disc examination.
The levels of agreement among observers for the assessment of the GONE Project disc set were described in a previous article.1 The first aim of the current study was to determine which optic disc characteristics were most influential in leading to a correct or incorrect assessment for glaucoma likelihood. A second aim was to identify the specific morphologic traits of optic discs that most commonly led to underestimation or overestimation of glaucoma likelihood.
Institutional review board approval from the Royal Victoria Eye and Ear Hospital was obtained for this project, and the research was conducted in accordance with the Declaration of Helsinki guidelines. Patients provided consent to their photographs being used in the GONE Project for teaching purposes. The GONE Project is an Internet-based self-assessment system that delivers a series of monoscopic optic disc photographs selected from a large set (n = 2500) of healthy and glaucomatous optic discs.1 The self-assessment system uses an initial set of 42 monoscopic disc images to represent a range of physiological and glaucomatous disc characteristics. Participants were asked to grade 9 topographic features of the disc and RNFL, including disc size, disc shape, disc tilt, peripapillary atrophy (PPA), vertical cup-disc ratio (VCDR) assessed by contour, cup shape (rim loss), cup depth, RNFL defects, and the presence or absence of hemorrhage. All variables were graded according to a categorical grading scale (Table 1). For the final question, participants were asked to assess glaucoma likelihood according to a 4-point ordinal scale, which included the following choices: unlikely, possible, probable, or certain. Participants had a time limit of 90 seconds for assessment of each disc image. All answers were recorded using the online system.1
Participants were self-selected having learned of the GONE Project through conference presentations, publications, and word of mouth among ophthalmology professionals throughout the Americas, Europe, Australia, Asia, and the Pacific Rim. Ophthalmology trainees and ophthalmologists who self-registered for the GONE Project from December 1, 2008 through June 30, 2010 through the custom-designed website (http://www.gone-project.com)1 and completed the full set of 42 discs were included in the study. At the time of data analysis, 197 participants had fully completed the evaluation, of which there were 109 ophthalmology trainees, 51 comprehensive ophthalmologists, and 37 glaucoma subspecialists. Participants who did not fully complete the assessment of all 42 discs were not included in the analysis. All participants received the same instructions posted on the GONE Project website before commencing the assessment. The instructions included a sample disc to familiarize themselves with the process for disc grading. No additional clinical details, ancillary ophthalmic investigations, or other test results were provided because the grading was intended to be based solely on the disc photograph.1
Statistical assessment was performed after consultation with an in-house statistician and in keeping with the prior GONE Project publication.1 Interobserver agreement was calculated using κ for nominal variables (disc shape, tilt, cup shape [rim loss], RNFL loss, and hemorrhage) and weighted κ (κw) with linear weighting for ordinal variables. The κ is a numerical value that can range from −1 (complete disagreement) to +1 (total agreement). A value of 0 represents chance agreement. Multivariable logistic regression analysis was used to study the relationship between κ statistics for glaucoma likelihood assessment and other disc features to further understand the factors that contribute to high or low agreement with the reference answer.
Factors that contribute to increased risk of underestimation or overestimation of glaucoma likelihood were assessed using multivariable logistic regression analysis with particular disc features serving as the independent variable. Similarly, the effect of assessing an optic disc parameter inaccurately on the risk of underestimation or overestimation of glaucoma likelihood was studied using multivariate logistic analysis. The κ and κw were calculated using MedCalc (MedCalc Software, Mariakerke, Belgium). All statistical analysis, other than κ statistics, was performed with SPSS software, version 15 (SPSS Inc).
The mode of the grades by the 37 glaucoma subspecialists was used to form reference answers for comparison with comprehensive ophthalmologists and ophthalmology trainees. Grades within 1 scale point from the reference answer were considered correct. Underestimation was defined as rating glaucoma likelihood at least 1 or more scale points lower than the reference answer, and overestimation was defined as rating glaucoma likelihood at least 1 or more scale points higher than the reference answer (Table 2). Overestimations and underestimations were defined similarly for ophthalmology trainees, comprehensive ophthalmologists, and glaucoma subspecialists.
Quiz Ref IDOverall, ophthalmology trainees and comprehensive ophthalmologists were twice as likely to underestimate glaucoma likelihood as overestimate it. In optic discs that glaucoma subspecialists graded as probable or definite glaucoma, ophthalmology trainees and comprehensive ophthalmologists underestimated glaucoma likelihood in a mean (SD) of 22.1% (1.6%) and 23.8% (1.8%), respectively. In optic discs graded by glaucoma subspecialists as possible or unlikely glaucoma, ophthalmology trainees and comprehensive ophthalmologists overestimated glaucoma likelihood in a mean (SD) of 13.0% (1.2%) and 8.9% (1.3%), respectively. Examples of the optic discs most frequently underestimated and overestimated for glaucoma likelihood are shown in the Figure.
Quiz Ref IDWe used multivariable linear regression analysis to determine the optic disc features that, when assessed correctly, had the greatest effect on leading to a correct assessment of glaucoma likelihood (defined as within 1 grade from the reference answer). Comparing standardized coefficient β, we identified that RNFL loss, rim loss, hemorrhage, and VCDR (P < .01) were the features that most influence the assessment of glaucoma likelihood. When all 4 of these disc features were incorrectly assessed by ophthalmology trainees and comprehensive ophthalmologists, underestimation of glaucoma likelihood increased to a mean (SD) of 43.0% (8.0%) and 42.9% (13.7%), respectively, but overestimation did not change significantly (10.7% [6.0%] and 15.0% [10.7%], respectively). When ophthalmology trainees and comprehensive ophthalmologists accurately assessed all 4 characteristics, underestimation of glaucoma likelihood significantly decreased to 10.9% (1.9%) and 3.1% (1.0%) and overestimation to 2.9% (0.6%) and 1.6% (0.7%), respectively.
Using multivariable logistic regression analysis, we found that among ophthalmology trainees and comprehensive ophthalmologists underestimation of glaucoma likelihood was associated with underestimation of the VCDR and failure to correctly identify RNFL loss, disc hemorrhage, or rim loss. Interestingly, among comprehensive ophthalmologists, it was also associated with underestimation of PPA (Table 3).
Quiz Ref IDDisc features that most predisposed ophthalmology trainees to underestimate glaucoma likelihood included horizontal ovoid-shaped discs and the presence of moderate PPA. Among comprehensive ophthalmologists, the presence of large PPA was associated with underestimation (Table 3).
Quiz Ref IDAmong ophthalmology trainees and comprehensive ophthalmologists, multivariable logistic regression analysis revealed that overestimation of glaucoma likelihood was associated with overestimation of RNFL loss, rim loss, VCDR, cup depth and hemorrhage, and incorrect assessment of disc tilt (Table 4). Among comprehensive ophthalmologists, overestimation of glaucoma likelihood was also associated with underestimation of PPA.
Large discs were more likely to be overestimated as glaucomatous among ophthalmology trainees and comprehensive ophthalmologists. In addition, horizontal or vertical disc tilt also predisposed ophthalmology trainees to overestimate glaucoma likelihood, and among comprehensive ophthalmologists moderate PPA was associated with overestimation (Table 4).
In this study, we report data from a large, comprehensive study of optic disc assessment conducted using an Internet-based self-assessment system that examined monoscopic optic disc photographs. We investigated how the assessment of key morphologic features of the optic disc and RNFL influenced the assessment of glaucoma likelihood in ophthalmology trainees and comprehensive ophthalmologists. Ophthalmology trainees and comprehensive ophthalmologists underestimated glaucoma likelihood in approximately 1 in 5 discs and were twice as likely to underestimate rather than overestimate glaucoma likelihood. Errors in the assessment of VCDR, RNFL, rim loss, and disc hemorrhage were the most influential factors that led to incorrect estimation of glaucoma likelihood. When all 4 parameters were incorrectly assessed, underestimation increased to approximately 40% in both groups.
Previous studies2- 7 with smaller numbers of participants have found significant variability in the subjective assessment of optic disc photographs for glaucoma. Even among glaucoma subspecialists, a high degree of variability has been reported for optic disc assessment.6- 8
To minimize the effect of interobserver variation, we used the mode of the answers given by the 37 glaucoma subspecialists as the criterion standard response for each of the disc characteristics studied and against which comparisons were made. This large panel of glaucoma subspecialists ensured inclusion of differing opinions on a given disc parameter.
Previous studies4,8- 10 that examined the effect of disc assessment on diagnostic accuracy have usually focused on assessment of a single disc characteristic, such as VCDR or optic disc notching, and simply whether the disc was glaucomatous or not. A European study11 that examined comprehensive ophthalmologists and glaucoma specialists found that, despite moderate sensitivity and specificity in identifying glaucomatous damage from stereoscopic optic disc images, variation in observer diagnostic accuracy existed. Another study found that discs of smaller size and with milder glaucomatous loss were most difficult to classify correctly by comprehensive ophthalmologists.12 However, the influence of other morphologic features was not reported. In our study, we found that assessment of VCDR, RNFL, rim loss, and disc hemorrhages was most influential on diagnostic accuracy. When all 4 of these disc features were inaccurately assessed, underestimation among ophthalmology trainees and comprehensive ophthalmologists reached a mean (SD) of 43.0% (8.0%) and 42.9% (13.7%), respectively. When accurately assessed, underestimation decreased significantly to 10.9% (1.9%) and 3.1% (1.0%), respectively. The reduction in underestimation among comprehensive ophthalmologists was significantly greater than for ophthalmology trainees (P < .01).
Certain phenotypic disc characteristics, such as the shape and size of the optic disc and the presence or absence of PPA, can influence the assessment of glaucoma likelihood and make accurate estimation of glaucoma likelihood more difficult.13,14 β-zone PPA is more frequent in glaucoma, tends to enlarge over time, and can be predictive of visual field progression.15- 17 We found that, among ophthalmology trainees, discs with moderate PPA (P = .01) and, among comprehensive ophthalmologists, discs with large amounts of PPA (P = .007) were more likely to have glaucoma likelihood underestimated. This finding may have been secondary to the effect the PPA has on obscuring identification of the outer margin of the optic disc.
The mean optic disc is vertically oval,15,18,19 and the normal optic cup is horizontally oval.15 Therefore, in the healthy eye the mean VCDR should be slightly larger in the horizontal than the vertical axis. In our study, horizontally ovoid optic discs were at a higher risk of having glaucoma likelihood underestimated by ophthalmology trainees (P < .001). This finding may have been secondary to the influence of disc shape on other VCDR or cup shape in these discs.
Tilted discs are associated with increased PPA, moderate myopia, and astigmatism and can be associated with visual field defects in nonglaucomatous eyes.20 In our study, both horizontal and vertical tilting of the disc caused overestimation of glaucoma likelihood by ophthalmology trainees only (P < .001). This finding may be secondary to difficulty in accurately differentiating the outer margin of the disc or assessing the optic cup in these optic nerves13 and a lack of experience in examining such discs given that it did not increase the risk among comprehensive ophthalmologists.
The positive association between optic disc and optic cup diameter is well established.15,18,19 This study confirms previous work in which large optic discs were more likely to have glaucoma likelihood overestimated secondary to their large, although physiologic, VCDR.21,22 Surprisingly, however, in this series, small disc size did not increase underestimation of glaucoma likelihood as previously reported.23
Potential limitations to this study include the use of monoscopic disc images, a limited variety of disc phenotypes, and differences in the selection of the expert group (invited) and participants (self-selected). Depth perception is not possible on monoscopic images, and estimates of cup depth rely on 2-dimensional cues, such as vessel contour, which may have reduced agreement for certain disc parameters. We chose monoscopic images because they facilitated widespread distribution of the GONE Project without the need for stereoscopic viewing apparatus. Some studies4,7,9 have found stereoscopic images to have a small measurable benefit in the assessment of optic disc features over monoscopic images. However, other studies24,25 refute this finding.
The GONE Project uses a set of 42 discs that may not fully reflect the phenotypic variation seen in healthy and glaucomatous discs. Although we have selected discs to highlight a wide range of optic disc features, certain disc characteristics, such as small disc size, disc hemorrhage, and high VCDR, are less well represented (Table 1).
Ophthalmology trainees and comprehensive ophthalmologists underestimate glaucoma likelihood in 1 in 5 discs and are twice as likely to underestimate as overestimate glaucoma likelihood. Moderate or large PPA, ovoid horizontal disc shape, and error in the assessment of VCDR, RNFL, cup shape (rim loss), and disc hemorrhage lead to underestimation. When all 4 parameters are poorly assessed, underestimation increases to almost 1 in 2. The results of this study will facilitate creation of targeted teaching tools for ophthalmology trainees and continual medical education modules for comprehensive ophthalmologists, focusing on specific aspects of disc examination. We recommend that the assessment parameters be highlighted in optic nerve head examination teaching and continuing medical education modules.
Submitted for Publication: July 4, 2013; final revision received November 9, 2013; accepted November 22, 2013.
Corresponding Author: Jonathan G. Crowston, PhD, FRANZCO, Centre for Eye Research Australia, University of Melbourne, Royal Victorian Eye and Ear Hospital, 32 Gisborne St, Victoria 3002, Australia.
Published Online: April 3, 2014. doi:10.1001/jamaophthalmol.2014.96.
Author Contributions: Drs Kong and Pandav had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs O’Neill and Gurria contributed equally to the manuscript and share first authorship.
Study concept and design: O’Neill, Gurria, Coote, Crowston.
Acquisition, analysis, and interpretation of data: O’Neill, Gurria, Pandav, Kong, Brennan, Xie, Coote.
Drafting of the manuscript: O’Neill, Gurria, Coote.
Critical revision of the manuscript for important intellectual content: O’Neill, Pandav, Kong, Brennan, Xie, Coote, Crowston.
Statistical analysis: O’Neill, Pandav, Kong, Xie, Coote.
Obtained funding: Gurria, Coote.
Administrative, technical, or material support: Brennan, Coote.
Study supervision: O’Neill, Pandav, Coote, Crowston.
Conflict of Interest Disclosures: None reported.
Funding/Support: The Centre for Eye Research Australia receives Operational Infrastructure Support from the Victorian Government. This project was supported by an unrestricted grant from Allergan Inc, The Helen McPherson Trust Grant, The Cranbourne Trust, The National Health and Medical Research Council Practitioner Fellowship, and The Dorothy Adele Edols Charitable Trust.
Role of the Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The views expressed in this publication are those of the authors and not necessarily those of the US Department of Health.