Standard Care vs Corticosteroid for Retinal Vein Occlusion (SCORE) Study System for Evaluation of Stereoscopic Color Fundus Photographs and Fluorescein Angiograms: SCORE Study Report 9

Objective  To describe the procedures and reproducibility for grading stereoscopic color fundus photographs and fluorescein angiograms of participants in the SCORE Study.

Methods  Standardized stereoscopic fundus photographs and fluorescein angiograms taken at 84 clinical centers were evaluated by graders at a central reading center. Type of retinal vein occlusion (RVO), area of retinal thickening, and area of retinal hemorrhage are evaluated from fundus photographs; area of fluorescein leakage and area of capillary nonperfusion are measured on fluorescein angiography. Temporal reproducibility consisted of annual regrading of a randomly selected dedicated subset of fundus photographs (60 subjects) and fluorescein angiograms (40 subjects) for 3 successive years. Contemporaneous reproducibility involved monthly regrading of a 5% random selection of recently evaluated fundus photographs (n = 73).

Results  The intergrader agreement for RVO type and presence of retinal thickening was greater than 90% in the 3 annual regrades. The intraclass correlation (ICC) for area of retinal thickening in the 3 years ranged from 0.39 to 0.64 and for area of retinal hemorrhage, 0.87 to 0.96. The ICC for area of fluorescein leakage ranged from 0.66 to 0.75 and for capillary nonperfusion, 0.94 to 0.97. The contemporaneous reproducibility results were similar to those of temporal reproducibility for all variables except area of retinal thickening (ICC, 0.84).

Conclusions  The fundus photography and fluorescein angiography grading procedures for the SCORE Study are reproducible and can be used for multicenter longitudinal studies of RVO. A systematic temporal drift occurred in evaluating area of retinal thickening.

The Standard Care vs Corticosteroid for Retinal Vein Occlusion (SCORE) Study consists of 2 multicenter randomized trials of 682 participants that investigate the safety and efficacy of standard of care vs intravitreal triamcinolone in the treatment of macular edema secondary to central retinal vein occlusion (CRVO) or branch retinal vein occlusion (BRVO). The design of this trial and the baseline data are provided in a previous article.¹ Stereoscopic color fundus photographs, fluorescein angiograms, and optical coherence tomographic images are taken to document retinal thickening (macular edema), retinal hemorrhage, fluorescein leakage, capillary nonperfusion, and other features of retinal vein occlusion (RVO). These images are evaluated by trained and certified graders at the Fundus Photograph Reading Center, University of Wisconsin. In this article, we describe the SCORE Study grading procedures and the reproducibility of the SCORE Study grading system for color fundus photographs and fluorescein angiograms. Optical coherence tomographic procedures and reproducibility are described in a separate article.² All grading forms used for the study are available from the National Technical Information Service.³

The SCORE Study uses stereoscopic pairs of color fundus photographs taken in a standardized fashion by certified photographers.⁴ Seven-field 30° fundus photographs of the study eye are taken at baseline and months 12, 24, and 36, and 3-field photographs are taken at months 4, 8, 16, and 20. Fluorescein angiograms may be obtained in either film or digital format and are taken at baseline and months 4, 12, 24, and 36. The imaging protocol for both color fundus photographs and fluorescein angiograms is available at the National Technical Information Service Web site. All images are deidentified in compliance with Health Insurance Portability and Accountability Act (HIPAA) regulations.

Film images of stereoscopic pairs of color fundus photographs and fluorescein angiograms are mounted in plastic sheets and viewed against a light box using a Donaldson stereo viewer (×5 magnification). A quadrant grid was designed for the SCORE Study and is centered on the optic disc. The grid consists of 4 spokes at 12-, 3-, 6-, and 9-o’clock positions, dividing the fundus into 4 quadrants: superotemporal, superonasal, inferotemporal, and inferonasal (Figure 1A), and helps the grader determine the type of RVO and the location (superior vs inferior) of a BRVO. For macular abnormalities seen on color fundus photographs and fluorescein angiograms, the SCORE Study used a modified Early Treatment for Diabetic Retinopathy Study (ETDRS) grid centered on the macula (Figure 1B). The grid is composed of 9 subfields with a total of 16 disc areas and is used to determine the area and location of retinal thickening and retinal hemorrhage.⁴ The grid consists of black lines on transparent acetate stock and is placed over the color photograph or film angiographic image that is centered on the macula (field 2). For digital fluorescein angiograms, the grid is overlaid on the image electronically.

The Fundus Photograph Reading Center defines the type of RVO as follows:

• Central retinal vein occlusion. A CRVO is present when retinal abnormalities, typically retinal hemorrhages and dilated retinal veins, are found in all 4 quadrants. Although it is common to find hemorrhages in all 4 quadrants, there are eyes with CRVO where the blood has cleared and macular edema has persisted. In the absence of retinal hemorrhages, a CRVO may be identified by the presence of collateral vessels on the disc. In the retina, there may be venous dilation and tortuosity, perivenous sheathing, and arteriolar narrowing. Chronic macular abnormalities such as cystoid macular edema and retinal pigment epithelial changes may also provide clues to the presence of a CRVO.

• Branch retinal vein occlusion. A BRVO is present when dilated retinal veins and retinal hemorrhages occupy 1, 2, or 3 quadrants of the retina. A typical BRVO occupies less than 1 quadrant of the retina and has a wedge-shaped appearance in the inferior or superior hemisphere. The branch vein occlusion may demonstrate macular edema, cotton-wool spots, and both deep and superficial retinal hemorrhages confined to the quadrant fed by the branch vein. Like CRVO, participants with BRVO may no longer have visible retinal hemorrhage and, in such cases, chronic macular findings are again used to determine BRVO.

• Hemiretinal retinal vein occlusion. A hemiretinal vein occlusion is present when there are features intermediate in extent between central and branch vein occlusions. A hemiretinal retinal vein occlusion affects either the superior or inferior retinal hemispheres, and the retinal hemorrhages involved are nearly equal in 2 altitudinal quadrants (the nasal and temporal aspects) of the involved hemisphere.

The SCORE Study system for grading color fundus photographs includes evaluation of the type and location of the vein occlusion, the presence and area of retinal thickening within the ETDRS grid, the presence and area of retinal hemorrhage within the ETDRS grid, as well as other features of vein occlusion. The type and location of RVO is evaluated at baseline by 2 graders and any disagreement is resolved by the SCORE Study reading center principal investigator (Dr Blodi). Further baseline evaluation and all follow-up visit evaluations are performed by a single grader. Graders are masked to treatment assignment but are not masked to the appearance of the fellow eye. Fundus photographs from all visits are graded independently of each other. All data are entered electronically into a password-protected database.

At each visit, the field 2 photographs are used to estimate the percentage of retinal thickening in each of the 9 subfields of the ETDRS grid. Based on the percentage of involvement of each subfield, an electronic algorithm calculates the disc areas of retinal thickening within the entire grid. The area of retinal hemorrhage (both intraretinal and subretinal) is graded in a similar fashion. Hemorrhages that are anterior to the retina (ie, preretinal and vitreous hemorrhages) are excluded from the measurement of the area of retinal hemorrhage. eFigures 123 describe the methodology in detail.

Other RVO features evaluated in field 2 include the presence of intraretinal cystoid spaces, hard exudates, epiretinal membrane, pigmentary disturbance and atrophy, macular fibrosis, retinal detachment, and macular photocoagulation scars. When 7-field fundus photographs are available, the presence of cotton-wool spots, collateral vessels, new vessels, fibrous proliferation, preretinal hemorrhage, vitreous hemorrhage, and scatter photocoagulation scars are also documented from peripheral fields. The optic disc evaluation includes the presence of disc swelling, collateral vessels, new vessels, and fibrous proliferation.

The SCORE Study system for grading fluorescein angiography is based on the ETDRS fluorescein angiographic grading protocol.⁵ The area of fluorescein leakage and area of capillary nonperfusion are quantified and the change in these areas between baseline and follow-up visits is estimated. A single grader evaluates each visit and has access to previous angiograms and previous grading data when grading follow-up angiograms. This longitudinal system of grading angiograms allows graders to overcome differences in quality of images over subsequent visits.

The presence and area of fluorescein leakage is estimated by using a stereo pair of field 2 images in the late phase of the angiogram and comparing them with images from the earlier phase. The grader excludes fluorescein leakage presumed to be from within the choroid or anterior to the retina (if retinal neovascularization is present). Areas of blocked fluorescence due to retinal hemorrhages surrounded by fluorescein leakage are included in the area of leakage. The percentage of leakage in each of the 9 subfields of the ETDRS grid (Figure 2) is assessed and an algorithm is applied to yield the disc areas of leakage within the entire grid. For digital images, the macular grid and planimetry software are used to give area measurements in millimeters squared or disc areas. The presence and area of cystoid changes are evaluated from the late phase of the angiogram as well.

Capillary nonperfusion within the grid is estimated from field 2 as a percentage within 8 subfields; capillary nonperfusion is not graded within the center circle, as this represents the foveal avascular zone. Capillary nonperfusion is defined as the absence of retinal arterioles and/or capillaries and is detected by characteristics such as a pruned appearance of adjacent arterioles and a darker appearance of the choroid. Areas within the grid that contain intraretinal blood may prevent the assessment of capillary nonperfusion, and in these eyes the area is recorded as ungradable. eFigures 4 and 5 describe the methodology in detail.

A 3-tier quality-control program was used for the SCORE Study: temporal reproducibility, contemporaneous reproducibility, and ongoing quality training. Temporal reproducibility was assessed on an established set of SCORE Study images with the intent to have them regraded annually through the course of the SCORE Study. Fundus photographs of 60 subjects and fluorescein angiograms of 40 subjects were randomly selected from the SCORE Study images that had grading completed by July 2006, irrespective of the visit. Images were distributed among the same 5 to 7 graders for each annual reproducibility exercise. The graders did not have access to the original grade of record and entered the regrade data into a quality-control database. Three temporal reproducibility exercises have been completed in the SCORE Study in successive years.

Contemporaneous reproducibility of color fundus photograph gradings was performed monthly during the second and third years of the SCORE Study. A random selection of approximately 5% of the SCORE Study fundus photographs graded within the prior 30 days was regraded by a second grader. Contemporaneous reproducibility was analyzed in the same manner as temporal reproducibility by comparing the quality-control regrade with the original grade of record. Ongoing quality training includes frequent case review meetings to discuss difficult cases from current work, review the grading protocol, and confer the results of the quality-control exercises.

For both the temporal and contemporaneous reproducibility, intergrader agreement was assessed by means of percent of exact agreement of categorical variables and intraclass correlations (ICCs) for continuous variables between each regrade and the original grade of record. κ Statistics were not calculated for categorical variables with skewed data (eg, the temporal reproducibility sample was predominantly from baseline and all eyes had retinal thickening present for the variable presence of retinal thickening). Exact agreement of greater than 90% was considered excellent reproducibility and greater than 70%, moderate reproducibility. An ICC of more than 0.75 was considered excellent reproducibility and greater than 0.4 was moderate reproducibility.

Table 1 and Table 2 represent the results of the 3 temporal reproducibility exercises where the same set of fundus photographs and fluorescein angiograms was regraded annually. The results of each annual regrade were compared with the original grading. In Table 1, the intergrader agreement for the type of RVO was 93% at year 1, 96% at year 2, and 97% at year 3. The agreement for the presence of retinal thickening at the center of the macula was 97%, 98%, and 96%, respectively. The ICC for the area of retinal thickening on color photographs was 0.64, 0.39, and 0.52, respectively. The ICC for area of retinal hemorrhage was 0.87, 0.88, and 0.96, respectively. On fluorescein angiography, the ICC for area of fluorescein leakage within the grid was 0.75, 0.66, and 0.69, respectively; the ICC for area of capillary nonperfusion within the grid was 0.94, 0.97, and 0.97, respectively. In Table 2, the mean areas obtained at each annual regrade are compared with the original grade. Figure 3 represents the scatterplots comparing the 3 annual regrades with the original grade of record for area of retinal thickening, area of retinal hemorrhage, area of fluorescein leakage, and area of capillary loss.

Fundus photographs of 73 subjects have been regraded for contemporaneous reproducibility during a 10-month period. The presence of retinal thickening, when regraded by a second grader within the same month, showed an exact agreement of 82%. The ICC for area of retinal thickening was 0.84 and for area of retinal hemorrhage, 0.89. The scatterplots are represented in Figure 3.

The SCORE Study is one of the largest multicenter studies in RVO since the Central Vein Occlusion Study and Branch Vein Occlusion Study⁶ and provides an excellent opportunity to evaluate in detail the fundus and fluorescein angiography features in RVO. This article describes the procedures and reproducibility of evaluating stereoscopic color fundus photographs and fluorescein angiograms developed for the assessment of RVO features in the SCORE Study. The features graded on color fundus photographs and fluorescein angiograms are important outcome measures in the SCORE Study.

The temporal reproducibility exercises evaluate the reproducibility of a reserved sample of images throughout the course of a study (similar to a laboratory calibration). These exercises help identify fluctuations in the grading methodology over the years of a clinical trial owing to changes in grading personnel, changes in technology, and experience gained over the course of the study. On the other hand, contemporaneous reproducibility gives immediate feedback regarding grader performance.

Agreement on the type of RVO at baseline by trained nonphysician graders using the definitions developed for the SCORE Study shows excellent temporal reproducibility (93%-96%) during 3 years. In addition, the graders are in good agreement (92%) with ophthalmologists' classification of RVO.⁷

The temporal reproducibility is also excellent for the detection of presence of retinal thickening (96%-98%) and is moderate (>0.4 to <0.75) for determining the area of retinal thickening (ICC, 0.39-0.64). As shown in Table 2 and Figure 3, the area of thickening in the yearly regrades showed a larger area in all 3 annual regrades compared with the original grade. However, the means (Table 2) are within the grading margin of error. The SCORE graders were all experienced in evaluating diabetic macular edema, which tends to have focal areas of retinal thickening, whereas areas of thickening among RVO eyes are typically larger and more diffuse. We speculate that the SCORE Study graders may have initially been conservative in grading the area of retinal thickening in all eyes. Experience over the course of the study changed the approach to grading RVO to include larger areas of edema, especially for the acute phase of RVO.

Assessment of retinal thickening using color fundus photographs requires good stereoscopic images, since thickening is identified by retinal elevation with or without partial loss of transparency.⁴ Use of an integrated approach in which color fundus photographs are graded concurrently with optical coherence tomographic images may improve reproducibility for evaluation of retinal thickening.⁸ This procedure was not used in the SCORE Study in which fundus photographs and optical coherence tomographic images were evaluated separately.

In the SCORE Study, agreement on area of retinal hemorrhages on color fundus photographs showed excellent reproducibility (0.87-0.96). Although easy to detect in acute cases of RVO, resolving hemorrhages are often difficult to distinguish from the background choroidal pattern in less acute cases. Image quality, especially red saturation, can interfere with detection of hemorrhages. In the digital environment, adjusting the color and tonal balance (optimization) of images removes some of the image quality–related issues and may improve the reproducibility further.⁹

Most eyes in the temporal drift sample did not have capillary nonperfusion on fluorescein angiography. Temporal reproducibility was excellent in the few eyes that had minimal capillary nonperfusion. Reproducibility of fluorescein leakage is moderate with area of leakage consistently larger on all annual regrades. Detection of fluorescein leakage can be difficult owing to the indistinct borders of the area of leakage and the strong influence of photograph quality.

The SCORE Study grading system for RVO provides clear definitions and evaluation methodology for color fundus photographs and fluorescein angiograms. The SCORE Study grading system provides morphologic characterization of the disease in terms of the type of RVO and the area of retinal thickening, retinal hemorrhage, fluorescein leakage, and capillary nonperfusion. A systematic temporal drift occurred in evaluating area of retinal thickening as the reading center gained experience in grading RVO studies. Overall, the reproducibility and quality-control exercises performed in the SCORE Study grading system demonstrate the reliability of assessment of RVO with color fundus photographs and fluorescein angiography. The SCORE Study grading system is reproducible and can be used for future multicenter longitudinal studies of RVO.

Correspondence: Barbara A. Blodi, MD, Department of Ophthalmology and Visual Sciences, University of Wisconsin–Madison, 2870 University Ave, Room 206, Madison, WI 53705-3611.

Submitted for Publication: October 14, 2009; final revision received February 23, 2010; accepted March 3, 2010.

Author Contributions: As the principal investigator for the Fundus Photograph Reading Center, Dr Blodi had full access to all of the reading center data in the study and takes responsibility for the integrity of that data and the accuracy of the data analysis.

Financial Disclosure: None reported.

Funding/Support: The SCORE Study was supported by grants 5U10EY014351, 5U10EY014352, and 5U10EY014404 from the National Eye Institute, National Institutes of Health, Department of Health and Human Services. Support was also provided in part by Allergan Inc through donation of investigational drug and partial funding of site monitoring visits and secondary data analyses.

Online-Only Material: The eFigures are available at http://www.archophthalmol.com.