Results and Repeatability of Retinal Thickness Measurements From Certification Submissions

Objectives  To present the results for subgroups defined by center point (CP) measurement and to assess the repeatability of the Fast Retinal Thickness Map analysis results from the Stratus OCT3 machine.

Methods  Two hundred eighty-one replicate OCT3 scans from 134 operators' certification submissions to a reading center were analyzed, including scans from eyes that were reported to be normal and eyes with exudative age-related macular degeneration and with macular edema due to diabetic retinopathy or retinal vascular occlusion.

Results  The mean (SD) of the CP was 284 (150) μm and the center subfield (CC) was 301 (130) μm. The CP coefficient of repeatability (CR) was 49 μm and the CC CR was 27 μm. The CR increased by increasing retinal thickness for the CP and the CC within arbitrarily defined subgroups. For the 87 eyes with a session 1 CP of 175 μm or less, the CP CR was 17 μm and the CC CR was 10 μm.

Conclusions  Among experienced operators, given the same operator, machine, and eye at the same sitting, OCT3 retinal thickness maps appear to have a CR that is likely to be less than the clinically important difference.

Optical coherence tomography (OCT) was introduced commercially as a new retinal imaging modality in 1995. It has quickly become a standard tool for clinical diagnostic and research purposes for glaucoma and macular diseases. The 10-μm axial resolution of third-generation OCT allows objective measurement of retinal thickness, assessment of retinal morphology, and detection of subretinal fluid. Clinicians and researchers have come to rely on the numeric measurements generated by the manufacturer's scanning software for detection and longitudinal assessment of macular thickening and for clinical research study eligibility and outcomes. Optical coherence tomography is also used for the evaluation of choroidal neovascularization due to age-related macular degeneration (AMD) and other causes. The study of retinal morphology and topography using OCT in exudative AMD is more commonly adjunctive to other assessments, such as fluorescein angiography. Laser light attenuation from OCT scanning deep to the retina renders choroidal features less demonstrable than retinal morphology.

The majority of the current literature reporting OCT measurements in normal and diseased maculas describes first- and second-generation machines. Measurement repeatability of the third-generation device is expected to be improved slightly over earlier versions because of greater rapidity of scan acquisition (which avoids movement artifact) and software enhancements.

We assessed measurement repeatability in the setting of a large series of operator certification submissions to a centralized reading center from multiple clinical sites. These certification submissions consisted of consecutive scans of the same patient at the same visit taken by the same technician using the same equipment. Operators were applying for certification to participate in clinical trials of exudative AMD or macular edema (ME) due to diabetic retinopathy or retinal vascular occlusion.

The clinical site equipment was the commercially available third-generation Stratus OCT3 (Carl Zeiss Meditec, Inc, Dublin, California). The scan protocol included the Fast Macular Thickness Map algorithm, which features 6 scan lines centered at the fovea equally spaced 30° apart with an acquisition time of 0.32 s/line and scan density (A-scans per scan) of 128. The analysis of this scan yields the Fast Macular Thickness Map, consisting of 9 sectoral thickness values in 3 concentric circles with diameters of 1, 3, and 6 mm. The sectoral measurements reported in the map are the average of the 6 linear scans for that sector. The center point (fovea minimum parameter) is the mean of the 6 measurements from the intersection of the scan lines and is reported with its standard deviation.

These certification submissions consisted of replicate scans of the same eye of the same patient at the same visit taken by the same technician using the same equipment. The scans may or may not have been consecutive, since the operator may have discarded suboptimal maps. The submissions included prints of the 6 underlying radial scans from the fast macular algorithm and a seventh print with the retinal thickness map measurements.

Each scan was evaluated by staff at a central reading center for accuracy of boundary delineation by the instrument and for completeness of the submission. Eyes were excluded from analysis if (1) there were boundary line artifacts in more than 1 of the 6 radial scans affecting the outer subfields only, (2) there was a boundary line artifact affecting the center subfield in any scan, or (3) more than 2 scan prints were missing from the submission and there was artifact suggested on the retinal map print.¹

Two “normal” eyes and 2 eyes with the disease under investigation were submitted for certification. The selection of patients was at the discretion of the technician. No other patient data were submitted (eg, history, examination, or visual acuity). The value of the 9 subfield regions and the mean (SD) of the center point along with the operator's designation of normal or disease (options were AMD or ME associated with diabetic retinopathy or retinal vascular occlusion) were entered into an Access database (Microsoft, Redmond, Washington).

Because we have the same technician, the same machine, the same eye, and the same visit, we are assessing repeatability, not reproducibility (the closeness of agreement between independent results obtained with the same method on identical test material but under different conditions [eg, different operators, different apparatus, different laboratories, and/or after different intervals]).^2,3 The coefficient of repeatability is 1.96 times the standard deviation of the difference of session 2 and session 1 measurements, a clinically important number because it tells the reader, based on this subset of submissions, how different 2 measurements must be before one can conclude that, with 95% probability, a change has occurred by natural history or by some intervention. Given the exploratory nature and multiple comparisons, only test statistics with P < .001 were considered statistically significant. Statistical analyses were performed using R² and SAS (SAS Institute Inc, Cary, North Carolina).

From May 2003 through November 2004, there were 314 OCT3 submissions for operator certification. Of these, 281 scans (139 normal eyes and 142 diseased eyes [66 AMD, 76 ME]) from 134 operators met the aforementioned accuracy and completeness criteria.

Subgroups defined by the center point (CP) at session 1 were created arbitrarily to facilitate data presentation and evaluation (Table 1). All of the scans with a CP of 325 μm or more were from diseased eyes; 95% of the scans with a CP of 175 μm or less were from normal eyes.

For the 139 normal scans, the mean (SD) CP was 173 (8) μm; for the 66 AMD scans, 383 (18) μm; and for the 76 ME scans, 399 (22) μm. Instructions for the Stratus OCT3 retinal thickness tabular output note the fovea minimum parameter with a normal range of 135 to 215 μm; 11 of the normal eyes exceeded this range and 11 of the diseased eyes (6 AMD, 5 ME) were within this range.

For each of the 281 eyes, the session 1 CP vs session 2 CP was plotted (Figure 1A). As summarized in Table 2, the CP coefficient of repeatability was 49 μm and the mean difference between session 1 and session 2 CP was 2 μm with a 95% confidence interval of −1 to 5 μm. There was no difference in the CP coefficient of repeatability between eyes with AMD and ME (data not shown). The CP coefficient of repeatability increased as the mean CP of the subgroup defined by the CP at session 1 increased. The average of session 1 CP and session 2 CP was plotted against the difference of session 1 minus session 2 (Figure 1B), and the CP coefficient of reliability for all 281 eyes and by subgroup are shown. The CP coefficient of reliability (17 μm) for the subgroup with a CP of 175 μm or less was different from the subgroup with a CP of 176 to 225 μm (28 μm) and that was different from the subgroup with a CP of 226 to 325 (57 μm) (P < .001).

For each of the 281 eyes, the session 1 CP vs the session 2 CP was plotted with a line segment whose length is 2 SDs and the midpoint is the CP (Figure 2). As extreme examples, in Figure 2, there is an eye with a session 1 CP of 441 μm (SD = 10 μm) and a session 2 CP of 286 μm (SD = 115 μm) and there is another eye with a session 1 CP of 571 μm (SD = 20 μm) and a session 2 CP of 428 μm (SD = 167 μm). These 2 eyes and the 43 other eyes whose CP at session 1 was 426 μm or more had a CP coefficient of reliability of 78 μm and a mean difference of 12 μm with a 95% confidence interval of 0 to 24 including zero (Table 2). Two subgroups were arbitrarily defined by the relationship of the standard deviation to the CP, 1 subgroup with more than 5% and the other, 5% or less (eg, for a CP of 150 μm, an SD of 7.5 μm or less). In the eyes with a CP of 426 μm or more, 80% (36 of 45 eyes) had a standard deviation of 5% or less of the CP. Comparison of these 2 subgroups did not show any difference in the CP reliability coefficient (47 μm vs 53 μm) (Table 2). For the eyes with a CP of 175 μm or less, the subset of 62 eyes with a standard deviation of 5% or less of the CP differed in the CP reliability coefficient from the subset of 25 eyes with a standard deviation of more than 5% of the CP (CP reliability coefficients of 12 μm and 26 μm [P < .001]).

The CP is the measurement at the center of the fovea where radial scan lines intersect, in contrast to the center subfield, which is an average of the 128 sampled points encompassing a wider area rather than a single point. The center subfield coefficient of repeatability was 27 μm and the center subfield mean difference between session 1 and session 2 was 2 μm with a 95% confidence interval of −1 to 2 μm (Table 3). There was no difference in the center subfield coefficient of repeatability between eyes with AMD and ME (data not shown). The center subfield coefficient of repeatability for the subgroup with a CP of 175 μm or less (10 μm) was different from the subgroup with a CP of 176 to 225 μm (18 μm) (P < .001). For the subgroup with a CP of 175 μm or less, the subset of 62 eyes with a standard deviation of 5% or less of the CP did not differ in the center subfield repeatability coefficient from the subset of 25 eyes with a standard deviation of more than 5% of the CP (center subfield coefficients of reliability of 8 μm and 13 μm).

For each of the 281 eyes, the session 1 CP and center subfield were plotted (Figure 3). These measurements were not independent. The estimate of how much larger the center subfield was than the CP was 57 μm for all 281 eyes and 75 μm for the subset with a CP of 325 μmor less. The solid line is the CP plus a constant (arbitrarily chosen as 25 μm), included as a common frame of reference in Figure 3, and indicates that as the CP increases, the estimated line for the center subfield differs (largest visual discrepancy in the upper right portion of Figure 3).

For each of the retinal thickness subgroups, the macular map sectoral thickness diagram summarizes the session 1 mean (SD) CP, followed by the coefficient of repeatability and the 95% confidence interval on mean difference (session 1 CP minus session 2 CP) (Figures 4, 5, 6, 7, and 8).

The numeric measurements generated by OCT scanning software are used by clinicians and researchers for detection and longitudinal assessment of macular thickening and for clinical research study eligibility and outcomes. However, the peer-reviewed literature provides relatively few data on the repeatability of such measurements. In contrast to fundus photography in the 1980s and 1990s, in recent years OCT has been in a more dynamic environment of evolving equipment and software.

Published reports of OCT3 reproducibility are limited by small samples and variable methods. In 1 study, 10 normal subjects had 1 eye imaged 6 times per day on 3 different occasions.⁴ High reproducibility with the Fast Macular Thickness Map was reported, with an intraclass correlation of 88% and intervisit/intravisit standard deviations of less than 4 μm at the center subfield. In another study using OCT3, 10 normal subjects had 1 eye imaged twice by different operators on 3 separate occasions.⁵ Analysis of variance demonstrated no difference in macular measurements. In 22 eyes with diabetic ME, repeated scanning with OCT3 yielded a reproducibility coefficient of 37 μm or less for the center subfield, which was considered good.⁶ Studies of reproducibility of macular scanning with earlier-generation OCT machines also are limited by small samples and variable methods, but with generally similar results.^7-9 Interoperator variability has been noted to be low in 2 studies of 10⁹ and 25 patients,¹⁰ whereas 1 study of 20 eyes suggested systematic difference based on macular thickness.¹¹ In a study using a single OCT3 machine and a single operator, Polito et al¹² report 9 μm as the coefficient of repeatability for the CP in healthy retinas (n = 10; mean CP, 180 μm) and 20 μm, in eyes with clinical diabetic ME (n = 15; mean CP, 394 μm). Given that these results are from not 1 but 134 different operators and not 1 but multiple clinical sites and that the patient groups were not directly comparable, it is not surprising that these coefficients of repeatability are larger (ie, 17 μm in the 87 eyes with a CP of 175 μm or less and 49 μm in all 281 eyes).

The results of the current study might be considered an ideal scenario to document repeatability of the OCT3 in a multicenter context; we examined replicate imaging studies, including diverse disease representation, a large number of trained and experienced OCT operators, and a situation whereby operators would be expected to submit only their highest-quality efforts (since these submissions would be used to determine certification for performance of OCT in clinical trials).

The critical issue is specification of the clinically acceptable difference and whether it is expected to increase as the CP and/or center subfield increases. The coefficient of repeatability, based on this subset of submissions, provides an estimate of how different 2 measurements must be before one can conclude that, with 95% probability, a change has occurred by natural history or by some intervention.

Using arbitrarily defined subgroups based on session 1 CP, the CP coefficient of repeatability was at most 10% of the CP for both the subgroup with a CP of 175 μm or less (coefficient of repeatability, 17 μm) and the subgroup with a CP of 176 to 225 μm (28 μm) and then increased to approximately 20% of the CP for other subgroups (coefficient of repeatability, 57 μm, 61 μm, and 78 μm). For eyes in the subgroup with a CP of 175 μm or less that also had a standard deviation of 5% or less of the CP, the CP coefficient of repeatability was 12 μm.

Using the arbitrarily defined subgroups based on session 1 CP, the center subfield coefficient of repeatability was 10 μm for the subgroup with a CP of 175 μm or less and increased to 18 μm for the subgroup with a CP of 176 to 225 μm, 28 μm for the subgroup with a CP of 226 to 325 μm, 40 μm for the subgroup with a CP of 326 to 425 μm, and 38 μm for the subgroup with a CP of 426 μm or more. These coefficient of repeatability estimates seem plausible given the report of 22 eyes with diabetic ME that had a reproducibility coefficient of 37 μm or less for the center subfield, which was considered good.⁵

The standard deviation of the CP is calculated from the 6 CP measurements from each of the underlying scans of the retina map. Therefore, it is expected that when the underlying scans differ from one another enough to produce a relatively large standard deviation, this reflects variability throughout the study. Causes of such variability include eye movement due to fixation or motor instability, or variations in boundary line detection, which are more likely to occur with poor signal strength or with very thick retinas with abnormal anatomy.¹ Approximately two-thirds of the submissions had a standard deviation of 5% or less of the CP; this was 80% in the subgroup with a CP of 426 μm or more and testifies to the excellent technique of the operators and cooperation of the patients. As an internal metric in these data, a standard deviation of more than 5% of the CP was most useful in the subgroup of eyes with a CP of 175 μm or less; the CP coefficient of repeatability for those 25 eyes was 26 μm compared with 12 μm for the subset of 62 eyes with a standard deviation of 5% or less of the CP.

Based on these analyses in a multicenter context of the same eye at the same visit using the same equipment and the same operator, a single OCT3 scan without artifact obtained by an experienced technician seems to have a coefficient of repeatability that is likely to be less than the clinically important difference.

Correspondence: Ronald P. Danis, MD, Department of Ophthalmology and Visual Sciences, University of Wisconsin–Madison, 406 Science Dr, Ste 400, Madison, WI 53711 (rdanis@rc.ophth.wisc.edu).

Submitted for Publication: December 13, 2005; final revision received March 5, 2007; accepted March 18, 2007.

Financial Disclosure: None reported.

Additional Contributions: Tim Hess, MS, provided the R programming for Figures 4, 5, 6, 7, and 8. We appreciate the insightful comments from the reviewer.