Repeatability and Reproducibility of Superficial Macular Retinal Vessel Density Measurements Using Optical Coherence Tomography Angiography En Face Images

Key Points

Question  What is the repeatability and reproducibility of quantitative superficial macular retinal vessel density data from optical coherence tomographic angiography?

Findings  In this cross-sectional study of 21 healthy volunteers and 22 patients with retinal disease, the coefficient of variation was less than 6% among consecutive scans and among different devices. The agreement was highest for the 3 × 3-mm scan pattern for the inner ring and 6 × 6-mm scan pattern for the outer ring, and signal strength and scan pattern had a significant influence on the measurements.

Meaning  The findings of this study might provide useful information for future clinical trials involving optical coherence tomographic angiography.

Importance  The repeatability and reproducibility of quantitative metrics from optical coherence tomographic angiography (OCTA) must be assessed before these data can be confidently interpreted in clinical research and practice.

Objective  To evaluate the repeatability and reproducibility of OCTA-derived retinal vascular quantitative metrics.

Design, Setting and Participants  In this cross-sectional study, 21 healthy volunteers (42 eyes) and 22 patients with retinal disease (22 eyes), including 14 with age-related macular degeneration, 3 with epiretinal membrane, 2 with diabetic retinopathy, 2 with myopic macular degeneration, and 1 with retinal vein occlusion, were enrolled. Participants were recruited from September 1 through November 31, 2016. Each eye underwent 3 repeated scans with 3 instruments for a total of 9 acquisitions. Eyes were randomly assigned to scanning with a 3 × 3-mm or 6 × 6-mm pattern. Eyes were excluded from subsequent analysis if any acquisition had a signal strength of less than 7. Repeatability (defined as the agreement in measurements within a device) and reproducibility (defined as the agreement between devices of the same type) were assessed by intraclass correlation coefficient (ICC) and coefficient of variation.

Exposures  All eyes underwent scanning using 3 separate devices.

Main Outcomes and Measures  Vessel length density (VLD) and perfusion density (PD) of the superficial retinal vasculature.

Results  A total of 21 healthy volunteers (8 men and 13 women; mean [SD] age, 36 [6] years) and 22 patients with retinal disease (15 men and 7 women; mean [SD] age, 79 [9] years) underwent evaluation. Of these, 40 of 42 normal eyes and 15 of 22 eyes with retinal disease met signal strength criteria and were included in this analysis. The ICC among the 3 consecutive scans ranged from 0.82 to 0.98 for VLD and from 0.83 to 0.95 for PD. The coefficient of variation (CV) ranged from 2.2% to 5.9% for VLD and from 2.4% to 5.9% for PD. For reproducibility, the ICC ranged from 0.62 to 0.95 and the CV was less than 6% in all groups. The agreement was highest for the 3 × 3-mm pattern in the inner ring (ICC range, 0.92 [95% CI, 0.85-0.96] to 0.96 [95% CI, 0.93-0.98]) and 6 × 6-mm pattern in the outer ring (ICC range, 0.93 [95% CI, 0.86-0.97] to 0.96 [95% CI, 0.92-0.98]).

Conclusions and Relevance  Vessel length density and PD of the superficial retinal vasculature can be obtained from OCTA images with high levels of repeatability and reproducibility but can vary with scan pattern and location.

Optical coherence tomography angiography (OCTA) is a rapidly evolving technology that enables noninvasive visualization of the microvascular flow in the retina by comparing the signal in consecutive B-scans at the same location to reveal the motion of erythrocytes. Because of its high-axial resolution, OCTA enables visualization of fine retinal vasculature in multiple layers that is difficult to achieve with conventional dye-based angiography.^1,2 Optical coherence tomography angiography further provides an opportunity to generate quantitative metrics to describe the status of the retinal vasculature. Optical coherence tomography angiography–derived metrics described in previous reports include vessel density or perfusion density (PD),³ blood flow index,³ skeletonized density or vessel length density (VLD),⁴ and fractal dimension.⁵

Automatic measurement tools have also been developed to generate many of these metrics.^5-7 Before one can confidently use these metrics in clinical trials or clinical practice, establishing repeatability and reproducibility of these measurements⁸ first is critical to judge whether differences between individuals or changes in these measurements are of significance. Repeatability refers to agreement of measurements from different sessions of scans using the same device, same eye, and same operator within a short period. Repeatability establishes the values for a margin in a measurement over time. Reproducibility, however, refers to agreement among different devices and operators with the same eye within a short period. Reproducibility has particular implications for comparisons of data obtained at different sites, for example, in the context of multicenter clinical trials.

In the present study, we determined the repeatability and reproducibility of an investigational measurement tool in measuring VLD and PD on OCTA en face images of superficial capillary plexus (SCP). We selected these specific metrics because they are provided by many OCTA commercial devices and may be of relevance to retinal vascular diseases.⁹

In this cross-sectional study at the Doheny UCLA Eye Centers, we recruited 2 cohorts from September 1 to November 31, 2016. One cohort consisted of 42 eyes from 21 healthy volunteers (11 white, 9 Asian, and 1 African) with no systemic or ocular disease, as confirmed by history and ophthalmic examination results. The second cohort consisted of 22 eyes from 22 patients (17 white, 2 Asian, and 1 African) with various retinal diseases, including 8 eyes with nonneovascular age-related macular degeneration, 6 eyes with neovascular age-related macular degeneration, 3 eyes with epiretinal membrane, 2 eyes with nonproliferative diabetic retinopathy, 2 eyes with myopic macular degeneration, and 1 eye with central retinal vein occlusion. The only requirement for the cohort with disease was that they have media of sufficient clarity to obtain OCT imaging. Written informed consent was obtained from all individuals before study participation. The study was approved by the institutional review board of UCLA (University of California, Los Angeles) and conducted in accordance with the ethical standards stated in the Declaration of Helsinki¹⁰ and in compliance with the regulations of the Health Insurance Portability and Accountability Act.

Three same-model OCTA devices (AngioPlex software, version 9.5, and Cirrus HD-OCT model 5000; Carl Zeiss Meditec, Inc) were used to acquire images for this study. The enrolled eyes underwent imaging with all 3 devices with 3 consecutive scans on each device in a randomly arranged sequence, with a total of 9 acquisitions for each eye. A different operator (3 of us: J.L., S.B., and E.B.) was assigned to each OCT instrument. Artificial tears were not given between scans. Two OCTA scan patterns (3 × 3-mm and 6 × 6-mm) were evaluated in this study. For the healthy cohort, 1 eye (randomly selected) was evaluated with one scan pattern and the other eye with the second scan pattern. For the cohort with retinal disease, only 1 eye was included as the study eye and was randomly assigned to 1 scan pattern. The operators were specifically trained on the study protocol and were required to check the en face OCTA images after each acquisition to ensure good focus, minimal saccades (identified by horizontal misalignment of vessel segments on en face images), and good centration. For an eye to be included in the final analysis, a signal strength (SS) of at least 7 (maximal SS for the AngioPlex software is a value of 10) had to be achieved across all 9 scans.

All scans were analyzed using en face OCTA images generated automatically by the optical microangiography algorithm used in the Cirrus OCTA software (AngioPlex software, version 10.0; clearance by the US Food and Drug Administration pending). Vessel length density (defined as the total length of perfused vasculature per unit area in a region of measurement) and PD (defined as the total area of perfused vasculature per unit area in a region of measurement) were measured automatically by the software that quantified vessel density of a local region of tissue according to Early Treatment of Diabetic Retinopathy Study subfields (Figure). At present, the software only reports values for the SCP, which is defined by the instrument to span from the internal limiting membrane to an estimated boundary of the inner plexiform layer. This inner plexiform layer boundary is calculated as 70% of the distance from the internal limiting membrane to an estimated boundary of the outer plexiform layer, which is determined as being 110 μm above the retinal pigment epithelium boundary as automatically detected by the software. In the present study, we evaluated the inner ring (mean value of subfields 2 through 5) on both scan patterns and the outer ring (mean value of subfields 6 through 9) on the 6 × 6-mm scan pattern (Figure).

All analyses were performed using SPSS software (version 23.0; IBM Corporation). P < .05 indicates statistical significance.

Intraclass correlation coefficient (ICC) and coefficient of variation (CV) for VLD and PD of the inner ring on both scan patterns and the outer ring on the 6-mm pattern were calculated among the 3 consecutive acquisitions for each device. For evaluation of the inner ring, we separated the healthy from the cohorts with retinal disease and the 3-mm from the 6-mm scan patterns. Independent-sample t test (comparison between 3- and 6-mm patterns) and paired-sample t test (comparison between the inner and outer ring of the 6-mm pattern) were performed for comparison of CV.

To test the agreement among the 3 devices, we first calculated the mean values of the 3 consecutive scans for each device. We then calculated the ICC and CV for mean VLD and PD among the 3 devices. This analysis was performed for the same cohorts and locations as for the repeatability analysis.

The 3 scans within each device were numbered from 1 to 3 according to the sequence of their acquisition time. A general linear model for repeated-measurements analysis described by Cnaan et al¹¹ was applied for this data set to test the effect of several candidate factors. Different devices and scans were considered to be within-eye factors, whereas different groups (healthy or retinal disease cohorts) and scan pattern (6 × 6- or 3 × 3-mm) were regarded as between-eye factors. The effects of those factors on SS were also evaluated using a general linear model.

For each eye, the acquired scans were given a numerical label from 1 to 9. The first scan in the sequence from 1 to 9 with the maximum SS and the first scan in the sequence from 9 to 1 with the minimum SS were chosen for comparison. The difference in SS between these 2 extremes was defined as the ΔSS (effectively the range of SS for that eye). The VLDs and PDs corresponding to these maximum and minimum SS scans were compared and the differences in VLD and PD were termed ΔVLD and ΔPD. Pearson correlation and linear regression were used to test the association between ΔSS and ΔVLD or ΔPD.

A total of 21 healthy volunteers (8 men and 13 women; mean [SD] age, 36 [6] years) and 22 patients with retinal disease (15 men and 7 women; mean [SD] age, 79 [9] years) underwent OCTA for this study. Among the 576 OCT scan acquisitions obtained, 13 (2.3%) from 9 eyes (2 normal eyes and 7 with retinal disease) had an SS of less than 7. After excluding these cases, the remaining 55 eyes (40 normal eyes from 20 healthy volunteers and 15 eyes with retinal disease from 15 patients) were included in the analysis.

The ICC and CV of VLD and PD among the 3 scans within each device are shown in Table 1. The CV ranged from 2.2% to 5.9% for VLD and from 2.4% to 5.9% for PD in all groups, with ICC ranging from 0.82 to 0.98 for VLD and from 0.83 to 0.95 for PD. Compared with the 3-mm inner ring, the ICC of the 6-mm inner ring was lower (without overlapping of the 95% CI) on device 2. The mean (SD) CV for VLD for the 6-mm outer ring was lower than that of the 6-mm inner ring on devices 1 (3.3 [2.7] vs 2.2 [1.6]; P = .009) and 3 (4.1 [4.7] vs 31. [3.3]; P = .04). The mean (SD) CV for PD was also lower in the outer ring on device 1 (2.4 [1.7] vs 3.5 [3.7]; P = .008). The mean (SD) CV of the healthy cohort was lower than that of the cohort with retinal disease for VLD on devices 1 (2.6 [1.9] vs 4.9 [3.0]; P = .02) and 2 (2.9 [3.6] vs 5.8 [4.8]; P = .02) and PD on device 1 (2.8 [1.8] vs 4.9 [3.1]; P = .03). The ICC for VLD was higher in the healthy cohort.

The ICC and CV of VLD and PD among the 3 different machines are shown in Table 2. The CV was less than 6% in all the groups, and the ICC ranged from 0.62 (95% CI, 0.29-0.82) to 0.95 (95% CI, 0.91-0.98). The ICC among different machines was lowest for the 6-mm inner ring (0.70 [95% CI, 0.43-0.85] for VLD and 0.62 [95% CI, 0.29-0.82] for PD).

We compared the mean VLD and PD values within the Early Treatment of Diabetic Retinopathy Study inner ring for the SCP in different subgroups. The estimated marginal means (SEs) for VLD (22.6 [0.2] vs 18.2 [0.4] mm⁻¹ for healthy cohort and 18.2 [0.4] vs 16.7 [0.3] mm⁻¹ for retinal disease group) and the PD was lower (0.403 [0.004] vs 0.437 [0.005] for healthy cohort and vs 0.343 [0.008] vs 0.405 [0.007] for retinal disease cohort; P < .001) was higher for the 3-mm scan pattern compared with the 6-mm scan pattern. The estimated mean SS was also higher in the healthy group (9.8) than in the retinal disease group (9.2; P < .001). The estimated mean SS for the 3-mm and 6-mm scan types were 9.5 and 9.6, respectively (P = .61).

The effects of certain factors and their interactions on the VLD and PD measurements are shown in Table 3. The scan order (ie, whether the scan was the first, second, or third obtained on a device) was observed to have an effect on PD (F = 4.53; P = .01). Thus, a further pairwise comparison was performed that revealed that the PD of the first scan was larger than that of the second (mean difference, 0.005; P = .03) and third (mean difference, 0.006; P = .005) scans. The order of scan acquisition had no effect on VLD (F = 2.04; P = .15).

We found a positive correlation between ΔSS and ΔVLD (r = 0.759; P < .001) and between ΔSS and ΔPD (r = 0.698; P < .001). Linear regression showed that the regression coefficient (B) for ΔVLD was 1.44 (95% CI, 1.10-1.78; P < .001) and the B for ΔPD was 0.03 (95% CI, 0.02-0.04; P < .001).

In this study, good repeatability and reproducibility were observed for VLD and PD of the SCP obtained using the Cirrus AngioPlex software, version 10.0, for image analysis. Because of the interest in using OCTA-derived quantitative metrics in clinical trials and clinical practice, establishing the margins of repeatability and reproducibility is important. Reproducibility (as defined in this study) is of particular importance for multicenter clinical trials because measurements obtained at multiple sites with different operators and devices will be evaluated.

Repeatability has previously been reported for other OCT devices. For the RTVue device (version 2014.2.0.65; Optovue Inc) and its included measurement software, the intravisit CV for PD was observed to range from 2.1% to 6.8% when measuring vessel density in different quadrants, with ICCs ranging from 0.3 to 0.8.¹² Similarly, the investigators also found that repeatability was higher in the 3-mm scan mode. In another study of healthy participants using the RTVue device and an automated measurement tool (AngioAnalytics software; Optovue),⁶ interoperator (only 1 device was used) agreement was 0.78 to 0.99 and intraoperator agreement was 0.64 to 0.93 for the SCP. Another study¹³ also previously evaluated the repeatability of automated PD measurements for the RS-3000 Advance OCTA device (Nidek). The CV for the superficial retinal layer was 5.2% with an ICC of 0.90. The investigators did not, however, assess reproducibility (between instruments and operators) in that prior study.¹³ In addition, in the present study, we observed that agreement was better for the outer ring compared with the inner ring with the 6-mm scan pattern. We suspect that precision of macular centration may have a greater influence on the inner ring when using the 6-mm scan pattern.

Despite the relatively high level of repeatability and reproducibility in our study, we found several factors that appeared to affect VLD and PD measurements. For example, the VLD and PD values in the cohort with retinal disease were lower compared with those in the healthy cohort. Because some eyes in the cohort with retinal disease had vascular disease, areas of nonperfusion in those eyes may have contributed to lower VLD and PD in this cohort. The cohort with retinal disease, however, was older than the healthy cohort; this variation also may have contributed to the difference.^6,14 Despite the lower VLD and PD in the disease cohort, the repeatability and reproducibility within this cohort remained good, with a CV of less than 6%. Because relatively few eyes had each specific disease, however, we were not able to correlate level of reproducibility with type of disease.

We also found that scan pattern had an influence on the outcome measurements, although the direction of the difference varied for VLD vs PD. For VLD, the value was greater for the 3 × 3-mm scan pattern, but it was lower for PD. This same dichotomy was also observed in the previous study with the RTVue device.¹¹ The larger VLDs with the 3 × 3-mm scans may be attributable to the 3 × 3-mm scan having better lateral resolution and thus having the ability to distinguish or resolve more capillaries. The larger PD for the 6 × 6-mm scans may be attributable to the capillary size being overestimated because of the lower resolution. These differences highlight the critical need to use the same and consistent scan pattern if VLD and PD measurements are compared between participants or over time in the same participant.

Another factor that appeared to affect measurements in our study was the scan order or the sequence with which scans were obtained. This factor only had an influence on PD. The difference in PD occurred primarily between the first scan and the other two scans. We found no significant difference between the second and third scans. Artificial tears were not used between individual scan acquisitions on a particular device in our study. Tear film instability and some diminution of the quality of the ocular surface may have contributed to the lower values for the second and third scans. The estimated mean SS for the first scan (all participants) was slightly higher (9.6) than the SS for the second and third scans (9.5 and 9.5, respectively), but this difference in SS was not statistically significant. Despite this apparent difference in measurements with scan order, the magnitude of the difference in PD was small (0.005 to 0.006), equating to a difference of less than 2%. This finding could support the use of artificial tears between scans to maximize SS and reduce another source of variability.

Another finding from our study was the influence of SS on the quantitative measurements. Previous studies have also suggested that quantitative metrics could be affected by OCT SS.^15,16 Gao et al¹⁷ reported that compensating for reflectance variation would result in more reliable vessel density quantification on OCTA. For this reason, manufacturers might recommend a minimum (and high) SS for OCTA acquisitions. Despite including only high-quality (SS≥7) scans, we observed that a difference in SS between scans was positively correlated with a difference in VLD and PD measurements for the whole sample. In particular, we observed that for an SS of at least 7, if the SS increased by 1 unit, the VLD increased by 1.4 mm⁻¹ and the PD increased by 0.03. This finding highlights the need not only to optimize SS when acquiring OCTA scans but also to maintain a consistent SS over time. If the magnitude of this association can be replicated in future studies, an opportunity to adjust or correct OCTA measurements for these differences due to SS may exist.

Our study has several limitations that must be considered. The overall cohort sizes were relatively small, particularly for the cohort with retinal disease, in which several cases had to be excluded owing to failure to achieve manufacturer-recommended SS for all scans. In addition, because we included a variety of different diseases, with small numbers of cases of individual diseases, we were not able to establish whether the repeatability and reproducibility were affected by the type of disease. This topic may be of interest for future studies. Second, we focused on only the superficial retinal vasculature (SCP), primarily because this was the only automated measurement currently provided by the automated software. The SCP may be less likely to be affected by OCTA artifacts, such as projection artifact, which can confound OCTA assessments. A previous study¹³ evaluated the repeatability of automated deep capillary plexus measurements in normal eyes with another OCTA device, but reproducibility still needs to be defined. Finally, we did not evaluate the reliability of other OCTA-derived vessel metrics, such as the foveal avascular zone area and fractal dimension.

In this study with a sample of 40 normal eyes and 15 with retinal disease, we observed that automated OCTA measurements (VLD and PD) of the SCP could be obtained with a high level of repeatability and reproducibility using the Cirrus AngioPlex software (version 10.0) for 3 × 3-mm and 6 × 6-mm scan patterns in both populations. The repeatability might differ, however, based on different scan patterns and locations (inner vs outer ring). The measurements also differed between scan patterns and with differences in SS between scans, highlighting the need for consistency in acquisition procedures for OCTA.

Corresponding Author: Srinivas R. Sadda, MD, Doheny Image Reading Center, Doheny Eye Institute, 1350 San Pablo St, Room DVRC211, Los Angeles, CA 90033 (ssadda@doheny.org).

Accepted for Publication: July 27, 2017.

Published Online: September 14, 2017. doi:10.1001/jamaophthalmol.2017.3431

Author Contributions: Drs Lei and Sadda had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Durbin, Sadda.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Lei, Uji, Baghdasaryan.

Critical revision of the manuscript for important intellectual content: Durbin, Shi, Balasubramanian, Al-Sheikh, Sadda.

Statistical analysis: Lei, Shi, Balasubramanian, Al-Sheikh.

Obtained funding: Sadda.

Administrative, technical, or material support: Shi, Uji, Balasubramanian, Al-Sheikh, Sadda.

Study supervision: Balasubramanian, Sadda.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Durbin reports receiving funding from Carl Zeiss Meditec, Inc. Dr Sadda reports receiving funding from Carl Zeiss Meditec, Inc, Optos, Allergan, and Genentech and consulting for Optos, Allergan, Genentech, Alcon, Novartis, Roche, Regeneron, Bayer, Thrombogenics, Stemm Cells, Inc, and Avalanche. No other disclosures were reported.