Estimating Retinal Sensitivity Using Optical Coherence Tomography With Deep-Learning Algorithms in Macular Telangiectasia Type 2

IMPORTANCE As currently used, microperimetry is a burdensome clinical testing modality for testing retinal sensitivity requiring long testing times and trained technicians. OBJECTIVE To create a deep-learning network that could directly estimate function from structure de novo to provide an en face high-resolution map of estimated retinal sensitivity. DESIGN, SETTING, AND PARTICIPANTS A cross-sectional imaging study using data collected betweenJanuary1,2016,andNovember30,2017,fromtheNaturalHistoryObservationandRegistry of macular telangiectasia type 2 (MacTel) evaluated 38 participants with confirmed MacTel from 2 centers. MAIN OUTCOMES AND MEASURES Mean absolute error of estimated compared with observed retinal sensitivity. Observed retinal sensitivity was obtained with fundus-controlled perimetry (microperimetry). Estimates of retinal sensitivity were made with deep-learning models that learned onsuperpositionsofhigh-resolutionopticalcoherencetomography(OCT)scansandmicroperimetry results. Those predictions were used to create high-density en face sensitivity maps of the macula. Training, validation, and test sets were segregated at the patient level. RESULTS Atotalof2499microperimetrysensitivitiesweremappedonto1708OCTB-scansfrom63


Introduction
A major impediment to the development of novel interventions for rare macular diseases is the lack of sensitive biomarkers that can be used as surrogate clinical trial end points.Particularly when the disease progression is slow, surrogate markers are necessary to overcome the need for large sample sizes and long follow-up duration, which increases the cost and burden of clinical trials and decreases the rate at which new therapeutic interventions can be evaluated.Visual acuity remains a poor functional end point for many macular diseases because the central fovea may be unaffected until late stages of the disease.Paradigmatic for such a disease course is macular telangiectasia type 2 (MacTel), a neurodegenerative disease in which the parafoveal region becomes disproportionately affected compared with the central fovea. 1 Fundus-controlled perimetry (ie, microperimetry) has the potential to overcome some of these limitations by projecting a localized visual stimulus directly onto exact positions of the patient's retina.Microperimetry tests visual function outside the central fovea, and thus is better suited for monitoring progression of extrafoveal vision loss, such as in MacTel. 2,3Microperimetry can be challenging because of the test duration and the requirement of good fixation and experienced examiners.5][6] Most acquisition protocols are limited to fewer than 50 spots as a practical compromise between the tolerable test time and the depth and quality of the data.
Therefore, subtle losses of sensitivity between the testing areas may be missed.In a clinical trial, the data contained in those areas might make the difference between significant changes of a therapeutic intervention for a slowly progressive disease and a failed treatment trial.
Optical coherence tomography (OCT) is a widely available noninvasive imaging modality capable of providing dense volumetric imaging of the retinal microstructures.Since the advent of OCT, attempts have been made to identify structures on OCT scans that correlate with measured retinal sensitivities in microperimetry.In a conventional structure-function correlation approach, the retinal structure of interest is selected a priori by the researchers, measured, and correlated with the retinal sensitivity at this location. 7With this approach, a correlation of ellipsoid zone loss on OCT with microperimetry data has been found. 8,9The phase 2 clinical trial of ciliary neurotrophic factor for MacTel was unique for ocular disease in that the US Food and Drug Administration agreed that visual acuity would not be the primary outcome. 10Ellipsoid zone loss on OCT was the primary outcome, and functional markers, such as visual acuity, microperimetry, and reading speed, were secondary outcomes.Structures other than the ellipsoid zone may be equally or more suitable as a surrogate measure for retinal function and those structures may vary between diseases.In addition, highresolution maps of estimated perimetry may provide a functionally more sensitive surrogate end point measure.
Deep learning is a recent advance in artificial intelligence and computer vision in which many layers of convolutional neural networks are stacked to perform end-to-end data-driven learning.
2][13][14][15][16][17][18][19] We sought to create a deep-learning network that estimates function from structure de novo using the OCT and microperimetry data to provide an en face highresolution map of estimated retinal sensitivity.

Image Acquisition
The study population consisted of participants of the Natural History Observation and Registration study of MacTel.The study protocol has been published. 20Optical coherence tomographic scans were performed (Spectralis OCT; Heidelberg Engineering) with a volume scan of 97 B-scans. 7croperimetry, which is not part of the standard study protocol, was performed (MAIA; Centervue)

Image Processing
The microperimetry results were exported as raw files.Using Adobe Photoshop CS5 12.0 (Adobe Systems Inc), the MAIA infrared scanning laser ophthalmoscope images were superimposed with the Spectralis infrared scanning laser ophthalmoscope images from the volume scan.Retinal vessels were used as landmarks.The image size was changed with constrained proportions and the image was rotated until the vessels were superimposed as closely as possible.No other image distortions were performed.
Spectralis OCT B-scans were selected to intersect the center of each MAIA stimulus.The stimuli intersected by the B-scan were superimposed onto the B-scan aligning their horizontal location on stimuli.The scans above and below this reference scan were also saved, without a MAIA stimulus.
As described previously, 11 we created vertical slices of 32 × 496-pixel windows around each perimetry point and used the slice above and below the registered B-scan as additional examples.No other image preprocessing was performed before input into the deep-learning model.No overlapping vertical OCT windows were created to establish the validation and test sets.

Deep-Learning Approach
Our model is implemented as a deep convolutional neural network with a regression output.To perform regression, a localized vertical slice of the OCT was used as input into the neural network, and the output from the model was set to the microperimetry-measured retinal sensitivity at that location as a continuous variable.By setting the anatomic structure as an input and an objective functional measurement as output, we hoped to create a neural network that could estimate function from structure directly de novo.At inference time, the trained network would provide an en face projection of a high-resolution map of estimated retinal sensitivity.
The network architecture is schematically shown in Figure 1A.We used a modified version of the Visual Geometry Group (VGG) 16 convolutional neural network. 22The input image is passed through a stack of 3 VGG blocks followed by 2 fully connected layers.The VGG blocks consists of 128, 256, and 256 convolution layers.For better generalization, dropout units were applied to every convolution layer except the last one, with a ratio of 0.1, and each of the fully connected layers with a ratio of 0.5.The total model size was 49.9 million variables.
Weights were initialized using the Xavier algorithm. 23The model was trained with mean squared error loss function using Adam optimizer.We chose a batch size of 64.The learning rate was initially set to 1 × 10 −5 , and decay over each update was set to initial learning ratio divided by epochs.
All inputs were normalized to a range between 0 and 1, and outputs were normalized to −1 to 1.We assessed the performance of neural network using cross validation with the validation set and evaluated the generalizability with an independent data set.Samples were divided into training sets at 60%, validation sets at 20%, and test sets at 20%.The training, validation, and test sets contained images from mutually exclusive groups of patients (ie, no single patient contributed images to >1 set).
To reduce overfitting on image data, we used data augmentation that consisted of generating image translations, rotations, and horizontal reflections.For translations and rotations, we first identified an approximate position of membrane using Canny edge detector 24 together with median filter, and vertically shifted and/or rotated the image within an acceptable range.To verify how our deep-learning algorithm predicted the sensitivity, we used Activation Map 25 to find spatial support regions for the prediction.Keras 26 and Python 27 were used for deep learning.All training occurred using the NVIDIA Pascal Titan X Graphics processing unit with NVIDIA cuda, version 8.0, and cu-dnn, version 5.5.1, libraries. 28

Statistical Analysis
All statistical analyses were performed using R, version 3.5.1.The statistical significance level was .05,and 2-tailed paired, nonparametric testing was performed.

Estimation of Retinal Sensitivity
The estimated sensitivity values were used to create high-resolution en face OCT projection maps of estimated retinal sensitivity (Figure 3).After manual registration of the microperimetry to the OCT volume (Figure 3A), the model created an estimate for each vertical line of a B-scan resulting in dense estimation lines for each B scan (Figure 3B).In the next step, those lines were stitched together to en face OCT projection maps (Figure 3C).Our model successfully delineated regions of normal against reduced retinal sensitivities.Figure 3D shows examples of activation map outputs, giving visual explanations from our regression model for which regions on the B scans were important for estimates.
We explored the possibility of applying our method to analyze progression of vision loss in patients with MacTel.For this purpose, we processed OCT volume scans taken approximately 12

JAMA Network Open | Ophthalmology
Estimating  4).The areas of low retinal sensitivities correlated with each other in location and shape and showed a growth over time that was in keeping with previous studies of natural scotoma progression. 2,8 compared our model with linear regression and LeNet as baseline models.We trained these models using the same data set and optimization method that we used for our model.Mean absolute error results of these models were 4.51 dB (95% CI, 4.36-4.65 dB) when using linear regression and 3.66 dB (95% CI, 3.53-3.78dB) when using LeNet.Using a 49.9 million-variable deep-learning model, a mean absolute error of 3.36 dB (95% CI, 3.25-3.48dB) of retinal sensitivity for validation and test was achieved.Correlation showed a high degree of agreement (Pearson correlation r = 0.78).By paired Wilcoxon rank sum test, our model significantly outperformed these 2 baseline models (P < .001)(Figure 5).

Discussion
One of the hallmarks of MacTel is the structural and functional restriction of disease to a specific macular area. 1,30Functional loss arises as a sharply delineated scotoma that progresses over time until it reaches a maximum within the MacTel area. 2,31This spatial and functional limitation is not only ideal for microperimetry, but also proved to be adaptable to our model, as both diseased and healthy retinal areas are found in the same OCT B-scans, thus offering internal controls.The available sample had a strong left skew with fewer samples of low than high sensitivity scores, affecting the ability of the model to estimate lower sensitivities.Therefore, the variance of deviation between estimated and tested sensitivity was higher for lower tested sensitivities, which might be the reason the estimated areas of functional loss look homogeneous (Figure 3C and Figure 4).However, this deviation was small enough to allow for correct identification of the sharp borders of functional loss and reliably discern healthy from diseased retina (Figure 3B and C).
Microperimetry can be demanding for both patients and examiners.It requires not only time (a correctly performed examination can take 30 minutes for 2 eyes) and intense concentration, but The en face estimations for our model.J, Mean absolute error (95% CIs).Our model achieved significantly better results over these baseline models (P < .001,paired Wilcoxon rank sum test).
also skilled and motivated examiners and patients, increasing the costs of this method and limiting its use to special cases or questions and selected patient groups.By contrast, an OCT scan requires much less time and minimal patient cooperation.High-quality scans can be obtained in children, adults with disability, and a wide variety of ocular conditions, even with poor visual acuity or poor fixation on a test target.Our approach has the potential of creating dense en face maps of estimated retinal sensitivity, as the OCT scan can cover the retina densely (10-μm interscan distances and widefield scans are commercially available).This high-resolution estimated perimetry map overcomes the natural limitation of every perimetry examination, which can sample only limited areas of the retina; the spatial resolution of the test points, which are limited owing to imprecision of eye tracking and stimulus projections; and the inherent trade-off of an increase in examination duration with lessened patient cooperation for each added test location.These high-resolution maps make the model potentially useful for measuring disease progression (Figure 4); any structural change on OCT scans will be readily translated into functional change by the model.
Several cross-sectional and longitudinal studies have shown a good correlation between retinal function and structure in MacTel, as seen on OCT. 8,9,32,33Because those studies only examined the ellipsoid zone (EZ), there might be a potential bias at the expense of other retinal structures that might be functionally more relevant than the EZ, for example, the outer nuclear layer.The model had no human input other than the retinal sensitivity at each microperimetry location, and was therefore unbiased as to which structure to look at on the OCT.The most important structures identified on the activation maps for retinal function were identified within the outer retina in MacTel.The model was able to positively estimate healthy retinal function based on the presence of intact outer retinal layers, in particular, the EZ.
The activation maps provided a close approximation of the en face zones of EZ loss derived from the OCT scans (Figure 3C).Loss of EZ has been recently approved by the US Food and Drug Administration as a primary outcome measure in investigational trials in MacTel (ClinicalTrials.govidentifiers NCT01949324, NCT03316300, and NCT03319849), justified by its correlation to retinal sensitivity measured with microperimetry.Therefore, our model could also be validated and then applied for measurements of surrogate measures in clinical trials of MacTel, as well as another way of monitoring disease progression, because it also takes into account other functionally relevant retinal structures.
Deepening the feature map in the first block did not improve our model performance.Highlevel structural information captured at a higher layer (second and third blocks of the network) is likely more important for the network to estimate retinal sensitivity than local feature, such as corners and edges captured at the lower layer (first block of the network).
Will this method obviate the need for microperimetry?Such a reliable and validated psychophysical method should not be abolished: retinal function is more meaningful to patients and clinicians than the subtle changes on their retinal scans.Trials are criticized for the use of surrogate end points as markers of therapeutic success; better end points are available.However, many clinics do not have a microperimetry device, whereas OCTs are available almost everywhere.Many patients are unable to perform demanding microperimetry scanning (children, elderly, or disabled patients) and are therefore excluded from clinical trial protocols.Some retinal diseases with wide areas of macular dysfunction are excluded from treatment protocols owing to the lack of a meaningful functional outcome.This model could broaden the inclusion criteria for some of these trials.We chose to focus on MacTel in this proof-of-principle study as a paradigmatic macular disease adaptable to the application of microperimetry owing to the focal and limited nature of its functional loss and anatomic changes.

Limitations
Although the findings offer the potential to train the model using other more common diseases, such as age-related macular degeneration and diabetic retinopathy, our proof-of-principle study has limitations.This study was based on 1 disease and the findings might not necessarily generalize to all MacTel has created fully automated and reliable approaches for binary segmentation of the EZ band on OCT B-scans. 19Our approach differs in that we used a functional measurement as the objective training target without selecting an anatomic feature beforehand.
Future studies could include using dense raster OCT volumes and 3-dimensional convolutions to further increase the model's accuracy.In addition, with higher-resolution microperimetry testing, U-net style architectures may be used to build deep-learning models for mapping structural OCT to retinal sensitivities.

Conclusions
We have developed high-resolution en face standard maps of estimated retinal sensitivities with direct relevance to standard microperimetry techniques in eyes with MacTel.The estimates were reliable and fast, and produced similar information compared with microperimetry, correctly delineating functionally healthy and impaired retina.In addition, retinal sensitivity in areas between the test points could be implied from the model.As a proof of principle, we plan to apply this model to other diseases.We believe the model may be useful to monitor structural and functional disease progression and has potential as a surrogate outcome measure in investigational treatment trials.

Figure 1 . 1 Dropout
Figure 1.Deep-Learning Model Structure and Learning Curves

Figure 2 .
Figure 2. Comparison of Observed vs Estimated Retinal Sensitivity

Figure 3 .
Figure 3. Microperimetry Results and Predicted High-Resolution Sensitivity Maps

Figure 4 .
Figure 4.Estimated Progression of Retinal Sensitivity Loss From Longitudinal Optical Coherence Tomographic Scans in 3 Patients With a Follow-up Time of 12 Months

Figure 5 .J
Figure 5.Comparison With Baseline Estimation Models Linear model A

Downloaded From: https://jamanetwork.com/ on 09/29/2023
29pth of feature maps for each layer was set to 128 in the first block and 256 at the second and third blocks.Lowering the depth of feature maps as 64, 128, and 256 in VGG caused the network to perform worse at evaluation time, and additional depth beyond 256 did not yield further improvements.Making the feature map in the first block deeper did not improve the model performance.Shallower networks, such as LeNet,29caused underfitting of data, and deeper networks, such as VGG16, resulted in overfitting owing to a limited amount of available training images.After 80 510 iterations per 50 epochs of training, the validation loss stabilized (Figure 1B).The training loss was larger than the validation loss in the first 4 or 5 epochs because the training set included augmented images, which were not included in the validation set.We achieved a mean absolute error of 1.70 dB for training, 3.02 dB for validation, and 3.83 dB for the test set.The model achieved 3.36 dB (95% CI, 3.25-3.48dB) for the validation and test set.The mean difference Retinal Sensitivity From Optical Coherence Tomography With Deep Learning JAMA Network Open.2019;2(2):e188029.doi:10.1001/jamanetworkopen.2018.8029(Reprinted) February 8, 2019 5/12 Downloaded From: https://jamanetwork.com/ on 09/29/2023 months apart (Figure