[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.161.175.236. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Download PDF
Figure 1.
Sensitivity and Specificity of Remote Nonexpert Diagnosis of Cytomegalovirus (CMV) Retinitis on Mosaic Fundus Photographs
Sensitivity and Specificity of Remote Nonexpert Diagnosis of Cytomegalovirus (CMV) Retinitis on Mosaic Fundus Photographs

Estimates are shown for each of 15 nonexperts and 3 CMV retinitis experts as well as the mean of the nonexperts. Graders had clinical experience (clinical), research experience (research), or neither clinical nor research experience (neither).

Figure 2.
Positive Predictive Values (PPVs) for Cytomegalovirus Retinitis Present (+CMV Retinitis) and Unknown, and Negative Predictive Values (NPVs) for CMV Retinitis Absent (−CMV Retinitis)
Positive Predictive Values (PPVs) for Cytomegalovirus Retinitis Present (+CMV Retinitis) and Unknown, and Negative Predictive Values (NPVs) for CMV Retinitis Absent (−CMV Retinitis)

Estimates are shown over a range of likely CMV retinitis prevalence values for each of the 15 nonexpert graders (solid thin lines), the mean of nonexpert graders (solid thick lines), and the mean of the expert graders (dashed lines).

Figure 3.
Intrarater Reliability for Nonexpert Telemedicine Diagnosis of Cytomegalovirus Retinitis
Intrarater Reliability for Nonexpert Telemedicine Diagnosis of Cytomegalovirus Retinitis

The Cohen κ value is shown for each nonexpert grader based on grades from 50 randomly selected duplicate images; graders had clinical experience (clinical), research experience (research), or neither clinical nor research experience (neither). Limit lines indicate 95% CI.

Figure 4.
Sensitivity and Specificity of Remote Nonexpert Grading to Diagnose Cytomegalovirus Retinitis Using Expert Consensus as the Reference Standard
Sensitivity and Specificity of Remote Nonexpert Grading to Diagnose Cytomegalovirus Retinitis Using Expert Consensus as the Reference Standard

Estimates are shown for each of 15 nonexperts as well as the mean of the nonexperts. Graders had clinical experience (clinical), research experience (research), or neither clinical nor research experience (neither).

Table.  
Summary of False-positive and False-negative Casesa
Summary of False-positive and False-negative Casesa
1.
Lee  V, Subak-Sharpe  I, Shah  S, Aitken  C, Limb  S, Pinching  A.  Changing trends in cytomegalovirus retinitis with triple therapy. Eye (Lond). 1999;13(pt 1):59-64.
PubMedArticle
2.
Doan  S, Cochereau  I, Guvenisik  N, Diraison  MC, Mousalatti  H, Thanh  HX.  Cytomegalovirus retinitis in HIV-infected patients with and without highly active antiretroviral therapy. Am J Ophthalmol. 1999;128(2):250-251.
PubMedArticle
3.
Deayton  JR, Wilson  P, Sabin  CA,  et al.  Changes in the natural history of cytomegalovirus retinitis following the introduction of highly active antiretroviral therapy. AIDS. 2000;14(9):1163-1170.
PubMedArticle
4.
Jacobson  MA, Stanley  H, Holtzer  C, Margolis  TP, Cunningham  ET.  Natural history and outcome of new AIDS-related cytomegalovirus retinitis diagnosed in the era of highly active antiretroviral therapy. Clin Infect Dis. 2000;30(1):231-233.
PubMedArticle
5.
Yust  I, Fox  Z, Burke  M,  et al.  Retinal and extraocular cytomegalovirus end-organ disease in HIV-infected patients in Europe: a EuroSIDA study, 1994-2001. Eur J Clin Microbiol Infect Dis. 2004;23(7):550-559.
PubMedArticle
6.
Sugar  EA, Jabs  DA, Ahuja  A,  et al; Studies of the Ocular Complications of AIDS Research Group.  Incidence of cytomegalovirus retinitis in the era of highly active antiretroviral therapy. Am J Ophthalmol. 2012;153(6):1016-1024.e5. doi:10.1016/j.ajo.2011.11.014.
PubMedArticle
7.
Ausayakhun  S, Watananikorn  S, Ittipunkul  N, Chaidaroon  W, Patikulsila  P, Patikulsila  D.  Epidemiology of the ocular complications of HIV infection in Chiang Mai. J Med Assoc Thai. 2003;86(5):399-406.
PubMed
8.
Pathanapitoon  K, Ausayakhun  S, Kunavisarut  P,  et al.  Blindness and low vision in a tertiary ophthalmologic center in Thailand: the importance of cytomegalovirus retinitis. Retina. 2007;27(5):635-640.
PubMedArticle
9.
Ausayakhun  S, Keenan  JD, Ausayakhun  S,  et al.  Clinical features of newly diagnosed cytomegalovirus retinitis in northern Thailand. Am J Ophthalmol. 2012;153(5):923-931.e1. doi:10.1016/j.ajo.2011.10.012.
PubMedArticle
10.
Tangmonkongvoragul  C, Ausayakhun  S.  Causes of visual acuity loss among HIV-infected patients with cytomegalovirus retinitis in the era of highly active antiretroviral therapy in Chiang Mai University Hospital. J Med Assoc Thai. 2012;95(suppl 4):S129-S135.
PubMed
11.
Estopinal  CB, Ausayakhun  S, Ausayakhun  S,  et al.  Access to ophthalmologic care in Thailand: a regional analysis. Ophthalmic Epidemiol. 2013;20(5):267-273.
PubMedArticle
12.
Heiden  D, Ford  N, Wilson  D,  et al.  Cytomegalovirus retinitis: the neglected disease of the AIDS pandemic. PLoS Med. 2007;4(12):e334. doi:10.1371/journal.pmed.0040334.
PubMedArticle
13.
Fijalkowski  N, Zheng  LL, Henderson  MT, Wallenstein  MB, Leng  T, Moshfeghi  DM.  Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): four-years of screening with telemedicine. Curr Eye Res. 2013;38(2):283-291.
PubMedArticle
14.
Chiang  MF, Keenan  JD, Starren  J,  et al.  Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124(3):322-327.
PubMedArticle
15.
Liesenfeld  B, Kohner  E, Piehlmeier  W,  et al.  A telemedical approach to the screening of diabetic retinopathy: digital fundus photography. Diabetes Care. 2000;23(3):345-348.
PubMedArticle
16.
Ausayakhun  S, Skalet  AH, Jirawison  C,  et al.  Accuracy and reliability of telemedicine for diagnosis of cytomegalovirus retinitis. Am J Ophthalmol. 2011;152(6):1053-1058.e1. doi:10.1016/j.ajo.2011.05.030.
PubMedArticle
17.
Castro  AF, Silva-Turnes  JC, Gonzalez  F.  Evaluation of retinal digital images by a general practitioner. Telemed J E Health. 2007;13(3):287-292.
PubMedArticle
18.
Williams  SL, Wang  L, Kane  SA,  et al.  Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus non-expert graders. Br J Ophthalmol. 2010;94(3):351-356.
PubMedArticle
19.
Cavallerano  JD, Silva  PS, Tolson  AM,  et al.  Imager evaluation of diabetic retinopathy at the time of imaging in a telemedicine program. Diabetes Care. 2012;35(3):482-484.
PubMedArticle
20.
Cavallerano  AA, Cavallerano  JD, Katalinic  P, Tolson  AM, Aiello  LP, Aiello  LM; Joslin Vision Network Clinical Team.  Use of Joslin Vision Network digital-video nonmydriatic retinal imaging to assess diabetic retinopathy in a clinical program. Retina. 2003;23(2):215-223.
PubMedArticle
21.
Sanchez  CR, Silva  PS, Cavallerano  JD, Aiello  LP, Aiello  LM.  Ocular telemedicine for diabetic retinopathy and the Joslin Vision Network. Semin Ophthalmol. 2010;25(5-6):218-224.
PubMedArticle
22.
Thailand AIDS Response Progress Report 2012; reporting period: 2011-2011. http://www.unaids.org/en/dataanalysis/knowyourresponse/countryprogressreports/2012countries/ce_TH_Narrative_Report[1].pdf. Accessed April 21, 2014.
23.
Conlin  PR, Fisch  BM, Cavallerano  AA, Cavallerano  JD, Bursell  SE, Aiello  LM.  Nonmydriatic teleretinal imaging improves adherence to annual eye examinations in patients with diabetes. J Rehabil Res Dev. 2006;43(6):733-740.
PubMedArticle
24.
Taylor  CR, Merin  LM, Salunga  AM,  et al.  Improving diabetic retinopathy screening ratios using telemedicine-based digital retinal imaging technology: the Vine Hill study. Diabetes Care. 2007;30(3):574-578.
PubMedArticle
25.
Boucher  MC, Desroches  G, Garcia-Salinas  R,  et al.  Teleophthalmology screening for diabetic retinopathy through mobile imaging units within Canada. Can J Ophthalmol. 2008;43(6):658-668.
PubMedArticle
26.
Mansberger  SL, Gleitsmann  K, Gardiner  S,  et al.  Comparing the effectiveness of telemedicine and traditional surveillance in providing diabetic retinopathy screening examinations: a randomized controlled trial. Telemed J E Health. 2013;19(12):942-948.
PubMedArticle
27.
Kumari Rani  P, Raman  R, Manikandan  M, Mahajan  S, Paul  PG, Sharma  T.  Patient satisfaction with tele-ophthalmology versus ophthalmologist-based screening in diabetic retinopathy. J Telemed Telecare. 2006;12(3):159-160.
PubMedArticle
28.
Kirkizlar  E, Serban  N, Sisson  JA, Swann  JL, Barnes  CS, Williams  MD.  Evaluation of telemedicine for screening of diabetic retinopathy in the Veterans Health Administration. Ophthalmology. 2013;120(12):2604-2610.
PubMedArticle
29.
Jackson  KM, Scott  KE, Graff Zivin  J,  et al.  Cost-utility analysis of telemedicine and ophthalmoscopy for retinopathy of prematurity management. Arch Ophthalmol. 2008;126(4):493-499.
PubMedArticle
30.
Maamari  RN, Keenan  JD, Fletcher  DA, Margolis  TP.  A mobile phone-based retinal camera for portable wide field imaging. Br J Ophthalmol. 2014;98(4):438-441.
PubMedArticle
31.
Haddock  LJ, Kim  DY, Mukai  S.  Simple, inexpensive technique for high-quality smartphone fundus photography in human and animal eyes. J Ophthalmol. 2013;2013:518479. doi:10.1155/2013/518479.
PubMedArticle
Original Investigation
September 2014

Telemedicine Diagnosis of Cytomegalovirus Retinitis by Nonophthalmologists

Author Affiliations
  • 1Francis I. Proctor Foundation, University of California, San Francisco
  • 2medical student, Icahn School of Medicine at Mount Sinai, New York, New York
  • 3Department of Ophthalmology, Faculty of Medicine, Chiang Mai University, Chiang Mai, Thailand
  • 4Department of Ophthalmology, Nakornping Hospital, Chiang Mai, Thailand
  • 5Department of Ophthalmology and Pacific Vision Foundation, California Pacific Medical Center, San Francisco
  • 6Department of Ophthalmology, University of California, Los Angeles
  • 7Department of Ophthalmology, University of California, San Francisco
JAMA Ophthalmol. 2014;132(9):1052-1058. doi:10.1001/jamaophthalmol.2014.1108
Abstract

Importance  Cytomegalovirus (CMV) retinitis continues to be a leading cause of blindness in many developing countries. Telemedicine holds the potential to increase the number of people screened for CMV retinitis, but it is unclear whether nonophthalmologists could be responsible for interpreting fundus photographs captured in a telemedicine program.

Objective  To determine the accuracy of nonophthalmologist photographic graders in diagnosing CMV retinitis from digital fundus photographs.

Design, Setting, and Participants  Fifteen nonexpert graders each evaluated 182 mosaic retinal images taken from the eyes of patients with AIDS who were evaluated at the Ocular Infectious Diseases Clinic at Chiang Mai University in Chiang Mai, Thailand.

Main Outcomes and Measures  Graders diagnosed each image as CMV retinitis present, CMV retinitis absent, or unknown. The results from each grader were compared with those of an indirect ophthalmoscopic examination from an experienced on-site ophthalmologist as well as with the consensus grade given by a panel of CMV retinitis experts.

Results  Relative to the on-site ophthalmologist, the sensitivity of remote CMV retinitis diagnosis by nonexpert graders ranged from 64.0% to 95.5% (mean, 84.1%; 95% CI, 78.6%-89.6%)), and the specificity ranged from 65.6% to 92.5% (mean, 82.3%; 95% CI, 76.6%-88.0%)). Agreement between nonexpert and expert graders was high: the mean sensitivity and specificity values of nonexpert diagnosis using expert consensus as the reference standard were 93.2% (95% CI, 90.6%-95.8%) and 88.4% (95% CI, 85.4%-91.1%), respectively. Mean intrarater reliability also was high (mean Cohen κ, 0.83; 95% CI, 0.78-0.87).

Conclusions and Relevance  The sensitivity and specificity of remote diagnosis of CMV retinitis by nonexpert graders was variable, although several nonexperts achieved a level of accuracy comparable to that of CMV retinitis experts. More intensive training and periodic evaluations would be required if nonexperts are to be used in clinical practice.

Cytomegalovirus (CMV) retinitis is a well-known complication of AIDS. Although the incidence of CMV retinitis has declined in the United States and Europe owing to widespread use of highly active antiretroviral therapy (HAART),16 it is still common in countries with a high burden of human immunodeficiency virus (HIV) and limited access to HAART. At a tertiary ophthalmologic center in Chiang Mai, Thailand, CMV retinitis was found in 33% of HIV-positive patients who received screening and was one of the leading causes of blindness among all patients seen at the clinic.7,8 In this resource-poor setting, patients often have severe disease when they first present to an ophthalmologist despite receiving HAART, which is suggestive of inadequate screening.9,10

There are significant challenges to screening the population at risk for CMV retinitis. Chief among these is a relative shortage of ophthalmologists. For example, in Thailand nearly half of all ophthalmologists practice in Bangkok, leaving the outlying provinces relatively understaffed.11 Uneven distribution of ophthalmologists is likely to be similar in other resource-poor settings. Improving coverage of CMV retinitis screening will require addressing the relative lack of ophthalmology care for patients with HIV.12

Telemedicine, which is used to screen for conditions such as diabetic retinopathy (DR) and retinopathy of prematurity (ROP),1315 is a promising method for screening at-risk patients with AIDS for CMV retinitis. A prior study16 showed that telemedicine diagnosis by expert graders has high sensitivity and specificity for CMV retinitis. However, ophthalmologists may have difficulty giving prompt diagnosis because of time demands on their schedules. Trained nonexpert graders at dedicated reading centers may provide faster turnaround at a fraction of the cost of an ophthalmologist. Previous research1719 found that nonexperts can be trained to remotely detect hypertensive retinopathy, DR, and ROP with high accuracy. The purpose of the present study was to evaluate how accurately individuals other than ophthalmologists were able to diagnose CMV retinitis using fundus photographs.

Methods
Ethics Statement

Approval for the study was obtained from the Committee on Human Research at the University of California, San Francisco, and the Faculty of Medicine Research Ethics Committee at Chiang Mai University. This study adhered to the tenets of the Declaration of Helsinki. All participants gave written informed consent. No financial compensation was provided.

Study Design

Fundus photography was performed in a series of patients with AIDS by a trained ophthalmic photographer, and indirect ophthalmoscopy was conducted by an experienced ophthalmologist. Digital fundus photographs were read by 3 CMV retinitis experts (D.H., G.N.H., and T.P.M.), as reported previously.16 In the present study, we performed a training session for 15 nonophthalmologists (including M.Y.) and subsequently asked them to evaluate the same set of fundus photographs reviewed by the CMV retinitis experts. We compared the diagnosis given by the nonophthalmologists with that of the in-clinic ophthalmologist and CMV retinitis experts to determine the accuracy of nonexpert diagnosis of CMV retinitis.

Patient Population

We used the same images from a prior study16 of patients being screened for CMV retinitis at the Ocular Infectious Diseases Clinic at Chiang Mai University. Between August 7, 2008, and April 9, 2009, we enrolled newly referred patients with AIDS who had evidence of CMV retinitis on indirect ophthalmoscopy, as well as patients who had received a CMV retinitis diagnosis in the month before the start of the study. From August 7, 2008, to January 7, 2009, we also enrolled patients with AIDS who had no evidence of CMV retinitis on indirect ophthalmoscopy. An experienced attending ophthalmologist performed indirect ophthalmoscopy on both eyes of each study participant. One-time fundus photography was performed on both eyes by a trained retinal photographer (TRC-NW6S digital fundus camera, Topcon). Using the camera’s preset internal fixation light to dictate the position of each picture, the photographer captured a standard set of 9 overlapping 45° fundus images. The photographer was masked to all clinical information, including whether the study participant had CMV retinitis and whether the patient was being evaluated for the first time or during a follow-up visit. Automated mosaics covering an 85° retinal field were created using software that was included with the camera. There were no adverse events associated with either the fundus photography or indirect ophthalmoscopy. Eyes were excluded from this study if the attending ophthalmologist was unable to determine with certainty the presence or absence of CMV retinitis.

Nonexpert Graders

Fifteen nonexpert graders from US and Thai medical centers were grouped into 3 categories based on prior clinical and research experience. The first group had no clinical or research experience and included 5 ophthalmology secretaries and 1 undergraduate university student. The second group consisted of individuals with research experience but no clinical experience and included 3 research assistants, 1 research coordinator, and 1 laboratory technician. Graders in the third group had clinical but no research experience and included 1 ophthalmic nurse and 3 medical students (including M.Y.). All graders attended one 2-hour training session that covered eye anatomy, retinal abnormalities, CMV retinitis, and reasons for ophthalmology referral. Trainings were conducted in English or Thai depending on the setting, and all participants viewed the same training materials. Graders were given a copy of the English-language training materials to use as reference.

Image Evaluation

Similar to the procedures in the prior study,16 graders examined the fundus photographs of each eye independently, in random order, without accompanying clinical information. Nonexpert graders assessed only the montage retinal images (ie, not the individual frames). Graders first evaluated the quality of each photograph, with good defined as an image in focus that included the entire 85° retinal field, acceptable defined as an image moderately out of focus or not including the entire retinal field but suitable for determination of the presence or absence of CMV retinitis, and poor defined as an image extremely out of focus or including such a small degree of the retinal field that the presence or absence of CMV retinitis could not be determined. Graders identified each image as CMV retinitis present, CMV retinitis absent, or unknown. All graders were masked to the diagnosis determined by the on-site ophthalmologist and to the diagnosis of the other graders.

Statistical Analysis

We calculated the sensitivity and specificity of CMV retinitis diagnosis for each nonexpert grader using the on-site ophthalmology examination as the reference standard. In practice, nonexpert graders would provide initial screening to determine which patients should be referred to an ophthalmologist for definitive diagnosis. Therefore, both CMV retinitis present and unknown counted as a positive diagnosis because both would warrant ophthalmology referral. To determine patterns of false-positive and false-negative photographic assessments, we identified eyes that were incorrectly categorized by most (≥8) of the nonexpert graders relative to the indirect ophthalmoscopic examination and compared the results with the consensus diagnosis of 3 expert graders from the prior study16 (ie, the majority diagnosis of the 3 experts). We calculated the positive predictive value for images graded as CMV retinitis present and unknown separately and the negative predictive value for images graded as CMV retinitis absent.

To compare the performance of nonexperts with that of experts, we calculated the sensitivity and specificity of each of the nonexperts using the expert consensus as the reference standard. Images were excluded from this analysis if there was no majority consensus or if the consensus was a diagnosis of unknown. We chose to express the agreement between expert and nonexpert graders as sensitivity and specificity because grades from these 2 groups of individuals could not reasonably be thought of as interchangeable data points—we would not expect nonexperts who received a brief training workshop to perform as well as expert ophthalmologists.

We used nonparametric tests (eg, Wilcoxon rank sum and Kruskal-Wallis) to perform comparisons between subgroups of graders. We used the Cohen κ coefficient to estimate (1) the intrarater reliability for each nonexpert grader from a set of 50 randomly selected and randomly presented duplicate images and (2) the interrater reliability among the 15 nonexpert graders. To assess the relationship between intrarater agreement and diagnostic accuracy, we calculated the Pearson correlation coefficient for intrarater agreement and a summary measure combining sensitivity and specificity (Youden index). To account for nonindependence of eyes from the same person, we calculated bootstrap 95% CIs for diagnostic test statistics with resampling at the patient level (9999 repetitions). All statistical analyses were performed with Stata, version 13 (StataCorp).

Results

Characteristics of the participants included in this study have been described previously.16 The on-site ophthalmologist was able to definitively determine the presence or absence of CMV retinitis in 182 distinct eyes of 94 patients; 167 of these eyes were evaluated as a first examination, and 15 eyes were seen during a follow-up visit. Of the 182 eyes included in the study, CMV retinitis was diagnosed in 89 (48.9%) by the attending ophthalmologist.16 Each nonexpert grader reviewed all 182 retinal photographs.

The nonexpert graders recorded a diagnosis of CMV retinitis present in a mean of 71 eyes, CMV retinitis absent in a mean of 91 eyes, and unknown in a mean of 20 eyes. Remote diagnosis of CMV retinitis by nonexperts had sensitivity ranging from 64.0% to 95.5% (mean, 84.1%; 95% CI, 78.6%-89.6%) and specificity ranging from 65.6% to 92.5% (mean, 82.3%; 95% CI, 76.6%-88.0%). In comparison, the CMV retinitis experts achieved sensitivity ranging from 88.8% to 91.0% and specificity ranging from 84.9% to 88.2% (P = .14 and P = .17, respectively, Wilcoxon rank sum test) (Figure 1).16 The 3 groups of nonexperts did not have markedly different mean sensitivities (85.4% for graders with clinical experience, 82.9% for graders with research experience, and 84.3% for graders with neither clinical nor research experience; P = .59, Kruskal-Wallis test) or specificities (86.6%, 82.6%, and 79.2%, respectively; P = .28).

Because the predictive value of a test depends on the prevalence of disease, we depicted the positive predictive value and negative predictive value over a range of disease prevalence estimates (Figure 2). As described previously,16 the prevalence of CMV retinitis in eyes being screened for that disease during the study period was 35.6%. Using this 35.6% prevalence, a nonexpert diagnosis of CMV retinitis present indicates an 86.4% probability that the patient has the disease, whereas a diagnosis of unknown indicates a 37.4% chance of the patient having CMV retinitis. At this same 35.6% prevalence, a diagnosis of CMV retinitis absent has a 90.5% chance of being a true negative. As shown in Figure 2, nonexpert graders had lower mean predictive values than did expert graders.

Intrarater agreement for the 15 nonexpert graders ranged from κ values of 0.48 to 1.00 (mean, 0.83; 95% CI, 0.78-0.87) (Figure 3). Mean intrarater agreement was 0.94 for the group of nonexperts with clinical experience, 0.78 for those with research experience, and 0.76 for those with neither clinical nor research experience (P = .18, Kruskal-Wallis test). Graders with low intrarater agreement tended to also have lower sensitivity and specificity (correlation coefficient between intrarater κ and Youden index, 0.82; 95% CI, 0.53-0.94). Interrater agreement between the 15 graders was substantial, with a κ value of 0.63 (95% CI, 0.57-0.68). For comparison, the previous study16 found intrarater κ values of 0.93 for each expert grader and an interrater κ of 0.86.

According to the expert consensus diagnosis, CMV retinitis was present in 73 eyes and absent in 91 eyes; no consensus could be established for the remaining 18 eyes. As shown in Figure 4, the diagnoses given by the 15 nonexperts were generally in agreement with the consensus diagnosis: mean sensitivity was 93.2% (95% CI, 90.6%-95.8%) and mean specificity was 88.4% (95% CI, 85.4%-91.1%). The 3 groups had similar mean sensitivities (94.5% for the group with clinical experience, 92.1% for the group with research experience, and 93.4% and for the group with neither clinical nor research experience; P = .54, Kruskal-Wallis test) and specificities (90.4%, 88.6%, and 87.0% respectively; P = .78).

We identified 9 eyes recorded as not having CMV retinitis by indirect ophthalmoscopy, which received a diagnosis of either CMV retinitis present or unknown by most nonexperts. An additional 9 eyes received a diagnosis of CMV retinitis by indirect ophthalmoscopy but not by most nonexperts (Table). For these discrepant cases, we compared the grades given by nonexperts with those given by the expert photographic graders and found that the diagnosis of most of the nonexperts agreed with that of the consensus of the experts (Table). Most of the false-positive nonexpert grades were the result of poor-quality photographs.

Discussion

Relative to an in-clinic ophthalmology examination by an attending ophthalmologist, nonexpert graders diagnosed CMV retinitis from fundus photographs with a mean sensitivity of 84.1% and specificity of 82.3%. Nonexperts usually had lower accuracy compared with a trio of CMV retinitis experts, although several of the nonexperts had sensitivity and specificity estimates that clustered near those of the expert graders (Figure 1). Among the 15 nonexperts, 7 had a sensitivity level and 7 had a specificity level equal to or greater than that of at least 1 of the experts. However, only 2 nonexperts had both sensitivity and specificity matching or exceeding that of 1 of the expert graders.

The results from this study are consistent with those from a study18 of telemedicine diagnosis of ROP, in which nonexpert graders exhibited considerable variability (sensitivity ranged from 73% to 87% and specificity ranged from 73% to 91%). Moreover, personnel other than ophthalmologists are already used in clinical practice at dedicated reading centers to stage DR.1921 In these cases, graders complete multiple-day training with strict oversight and quality assurance protocols.20 Given the variability exhibited in the present study, if nonexperts are to be used for remote diagnosis of CMV retinitis they likewise must be carefully selected and undergo more intensive training than was offered in this study. Moreover, given the inconsistent intrarater agreement, periodic testing of nonexperts would be mandatory. Nonetheless, this study demonstrates that at least some nonexperts were capable of grading fundus photographs for CMV retinitis at a level similar to that of ophthalmologist experts, suggesting that nonexpert assessment could be useful for a CMV retinitis telemedicine program.

The prevalence of CMV retinitis during the study period (35.6%) was lower than the proportion of eyes in the present study that received a diagnosis of CMV retinitis (48.9%) because patients without CMV retinitis were not enrolled in the later months of the study. Although this discrepancy should not affect our estimates for sensitivity and specificity, the site-specific prevalence will affect the estimates of the positive and negative predictive values. In fact, because the study site is a tertiary referral center, it is likely that even the 35.6% prevalence estimate is higher than what would be observed at a typical HIV clinic in Thailand.7,8,16 At a 10% prevalence, which is a more realistic estimate for a nontertiary health care setting, the mean positive predictive value would be 61.9% for a diagnosis of CMV retinitis present and 11.5% for a diagnosis of unknown, and the mean negative predictive value would be 97.9%. These results demonstrate that a telemedicine program with nonexpert graders would be effective in ruling out CMV retinitis in patients who do not have the disease. Such a program could increase the number of at-risk patients with AIDS referred for CMV retinitis screening while not overburdening a limited pool of ophthalmologists. This would be a great improvement over the current situation, in which at-risk patients with AIDS are not routinely screened for CMV retinitis.

In Thailand, it is estimated9,22 that 60% of patients with AIDS have CD4 cell counts less than 100/μL when they begin receiving HAART, but very few are screened for CMV retinitis before HAART is begun. Telemedicine diagnosis of CMV retinitis could therefore greatly increase access to screening, as it has for DR,2326 if instituted at the point of HIV care. Patients would benefit from not having a separate ophthalmology appointment, which would reduce the costs, time, and lost wages that would be incurred from an additional hospital visit.27 Providers of care for patients with AIDS would benefit by being able to offer CMV retinitis screening to more patients than is now feasible. Moreover, telemedicine would allow earlier diagnosis of CMV retinitis and hence result in fewer complications and better visual outcomes.

Telemedicine has been shown28,29 to be cost-effective in the United States for the diagnosis of DR and ROP. Economic considerations are even more important in resource-limited settings. Further studies are needed to formally assess the cost-effectiveness of using nonexpert photographic graders to remotely diagnose CMV retinitis. However, the present study suggests that nonexpert graders could be a cost-effective option, with the highest-performing nonexpert graders diagnosing CMV retinitis at a level of accuracy comparable to that of an expert ophthalmologist. However, photographic graders are only one component of a telemedicine system. The current lack of affordable fundus cameras and appropriately trained photographers may restrict the use of telemedicine for CMV retinitis screening. The development of a low-cost, easy-to-use fundus camera is crucial for screening efforts—not just for CMV retinitis but also for more prevalent conditions, such as DR.30,31

The present study had several limitations. The study was designed to determine the sensitivity and specificity of an average nonexpert grader and therefore was not powered to be able to detect a difference between expert and nonexpert graders. Although training was performed in the grader’s native language, training materials were in English; Thai graders may have had a more difficult time referencing these materials after the training session. Thai-language materials could easily be developed if a telemedicine grading system based on nonexpert graders was to be implemented. The study included both patients presenting for an initial examination as well as those who received the CMV retinitis diagnosis in the previous month and had begun receiving treatment. We believe it unlikely that photographers were biased by this because they did not participate in this clinic before the study and were masked to all patient information. Moreover, because CMV retinitis resolves slowly and leaves a scar, we do not believe that receiving several weeks of antiviral treatment would have greatly altered the sensitivity and specificity of fundus photographs for detecting CMV retinitis.

Conclusions

Some, but not all, nonexpert graders in the present study were able to diagnose CMV retinitis using fundus photographs at a level that closely matched that of an expert grader. Strict training protocols and methods of evaluation would be required to select the best nonexpert graders and ensure accurate and consistent diagnosis. Nonexpert graders could potentially be used in primary care settings to screen at-risk HIV-positive patients for CMV retinitis using telemedicine, with referral to an ophthalmologist for definitive diagnosis. Studies of the cost-effectiveness of using nonexpert graders in a telemedicine program for CMV retinitis screening would be helpful before such a program were implemented.

Back to top
Article Information

Submitted for Publication: January 9, 2014; final revision received February 26, 2014; accepted February 26, 2014.

Corresponding Author: Jeremy D. Keenan, MD, MPH, Department of Ophthalmology, University of California, San Francisco, Medical Sciences Bldg S334, PO Box 0412, 513 Parnassus Ave, San Francisco, CA 94143 (jeremy.keenan@ucsf.edu).

Published Online: June 19, 2014. doi:10.1001/jamaophthalmol.2014.1108.

Author Contributions: Dr Keenan had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Somsanguan Ausayakhun, Sakarin Ausayakhun, Jirawison, Heiden, Margolis, Keenan.

Acquisition, analysis, or interpretation of data: Yen, Chen, Holland, Margolis.

Drafting of the manuscript: Yen, Somsanguan Ausayakhun, Sakarin Ausayakhun, Margolis.

Critical revision of the manuscript for important intellectual content: Yen, Somsanguan Ausayakhun, Chen, Jirawison, Heiden, Holland, Keenan.

Statistical analysis: Yen, Keenan.

Obtained funding: Margolis.

Administrative, technical, or material support: Yen, Somsanguan Ausayakhun, Chen, Sakarin Ausayakhun, Jirawison, Heiden, Holland.

Study supervision: Sakarin Ausayakhun, Margolis.

Conflict of Interest Disclosures: Dr Margolis has pending intellectual property with the University of California describing a mobile phone camera for retinal imaging. At this time, this intellectual property has no financial value. No other disclosures are reported.

Funding/Support: This study was supported by the Gladstone Institute of Virology and Immunology Center for AIDS Research and the Research Evaluation and Allocation Committee, University of California, San Francisco; by grant K23EY019071 from the National Eye Institute; and by That Man May See, the Littlefield Trust, the Peierls Foundation, and the Doris Duke Charitable Foundation through a grant supporting the Doris Duke International Clinical Research Fellows Program at the University of California, San Francisco. Mr Yen is a Doris Duke International Clinical Research Fellow.

Role of the Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: We thank the nonexpert graders for their participation in this study: Atitaya Apivatthakakul, Thanyarat Chaichana, BS, Alissara Chaisong, CMA, Saringkarn Intachai. BA, Dueanwarang Inthawichai, BA, Nutchanun Kitsanasaranee, BA, Jaruwan Kongngern, BS, Phuriwat Maksongkhla, CMA, Chantawan Paspisanu, BSN, and Napatsawan Sangkun (Chiang Mai University); Sangita Annamalai, BA (University of California, Berkeley); and Kathleen Apakupakul, MS, Nicole Stoller, MPH, and Sophia Swanson, MD (University of California, San Francisco). Graders from Chiang Mai University were financially reimbursed for their contributions.

References
1.
Lee  V, Subak-Sharpe  I, Shah  S, Aitken  C, Limb  S, Pinching  A.  Changing trends in cytomegalovirus retinitis with triple therapy. Eye (Lond). 1999;13(pt 1):59-64.
PubMedArticle
2.
Doan  S, Cochereau  I, Guvenisik  N, Diraison  MC, Mousalatti  H, Thanh  HX.  Cytomegalovirus retinitis in HIV-infected patients with and without highly active antiretroviral therapy. Am J Ophthalmol. 1999;128(2):250-251.
PubMedArticle
3.
Deayton  JR, Wilson  P, Sabin  CA,  et al.  Changes in the natural history of cytomegalovirus retinitis following the introduction of highly active antiretroviral therapy. AIDS. 2000;14(9):1163-1170.
PubMedArticle
4.
Jacobson  MA, Stanley  H, Holtzer  C, Margolis  TP, Cunningham  ET.  Natural history and outcome of new AIDS-related cytomegalovirus retinitis diagnosed in the era of highly active antiretroviral therapy. Clin Infect Dis. 2000;30(1):231-233.
PubMedArticle
5.
Yust  I, Fox  Z, Burke  M,  et al.  Retinal and extraocular cytomegalovirus end-organ disease in HIV-infected patients in Europe: a EuroSIDA study, 1994-2001. Eur J Clin Microbiol Infect Dis. 2004;23(7):550-559.
PubMedArticle
6.
Sugar  EA, Jabs  DA, Ahuja  A,  et al; Studies of the Ocular Complications of AIDS Research Group.  Incidence of cytomegalovirus retinitis in the era of highly active antiretroviral therapy. Am J Ophthalmol. 2012;153(6):1016-1024.e5. doi:10.1016/j.ajo.2011.11.014.
PubMedArticle
7.
Ausayakhun  S, Watananikorn  S, Ittipunkul  N, Chaidaroon  W, Patikulsila  P, Patikulsila  D.  Epidemiology of the ocular complications of HIV infection in Chiang Mai. J Med Assoc Thai. 2003;86(5):399-406.
PubMed
8.
Pathanapitoon  K, Ausayakhun  S, Kunavisarut  P,  et al.  Blindness and low vision in a tertiary ophthalmologic center in Thailand: the importance of cytomegalovirus retinitis. Retina. 2007;27(5):635-640.
PubMedArticle
9.
Ausayakhun  S, Keenan  JD, Ausayakhun  S,  et al.  Clinical features of newly diagnosed cytomegalovirus retinitis in northern Thailand. Am J Ophthalmol. 2012;153(5):923-931.e1. doi:10.1016/j.ajo.2011.10.012.
PubMedArticle
10.
Tangmonkongvoragul  C, Ausayakhun  S.  Causes of visual acuity loss among HIV-infected patients with cytomegalovirus retinitis in the era of highly active antiretroviral therapy in Chiang Mai University Hospital. J Med Assoc Thai. 2012;95(suppl 4):S129-S135.
PubMed
11.
Estopinal  CB, Ausayakhun  S, Ausayakhun  S,  et al.  Access to ophthalmologic care in Thailand: a regional analysis. Ophthalmic Epidemiol. 2013;20(5):267-273.
PubMedArticle
12.
Heiden  D, Ford  N, Wilson  D,  et al.  Cytomegalovirus retinitis: the neglected disease of the AIDS pandemic. PLoS Med. 2007;4(12):e334. doi:10.1371/journal.pmed.0040334.
PubMedArticle
13.
Fijalkowski  N, Zheng  LL, Henderson  MT, Wallenstein  MB, Leng  T, Moshfeghi  DM.  Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): four-years of screening with telemedicine. Curr Eye Res. 2013;38(2):283-291.
PubMedArticle
14.
Chiang  MF, Keenan  JD, Starren  J,  et al.  Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124(3):322-327.
PubMedArticle
15.
Liesenfeld  B, Kohner  E, Piehlmeier  W,  et al.  A telemedical approach to the screening of diabetic retinopathy: digital fundus photography. Diabetes Care. 2000;23(3):345-348.
PubMedArticle
16.
Ausayakhun  S, Skalet  AH, Jirawison  C,  et al.  Accuracy and reliability of telemedicine for diagnosis of cytomegalovirus retinitis. Am J Ophthalmol. 2011;152(6):1053-1058.e1. doi:10.1016/j.ajo.2011.05.030.
PubMedArticle
17.
Castro  AF, Silva-Turnes  JC, Gonzalez  F.  Evaluation of retinal digital images by a general practitioner. Telemed J E Health. 2007;13(3):287-292.
PubMedArticle
18.
Williams  SL, Wang  L, Kane  SA,  et al.  Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus non-expert graders. Br J Ophthalmol. 2010;94(3):351-356.
PubMedArticle
19.
Cavallerano  JD, Silva  PS, Tolson  AM,  et al.  Imager evaluation of diabetic retinopathy at the time of imaging in a telemedicine program. Diabetes Care. 2012;35(3):482-484.
PubMedArticle
20.
Cavallerano  AA, Cavallerano  JD, Katalinic  P, Tolson  AM, Aiello  LP, Aiello  LM; Joslin Vision Network Clinical Team.  Use of Joslin Vision Network digital-video nonmydriatic retinal imaging to assess diabetic retinopathy in a clinical program. Retina. 2003;23(2):215-223.
PubMedArticle
21.
Sanchez  CR, Silva  PS, Cavallerano  JD, Aiello  LP, Aiello  LM.  Ocular telemedicine for diabetic retinopathy and the Joslin Vision Network. Semin Ophthalmol. 2010;25(5-6):218-224.
PubMedArticle
22.
Thailand AIDS Response Progress Report 2012; reporting period: 2011-2011. http://www.unaids.org/en/dataanalysis/knowyourresponse/countryprogressreports/2012countries/ce_TH_Narrative_Report[1].pdf. Accessed April 21, 2014.
23.
Conlin  PR, Fisch  BM, Cavallerano  AA, Cavallerano  JD, Bursell  SE, Aiello  LM.  Nonmydriatic teleretinal imaging improves adherence to annual eye examinations in patients with diabetes. J Rehabil Res Dev. 2006;43(6):733-740.
PubMedArticle
24.
Taylor  CR, Merin  LM, Salunga  AM,  et al.  Improving diabetic retinopathy screening ratios using telemedicine-based digital retinal imaging technology: the Vine Hill study. Diabetes Care. 2007;30(3):574-578.
PubMedArticle
25.
Boucher  MC, Desroches  G, Garcia-Salinas  R,  et al.  Teleophthalmology screening for diabetic retinopathy through mobile imaging units within Canada. Can J Ophthalmol. 2008;43(6):658-668.
PubMedArticle
26.
Mansberger  SL, Gleitsmann  K, Gardiner  S,  et al.  Comparing the effectiveness of telemedicine and traditional surveillance in providing diabetic retinopathy screening examinations: a randomized controlled trial. Telemed J E Health. 2013;19(12):942-948.
PubMedArticle
27.
Kumari Rani  P, Raman  R, Manikandan  M, Mahajan  S, Paul  PG, Sharma  T.  Patient satisfaction with tele-ophthalmology versus ophthalmologist-based screening in diabetic retinopathy. J Telemed Telecare. 2006;12(3):159-160.
PubMedArticle
28.
Kirkizlar  E, Serban  N, Sisson  JA, Swann  JL, Barnes  CS, Williams  MD.  Evaluation of telemedicine for screening of diabetic retinopathy in the Veterans Health Administration. Ophthalmology. 2013;120(12):2604-2610.
PubMedArticle
29.
Jackson  KM, Scott  KE, Graff Zivin  J,  et al.  Cost-utility analysis of telemedicine and ophthalmoscopy for retinopathy of prematurity management. Arch Ophthalmol. 2008;126(4):493-499.
PubMedArticle
30.
Maamari  RN, Keenan  JD, Fletcher  DA, Margolis  TP.  A mobile phone-based retinal camera for portable wide field imaging. Br J Ophthalmol. 2014;98(4):438-441.
PubMedArticle
31.
Haddock  LJ, Kim  DY, Mukai  S.  Simple, inexpensive technique for high-quality smartphone fundus photography in human and animal eyes. J Ophthalmol. 2013;2013:518479. doi:10.1155/2013/518479.
PubMedArticle
×