Comparison of Clinical Trial and Systematic Review Outcomes for the 4 Most Prevalent Eye Diseases

Key Points

Question  For the 4 most prevalent eye diseases (age-related macular degeneration, cataract, diabetic retinopathy, and glaucoma), what is the overlap between outcomes named in Cochrane reviews and outcomes reported in the trials included in the reviews?

Findings  In this cross-sectional study, for each disease the trials included a considerably greater number of outcomes than did the reviews, ranging from 2.9 times greater (glaucoma) to 4.9 times greater (cataract). Although most review outcomes were reported in the trials, most trial outcomes were not reported in the reviews.

Meaning  Inconsistency in trial outcomes impedes research synthesis efforts and indicates the need for disease-specific core outcome sets in ophthalmology.

Importance  Suboptimal overlap in outcomes reported in clinical trials and systematic reviews compromises efforts to compare and summarize results across these studies.

Objectives  To examine the most frequent outcomes used in trials and reviews of the 4 most prevalent eye diseases (age-related macular degeneration [AMD], cataract, diabetic retinopathy [DR], and glaucoma) and the overlap between outcomes in the reviews and the trials included in the reviews.

Design, Setting, and Participants  This cross-sectional study examined all Cochrane reviews that addressed AMD, cataract, DR, and glaucoma; were published as of July 20, 2016; and included at least 1 trial and the trials included in the reviews. For each disease, a pair of clinical experts independently classified all outcomes and resolved discrepancies. Outcomes (outcome domains) were then compared separately for each disease.

Main Outcomes and Measures  Proportion of review outcomes also reported in trials and vice versa.

Results  This study included 56 reviews that comprised 414 trials. Although the median number of outcomes per trial and per review was the same (n = 5) for each disease, the trials included a greater number of outcomes overall than did the reviews, ranging from 2.9 times greater (89 vs 30 outcomes for glaucoma) to 4.9 times greater (107 vs 22 outcomes for AMD). Most review outcomes, ranging from 14 of 19 outcomes (73.7%) (for DR) to 27 of 29 outcomes (93.1%) (for cataract), were also reported in the trials. For trial outcomes, however, the proportion also named in reviews was low, ranging from 19 of 107 outcomes (17.8%) (for AMD) to 24 of 89 outcomes (27.0%) (for glaucoma). Only 1 outcome (visual acuity) was consistently reported in greater than half the trials and greater than half the reviews.

Conclusions and Relevance  Although most review outcomes were reported in the trials, most trial outcomes were not reported in the reviews. The current analysis focused on outcome domains, which might underestimate the problem of inconsistent outcomes. Other important elements of an outcome (ie, specific measurement, specific metric, method of aggregation, and time points) might have differed even though the domains overlapped. Inconsistency in trial outcomes may impede research synthesis and indicates the need for disease-specific core outcome sets in ophthalmology.

Outcomes are measures or events used to assess the effectiveness and/or safety of clinical interventions.¹ In clinical trials and systematic reviews, researchers use outcomes as a basis for conclusions about whether interventions being tested will be effective and safe.

Worldwide, the 4 most prevalent eye diseases are age-related macular degeneration (AMD), cataract, diabetic retinopathy (DR), and glaucoma.² To improve the conditions of patients with these diseases, clinicians and patients should use evidence from trials and reviews to identify effective and safe interventions and treatment strategies. Determining which interventions and treatment strategies are the most effective and safe involves making comparisons across trials and reviews. However, suboptimal overlap in outcomes among these studies^3-5 compromises such comparisons. A systematic review⁶ has documented the problem of inconsistency in outcome use in various fields.

An example of this problem in ophthalmology was demonstrated in a Cochrane review⁷ of trials that compared nonsteroidal anti-inflammatory drugs with corticosteroids for controlling inflammation after uncomplicated cataract surgery. Although the review authors⁷ included 48 trials, none of the trials reported data for the review’s prespecified primary outcome: proportion of patients with intraocular inflammation at 1-week follow-up after surgery. Modifying the outcome to include mean amount of inflammation at 1-week follow-up would have allowed only 7 trials to be eligible. Including other follow-up time points would have allowed only 4 additional studies to be eligible.⁷ Studies have demonstrated that inconsistent outcome use is also a problem in AMD,⁸ glaucoma,⁹ uveitis,¹⁰ allergic conjunctivitis,¹¹ and intermittent exotropia.¹²

Inconsistent outcome use is also a problem in reviews in ophthalmology. A previous study³ examined all Cochrane reviews that addressed the 4 most prevalent eye diseases and found that researchers who evaluated interventions for the same disease considered different outcomes to be important, and when researchers considered the same outcome to be important, they usually used different measurements or analyzed the data differently or at different time points. Similarly, Ismail and colleagues¹³ identified inconsistency in outcomes examined in reviews that addressed glaucoma.

Our goal was to assess the extent of overlap in outcomes in reviews of the 4 most prevalent eye diseases and in the trials included in the reviews. Specifically, for each disease, our objectives were to examine the most frequent outcomes used in trials and reviews and the overlap between outcomes in the reviews and the trials included in the reviews.

In the present study, we identified the current versions of all Cochrane reviews that addressed AMD, cataract, DR, and glaucoma. We compared the outcomes in these reviews with the outcomes reported in the trials included in the reviews.

A completely specified outcome includes 5 elements: domain, specific measurement, specific metric, method of aggregation, and time points of interest.^3,14 We focused on the domain (eg, visual acuity, intraocular pressure). An example would be that measuring visual acuity using the Snellen chart or the Early Treatment Diabetic Retinopathy Study chart pertains to the outcome domain visual acuity. Similarly, we counted an outcome reported at multiple time points as pertaining to a single outcome domain. We classified outcome domains as specifically as possible (eg, we considered photopic contrast sensitivity and mesopic contrast sensitivity as 2 separate outcome domains).

We included all Cochrane reviews that addressed at least 1 of the 4 most prevalent eye diseases (AMD, DR, glaucoma, and cataract) and were published by Cochrane Eyes and Vision in the Cochrane Database of Systematic Reviews as of July 20, 2016. Because we were interested in the overlap in review and trial outcomes, we restricted this study to completed reviews (ie, we excluded reviews in the protocol stage) that included at least 1 trial (ie, we excluded reviews that did not include any studies, the so-called empty reviews). We assessed the overlap in outcomes within subgroups defined by disease. For each review, we abstracted all outcomes reported in the Methods section irrespective of whether they were also presented in the Results section.

We examined each trial that each eligible review included if the trial (1) compared at least 2 groups to which participants were randomly allocated and (2) was published as a peer-reviewed journal article (ie, we excluded conference abstracts). For each trial, we identified 1 journal article defined by the review authors as that trial’s primary publication, as conventionally indicated by an asterisk next to the citation information in the References to Studies Included in this Review section of Cochrane reviews.

We developed a data abstraction form in the Systematic Review Data Repository (https://srdr.ahrq.gov), an open repository of review data.^15,16 We conducted a pilot test of the form by using 10 trials and 10 reviews. The form included check-box items for predefined outcomes and free-text items for additional outcomes not previously identified. Two of us (I.J.S., K.L.) and a Cochrane eyes and vision methodologist (Sueko Ng, MHS) conducted data abstraction; 2 individuals independently abstracted data from each trial and review, resolving discrepancies through discussion.

From each trial’s primary publication, we abstracted all outcomes for which results were reported. We defined results as any quantitative data, including from statistical testing, that compared 2 or more interventions for efficacy or safety after trial baseline reported anywhere in the article’s text, tables, or figures.

We classified outcomes into specific domains by using a 2-step process. In step 1, the outcomes in the trials and reviews were initially coded (by 2 of us [I.J.S., K.L.] and Sueko Ng), and a prior classification system of outcomes in Cochrane reviews that addressed the same 4 eye diseases was updated.³ In step 2, for each disease, 2 clinician coauthors (D.D. and C.M. for AMD, R.S.C. and L.S.J. for cataract, D.V.D. and G.V. for DR, and A.L.C. and H.D.J. for glaucoma) with expertise in that disease verified the initial coding of abstracted outcomes. Within each pair, masked to each other’s and to the initial coding, each expert coded the reported outcomes in the trials and reviews. For each reported outcome, the expert (1) coded the outcome as an exact match to an existing outcome in the updated classification system of Saldanha et al³ or (2) suggested a new outcome to which the outcome pertained. After independent coding by the experts, disagreements were resolved through discussion. We considered the agreed-on classification by the experts as the final classification for each outcome.

For each disease, we adopted the following 3 approaches to examine the overlap of outcomes in trials and reviews. First, we constructed Venn diagrams for the number of outcomes in both trials and reviews and the numbers uniquely in each. Second, we constructed scatterplots of the proportion of trials and the proportion of reviews that examined each outcome (hypothetical scenarios explained in Figure 1). Third, we examined the overlap in the 7 most frequent outcomes in the trials and reviews. We chose 7 because Cochrane recommends including up to 7 outcomes in summary of findings tables in reviews.¹⁷

Among the 65 Cochrane reviews of AMD, cataract, DR, and glaucoma published as of July 2016, a total of 61 were completed (ie, 4 were in protocol stages), and 56 of these completed reviews included at least 1 trial (eFigure in the Supplement). A total of 54 of the 56 eligible and included reviews (96.4%) were published in 2008 or later (Table 1).

Overall, the 56 included reviews comprised 445 unique trials. We excluded 31 trials reported only as conference abstracts, thereby including 414 unique trials. Reviews incorporated a median of 5.0 trials each (interquartile range [IQR], 2.0-10.5; range 1.0–60.0). Most trials were in reviews that addressed glaucoma (142 of 414 [34.3%]) or cataract (138 of 414 [33.3%]).

We identified 262 total unique outcomes in the trials and reviews. Overall, the trials and reviews reported measuring similar numbers of outcomes (trials: median, 5.0 outcomes per trial; IQR, 3.0-8.0; range, 1.0-24.0; reviews: median, 5.0 outcomes per review; IQR, 4.0-6.0; range, 2.0-10.0).

For each disease, the trials included a greater number of outcomes than did the reviews, ranging from 2.9 times (89 vs 30 outcomes for glaucoma) to 4.9 times greater (107 vs 22 outcomes for AMD) (Figure 2). When considering all outcomes across trials and reviews that addressed a disease, the overlap between the outcomes measured in trials and reviews was limited, ranging from 19 of 110 outcomes (17.3%) (for AMD) to 24 of 95 outcomes (25.3%) (for glaucoma). For review outcomes, most outcomes were also reported in the trials, ranging from 14 of 19 outcomes (73.7%) (for DR) to 27 of 29 outcomes (93.1%) (for cataract). For trial outcomes, the overlap was small, ranging from 19 of 107 outcomes (17.8%) (for AMD) to 24 of 89 outcomes (27.0%) (for glaucoma).

In each scatterplot (Figure 3), most outcomes clustered in the lower left quadrant, indicating that, for each disease, most outcomes were reported in fewer than half the trials and fewer than half the reviews. Across the 4 diseases, only 1 outcome (visual acuity) was consistently named in greater than half the trials and greater than half the reviews.

For each disease, there was limited overlap in the 7 most frequent outcomes in the trials and reviews, with some noticeable differences (Table 2). For trials and reviews, visual acuity was the most frequent outcome for 3 diseases (AMD, cataract, and DR), whereas for glaucoma, it was among the 7 most frequent outcomes. Ocular adverse events also were among the 7 most frequent outcomes for trials and reviews except for reviews that addressed AMD. Some frequent review outcomes were not often reported in the trials. For example, general quality of life was among the 7 most frequent review outcomes for each disease but never among the 7 most frequent trial outcomes. Similarly, costs were among the 7 most frequent review outcomes for cataract, DR, and glaucoma but not among the 7 most frequent trial outcomes.

Table 2 examines whether certain outcomes may be frequently used by trialists and reviewers. None of the common outcomes were reported in all trials and reviews.

In this study, which focused on outcome domains, we found that trials included in Cochrane reviews of the 4 most prevalent eye diseases reported a greater number of outcomes than did the reviews. Although large proportions of review outcomes, ranging from 73.7% to 93.1%, were reported in the trials, smaller proportions of trial outcomes, ranging from 17.8% to 27.0%, were reported in the reviews.

Visual acuity was the most frequent outcome in trials and reviews for all diseases in our study. Visual acuity directly measures vision, the eye’s primary function and a mechanism that most eye diseases eventually affect. The measurement of visual acuity is relatively insensitive to the patient’s language fluency and educational level and is important because of its correlations with general and vision-related quality of life¹⁸ and activities of daily living.¹⁹ In addition, measurement of visual acuity is generally inexpensive and minimally invasive.

For each disease, the 7 most frequent outcomes in trials never included general quality of life, vision-related quality of life, or costs, outcomes recommended for Cochrane reviews.¹⁶ However, almost half (47.3%) the trials included in our sample were published before or during 2002; outcome selection for these trials likely occurred years earlier. Widespread recognition of the importance of quality of life as an outcome for clinical research is a more recent phenomenon. In recent trials, a possible reason for omission might be the additional resources and expertise needed to rigorously collect and analyze quality-of-life data compared with less subjective outcomes.^20,21

Ocular adverse events were frequent outcomes in trials and reviews that addressed cataract, DR, and glaucoma. Systemic adverse events were common in trials and reviews of DR but only in trials of AMD. However, in the Methods sections of AMD, cataract, and DR reviews, the authors mentioned the intention to examine adverse events without providing further detail; we therefore denoted these as adverse events (unspecified). Adverse events might be approached differently in trials and reviews. Trialists are often subject to strict regulations regarding reporting of individual adverse events, especially if the adverse events are severe and even if unrelated to the treatment. However, reviewers might consider specific adverse events to be of little interest if they are not a priori known to be associated with the treatment. Moreover, in ophthalmology, reviews generally have identified few trials, and the included trials often have small sample sizes and/or short follow-up durations; thus, low numbers of detected adverse events are reported.²² Reviews and meta-analyses in this field consequently do not often achieve sufficient power to make conclusions regarding specific adverse events.²²

Many of the 7 most frequent outcomes in trials but not in reviews were anatomical outcomes, such as retinal thickness and vitreous hemorrhage (in DR), choroidal neovascular membrane size (in AMD), and posterior capsular opacification (in cataract). Although Cochrane reviewers are encouraged to include patient-centered and functional outcomes, there may still be a need to continue examining anatomical outcomes in reviews.

Our current findings in ophthalmology are consistent with recent findings of small overlap in outcomes between Cochrane reviews that address human immunodeficiency virus (HIV) infection and AIDS and the trials included in those reviews.²³ These findings reflect discord among reviewers and trialists addressing the same disease, in addition to the increasing evidence of the inconsistency in outcome use among trials. Other systematic investigations of trials that addressed HIV infection and AIDS,²³ tinnitus,²⁴ cardiac arrest,²⁵ and critical care²⁶ have also demonstrated the absence of a single outcome that was reported across all trials. The proportion of outcomes reported in only 1 trial each has been reported to be high, ranging from 41% to 70%, for HIV infection and AIDS,²³ glaucoma,¹³ cardiothoracic surgery,²⁷ and audiology.²⁸ Multiplicity in outcomes can serve a purpose. It may represent the intention of trialists to capture nontraditional outcomes and can lead to new hypotheses and deeper understanding of potential effects of interventions on disease processes. However, when multiplicity in outcomes occurs to an extent that precludes reviews from achieving their goal (ie, combining results from trials), as in the example of the review¹² comparing nonsteroidal anti-inflammatory drugs with corticosteroids for patients with uncomplicated cataract surgery, evidence-based medicine may be undermined.

The small overlap in outcomes in trials and reviews highlights the urgent need to harmonize outcomes in ophthalmology. The Core Outcome Measures for Effectiveness Trials (COMET) and Outcome Measures in Rheumatology (OMERACT) initiatives have promoted consistency in outcome use, thereby aiming to facilitate meaningful comparisons across studies within specific disease areas.^29,30 These efforts have fostered core outcome set development in various fields.⁶ Core outcome sets refer to the minimum set of outcomes that must be measured in all clinical trials that address a given topic.²⁹

In ophthalmology, we are aware of available core outcome sets for AMD,^31,32 cataract,³³ cataract surgery,³⁴ glaucoma,³⁵ juvenile idiopathic arthritis–associated uveitis,³⁶ and thyroid eye disease.³⁷ The 7 most frequent outcomes in our sample of AMD trials and reviews include 3 outcomes (visual acuity, ocular adverse events, and vision-related quality of life) in common with one of the available AMD core outcome sets³¹ and 2 outcomes (visual acuity and ocular adverse events) in common with the other.³² For cataract, the available core outcome set includes 4 outcomes,³³ of which 3 (visual acuity, ocular adverse events, and vision-related quality of life) are common to the 7 most frequent outcomes in our sample of trials and reviews. Similarly, for glaucoma, the available core outcome set includes 4 outcomes,³⁵ of which 3 (intraocular pressure, visual field, and ocular adverse events) are common to the 7 most frequent outcomes in our sample of trials and reviews. In addition to published core outcome sets, core outcome sets are being developed for AMD, uveitis, DR, visual impairment after stroke, amblyopia, strabismus, and ocular motility.²⁹ To achieve greater consistency in outcomes, those developing core outcome sets should consider the views and priorities of all relevant stakeholders, including patients, clinicians, and others.

COMET suggests that core outcome set development should begin with a comprehensive review of the literature, including trials and reviews.⁴ We previously tested a framework for this approach for outcomes in trials and reviews that addressed HIV infection and AIDS.^23,38 Macefield and colleagues³⁹ also used a similar framework while identifying patient-reported core outcomes for esophageal cancer.

Our study has some limitations. First, we focused on 1 of the 5 elements of an outcome (ie, the domain). Therefore, the identification of overlap in outcomes that we found implies only that various researchers examining the same disease might be performing similar assessments and not that the reported results can be combined in meta-analyses. In ongoing work, we are exploring the specific overlapping outcome domains to establish whether the overlap represents the same outcome measured using the same measurement and with data aggregation and analyses performed in the same way at the same time point. If such overlap is not present, the inconsistency of outcomes may be greater than we have reported. Second, we excluded trials only reported in conference abstracts; therefore, some outcomes from unpublished trials may have been missed. Third, our study focused on Cochrane reviews. It is possible that the overlap in outcomes between trials and non-Cochrane reviews might be systematically different from the overlap reported in this article.

We compared all outcomes in all Cochrane reviews that addressed the 4 most prevalent eye diseases with outcomes in the trials included in those reviews. Although most review outcomes were reported in the trials, most trial outcomes were not reported in the reviews. Inconsistency in trial outcomes may impede research synthesis efforts and indicates the need for disease-specific core outcome sets in ophthalmology.

Corresponding Author: Ian J. Saldanha, MBBS, MPH, PhD, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe St, Room E6035, Baltimore, MD 21205 (isaldan1@jhmi.edu).

Accepted for Publication: June 9, 2017.

Published Online: August 3, 2017. doi:10.1001/jamaophthalmol.2017.2583

Author Contributions: Dr Saldanha had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Saldanha, Dickersin.

Acquisition, analysis, or interpretation of data: Saldanha, Lindsley, Do, Chuck, Meyerle, Jones, Coleman, Jampel, Virgili.

Drafting of the manuscript: Saldanha, Virgili.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Saldanha, Virgili.

Obtained funding: Chuck, Dickersin.

Administrative, technical, or material support: Lindsley, Chuck, Meyerle.

Study supervision: Saldanha.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: This study was funded by grant EY020140 from the National Institutes of Health and grant EY020522 from Cochrane Eyes and Vision (Dr Dickersin).

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.

Additional Contributions: Sueko Ng, MHS, Cochrane Eyes and Vision, assisted with data abstraction. She was financially compensated through the Cochrane Eyes and Vision grant for her work on this project.