aFive criteria used for classifying reliability of systematic reviews.
bThe denominator was 92 for systematic reviews with defined eligibility criteria.
cThe denominator was 54 for systematic reviews reporting 2 or more title/abstract screeners.
dThe denominator was 50 for systematic reviews reporting 2 or more methodologic quality assessors.
eThe denominator was 56 for systematic reviews reporting 2 or more data abstractors.
fThe denominator was 97 for systematic reviews including at least 1 primary study.
gThe denominator was 59 for systematic reviews reporting at least 1 meta-analysis.
eMethods. Search strategies for identifying systematic reviews in eyes and vision research
eTable 1. 2011 Academy for Ophthalmology (AAO) Preferred Practice Patterns (PPP) Table of Content for Cataract in the Adult Eye
eTable 2. Characteristics of 46 reliable systematic reviews on the management of cataract in the adult eye
eTable 3. Characteristics of 53 unreliable systematic reviews on the management of cataract in the adult eye
eTable 4. Management categories in the 2011 PPP with evidence gaps
eFigure. Assessment of reliability of 99 systematic reviews on the management of cataract in the adult eye by Cochrane affiliation
Customize your JAMA Network experience by selecting one or more topics from the list below.
Golozar A, Chen Y, Lindsley K, et al. Identification and Description of Reliable Evidence for 2016 American Academy of Ophthalmology Preferred Practice Pattern Guidelines for Cataract in the Adult Eye. JAMA Ophthalmol. 2018;136(5):514–523. doi:10.1001/jamaophthalmol.2018.0786
Which systematic reviews are reliable to inform the update of the American Academy of Ophthalmology’s Preferred Practice Pattern guidelines on cataract in the adult eye?
In a cross-sectional study, 99 systematic reviews on cataract in the adult eye were identified, and 46 (46%) were classified as reliable using prespecified criteria. All 46 reliable reviews were cited in the 2016 update of the Preferred Practice Pattern guidelines.
The partnership between Cochrane Eyes and Vision US Satellite and the American Academy of Ophthalmology facilitated access to reliable systematic review evidence to support the development of clinical practice guidelines for cataract in the adult eye.
Trustworthy clinical practice guidelines require reliable systematic reviews of the evidence to support recommendations. Since 2016, the American Academy of Ophthalmology (AAO) has partnered with Cochrane Eyes and Vision US Satellite to update their guidelines, the Preferred Practice Patterns (PPP).
To describe experiences and findings related to identifying reliable systematic reviews that support topics likely to be addressed in the 2016 update of the 2011 AAO PPP guidelines on cataract in the adult eye.
Design, Setting, and Participants
Cross-sectional study. Systematic reviews on the management of cataract were searched for in an established database. Each relevant systematic review was mapped to 1 or more of the 24 management categories listed under the Management section of the table of contents of the 2011 AAO PPP guidelines. Data were extracted to determine the reliability of each systematic review using prespecified criteria, and the reliable systematic reviews were examined to find whether they were referenced in the 2016 AAO PPP guidelines. For comparison, we assessed whether the reliable systematic reviews published before February 2010 the last search date of the 2011 AAO PPP guidelines were referenced in the 2011 AAO PPP guidelines. Cochrane Eyes and Vision US Satellite did not provide systematic reviews to the AAO during the development of the 2011 AAO PPP guidelines.
Main Outcomes and Measures
Systematic review reliability was defined by reporting eligibility criteria, performing a comprehensive literature search, assessing methodologic quality of included studies, using appropriate methods for meta-analysis, and basing conclusions on review findings.
From 99 systematic reviews on management of cataract, 46 (46%) were classified as reliable. No evidence that a comprehensive search had been conducted was the most common reason a review was classified as unreliable. All 46 reliable systematic reviews were cited in the 2016 AAO PPP guidelines, and 8 of 15 available reliable reviews (53%) were cited in the 2011 PPP guidelines.
Conclusions and Relevance
The partnership between Cochrane Eyes and Vision US Satellite and the AAO provides the AAO access to an evidence base of relevant and reliable systematic reviews, thereby supporting robust and efficient clinical practice guidelines development to improve the quality of eye care.
To decide among treatment options, health care professionals benefit from consistent guidance provided in clinical practice guidelines (CPGs).1 The Institute of Medicine standards for developing trustworthy CPGs recommend using evidence from high-quality systematic reviews to inform guideline recommendations.1 Guideline developers can perform systematic reviews themselves, or they can partner with groups specializing in evidence synthesis.1
The American Academy of Ophthalmology (AAO) partners with the Cochrane Eyes and Vision US Satellite (CEV@US) to update the Preferred Practice Pattern (PPP) guidelines.2 The Cochrane Eyes and Vision Editorial Base is located at the London School of Hygiene and Tropical Medicine in London, England. Activities based in the United States are coordinated by the CEV@US at Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. The National Eye Institute has supported CEV@US since 2002. Cochrane Eyes and Vision aims to prepare and promote access to systematic reviews of interventions for preventing or treating eye conditions and/or visual impairment and helping people adjust to visual impairment or blindness.
Cochrane Eyes and Vision US Satellite was tasked to identify up-to-date, reliable systematic reviews that can be used to inform the updates to the AAO PPP guidelines. In this article, we describe our experiences and findings related to identifying reliable systematic reviews that support topics of interest that were likely to be addressed in the 2016 update of the 2011 AAO PPP guidelines on cataract in the adult eye. The PPP guidelines for cataract in the adult eye were the first collaboration of this type between the AAO and CEV@US.
We included full-text journal articles and reports that claimed to be systematic reviews or meta-analyses anywhere in the text. We also included reports that met the definition of a systematic review or a meta-analysis when these terms were not used, as defined by the Institute of Medicine.3 For the 2016 AAO PPP guidelines for cataract in the adult eye, a systematic review was selected and evaluated when it addressed interventions for treating cataract and could be mapped to 1 of the 24 management categories covered in the Management section of the table of contents of the previous version of the guidelines, the 2011 AAO PPP guidelines for cataract in the adult eye (eTable 1 in the Supplement). The Management section was broadly divided into Nonsurgical Management and Surgical Management.
Cochrane Eyes and Vision US Satellite maintains a database of systematic reviews and meta-analyses in vision research and eye care in EndNote (Clarivate Analytics). The initial search for systematic reviews was conducted in March 2007; the search was updated in September 2009, April 2012, May 2014, and March 2016 (full search strategy available in eMethods in the Supplement).4,5 For systematic reviews retrieved up to the 2012 search, 2 people independently identified eligible cataract reviews from the search results. For systematic reviews retrieved in 2014 and 2016 searches, 1 person (Y.C. in 2014 and A.G. in 2016) identified eligible cataract reviews; eligibility was verified by a senior member of the team (B.S.H.). Differences of opinion were resolved through discussion or by a third team member when necessary.
We mapped each relevant systematic review to 1 or more management categories listed in the Management section of the table of contents of the 2011 AAO PPP guidelines. A systematic review could be mapped to multiple categories. One person (A.G.) initially mapped all reviews; a senior member of the team (B.S.H.) verified the mapping. Differences of opinion were resolved through discussion.
We adapted a data extraction form from one used by our team in previous studies.6-8 Data items on the forms came from the Critical Appraisal Skills Programme,9 the Assessment of Multiple Systematic Reviews,10 and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses.11 We pilot-tested the form and maintained the form for data entry in the Systematic Review Data Repository.12,13 Data extracted from each review included the characteristics of the systematic review (eg, the objectives, study population, interventions compared, outcomes examined, number of studies included, key findings). We also recorded the methods used for systematic reviews, including the search strategy, the number of people involved in various steps of the systematic review process, whether risk of bias in the studies included in the review had been assessed, and how meta-analysis was conducted. Pairs of individuals (A.G., Y.C., K.L., B.R., B.S.H., T.L., and other nonauthor researchers) independently extracted data from eligible reviews and resolved discrepancies through discussion.
By comparing the extracted data with published criteria,3,14,15 we classified a systematic review method as reliable when the authors of the systematic review had (1) reported eligibility criteria for including studies, (2) conducted a comprehensive literature search for eligible studies, (3) assessed risk of bias of included studies, (4) used appropriate methods for meta-analyses if meta-analysis was reported, and (5) drawn conclusions that were supported by the review findings. Definitions of the reliability assessment criteria are given in Table 1.8,9,14,16 We considered a systematic review unreliable when 1 or more of these criteria were not met. Our classifications were based on the methods reported in the review; we did not contact review authors to obtain additional information regarding our assessment criteria.
We tabulated the characteristics and our reliability classification of the eligible reviews. We compared review characteristics and the reliability assessment for systematic reviews published in the Cochrane Database of Systematic Reviews (Cochrane reviews) with those of systematic reviews published elsewhere (non-Cochrane reviews).
We sent citations for all reliable systematic reviews, mapped to the Management section of the table of contents of the 2011 AAO PPP guidelines, along with characteristics of the reliable reviews to the AAO PPP cataract panel in February 2016 and an updated list in June 2016. After publication of the 2016 AAO PPP guidelines in October 2016,2 we examined whether the reliable systematic reviews we contributed were referenced in the 2016 AAO PPP guidelines. For comparison, we assessed whether the reliable systematic reviews published before February 2010, the last search date of the 2011 AAO PPP guidelines, had been referenced in the 2011 AAO PPP guidelines, which were developed without the same degree of CEV@US participation.
Of 1863 systematic reviews on eyes and vision in our database as of March 2016, we identified 99 that evaluated management strategies for cataract in the adult eye and were eligible for this project (Figure 1).17-115 The earliest review was published in 1994, but more than half were published after 2012 (Table 2). Twenty of 99 reviews (20%) were published by CEV authors in The Cochrane Database of Systematic Reviews.20,22,27,30,39,41,44,45,50,52,60,64,67,75,80,87,88,93,98,111 Of the 79 non-Cochrane reviews,17-19,21,23-26,28,29,31-38,40,42,43,46-49,51,54-59,61-63,65,66,68-74,76-79,81-86,89-92,94-97,99-110,112-115 50 (63%) were published in specialty medical journals (eg, Ophthalmology). Intraocular lenses (IOL) implantation was the most commonly examined intervention (25 of 99 [25%]), followed by phacoemulsification (20 of 99 [20%]). In terms of outcomes, 59 of 99 reviews (60%) assessed visual acuity and 53 (54%) examined safety (Table 2).
Of the 99 systematic reviews assessed, 46 (46%) were classified as reliable, and the remaining 53 (54%) were classified as unreliable (Figure 2 and eTables 2 and 3 in the Supplement). Most of the unreliable systematic reviews (43 of 53 [81%]) fell short of more than 1 reliability assessment criterion (eTable 3 in the Supplement). Lack of reporting a comprehensive search (38 of 53 [71%]) was the most frequent reason for classifying a systematic review as unreliable.
Among all 99 cataract systematic reviews, the median (interquartile range) number of bibliographic databases searched per review was 4 (2-5). Almost all the authors of the systematic reviews searched PubMed (95 of 96 [96%]), and 68% (67 of 99) searched the Cochrane Central Register of Controlled Trials (Table 2). Thirty-nine reviews (38%) reported government funding, and 18 reviews (18%) reported foundation funding. Only 5 reviews (5%) reported receiving funding from industry. The median number of studies included in these systematic reviews was 12 (interquartile range, 7-25). Fifty-eight reviews (59%) reported on the number of participants included, and 17 reviews (17%) reported on the number of eyes included. Among reviews reporting on the number of participants and eyes, the median numbers included were 1313 (interquartile range, 655-4292) and 1573 (interquartile range, 722-3800), respectively.
Compared with reliable systematic reviews, unreliable reviews less often had been developed with government funding (16 [30%] vs 23 [50%]) and more often with funding from industry (5 [9%] vs 0%) (Table 2). All 20 Cochrane reviews were classified as reliable; they accounted for 43% (n = 20) of all reliable reviews. Compared with non-Cochrane reviews, Cochrane reviews more often had 2 or more review authors who independently selected studies, assessed risk of bias, and extracted data. Neither Cochrane nor non-Cochrane reviews scored well with respect to discussing limitations of the review at the review level (eg, incomplete retrieval of relevant studies, the potential effect of reporting bias on the review findings) (eFigure in the Supplement).
Twenty-seven of 46 reliable reviews (54%) reported that the intervention evaluated in the review was effective. Other reviews reported inconclusive findings (12 [26%]) or reported insufficient evidence of an intervention effect (7 [15%]) (eTable 2 in the Supplement). In about one-third of these 46 reviews (15 of 46 [33%]), different groups of reviewers had examined the same intervention. Among reviews investigating the same intervention, not all reported consistent findings. For example, 4 reviews compared sharp-edged IOL with round-edged IOL, and all favored sharp-edged IOLs over round-edged IOLs.24,45,68,76 As another example, in comparisons of manual small incision cataract surgery vs extracapsular cataract extraction, Ang et al22 reported inconclusive findings, but Riaz et al88 found manual small incision cataract surgery was more effective than extracapsular cataract extraction. This discordance may be due to the different visual acuity outcomes used to reach their conclusions: Ang et al22 used best-corrected visual acuity, and Riaz et al88 used uncorrected visual acuity. We provided the AAO with both reviews so that the panelists could decide whether and how the discordant findings affected their recommendations.
We sent references for the reliable systematic reviews to the AAO cataract guideline panel for use in updating the 2016 AAO PPP guidelines for cataract in the adult eye. These reviews were mapped to 18 of the 24 management categories (75%) of the table of contents of the 2011 AAO PPP guidelines. We did not identify any reliable systematic review for 6 of the 24 management categories (25%): indications for surgery, contraindications to surgery, biometry and IOL power calculation, toxic anterior segment syndrome, cataract surgery checklist, and discharge from surgical facility. There may be a need for randomized clinical trials and systematic reviews in these areas (eTable 4 in the Supplement). We identified reliable systematic reviews for 2 topics that were not covered in the 2011 AAO PPP guidelines: prophylaxis for cystoid macular edema after cataract surgery90,94,104,109 and timing for cataract surgery.38,54 All 46 reliable systematic reviews were cited in the 2016 AAO PPP guidelines. In contrast, before the close partnership between CEV@US and AAO was established, only 8 of 15 reliable systematic reviews (53%) available at that time were cited in the 2011 AAO PPP guidelines. Although CEV@US has never sent unreliable systematic reviews to the AAO PPP panel, we noticed that the 2016 AAO PPP guidelines cited 16 unreliable systematic reviews in various contexts. Two unreliable systematic reviews were used to inform treatment recommendations.51,113
We contributed 46 reliable systematic reviews, nearly half of all 99 eligible reviews we had identified, to the guideline panel charged with preparing the 2016 update of the AAO PPP guidelines on the management of cataract in the adult eye. Most of the unreliable reviews fell short on more than 1 methodologic criterion. No evidence of a comprehensive literature search was the most common reason for classifying a review as unreliable. Cochrane reviews constituted a fifth of all identified reviews and were all classified as reliable.
Achieving evidence-based health care involves an intense effort. First, evidence must be generated, and then the available evidence must be synthesized in a reliable way. Synthesized evidence must be further translated into policy, often manifested as evidence-based CPGs. Finally, the policy must be applied for the evidence to have an effect on care. It takes a coordinated effort among stakeholders to achieve the collective needs of patients, caregivers, and policy makers. Collaboration between systematic reviewers and guideline developers is necessary to target important topics to be addressed in systematic reviews and to improve the trustworthiness and validity of CPGs,1 as evidenced by the increased reference to reliable evidence in the 2016 AAO PPP guidelines. This close collaboration facilitates active dissemination of systematic review findings. We believe our approach of working directly with guideline developers aligns well with the 5 core areas for change outlined in the 2016 National Academies of Science, Engineering, and Medicine report Making Eye Health a Population Health Imperative: Vision for Tomorrow, specifically, generating evidence to guide policy decisions and evidence-based actions.116
As a result of this project, we identified 6 topics without a reliable published systematic review. Treatment recommendations in these management categories were made on the basis of the findings from individual studies or expert consensus. While some of the studies that supported treatment recommendations were well-designed and widely known randomized clinical trials, reliable systematic reviews of data available from all studies that have addressed the same research question offer a more comprehensive and compelling evidence base than individual studies. Furthermore, systematic reviews are particularly useful for evaluating consistency of findings across all studies of the same research question and for studying outcomes that are rare (eg, adverse events).117 Future collaborations between clinical researchers, CEV@US, and AAO could focus on important clinical questions in which there are a need for evidence generation and/or synthesis.
Our use of the word reliable refers to the reliability of the methods used by the systematic reviewers. The criteria we used to categorize reliable reviews were based on the standards set by the Institute of Medicine and the Methodological Expectations of Cochrane Intervention Reviews.3,15 However, our criteria were more modest than these standards as we used only a few important items from the full set of recommendations for assessing the reliability of reviews.
Consistent with our previous work on glaucoma and age-related macular degeneration7,8 and work in other areas,118-121 we found that a large proportion of published systematic reviews were redundant and unreliable. This constitutes a form of research waste.122 To ensure the production of reliable systematic reviews, methodologic and reporting standards must be followed. Other considerations include ensuring that systematic reviews are conducted by individuals with adequate training, and that reports of systematic reviews are reviewed by peer reviewers and editors knowledgeable in methods and reporting standards for systematic reviews.123 Cochrane Eyes and Vision is partnering with 10 major vision science journals, whereby a CEV methodologist serves as an editor for systematic review manuscripts submitted for publication.124
We believe that reliable reviews are more likely than unreliable reviews to be reproducible and provide high-quality summarized findings from research. The Institute of Medicine recommends that CPGs be based on high-quality systematic reviews.1 These reviews may be de novo reviews conducted or contracted by the guideline developers or already existing reviews. Although developers may want to conduct a new systematic review to ensure inclusion of the most up-to-date primary research or to address the questions deemed most important for the CPG, it may not be efficient or necessary to conduct a new review given the amount of primary literature to be examined for each review. Because most CPGs address a variety of clinically distinct scenarios, it would be an enormous undertaking to conduct a new systematic review for each clinical question, an undertaking likely impossible to complete within realistic time frames. For example, the AAO cataract PPP guidelines address a diverse number of questions, including nonsurgical management, surgical management, and anesthesia for surgery.3 In such cases, it may be preferable to use reliable existing reviews to inform recommendations, as the PPP panel did, and only conduct new systematic reviews of the primary literature when there are no existing reviews or the existing review is dangerously out-of-date.
Our investigation had strengths. The database of systematic reviews and meta-analyses in vision research and eye care was not compiled specifically for the purpose of informing the AAO panel charged with updating the PPP guidelines for management of cataract in adult eyes or any other team of CPG developers. It was assembled, updated, and coded to cover a broad range of eye and vision conditions and to serve researchers and users with diverse interests. Coding of reviews by condition and by area of relevance to cataract management was performed independently by 2 or more members of CEV. Our investigation also has limitations. First, we limited our investigation to systematic reviews published in English and Chinese, which 1 or more of us could read. We did not translate articles written in other languages. In addition, application of our reliability criteria was subjective and may have reflected our connections to Cochrane.
In conclusion, the collaboration between CEV@US and AAO is a positive step toward identifying reliable systematic reviews and using them to inform guideline recommendations. The increased proportion of available and reliable systematic reviews cited by the AAO panel between the 2011 and 2016 versions of the cataract PPP guidelines demonstrated the success of this close interaction between guideline developers and systematic review groups, as emphasized by the Institute of Medicine. We look forward to continuing and expanding this partnership, to partnering with others responsible for developing CPGs and formulating policy, and to identifying areas in need of evidence generation and synthesis.
Accepted for Publication: February 6, 2018.
Corresponding Author: Tianjing Li, MD, PhD, Center for Clinical Trials and Evidence Synthesis, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe St, Room E6011, Baltimore, MD 21205 (email@example.com).
Published Online: April 12, 2018. doi:10.1001/jamaophthalmol.2018.0786
Author Contributions: Drs Golozar and Li had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Golozar, Lindsley, Lum, Li.
Acquisition, analysis, or interpretation of data: Golozar, Chen, Lindsley, Rouse, Musch, Hawkins, Li.
Drafting of the manuscript: Golozar, Chen, Lindsley, Lum, Li.
Critical revision of the manuscript for important intellectual content: Golozar, Lindsley, Rouse, Musch, Hawkins, Li.
Statistical analysis: Golozar, Chen, Lindsley, Hawkins.
Administrative, technical, or material support: Lindsley, Lum, Hawkins, Li.
Study supervision: Lindsley, Li.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Golozar, Ms Chen, Ms Lindsley, Mr Rouse, Dr Hawkins, and Dr Li were affiliated with Cochrane Eyes and Vision US Satellite during the course of this study. Dr Musch is a member of the Cataract and Anterior Segment Preferred Practice Pattern Panel, which wrote the 2016 American Academy of Ophthalmology Cataract in the Adult Eye Preferred Practice Pattern. Dr Lum is the vice president of Quality and Data Science at the American Academy of Ophthalmology. No other disclosures are reported.
Funding/Support: This project was supported by the National Eye Institute, National Institutes of Health, and the US Department of Health and Human Services (grant 1 U01 EY020522, Kay Dickersin, PhD [nonauthor]).
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: We thank Sueko Matsumura Ng, MHS, Johns Hopkins University; Elizabeth Clearfield, MHS, Johns Hopkins University; and Andrew Law, MHS, Johns Hopkins University, for assistance with data acquisition. All received compensation for their work.
Create a personal account or sign in to: