Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Schumacher RC, Nguyen OK, Deshpande K, Makam AN. Evidence-Based Medicine and the American Thoracic Society Clinical Practice Guidelines. JAMA Intern Med. 2019;179(4):584–586. doi:10.1001/jamainternmed.2018.7461
The American Thoracic Society (ATS) issues clinical practice guidelines for the care of patients with pulmonary and critical care disease. The utility of ATS guidelines depends on the quality of the evidence base underpinning recommendations and whether the guidelines permit the practice of evidence-based medicine (EBM).1,2 However, the extent to which ATS guidelines are substantiated by high-quality evidence and can be used to promote EBM is unknown.
Two of 3 investigators (R.C.S., K.D., and A.N.M.) reviewed each ATS clinical practice guideline recommendations listed on the ATS website as of August 1, 2017, that pertained to adults.3 We abstracted the following domains necessary for evidence-based clinical decision making based on prior conceptual frameworks1,2: recommendation type, recommendation strength (using the Grading of Recommendations Assessment, Development, and Evaluation [GRADE] scoring system4 of strong [benefits clearly outweigh risks in most patients] vs low/conditional [benefits do not clearly outweigh risks in a substantial minority of patients]), quality of evidence (using GRADE categories4 of high [further research is unlikely to change estimate of effect] to very low [any estimate of effect is uncertain]), EBM measures, and patient context. Institutional review board approval was not needed because no human participants were included.
We defined recommendations as including the basic set of EBM measures if they included at least 1 measure of test performance for diagnostic recommendations (sensitivity, specificity, or likelihood ratio) and at least 1 measure of absolute benefit or harm for therapeutic recommendations (absolute risk reduction/increase, number needed to treat/harm, or relative risk with incidence of the outcome for the control group). For patient context, we ascertained whether the narrative text included any discussion of a person’s severity of illness or comorbidities, sociopersonal context, prognosis, or personal preference and how these domains might influence the recommendation.1 Differences between reviewers were resolved through negotiated consensus, aiming to achieve agreement using the most inclusive definitions. Two-sided P < .05 for descriptive statistics indicated significance.
Among 222 unique recommendations from 16 separate guidelines, 141 (63.5%) were based on low-quality evidence, whereas fewer than 1 in 10 (19 [8.6%]) were based on high-quality evidence (Table 1). Nonetheless, 86 (38.7%) were designated strong recommendations. Higher quality of evidence was associated with an increased probability of receiving a strong recommendation; 29 of 141 low-quality evidence recommendations (20.6%), 41 of 62 moderate-quality evidence recommendations (66.1%), and 16 of 19 high-quality evidence recommendations (84.2%) were strongly recommended (P < .001 for trend). However, most strong recommendations were not supported by high-quality evidence (16 of 86 [18.6%]).
Of 52 diagnostic testing recommendations, 26 (50.0%) presented the test’s sensitivity, specificity, or likelihood ratios. Of 165 therapeutic recommendations, 76 (46.1%) reported the treatment’s absolute benefits or harms. Overall, 101 recommendations (45.5%) included any discussion of patient context, and this did not differ by recommendation strength (61 [44.9%] for low/conditional vs 40 [46.5%] for strong recommendations; P = .81) or quality of evidence (65 [46.1%] for low-, 29 [46.8%] for medium-, and 7 [36.8%] for high-quality recommendations; P = .73). The most commonly discussed domains were severity of illness or comorbidities (84 [37.8%]) and sociopersonal context (51 [23.0%]). Prognosis (14 [6.3%]) and patient preference (3 [1.4%]) were rarely discussed.
Guidelines varied considerably in number, evidence base, and strength of recommendations (Table 2). The community-acquired pneumonia guideline had the most recommendations (n = 47), had the highest proportion supported by high-quality evidence (15 [31.9%]), and accounted for 15 of 19 total recommendations supported by high-quality evidence.
The ATS clinical practice guidelines are supported by suboptimal evidence and often are not presented in a manner suitable to optimize care for individual patients. Although 38.7% of recommendations are strong, fewer than 1 in 10 are supported by high-quality evidence (ie, a randomized controlled trial or a meta-analysis). Apart from community-acquired pneumonia, only 4 recommendations were supported by high-quality evidence among the 175 recommendations for all other pulmonary and critical care conditions combined.
The main limitation of our study was that we overestimated the frequency that recommendations included EBM measures and patient context domains, because we did not require meaningful and complete information in these areas. For example, if a recommendation mentioned consideration of costs, we credited the recommendation for including discussion of a patient’s sociopersonal context.
Our study has several implications. First, most ATS guidelines should be cautiously applied and should not be considered standard of care, given the paucity of high-quality evidence. Future randomized clinical trials could lead to reversal of many recommendations, even for tests and therapies that are currently strongly recommended.5 Further, standardizing poorly substantiated diagnostic tests or treatments can lead to inappropriate and harmful care.6 Second, future ATS guideline committees should strive to adhere to best practices for EBM, which include a transparent discussion of diagnostic test characteristics, treatment effectiveness, and relevant patient context domains to help guide clinical decision making.2 Third, our framework can be used by guideline development checklists and methodology reports to ensure future guidelines adheres to EBM principles.
Accepted for Publication: November 4, 2018.
Published Online: February 18, 2019. doi:10.1001/jamainternmed.2018.7461
Correction: This article was corrected on May 28, 2019, to fix a an author’s surname in the byline.
Corresponding Author: Anil N. Makam, MD, MAS, Division of Hospital Medicine, Zuckerberg San Francisco General Hospital, University of California, San Francisco, 1001 Potrero Ave, PO Box 0862, San Francisco, CA 94110 (email@example.com).
Author Contributions: Dr Makam had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Schumacher, Makam.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Schumacher, Deshpande, Makam.
Critical revision of the manuscript for important intellectual content: Schumacher, Nguyen, Makam.
Statistical analysis: Schumacher, Deshpande, Makam.
Administrative, technical, or material support: Nguyen, Deshpande, Makam.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported by grant K23HL133441 from the National Heart, Lung, and Blood Institute (Dr Nguyen) and grant K23AG052603 from the National Institutes on Aging (Dr Makam).
Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.