[Skip to Navigation]
Sign In
Table 1.  Summary of the Evidence Base for ATS Clinical Practice Guidelines
Summary of the Evidence Base for ATS Clinical Practice Guidelines
Table 2.  Summary of Evidence Base by Guideline
Summary of Evidence Base by Guideline
1.
Wyatt  KD, Stuart  LM, Brito  JP,  et al.  Out of context: clinical practice guidelines and patients with multiple chronic conditions: a systematic review.  Med Care. 2014;52(suppl 3):S92-S100. doi:10.1097/MLR.0b013e3182a51b3dPubMedGoogle ScholarCrossref
2.
Makam  AN, Nguyen  OK.  An evidence-based medicine approach to antihyperglycemic therapy in diabetes mellitus to overcome overtreatment.  Circulation. 2017;135(2):180-195. doi:10.1161/CIRCULATIONAHA.116.022622PubMedGoogle ScholarCrossref
3.
American Thoracic Society. ATS Official Documents.; https://www.thoracic.org/statements/index.php. Reviewed October 2017. Accessed August 1, 2017.
4.
Atkins  D, Best  D, Briss  PA,  et al; GRADE Working Group.  Grading quality of evidence and strength of recommendations.  BMJ. 2004;328(7454):1490. doi:10.1136/bmj.328.7454.1490PubMedGoogle ScholarCrossref
5.
Neuman  MD, Goldstein  JN, Cirullo  MA, Schwartz  JS.  Durability of class I American College of Cardiology/American Heart Association clinical practice guideline recommendations.  JAMA. 2014;311(20):2092-2100. doi:10.1001/jama.2014.4949PubMedGoogle ScholarCrossref
6.
Makam  AN, Auerbach  AD, Steinman  MA.  Blood culture use in the emergency department in patients hospitalized for community-acquired pneumonia.  JAMA Intern Med. 2014;174(5):803-806. doi:10.1001/jamainternmed.2013.13808PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Research Letter
    February 18, 2019

    Evidence-Based Medicine and the American Thoracic Society Clinical Practice Guidelines

    Author Affiliations
    • 1Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas
    • 2Department of Clinical Sciences, University of Texas Southwestern Medical Center, Dallas
    • 3Division of Hospital Medicine, Zuckerberg San Francisco General Hospital, University of California, San Francisco
    JAMA Intern Med. 2019;179(4):584-586. doi:10.1001/jamainternmed.2018.7461

    The American Thoracic Society (ATS) issues clinical practice guidelines for the care of patients with pulmonary and critical care disease. The utility of ATS guidelines depends on the quality of the evidence base underpinning recommendations and whether the guidelines permit the practice of evidence-based medicine (EBM).1,2 However, the extent to which ATS guidelines are substantiated by high-quality evidence and can be used to promote EBM is unknown.

    Methods

    Two of 3 investigators (R.C.S., K.D., and A.N.M.) reviewed each ATS clinical practice guideline recommendations listed on the ATS website as of August 1, 2017, that pertained to adults.3 We abstracted the following domains necessary for evidence-based clinical decision making based on prior conceptual frameworks1,2: recommendation type, recommendation strength (using the Grading of Recommendations Assessment, Development, and Evaluation [GRADE] scoring system4 of strong [benefits clearly outweigh risks in most patients] vs low/conditional [benefits do not clearly outweigh risks in a substantial minority of patients]), quality of evidence (using GRADE categories4 of high [further research is unlikely to change estimate of effect] to very low [any estimate of effect is uncertain]), EBM measures, and patient context. Institutional review board approval was not needed because no human participants were included.

    We defined recommendations as including the basic set of EBM measures if they included at least 1 measure of test performance for diagnostic recommendations (sensitivity, specificity, or likelihood ratio) and at least 1 measure of absolute benefit or harm for therapeutic recommendations (absolute risk reduction/increase, number needed to treat/harm, or relative risk with incidence of the outcome for the control group). For patient context, we ascertained whether the narrative text included any discussion of a person’s severity of illness or comorbidities, sociopersonal context, prognosis, or personal preference and how these domains might influence the recommendation.1 Differences between reviewers were resolved through negotiated consensus, aiming to achieve agreement using the most inclusive definitions. Two-sided P < .05 for descriptive statistics indicated significance.

    Results

    Among 222 unique recommendations from 16 separate guidelines, 141 (63.5%) were based on low-quality evidence, whereas fewer than 1 in 10 (19 [8.6%]) were based on high-quality evidence (Table 1). Nonetheless, 86 (38.7%) were designated strong recommendations. Higher quality of evidence was associated with an increased probability of receiving a strong recommendation; 29 of 141 low-quality evidence recommendations (20.6%), 41 of 62 moderate-quality evidence recommendations (66.1%), and 16 of 19 high-quality evidence recommendations (84.2%) were strongly recommended (P < .001 for trend). However, most strong recommendations were not supported by high-quality evidence (16 of 86 [18.6%]).

    Of 52 diagnostic testing recommendations, 26 (50.0%) presented the test’s sensitivity, specificity, or likelihood ratios. Of 165 therapeutic recommendations, 76 (46.1%) reported the treatment’s absolute benefits or harms. Overall, 101 recommendations (45.5%) included any discussion of patient context, and this did not differ by recommendation strength (61 [44.9%] for low/conditional vs 40 [46.5%] for strong recommendations; P = .81) or quality of evidence (65 [46.1%] for low-, 29 [46.8%] for medium-, and 7 [36.8%] for high-quality recommendations; P = .73). The most commonly discussed domains were severity of illness or comorbidities (84 [37.8%]) and sociopersonal context (51 [23.0%]). Prognosis (14 [6.3%]) and patient preference (3 [1.4%]) were rarely discussed.

    Guidelines varied considerably in number, evidence base, and strength of recommendations (Table 2). The community-acquired pneumonia guideline had the most recommendations (n = 47), had the highest proportion supported by high-quality evidence (15 [31.9%]), and accounted for 15 of 19 total recommendations supported by high-quality evidence.

    Discussion

    The ATS clinical practice guidelines are supported by suboptimal evidence and often are not presented in a manner suitable to optimize care for individual patients. Although 38.7% of recommendations are strong, fewer than 1 in 10 are supported by high-quality evidence (ie, a randomized controlled trial or a meta-analysis). Apart from community-acquired pneumonia, only 4 recommendations were supported by high-quality evidence among the 175 recommendations for all other pulmonary and critical care conditions combined.

    The main limitation of our study was that we overestimated the frequency that recommendations included EBM measures and patient context domains, because we did not require meaningful and complete information in these areas. For example, if a recommendation mentioned consideration of costs, we credited the recommendation for including discussion of a patient’s sociopersonal context.

    Our study has several implications. First, most ATS guidelines should be cautiously applied and should not be considered standard of care, given the paucity of high-quality evidence. Future randomized clinical trials could lead to reversal of many recommendations, even for tests and therapies that are currently strongly recommended.5 Further, standardizing poorly substantiated diagnostic tests or treatments can lead to inappropriate and harmful care.6 Second, future ATS guideline committees should strive to adhere to best practices for EBM, which include a transparent discussion of diagnostic test characteristics, treatment effectiveness, and relevant patient context domains to help guide clinical decision making.2 Third, our framework can be used by guideline development checklists and methodology reports to ensure future guidelines adheres to EBM principles.

    Back to top
    Article Information

    Accepted for Publication: November 4, 2018.

    Published Online: February 18, 2019. doi:10.1001/jamainternmed.2018.7461

    Correction: This article was corrected on May 28, 2019, to fix a an author’s surname in the byline.

    Corresponding Author: Anil N. Makam, MD, MAS, Division of Hospital Medicine, Zuckerberg San Francisco General Hospital, University of California, San Francisco, 1001 Potrero Ave, PO Box 0862, San Francisco, CA 94110 (anil.makam@ucsf.edu).

    Author Contributions: Dr Makam had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Schumacher, Makam.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Schumacher, Deshpande, Makam.

    Critical revision of the manuscript for important intellectual content: Schumacher, Nguyen, Makam.

    Statistical analysis: Schumacher, Deshpande, Makam.

    Administrative, technical, or material support: Nguyen, Deshpande, Makam.

    Supervision: Makam.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: This study was supported by grant K23HL133441 from the National Heart, Lung, and Blood Institute (Dr Nguyen) and grant K23AG052603 from the National Institutes on Aging (Dr Makam).

    Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    References
    1.
    Wyatt  KD, Stuart  LM, Brito  JP,  et al.  Out of context: clinical practice guidelines and patients with multiple chronic conditions: a systematic review.  Med Care. 2014;52(suppl 3):S92-S100. doi:10.1097/MLR.0b013e3182a51b3dPubMedGoogle ScholarCrossref
    2.
    Makam  AN, Nguyen  OK.  An evidence-based medicine approach to antihyperglycemic therapy in diabetes mellitus to overcome overtreatment.  Circulation. 2017;135(2):180-195. doi:10.1161/CIRCULATIONAHA.116.022622PubMedGoogle ScholarCrossref
    3.
    American Thoracic Society. ATS Official Documents.; https://www.thoracic.org/statements/index.php. Reviewed October 2017. Accessed August 1, 2017.
    4.
    Atkins  D, Best  D, Briss  PA,  et al; GRADE Working Group.  Grading quality of evidence and strength of recommendations.  BMJ. 2004;328(7454):1490. doi:10.1136/bmj.328.7454.1490PubMedGoogle ScholarCrossref
    5.
    Neuman  MD, Goldstein  JN, Cirullo  MA, Schwartz  JS.  Durability of class I American College of Cardiology/American Heart Association clinical practice guideline recommendations.  JAMA. 2014;311(20):2092-2100. doi:10.1001/jama.2014.4949PubMedGoogle ScholarCrossref
    6.
    Makam  AN, Auerbach  AD, Steinman  MA.  Blood culture use in the emergency department in patients hospitalized for community-acquired pneumonia.  JAMA Intern Med. 2014;174(5):803-806. doi:10.1001/jamainternmed.2013.13808PubMedGoogle ScholarCrossref
    ×