Comparison of Payment Changes and Choosing Wisely Recommendations for Use of Low-Value Laboratory Tests in the United States and Canada | Health Care Economics, Insurance, Payment | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Figure.  Trends in Low-Value Vitamin D Screening and Triiodothyronin Testing Rates Within Each Jurisdiction
Trends in Low-Value Vitamin D Screening and Triiodothyronin Testing Rates Within Each Jurisdiction

Solid lines indicate observed trends, and dashed lines indicate counterfactual trends. OHTAC indicates Ontario Health Technology Advisory Committee.

Table 1.  Sample Demographics for Each Jurisdiction and Measurea
Sample Demographics for Each Jurisdiction and Measurea
Table 2.  Estimated Changes in Use of Low-Value Vitamin D and Triiodothyronine Testing Rates Before and After Choosing Wisely Recommendationsa
Estimated Changes in Use of Low-Value Vitamin D and Triiodothyronine Testing Rates Before and After Choosing Wisely Recommendationsa
1.
Institute of Medicine.  Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: The National Academies Press; 2001.
2.
Levinson  W, Kallewaard  M, Bhatia  RS, Wolfson  D, Shortt  S, Kerr  EA; Choosing Wisely International Working Group.  ‘Choosing Wisely’: a growing international campaign.  BMJ Qual Saf. 2015;24(2):167-174. doi:10.1136/bmjqs-2014-003821PubMedGoogle ScholarCrossref
3.
Berwick  DM, Hackbarth  AD.  Eliminating waste in US health care.  JAMA. 2012;307(14):1513-1516. doi:10.1001/jama.2012.362PubMedGoogle ScholarCrossref
4.
Canadian Institute for Health Information. Unnecessary Care in Canada. Published April 2017. https://www.cihi.ca/sites/default/files/document/choosing-wisely-baseline-report-en-web.pdf. Accessed April 25, 2019.
5.
Bhatia  RS, Levinson  W, Shortt  S,  et al.  Measuring the effect of Choosing Wisely: an integrated framework to assess campaign impact on low-value care.  BMJ Qual Saf. 2015;24(8):523-531. doi:10.1136/bmjqs-2015-004070PubMedGoogle ScholarCrossref
6.
Colla  CH, Morden  NE, Sequist  TD, Schpero  WL, Rosenthal  MB.  Choosing wisely: prevalence and correlates of low-value health care services in the United States.  J Gen Intern Med. 2015;30(2):221-228. doi:10.1007/s11606-014-3070-zPubMedGoogle ScholarCrossref
7.
Schwartz  AL, Landon  BE, Elshaug  AG, Chernew  ME, McWilliams  JM.  Measuring low-value care in Medicare.  JAMA Intern Med. 2014;174(7):1067-1076. doi:10.1001/jamainternmed.2014.1541PubMedGoogle ScholarCrossref
8.
Segal  J, Bridges  J, Chang  H-Y,  et al.  Identifying possible indicators of systematic overuse of health care procedures with claims data.  Med Care. 2014;52(2):157-163. doi:10.1097/MLR.0000000000000052PubMedGoogle ScholarCrossref
9.
Mafi  JN, Russell  K, Bortz  BA, Dachary  M, Hazel  WA  Jr, Fendrick  AM.  Low-cost, high-volume health services contribute the most to unnecessary health spending.  Health Aff (Millwood). 2017;36(10):1701-1704. doi:10.1377/hlthaff.2017.0385PubMedGoogle ScholarCrossref
10.
Bhatia  RS, Bouck  Z, Ivers  NM,  et al.  Electrocardiograms in low-risk patients undergoing an annual health examination.  JAMA Intern Med. 2017;177(9):1326-1333. doi:10.1001/jamainternmed.2017.2649PubMedGoogle ScholarCrossref
11.
Hong  AS, Ross-Degnan  D, Zhang  F, Wharam  JF.  Small Decline in low-value back imaging associated with the ‘Choosing Wisely’ campaign, 2012-14.  Health Aff (Millwood). 2017;36(4):671-679. doi:10.1377/hlthaff.2016.1263PubMedGoogle ScholarCrossref
12.
Rosenberg  A, Agiro  A, Gottlieb  M,  et al.  Early trends among seven recommendations from the Choosing Wisely campaign.  JAMA Intern Med. 2015;175(12):1913-1920. doi:10.1001/jamainternmed.2015.5441PubMedGoogle ScholarCrossref
13.
Brett  J, Maust  DT, Bouck  Z,  et al.  Benzodiazepine use in older adults in the United States, Ontario, and Australia from 2010 to 2016.  J Am Geriatr Soc. 2018;66(6):1180-1185. doi:10.1111/jgs.15292PubMedGoogle ScholarCrossref
14.
Cassel  CK, Guest  JA.  Choosing wisely: helping physicians and patients make smart decisions about their care.  JAMA. 2012;307(17):1801-1802. doi:10.1001/jama.2012.476PubMedGoogle ScholarCrossref
15.
Choosing Wisely. Our Mission. https://www.choosingwisely.org/our-mission/. Accessed July 10, 2019.
16.
Choosing Wisely Canada. The necessary conference. Choosing Wisely Canada’s National Meeting. https://choosingwiselycanada.org/. Accessed July 10, 2019.
17.
Kerr  EA, Kullgren  JT, Saini  SD.  Choosing Wisely: how to fulfill the promise in the next 5 years.  Health Aff (Millwood). 2017;36(11):2012-2018. doi:10.1377/hlthaff.2017.0953PubMedGoogle ScholarCrossref
18.
Ferrari  R, Prosser  C.  Testing vitamin D levels and choosing wisely.  JAMA Intern Med. 2016;176(7):1019-1020. doi:10.1001/jamainternmed.2016.1929PubMedGoogle ScholarCrossref
19.
Naugler  C, Hemmelgarn  B, Quan  H,  et al.  Implementation of an intervention to reduce population-based screening for vitamin D deficiency: a cross-sectional study.  CMAJ Open. 2017;5(1):E36-E39. doi:10.9778/cmajo.20160073PubMedGoogle ScholarCrossref
20.
Felcher  AH, Gold  R, Mosen  DM, Stoneburner  AB.  Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing.  J Am Med Inform Assoc. 2017;24(4):776-780. doi:10.1093/jamia/ocw182PubMedGoogle ScholarCrossref
21.
Petrilli  CM, Henderson  J, Keedy  JM,  et al.  Reducing unnecessary vitamin D screening in an academic health system: what works and when.  Am J Med. 2018;131(12):1444-1448. doi:10.1016/j.amjmed.2018.06.025PubMedGoogle ScholarCrossref
22.
Colla  CH, Mainor  AJ, Hargreaves  C, Sequist  T, Morden  N.  Interventions aimed at reducing use of low-value health services: a systematic review.  Med Care Res Rev. 2017;74(5):507-550. doi:10.1177/1077558716656970PubMedGoogle ScholarCrossref
23.
Choosing Wisely. American Society for Clinical Pathology. Don’t perform population based screening for 25-OH-vitamin D deficiency. Released February 21, 2013. http://www.choosingwisely.org/clinician-lists/american-society-clinical-pathology-population-based-screening-for-vitamin-d-deficiency/. Accessed April 2, 2019.
24.
College of Family Physicians of Canada. Choosing Wisely Canada. Family Medicine. Thirteen things physicians and patients should question. Last updated, July 2019. https://choosingwiselycanada.org/family-medicine/. Accessed March 21, 2019.
25.
Canadian Association of Pathologists. Choosing Wisely Canade. Pathology. Five things physicians and patients should question in pathology. Last updated, June 2017. https://choosingwiselycanada.org/pathology/. Accessed April 2, 2019.
26.
Medical Advisory Secretariat. Clinical utility of vitamin D testing: an evidence-based analysis.  Ont Health Technol Assess Ser. 2010;10(2):1-93.PubMed
27.
Provincial Programs Branch. OHIP-insured vitamin D testing. 2010. http://www.health.gov.on.ca/en/pro/programs/ohip/bulletins/4000/bul4522.pdf. Accessed August 19, 2019.
28.
Truven Health Marketplace Database. MarketScan Commercial Claims and Encounters Research Database. 2009-2015.
29.
Ministry of Health and Long Term Care. Schedule of benefits: physician services under the Health Insurance Act. Amended October 1, 2005. http://www.health.gov.on.ca/en/pro/programs/ohip/sob/physserv/sob_master11062015.pdf. Accessed November 19, 2019.
30.
Ontario Ministry of Health. Ministry of Long-Term Care. Primary care payment models in Ontario. Last modified July 11, 2019. http://www.health.gov.on.ca/en/pro/programs/pcpm/. Accessed November 19, 2019.
31.
Sullivan  T, Gordon  P, Minto  S. Laboratory Services Expert Panel. November 12, 2015. http://www.health.gov.on.ca/en/common/ministry/publications/reports/lab_services/labservices.pdf. Accessed November 19, 2019.
32.
Health Quality Ontario. Quality improvement. My practice: primary care. https://www.hqontario.ca/Quality-Improvement/Guides-Tools-and-Practice-Reports/Primary-Care. Accessed July 10, 2019.
33.
Centers for Medicare & Medicaid Services. Evaluation and Management Services. December 2018. https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/MLN-Publications-Items/CMS1243514.html. Accessed August 19, 2019.
34.
Choosing Wisely. Endocrine Society. Don’t order a total or free T3 level when assessing levothyroxine (T4) dose in hypothyroid patients. https://www.choosingwisely.org/clinician-lists/endocrine-society-total-or-free-t3-level-when-assessing-levothryroxine-dose-in-hyperthyroid-patients/. Accessed March 26, 2019.
35.
Canadian Society of Endocrinology and Metabolism. Choosing Wisely Canada. Five things physicians and patients should question. Last updated June 2017. https://choosingwiselycanada.org/endocrinology-and-metabolism/. Accessed April 2, 2019.
36.
Norton  EC, Dowd  BE, Maciejewski  ML.  Marginal effects–quantifying the effect of changes in risk factors in logistic regression models.  JAMA. 2019;321(13):1304-1305. doi:10.1001/jama.2019.1954PubMedGoogle ScholarCrossref
37.
Keats JP. Curtailing utilization of low-value medical care. Am J Accountable Care. 2019; 7(2):24-25.
38.
Kerr  EA, Chen  J, Sussman  JB, Klamerus  ML, Nallamothu  BK.  Stress testing before low-risk surgery: so many recommendations, so little overuse.  JAMA Intern Med. 2015;175(4):645-647. doi:10.1001/jamainternmed.2014.7877PubMedGoogle ScholarCrossref
39.
Burke  JF, Kerr  EA, McCammon  RJ, Holleman  R, Langa  KM, Callaghan  BC.  Neuroimaging overuse is more common in Medicare compared with the VA.  Neurology. 2016;87(8):792-798. doi:10.1212/WNL.0000000000002963PubMedGoogle ScholarCrossref
40.
Winchester  DE, Meral  R, Ryals  S, Beyth  RJ, Shaw  LJ.  Appropriate use of myocardial perfusion imaging in a veteran population: profit motives and professional liability concerns.  JAMA Intern Med. 2013;173(14):1381-1383. doi:10.1001/jamainternmed.2013.953PubMedGoogle ScholarCrossref
41.
Ubel  PA, Rosenthal  MB.  Beyond nudges–when improving ehealth calls for greater assertiveness.  N Engl J Med. 2019;380(4):309-311. doi:10.1056/NEJMp1806371PubMedGoogle ScholarCrossref
42.
US Preventive Services Task Force. Appendix I. Congressional mandate establishing the US Preventive Services Task Force. Current as of July 2017. https://www.uspreventiveservicestaskforce.org/Page/Name/appendix-i-congressional-mandate-establishing-the-us-preventive-services-task-force. Accessed November 4, 2019.
43.
Colla  CH.  Swimming against the current–what might work to reduce low-value care?  N Engl J Med. 2014;371(14):1280-1283. doi:10.1056/NEJMp1404503PubMedGoogle ScholarCrossref
44.
Washington Health Alliance. Less Harm. Less Waste. Choosing Wisely in Washington State. August 2016. https://wahealthalliance.org/wp-content/uploads/2016/08/2016-choosing-wisely-washington-state-report.pdf. Accessed July 9, 2019.
45.
Northern New England Practice Transformation Network. https://qualidigm.org/our-work/nneptn/. Accessed January 2, 2020.
46.
Ivers  MN, Desveaux  L.  De-implementation of low-value care: audit and feedback wisely.  Healthc Pap. 2019;18(1):41-47.Google Scholar
47.
Buxbaum JD, Mafi JN, Fendrick AM. Tackling low-value care: a new “top five” for purchaser action. Health Aff. November 21, 2017. https://www.healthaffairs.org/do/10.1377/hblog20171117.664355/full/. Accessed October 28, 2019.
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Less Is More
    February 10, 2020

    Comparison of Payment Changes and Choosing Wisely Recommendations for Use of Low-Value Laboratory Tests in the United States and Canada

    Author Affiliations
    • 1Consulting for Statistics, Computing and Analytics Research, University of Michigan, Ann Arbor
    • 2Veterans Affairs Center for Clinical Management Research, Ann Arbor, Michigan
    • 3Institute for Health Systems Solutions and Virtual Care, Women’s College Hospital, Toronto, Ontario, Canada
    • 4Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
    • 5Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
    • 6Department of Internal Medicine, University of Michigan, Ann Arbor
    • 7Michigan Program on Value Enhancement, Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor
    JAMA Intern Med. 2020;180(4):524-531. doi:10.1001/jamainternmed.2019.7143
    Key Points

    Question  Did use of low-value blood tests for vitamin D and triiodothyronine levels change after implementation of a payment policy change or Choosing Wisely recommendations in Canada or the United States?

    Finding  In this cross-sectional study of administrative claims data, a greater reduction in low-value vitamin D screening was associated with a payment policy change and related recommendations in Ontario, Canada, compared with Choosing Wisely recommendations in both Canada and the United States. Reductions in low-value triiodothyronine level testing after relevant recommendations were not observed.

    Meaning  The findings suggest that recommendations alone may be insufficient for reducing use of low-value services at a national or regional level.

    Abstract

    Importance  Evidence comparing the consequences of Choosing Wisely recommendations across health systems or with the consequences of recommendations plus policy change is lacking.

    Objectives  To compare changes in the use of 2 low-value laboratory tests after the release of Choosing Wisely recommendations across 3 health care jurisdictions and changes associated with a related policy change.

    Design, Setting, and Participants  This cross-sectional study was a population-based interrupted time series of adult patients (aged 18-64 years) who had primary care visits between January 1, 2010, and June 30, 2015, or established hypothyroidism between January 1, 2012, and June 30, 2015, across 3 health care delivery jurisdictions: Ontario, Canada; the US Veterans Health Administration; and the US employer-sponsored insurance market. Data analysis was performed from March 21, 2018, to October 31, 2019.

    Exposures  A December 2010 payment policy change that eliminated reimbursement of vitamin D screening in Ontario, Canada, and the subsequent release of Choosing Wisely recommendations against low-value use of vitamin D tests in February 2013 and triiodothyronine tests in October 2013 in the United States and both tests in October 2014 in Canada.

    Main Outcomes and Measures  Relative marginal effects (RMEs) comparing low-value testing rates after the release of Choosing Wisely recommendations with rates expected based on prerelease trends and the associated change in low-value vitamin D testing after the 2010 payment policy change in Ontario, Canada.

    Results  Of 54 223 448 total persons, 28 504 576 (52.6%) were female, with 17 895 458 persons (33.0%) aged 18 to 34 years, 11 101 985 (20.5%) aged 35 to 44 years, and 25 226 005 (46.5%) aged 45 to 64 years. The December 2010 policy eliminating reimbursement for low-value vitamin D screening in Ontario, Canada, was associated with a 92.7% (95% CI, 92.4%-93.0%) relative reduction in such screening. Corresponding Choosing Wisely recommendations were associated with smaller reductions: 4.5% (95% CI, 2.6%-6.3%) in Ontario, 13.8% (95% CI, 11.8%-15.9%) for US Veterans Health Administration, and 14.0% (95% CI, 12.8%-15.2%) for US employer-sponsored insurance. In contrast, low-value use of triiodothyronine testing did not change significantly in Ontario, Canada (RME, 0.3%; 95% CI, –1.4% to 2.0%) or the US Veterans Health Administration (RME, 0.7%; 95% CI, –4.7% to 6.4%) and increased (RME, 3.0%; 95% CI, 1.6%-4.4%) for US employer-sponsored insurance.

    Conclusions and Relevance  In this study, marginal reductions in the use of 2 low-value laboratory tests were associated with the release of related Choosing Wisely recommendations but a greater reduction in low-value vitamin D screening was associated with a previous payment policy change implemented in Ontario, Canada. These findings suggest that recommendations alone may be insufficient to significantly reduce use of low-value services and that pairing recommendations with policy changes may be more effective.

    Introduction

    Numerous cross-sectional studies have established that low-value services, commonly defined as tests, treatments, or procedures providing minimal benefit relative to cost or risks of harm,1,2 are prevalent and costly in both the United States and internationally.3-10 However, less attention has been given to how and why the use of low-value services changes over time and whether there are significant differences by system or country in response to recommendations for reducing low-value care.11-13

    Studies of trends in performance of low-value services are particularly relevant considering ongoing efforts to better align medical evidence with practice. For example, the Choosing Wisely initiative has partnered with medical professional societies since 2012 to develop and promote recommendations about low-value services that should be discontinued.14,15 Choosing Wisely has since expanded to more than 20 countries, including Canada.2,16 Yet, despite increasing awareness of this campaign,17 available evidence on the consequences of these recommendations suggests that they are associated at a regional or national level with modest reductions in the use of low-value services.11,12 In contrast, changes to payers’ policies and targeted interventions in health systems have been associated with significant reductions in the use of specific low-value services, including population-based screening for vitamin D deficiency.18-22 Given that Choosing Wisely is now an international campaign, there is need and opportunity to better understand how to improve the broad uptake of Choosing Wisely recommendations and to assess whether some countries and health care systems are more successful in this objective. In particular, to what extent uptake of recommendations can be enhanced through specific policy changes or motivated by attributes of health care systems should be assessed.

    The present study addressed these challenges by longitudinally examining use of low-value laboratory testing—specifically, vitamin D screening and triiodothyronine (T3) level testing—across 3 jurisdictions: government-funded health coverage in Ontario, Canada (CA-Ontario); Veterans Health Administration (VHA) coverage provided to eligible US military veterans (US-Veterans); and the US employer-sponsored insurance market (US-Commercial). Choosing Wisely campaigns in the United States and Canada endorse recommendations against both population-based vitamin D screening (because screening is not associated with improved outcomes)23-25 and T3 level testing for monitoring among patients with established hypothyroidism (in favor of thyroid-stimulating hormone level tests alone). Furthermore, a 2010 policy in Ontario, Canada, consistent with the aforementioned Choosing Wisely recommendations and following an earlier recommendation from an advisory committee, eliminated reimbursement for population-based vitamin D screening supported by the government-sponsored health plan.26,27 However, similar payment policy changes were not implemented in the United States for vitamin D screening or in either country for T3 level testing. By examining rates of vitamin D screening and T3 level testing, which both have contemporaneous recommendations but only 1 with a payment policy change, across 3 jurisdictions, we sought to help clarify how policies, recommendations, and systems of care are associated with the use of low-value services at regional and national levels.

    Methods
    Study Design and Data Sources

    We conducted a retrospective cross-sectional study of administrative claims data to examine the use of low-value vitamin D screening and T3 level testing across CA-Ontario, US-Veterans, and US-Commercial. Specifically, we compared low-value utilization rates of the target services among all beneficiaries aged 18 to 64 years using population data for CA-Ontario and US-Veterans and a large claims database for the US-Commercial population. Use of VHA data for this study was approved by the institutional review board of the Veterans Affairs Ann Arbor Healthcare System, with a waiver of informed consent and Health Insurance Portability and Accountability Act authorization because of the impracticality of consenting millions of patients and the minimal risk presented by this study. Use of Marketscan Commercial Claims and Encounters Research data28 was approved as not regulated by the institutional review board of the University of Michigan Medical School; and use of administrative data from Ontario, Canada was approved under §44 of the province’s Personal Health Information Protection Act, which does not require review by a research ethics board. Data analysis was performed from March 31, 2018, to October 31, 2019.

    In Ontario, Canada, medically necessary services, as detailed in the schedule of benefits, including physician visits and laboratory blood work are covered with no out-of-pocket costs to patients under the publicly funded Ontario Health Insurance Plan (OHIP), which is run by the provincial government.29 The OHIP pays for primary care physician visits by either fee-for-service or capitated payments (which cover visits and care coordination but not diagnostic or screening tests or procedures), whereas specialists receive fee-for-service payment.30 For outpatient laboratory tests, OHIP contracts with a network of community laboratories that receive fee-for-service payments for tests performed, subject to a global cap on spending.31 For CA-Ontario, claims for eligible outpatient care were identified from OHIP databases. For US-Veterans, data were sourced from a population of active VHA patients, defined here as those with at least 2 visits to VHA facilities in the previous 2 years. Estimates for the US-Commercial market were based on the Marketscan Commercial Claims and Encounters Research database, which consists of health care claims from a national cross-section of individuals with employer-sponsored health insurance, including employees, spouses, and dependents. Estimates based on Marketscan data were poststratified to reflect the broader US employer-sponsored insurance market in terms of age, sex, census region, and employer relationship.28 In addition to identifying services claimed, each database also contains demographic information, including age, sex, and health region.

    Vitamin D Screening

    Population-based screening for vitamin D deficiency absent high-risk conditions warranting aggressive monitoring and treatment (eg, metabolic disorder, renal disease) has not been found to be associated with improved outcomes. In February 2010, the Ontario Health Technology Advisory Committee (OHTAC) published an evidence-based analysis and recommended against testing serum vitamin D levels in average-risk individuals.26 The OHTAC is charged with making recommendations to Health Quality Ontario, which then decides which services should be publicly funded. As a result, the government-sponsored health plan in Ontario, Canada, eliminated reimbursement to laboratories for population-based vitamin D screening, a policy change that resulted in laboratories no longer processing vitamin D screening tests and that signaled to primary care clinicians to stop ordering the screening tests. The change to the governing regulation was posted and advertised in November 2010 and took effect in December 2010.27 Subsequently, both Choosing Wisely USA and Choosing Wisely Canada endorsed recommendations against population-based vitamin D screening.23-25 To measure rates of low-value vitamin D screening, we identified all claims for an outpatient primary care visit occurring between January 1, 2010, and June 30, 2015, per jurisdiction using procedure codes for evaluation and management visits and similar OHIP fee codes.32,33 We excluded visits involving beneficiaries with 1 or more claims containing a diagnosis code potentially justifying vitamin D testing as appropriate (eg, metabolic disorder, malabsorption syndrome) during the previous year. We derived exclusions from existing Ontario Ministry of Health policy and other guidelines,21,27 taking an inclusive approach to construct a measure emphasizing specificity over sensitivity in identifying low-value tests. The most common reason for exclusion was renal disease (eTable 1 in the Supplement). For beneficiaries with multiple qualifying visits per month, we selected only their first visit. Among qualifying visits, we identified those followed by a low-value (ie, 25-hydroxy or 1,25-dihydroxy) vitamin D test within 30 days. Rates were then expressed as the number of qualifying visits with a subsequent vitamin D test per 100 qualifying visits in a month.

    Triiodothyronine Testing

    Triiodothyronine testing used for monitoring patients with established hypothyroidism is not recommended. Recommendations endorsed by both Choosing Wisely USA (October 2013) and Choosing Wisely Canada (October 2014) affirm that use of a more appropriate test, thyroid-stimulating hormone level, is favored.34,35 To calculate rates of low-value T3 testing between January 1, 2012, and June 30, 2015, we identified beneficiaries with established hypothyroidism during this period. Consistent with the Chronic Conditions Data Warehouse34 definition, we reviewed up to 3 years (January 1, 2009) to identify beneficiaries with 2 or more outpatient claims or a single inpatient claim with a hypothyroidism diagnosis. Beneficiaries were included in the measure denominator beginning the month after hypothyroidism was first established and remained eligible for the denominator for 3 years after each qualifying hypothyroidism diagnosis if they also remained eligible for services. We expressed monthly rates of low-value T3 testing as the number of beneficiaries receiving T3 testing per 100 eligible beneficiaries with hypothyroidism.

    Target Population

    Our analyses began in January 1, 2010, for vitamin D and January 1, 2012, for T3 based on the availability of Marketscan data used to assess rates for the US-Commercial population and the need for a 1-year review period to establish eligibility for vitamin D level testing and a 3-year review period for T3 tests. We further limited our analyses to extend only through June 30, 2015, because of the October 2015 transition from International Classification of Diseases, Ninth Revision to International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) diagnosis codes in the United States. Although earlier adoption of ICD-10 in Ontario led us to specify measures using both code sets, we intended to limit the potential confounding associated with changes in diagnostic coding between the use of low-value testing and the release of Choosing Wisely recommendations. We summarize the coding systems used in eTable 2 in the Supplement.

    Statistical Analysis

    We identified low-value vitamin D and T3 tests within each jurisdiction and quantified these as monthly rates per 100 eligible events (ie, qualifying visits or persons at risk for the vitamin D and T3 measures, respectively). We then independently modeled monthly counts of each low-value test per jurisdiction (ie, 6 unique time series) in a multivariable quasi–Poisson regression with the number of eligible beneficiaries as an offset using generalized estimating equations to account for overdispersion and autocorrelation. Each model included seasonal effects for calendar month and segmented log-linear time trends, which were allowed to differ by strata based on age (18-34 years, 35-44 years, 45-64 years), sex (male or female), health region (4 per jurisdiction), and, for US-Commercial only, relationship to employer (employee or dependent). The resulting 24 total strata per time series (48 for US-Commercial) were used to specify clusters in the generalized estimating equations models. Log-linear time trends were segmented to allow assessment of changes after the release of Choosing Wisely recommendations in the United States (vitamin D in February 2013 and T3 in October 2013) and Canada (both October 2014), and the vitamin D policy change in Canada (December 2010). Differential trends for 2010 were included (post hoc) in models for the 2 US jurisdictions to improve model fit. Generalized estimating equations model parameters and variances were estimated using SAS, version 9.4 (SAS Institute Inc).

    Primary outcomes were the relative marginal effects (RMEs) and absolute marginal effects of the Choosing Wisely recommendations during the postrecommendation periods within each jurisdiction.36 The absolute marginal effects of the recommendations are the differences between the observed rates after exposure and those expected based on prior trends, whereas the RMEs express these differences as a percentage of the expected postexposure rates. Because each of these measures were aggregated across demographic strata and time, we used the delta method to obtain SEs for constructing 95% CIs and significance tests. For the US-Commercial population, jurisdiction-level aggregates used poststratification weights derived from the Medical Expenditure Panel Survey.28 The weights are distributed with the data set and described in an accompanying user manual.

    As a secondary outcome, we compared pre- and postrecommendation trends. Strata-level trends in the regression models described above capture the relative rate of change for each low-value measure within strata. To summarize the relative rate of change within each jurisdiction and period, we estimated the average annual percentage change (AAPC) by first computing monthly jurisdiction-level estimates using a weighted average of model predictions, forming year-over-year rate ratios for each month, averaging, and then transforming from the ratio to percentage change scale. We also expressed jurisdiction-level trends on the absolute scale using average marginal effects. To compare pre- and postrecommendation trends, we used ratios of estimated AAPC values, interpretable as relative risk ratios. Marginal effects, AAPC, and AAPC ratios were computed using R, version 3.4.4 (R Foundation for Statistical Computing). Additional details are available in the eMethods in the Supplement.

    Results
    Sample Characteristics

    For the vitamin D measure, we identified 363 915 220 qualifying primary care visits from 50 824 135 unique individuals between January 1, 2010, and June 30, 2015, with 7 245 271 (2.0%) visits associated with a low-value vitamin D screening test. For the T3 measure, we examined 76 671 630 person-months of follow-up from 3 399 313 unique patients with hypothyroidism and identified 1 904 815 (2.2%) low-value T3 tests. Of 54 223 448 total persons, 28 504 576 (52.6%) were female, 17 895 458 persons (33.0%) aged 18 to 34 years, 11 101 985 (20.5%) aged 35 to 44 years, and 25 226 005 (46.5%) aged 45 to 64 years. Detailed demographic information for each jurisdiction appears in Table 1 and eTable 3 in the Supplement.

    Low-Value Vitamin D Screening Rates

    In early 2010, rates of low-value vitamin D screening were similar across all 3 jurisdictions. Between January 1, 2010, and June 30, 2010, there were 2.25 (US-Veterans), 2.17 (CA-Ontario), and 2.07 (US-Commercial) low-value vitamin D screening events for every 100 qualifying primary care visits. However, rates of low-value screening subsequently diverged with rates of 4.36 (US-Veterans), 0.61 (CA-Ontario), and 2.40 (US-Commercial) from 2014 to 2015. Trends are depicted in the Figure.

    The large reduction observed in Ontario may have been associated with anticipation of the December 2010 payment policy change to halt reimbursement of population-based vitamin D testing. When comparing January and November 2011 with the corresponding months in 2010, this intervention was associated with relative reductions of 92.7% (95% CI, 92.4%-93.0%) and 67.2% (95% CI, 66.3%-68.1%), respectively. The former estimate shows changes associated with all events in 2010 leading up to the December policy change, and the latter partially discounts this estimate for the effects of the earlier February 2010 recommendation that would later lead to the policy change. After this decrease, rates of low-value vitamin D screening trended upward in CA-Ontario but subsequently remained below 2010 levels.

    Excluding changes associated with the December 2010 policy change in Ontario, low-value use of vitamin D screening trended upward in all jurisdictions before the release of Choosing Wisely recommendations on February 13, 2013 (United States), and October 29, 2014 (Canada) (eTable 4 in the Supplement). However, in the postrecommendation periods, the annual relative rates of growth decreased from 30% to 25% for CA-Ontario, from 17% to 8% for US-Veterans, and from 6% to −3% for US-Commercial. As a result, during the postrecommendation periods, there were fewer than expected low-value screenings in all jurisdictions: 4.5% (95% CI, 2.5%-6.3%) fewer for CA-Ontario, 13.8% (95% CI, 11.8%-15.9%) fewer for US-Veterans, and 14.0% (95% CI, 12.8%-15.2%) fewer for US-Commercial (Table 2). Additional details on trends are available in eTable 4 in the Supplement.

    Compared with the 92.7% reduction (2.1 fewer tests per 100 visits) in rates of low-value vitamin D screening in Ontario associated with the December 2010 policy change eliminating reimbursement for such screenings, estimated reductions associated with Choosing Wisely recommendations were small. Although the relative rate of change slowed in CA-Ontario and among US-Veterans, these decelerations were associated with absolute marginal effects of –0.03 (95% CI, –0.04 to –0.02) screenings per 100 visits in CA-Ontario and –0.68 (95% CI, –0.79 to –0.56) screenings per 100 visits for US-Veterans, with the small effect in Ontario potentially reflecting the already low baseline after the earlier payment policy change. Although a small absolute decrease was observed for the US-Commercial population, the estimated total marginal effect for the low-value screening rate was –0.41 (95% CI, –0.45 to –0.37) tests per 100 visits.

    Low-Value T3 Testing Rates

    Although Choosing Wisely recommendations against population-based screening for vitamin D levels were associated with modest reductions in the rate of growth for low-value testing, comparable recommendations against T3 testing among patients with established hypothyroidism (on October 16, 2013, for the United States and October 29, 2014, for Canada) were not associated with changes in CA-Ontario (RME, 0.3%; 95% CI, –1.4% to 2.0%) or US-Veterans (RME, 0.7%; 95% CI, –4.7% to 6.4%) and were associated with a small increase for the US-Commercial population (RME, 3.0%; 95% CI, 1.6% to 4.4%) (Table 2).

    Discussion

    In the 3 jurisdictions examined, Choosing Wisely recommendations were associated with only limited reductions in use of low-value vitamin D screenings and were not associated with reduced use of low-value T3 testing. For vitamin D screenings, the recommendations were associated with slowing of trends toward increased overuse. The limited reductions associated with these recommendations alone are notable when compared with the 93% reduction in low-value vitamin D screening associated with the 2010 policy change and preceding recommendation in Ontario. This finding is consistent with the greater than 90% reduction associated with a similar intervention in 2015 in Alberta, Canada, that required a special requisition to test vitamin D levels.19 Moreover, although screening rates increased slightly in Ontario after 2011, they remained significantly lower than rates in the US jurisdictions. For instance, in June 2015, screening rates in the US-Veterans and US-Commercial populations were 6.05 times and 3.22 times higher, respectively, than in CA-Ontario. That is, if low-value vitamin D screening rates in the United States were the same as the highest regional rate in Ontario from 2011 to 2015, an average of 213 000 unnecessary screenings each year could have been avoided among US veterans and 4.4 million each year in the US-Commercial market. Perhaps recognizing the potential savings and a responsibility toward resource stewardship, at least 1 third-party US payer recently moved to eliminate reimbursement for low-value vitamin D screenings.37

    We also explored whether both absolute rates and change in rates of overuse because of Choosing Wisely recommendations varied by jurisdiction or system. Although we found that reductions in screening and testing owing to Choosing Wisely recommendations were consistently small across jurisdictions, relative rates of overuse of the 2 tests were not consistent. In particular, vitamin D screening in the US-Veterans population was higher than that in the US-Commercial population, whereas T3 testing in the US-Veterans population was substantially lower than that in the US-Commercial or CA-Ontario population. Previous studies have shown lower or similar rates of overuse in the VHA compared with Medicare.38-40 This lower rate of overuse is in part unsurprising because the VHA is a capitated system and neither the VHA facilities nor clinicians receive additional payments for performing laboratory tests. Under such conditions, focusing on saving resources for the system as a whole, policies that restrict test ordering, and evidence-based clinical decision support and/or other behavioral change interventions41 may better yield desired decreases in low-value services.20,21 In addition, it is possible that the relatively high vitamin D screening rates at the VHA are at least partly associated with the lower rates of coding of comorbidities, such as renal disease and vitamin D deficiency, because of lack of financial incentives for complete coding capture. Such undercoding would result in overestimates in provision of low-value screening because patients appropriately undergoing vitamin D testing would not be excluded from the denominator. We further hypothesized that rates of low-value T3 testing were lower because of less frequent use of bundled thyroid function testing in the VHA and because the VHA has a higher proportion of males, for whom low-value T3 testing is less common.

    As shown in this analysis, changes in payment policies were associated with broad reductions in use of low-value care. Of note, the Affordable Care Act presently provides authority to deny Medicare payments for medically unnecessary services.42 Moreover, financial incentives can be implemented at the system, clinician, or patient levels.43 However, recommendations to reduce low-value services often rely on an understanding of a patient’s complex clinical status, and payment policy change may be too prohibitive to ensure needed services can still be performed. Furthermore, payment policies are particular to the care delivery or insurance system. For example, payment policies that do not reimburse for delivered services may work well in fee-for-service environments but are less applicable in capitated systems. Therefore, to accelerate the broad uptake of Choosing Wisely recommendations without promoting underuse, promotion of implementation of effective and diverse interventions22 tailored to both clinical and health system context are needed as well as interventions with regional or national policies or incentives that can be broadly but safely applied by health care clinicians in a given region. In the United States and Canada, for example, such collaborations have begun at the state and province levels.44-46 For long-term sustainability, the goal of such collaborations should be not only to reduce specific instances of low-value care but also to catalyze and sustain momentum toward a culture emphasizing the responsibility of health care institutions, clinicians, and patients to provide and seek high-value, evidence-based care while avoiding low-value services.17,47

    Limitations

    Our study has several limitations. Although we attempted to create measures that were highly specific, administrative data lack the clinical information necessary to label with complete confidence individual instances of these laboratory tests as low-value care, nor did we examine whether rates of necessary testing decreased. Moreover, because we relied on administrative data from systems with different financial incentives for medical coding, we were unable to calibrate the extent to which absolute differences between jurisdictions were associated with differences in coding vs differences in care delivery. Similarly, our analysis did not separate changes in test use from potential changes in diagnostic coding associated with recommendation or payment policy exposures.

    In addition, although an interrupted time series design is useful for assessing associations between key events and changes in trends, it should not be interpreted as establishing causality. In addition, we had limited data points before the delisting of vitamin D screening tests in Ontario, and therefore, there may have been secular trends in low-value vitamin D screenings before the policy change in Ontario. In particular, we saw that reductions in screening rates began in Ontario before the December 2010 policy change, which may have been partially associated with the February 2010 OHTAC recommendation and the November 2010 posting that advertised the upcoming change. However, it is unlikely that this decline began before 2010 because the OHTAC analysis reported a 24-fold increase in the raw number of vitamin D tests between 2004 and 2009. Also, although we stratified by important demographic variables, our estimated trends are ecological rather than at the person level and should be interpreted accordingly.

    Conclusions

    In this study, marginal reductions in the use of 2 low-value laboratory tests were associated with the release of related Choosing Wisely recommendations but a greater reduction in low-value vitamin D screening was associated with a prior payment policy change implemented in Ontario, Canada. These findings suggest that recommendations alone may be insufficient to significantly reduce use of low-value services and that pairing recommendations with policy changes may be more effective.

    Back to top
    Article Information

    Accepted for Publication: December 10, 2019.

    Corresponding Author: James Henderson, PhD, Consulting for Statistics, Computing and Analytics Research, University of Michigan, 915 E Washington, 3550 Rackham, Ann Arbor, MI 48010 (jbhender@umich.edu).

    Published Online: February 10, 2020. doi:10.1001/jamainternmed.2019.7143

    Author Contributions: Dr Henderson had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Henderson, Bouck, Chu, Bhatia, Kerr.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Henderson, Bouck, Chu, Santiago.

    Critical revision of the manuscript for important intellectual content: Henderson, Bouck, Holleman, Chu, Klamerus, Bhatia, Kerr.

    Statistical analysis: Henderson, Bouck, Holleman, Chu, Bhatia.

    Obtained funding: Kerr.

    Administrative, technical, or material support: Bouck, Chu, Klamerus.

    Supervision: Bhatia, Kerr.

    Conflict of Interest Disclosures: Dr Santiago reported receiving grants from Choosing Wisely Canada and Women's College Hospital during the conduct of the study. Dr Kerr reported receiving grants from the Department of Veterans Affairs and the Donaghue Foundation during the conduct of the study, receiving personal fees from Bind outside the submitted work, and serving on the Secretariat for the Choosing Wisely International Initiative. No other disclosures were reported.

    Disclaimer: The opinions expressed represent the views of the authors and do not reflect the US Department of Veterans Affairs.

    References
    1.
    Institute of Medicine.  Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: The National Academies Press; 2001.
    2.
    Levinson  W, Kallewaard  M, Bhatia  RS, Wolfson  D, Shortt  S, Kerr  EA; Choosing Wisely International Working Group.  ‘Choosing Wisely’: a growing international campaign.  BMJ Qual Saf. 2015;24(2):167-174. doi:10.1136/bmjqs-2014-003821PubMedGoogle ScholarCrossref
    3.
    Berwick  DM, Hackbarth  AD.  Eliminating waste in US health care.  JAMA. 2012;307(14):1513-1516. doi:10.1001/jama.2012.362PubMedGoogle ScholarCrossref
    4.
    Canadian Institute for Health Information. Unnecessary Care in Canada. Published April 2017. https://www.cihi.ca/sites/default/files/document/choosing-wisely-baseline-report-en-web.pdf. Accessed April 25, 2019.
    5.
    Bhatia  RS, Levinson  W, Shortt  S,  et al.  Measuring the effect of Choosing Wisely: an integrated framework to assess campaign impact on low-value care.  BMJ Qual Saf. 2015;24(8):523-531. doi:10.1136/bmjqs-2015-004070PubMedGoogle ScholarCrossref
    6.
    Colla  CH, Morden  NE, Sequist  TD, Schpero  WL, Rosenthal  MB.  Choosing wisely: prevalence and correlates of low-value health care services in the United States.  J Gen Intern Med. 2015;30(2):221-228. doi:10.1007/s11606-014-3070-zPubMedGoogle ScholarCrossref
    7.
    Schwartz  AL, Landon  BE, Elshaug  AG, Chernew  ME, McWilliams  JM.  Measuring low-value care in Medicare.  JAMA Intern Med. 2014;174(7):1067-1076. doi:10.1001/jamainternmed.2014.1541PubMedGoogle ScholarCrossref
    8.
    Segal  J, Bridges  J, Chang  H-Y,  et al.  Identifying possible indicators of systematic overuse of health care procedures with claims data.  Med Care. 2014;52(2):157-163. doi:10.1097/MLR.0000000000000052PubMedGoogle ScholarCrossref
    9.
    Mafi  JN, Russell  K, Bortz  BA, Dachary  M, Hazel  WA  Jr, Fendrick  AM.  Low-cost, high-volume health services contribute the most to unnecessary health spending.  Health Aff (Millwood). 2017;36(10):1701-1704. doi:10.1377/hlthaff.2017.0385PubMedGoogle ScholarCrossref
    10.
    Bhatia  RS, Bouck  Z, Ivers  NM,  et al.  Electrocardiograms in low-risk patients undergoing an annual health examination.  JAMA Intern Med. 2017;177(9):1326-1333. doi:10.1001/jamainternmed.2017.2649PubMedGoogle ScholarCrossref
    11.
    Hong  AS, Ross-Degnan  D, Zhang  F, Wharam  JF.  Small Decline in low-value back imaging associated with the ‘Choosing Wisely’ campaign, 2012-14.  Health Aff (Millwood). 2017;36(4):671-679. doi:10.1377/hlthaff.2016.1263PubMedGoogle ScholarCrossref
    12.
    Rosenberg  A, Agiro  A, Gottlieb  M,  et al.  Early trends among seven recommendations from the Choosing Wisely campaign.  JAMA Intern Med. 2015;175(12):1913-1920. doi:10.1001/jamainternmed.2015.5441PubMedGoogle ScholarCrossref
    13.
    Brett  J, Maust  DT, Bouck  Z,  et al.  Benzodiazepine use in older adults in the United States, Ontario, and Australia from 2010 to 2016.  J Am Geriatr Soc. 2018;66(6):1180-1185. doi:10.1111/jgs.15292PubMedGoogle ScholarCrossref
    14.
    Cassel  CK, Guest  JA.  Choosing wisely: helping physicians and patients make smart decisions about their care.  JAMA. 2012;307(17):1801-1802. doi:10.1001/jama.2012.476PubMedGoogle ScholarCrossref
    15.
    Choosing Wisely. Our Mission. https://www.choosingwisely.org/our-mission/. Accessed July 10, 2019.
    16.
    Choosing Wisely Canada. The necessary conference. Choosing Wisely Canada’s National Meeting. https://choosingwiselycanada.org/. Accessed July 10, 2019.
    17.
    Kerr  EA, Kullgren  JT, Saini  SD.  Choosing Wisely: how to fulfill the promise in the next 5 years.  Health Aff (Millwood). 2017;36(11):2012-2018. doi:10.1377/hlthaff.2017.0953PubMedGoogle ScholarCrossref
    18.
    Ferrari  R, Prosser  C.  Testing vitamin D levels and choosing wisely.  JAMA Intern Med. 2016;176(7):1019-1020. doi:10.1001/jamainternmed.2016.1929PubMedGoogle ScholarCrossref
    19.
    Naugler  C, Hemmelgarn  B, Quan  H,  et al.  Implementation of an intervention to reduce population-based screening for vitamin D deficiency: a cross-sectional study.  CMAJ Open. 2017;5(1):E36-E39. doi:10.9778/cmajo.20160073PubMedGoogle ScholarCrossref
    20.
    Felcher  AH, Gold  R, Mosen  DM, Stoneburner  AB.  Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing.  J Am Med Inform Assoc. 2017;24(4):776-780. doi:10.1093/jamia/ocw182PubMedGoogle ScholarCrossref
    21.
    Petrilli  CM, Henderson  J, Keedy  JM,  et al.  Reducing unnecessary vitamin D screening in an academic health system: what works and when.  Am J Med. 2018;131(12):1444-1448. doi:10.1016/j.amjmed.2018.06.025PubMedGoogle ScholarCrossref
    22.
    Colla  CH, Mainor  AJ, Hargreaves  C, Sequist  T, Morden  N.  Interventions aimed at reducing use of low-value health services: a systematic review.  Med Care Res Rev. 2017;74(5):507-550. doi:10.1177/1077558716656970PubMedGoogle ScholarCrossref
    23.
    Choosing Wisely. American Society for Clinical Pathology. Don’t perform population based screening for 25-OH-vitamin D deficiency. Released February 21, 2013. http://www.choosingwisely.org/clinician-lists/american-society-clinical-pathology-population-based-screening-for-vitamin-d-deficiency/. Accessed April 2, 2019.
    24.
    College of Family Physicians of Canada. Choosing Wisely Canada. Family Medicine. Thirteen things physicians and patients should question. Last updated, July 2019. https://choosingwiselycanada.org/family-medicine/. Accessed March 21, 2019.
    25.
    Canadian Association of Pathologists. Choosing Wisely Canade. Pathology. Five things physicians and patients should question in pathology. Last updated, June 2017. https://choosingwiselycanada.org/pathology/. Accessed April 2, 2019.
    26.
    Medical Advisory Secretariat. Clinical utility of vitamin D testing: an evidence-based analysis.  Ont Health Technol Assess Ser. 2010;10(2):1-93.PubMed
    27.
    Provincial Programs Branch. OHIP-insured vitamin D testing. 2010. http://www.health.gov.on.ca/en/pro/programs/ohip/bulletins/4000/bul4522.pdf. Accessed August 19, 2019.
    28.
    Truven Health Marketplace Database. MarketScan Commercial Claims and Encounters Research Database. 2009-2015.
    29.
    Ministry of Health and Long Term Care. Schedule of benefits: physician services under the Health Insurance Act. Amended October 1, 2005. http://www.health.gov.on.ca/en/pro/programs/ohip/sob/physserv/sob_master11062015.pdf. Accessed November 19, 2019.
    30.
    Ontario Ministry of Health. Ministry of Long-Term Care. Primary care payment models in Ontario. Last modified July 11, 2019. http://www.health.gov.on.ca/en/pro/programs/pcpm/. Accessed November 19, 2019.
    31.
    Sullivan  T, Gordon  P, Minto  S. Laboratory Services Expert Panel. November 12, 2015. http://www.health.gov.on.ca/en/common/ministry/publications/reports/lab_services/labservices.pdf. Accessed November 19, 2019.
    32.
    Health Quality Ontario. Quality improvement. My practice: primary care. https://www.hqontario.ca/Quality-Improvement/Guides-Tools-and-Practice-Reports/Primary-Care. Accessed July 10, 2019.
    33.
    Centers for Medicare & Medicaid Services. Evaluation and Management Services. December 2018. https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/MLN-Publications-Items/CMS1243514.html. Accessed August 19, 2019.
    34.
    Choosing Wisely. Endocrine Society. Don’t order a total or free T3 level when assessing levothyroxine (T4) dose in hypothyroid patients. https://www.choosingwisely.org/clinician-lists/endocrine-society-total-or-free-t3-level-when-assessing-levothryroxine-dose-in-hyperthyroid-patients/. Accessed March 26, 2019.
    35.
    Canadian Society of Endocrinology and Metabolism. Choosing Wisely Canada. Five things physicians and patients should question. Last updated June 2017. https://choosingwiselycanada.org/endocrinology-and-metabolism/. Accessed April 2, 2019.
    36.
    Norton  EC, Dowd  BE, Maciejewski  ML.  Marginal effects–quantifying the effect of changes in risk factors in logistic regression models.  JAMA. 2019;321(13):1304-1305. doi:10.1001/jama.2019.1954PubMedGoogle ScholarCrossref
    37.
    Keats JP. Curtailing utilization of low-value medical care. Am J Accountable Care. 2019; 7(2):24-25.
    38.
    Kerr  EA, Chen  J, Sussman  JB, Klamerus  ML, Nallamothu  BK.  Stress testing before low-risk surgery: so many recommendations, so little overuse.  JAMA Intern Med. 2015;175(4):645-647. doi:10.1001/jamainternmed.2014.7877PubMedGoogle ScholarCrossref
    39.
    Burke  JF, Kerr  EA, McCammon  RJ, Holleman  R, Langa  KM, Callaghan  BC.  Neuroimaging overuse is more common in Medicare compared with the VA.  Neurology. 2016;87(8):792-798. doi:10.1212/WNL.0000000000002963PubMedGoogle ScholarCrossref
    40.
    Winchester  DE, Meral  R, Ryals  S, Beyth  RJ, Shaw  LJ.  Appropriate use of myocardial perfusion imaging in a veteran population: profit motives and professional liability concerns.  JAMA Intern Med. 2013;173(14):1381-1383. doi:10.1001/jamainternmed.2013.953PubMedGoogle ScholarCrossref
    41.
    Ubel  PA, Rosenthal  MB.  Beyond nudges–when improving ehealth calls for greater assertiveness.  N Engl J Med. 2019;380(4):309-311. doi:10.1056/NEJMp1806371PubMedGoogle ScholarCrossref
    42.
    US Preventive Services Task Force. Appendix I. Congressional mandate establishing the US Preventive Services Task Force. Current as of July 2017. https://www.uspreventiveservicestaskforce.org/Page/Name/appendix-i-congressional-mandate-establishing-the-us-preventive-services-task-force. Accessed November 4, 2019.
    43.
    Colla  CH.  Swimming against the current–what might work to reduce low-value care?  N Engl J Med. 2014;371(14):1280-1283. doi:10.1056/NEJMp1404503PubMedGoogle ScholarCrossref
    44.
    Washington Health Alliance. Less Harm. Less Waste. Choosing Wisely in Washington State. August 2016. https://wahealthalliance.org/wp-content/uploads/2016/08/2016-choosing-wisely-washington-state-report.pdf. Accessed July 9, 2019.
    45.
    Northern New England Practice Transformation Network. https://qualidigm.org/our-work/nneptn/. Accessed January 2, 2020.
    46.
    Ivers  MN, Desveaux  L.  De-implementation of low-value care: audit and feedback wisely.  Healthc Pap. 2019;18(1):41-47.Google Scholar
    47.
    Buxbaum JD, Mafi JN, Fendrick AM. Tackling low-value care: a new “top five” for purchaser action. Health Aff. November 21, 2017. https://www.healthaffairs.org/do/10.1377/hblog20171117.664355/full/. Accessed October 28, 2019.
    ×