Changes in Low-Value Services in Year 1 of the Medicare Pioneer Accountable Care Organization Program

Importance  Wasteful practices are widespread in the US health care system. It is unclear if payment models intended to improve health care efficiency, such as the Medicare accountable care organization (ACO) programs, discourage the provision of low-value services.

Objective  To assess whether the first year of the Medicare Pioneer ACO program was associated with a reduction in use of low-value services.

Design, Setting, and Participants  In a difference-in-differences analysis, we compared use of low-value services between Medicare fee-for-service beneficiaries attributed to health care provider groups that entered the Pioneer program (ACO group) and beneficiaries attributed to other health care providers (control group) before (2009-2011) vs after (2012) Pioneer ACO contracts began. Data analysis was conducted from December 1, 2014, to June 27, 2015. Comparisons were adjusted for beneficiaries’ sociodemographic and clinical characteristics as well as for geography. We decomposed estimates according to service characteristics (clinical category, price, and sensitivity to patient preferences) and compared estimates between subgroups of ACOs with higher vs lower baseline use of low-value services.

Main Outcomes and Measures  Use of, and spending on, 31 services in instances that provide minimal clinical benefit, measured as annual service counts per 100 beneficiaries and price-standardized annual service spending per 100 beneficiaries.

Results  During the precontract period, trends in the use of low-value services were similar for the ACO and control groups. The first year of ACO contracts was associated with a differential reduction (95% CI) of 0.8 low-value services per 100 beneficiaries for the ACO group (−1.2 to −0.4; P < .001), corresponding to a 1.9% differential reduction in service quantity (−2.9% to −0.9%) and a 4.5% differential reduction in spending on low-value services (−7.5% to −1.4%; P = .004). Differential reductions were similar for services less sensitive vs more sensitive to patient preferences and for higher- vs lower-priced services. The ACOs with higher than their markets’ mean baseline levels of low-value service use experienced greater service reductions (−1.2 services per 100 beneficiaries; −1.7 to −0.7; P < .001) than did ACOs with use below the mean (−0.2 services per 100 beneficiaries, −0.6 to −0.2; P = .41; P = .003 for test of difference between subgroups).

Conclusions and Relevance  During its first year, the Pioneer ACO program was associated with modest reductions in low-value services, with greater reductions for organizations providing more low-value care. Accountable care organization–like risk contracts may be able to discourage use of low-value services even without specifying services to target.

Reducing unnecessary health care utilization, a source of substantial spending,¹ is a central goal of many government^2-4 and private^5,6 initiatives. Recent efforts, such as the American Board of Internal Medicine’s Choosing Wisely campaign,⁷ have drawn attention to specific services that provide minimal clinical benefit.^8,9 However, little is known about what strategies can effectively discourage the use of these services. Distinguishing high-value from low-value use of the same service is often challenging because value often depends on clinical context. As a result, efforts to directly limit overuse of specific services through coverage restrictions or other payment incentives may produce unintended consequences or achieve minimal gains.^8,10,11

Other strategies intended to improve health care efficiency do not target specific services. Among the most prominent of these strategies are alternative payment models, such as the model used in the Medicare Pioneer accountable care organization (ACO) program, which places spending for all services under a global budget with incentives to stay within the budget and improve performance on quality measures. This approach has been associated with lower overall spending and improved or stable performance on standard quality measures.^12-16

However, it is unknown whether payment reforms such as these are associated with disproportionate reductions in the use of low-value services. Although ACO-like payment models are intended to discourage the provision of services that contribute to spending but not to health, the combination of lower overall spending and improved performance on quality measures that has been observed may have resulted from reductions in high-value services affecting unmeasured dimensions of quality rather than from reductions in low-value services. More generally, because risk-based contracts do not incentivize reductions in overuse directly, it is unclear whether providers under these contracts are targeting low-value services in their broader efforts to control overall spending. If ACO-like payment models succeed in reducing the use of low-value services, there should be observable reductions in the delivery of the low-value services that can be measured directly. Moreover, if health care providers respond to ACO contracts by targeting low-value services, their efforts should result in greater reductions in spending on low-value services than in overall spending.

We constructed 31 claims-based measures of low-value services (ie, services that provide minimal clinical benefit on average). Using these measures and 2009-2012 Medicare fee-for-service claims, we conducted a difference-in-differences analysis comparing the use of low-value services between beneficiaries served by Pioneer ACOs and beneficiaries served by non-ACO providers before vs after the start of Pioneer contracts in 2012.

In 2012, a total of 32 health care provider organizations volunteered to participate in the Medicare Pioneer ACO program in which participating organizations receive a bonus payment or are penalized if overall spending for an attributed patient population falls sufficiently below or above a financial benchmark, respectively. Performance on 33 quality measures determines the proportion of savings or losses shared by the ACO, although ACOs were only required to report on these measures to be eligible for maximum savings in 2012.¹⁷ None of the quality measures in Medicare ACO contracts assesses overuse of medical services. Data analysis was conducted from December 1, 2014, to June 27, 2015. The Harvard University Committee on the Use of Human Subjects in Research and the National Bureau of Economic Research institutional review board approved the study and waived the requirement of informed consent.

We examined services provided from 2009 to 2012 using Medicare claims for a random 20% sample of beneficiaries; in a given year, this sample includes members from the previous year plus a 20% sample of newly eligible beneficiaries. For each year, we included beneficiaries in the study sample if they were continuously enrolled in Parts A and B of traditional Medicare while alive during that year and the entire prior year. We used the previous year of claims to collect diagnoses and procedures used for case-mix adjustment or for assessing the appropriateness of service use. In each study year, beneficiaries were excluded if they did not receive primary care services necessary for attribution to provider organizations or if the beneficiaries were attributed to any of the 114 organizations that entered the Medicare Shared Savings Program later in 2012. Medicare Shared Savings Program ACOs faced weaker incentives than Pioneer ACOs to reduce spending and were active for only part of 2012. Thus, if Medicare Shared Savings Program ACOs took early steps to limit low-value services, inclusion of their beneficiaries in the control group could have biased our estimates.

Each of the 32 organizations that entered the Pioneer ACO program was defined as the collection of National Provider Identifiers for physicians listed by the ACO as participating in the ACO contract (eMethods in the Supplement). Our definition of ACOs as sets of National Provider Identifiers reflects the organizations’ ability to include only a subset of affiliated physicians in their ACO contracts. Following the Medicare Shared Savings Program attribution rules and previously described methods,¹⁶ for each year in the study period, each beneficiary was assigned to the ACO (ACO group) or non-ACO (control group) practice that accounted for the greatest fraction of that beneficiary’s annual allowed charges for primary care services (eMethods in the Supplement). Non-ACO practices were defined by taxpayer identification numbers, which identify the billing practice, provider organization, or individual physician.

We constructed 31 claims-based measures of services that are low value, which we defined as providing minimal or no average clinical benefit in specific clinical scenarios. The measures, 26 of which were drawn from a prior study,⁸ were derived from evidence-based lists of low-value services. As described previously,⁸ we surveyed the following sources for candidate low-value services: the American Board of Internal Medicine Foundation’s Choosing Wisely initiative,¹⁸ the US Preventive Services Task Force D recommendations,¹⁹ the Canadian Agency for Drugs and Technologies in Health technology assessments,²⁰ and peer-reviewed medical literature.²¹ Services selected for measure development met 3 criteria: the service was relevant to the Medicare population, evidence that the service confers minimal clinical benefit had been established before the start of the study period, and claims and enrollment data were sufficient to distinguish high-value use from low-value use with reasonable accuracy.

For each measure, we created an operational definition of low-value service occurrence based on characteristics of patients and the service they received. Relevant patient variables included demographic characteristics and diagnoses present in concurrent or past claims. In addition to the type of service received, some measure definitions incorporated the timing of the service (eg, time since an inpatient discharge) and the site of care. We defined low-value services conservatively, opting for more specific definitions that reduced the likelihood of classifying a high-value service as low value.⁸ We detected service occurrences meeting these definitions on the basis of claims data elements, including Current Procedural Terminology (CPT) service codes,²²International Classification of Diseases, Ninth Revision (ICD-9)²³ patient diagnosis codes, data from Medicare enrollment files, and condition indicators from the Chronic Condition Data Warehouse.²⁴ Details regarding service identification, including codes used for service detection, are presented in the eMethods and eTable in the Supplement. To avoid duplicative counting of services, we did not include any service instances occurring within 7 days of the same service.

The primary outcome of this study was use of low-value services, defined as the annual count of all measured services. We chose this measure as our primary outcome because overall service counts equally weight clinical decisions to provide different services, whereas spending on low-value services is influenced heavily by use of more expensive services. We examined price-standardized spending on low-value services as a secondary outcome to compare changes in low-value spending with changes in overall spending associated with the Pioneer ACO program that were estimated previously using similar methods (see eMethods in the Supplement for price standardization methods).¹⁶

To assess whether any changes in low-value service use associated with Pioneer ACO contracts were concentrated in a specific clinical area or evident in multiple areas, we categorized the 31 low-value services into the following clinical categories: cancer screening, diagnostic and preventive testing, preoperative testing, imaging, cardiovascular testing and procedures, and other invasive procedures. We also categorized services as being priced higher (standardized price, $180-$13 331) or lower ($5-$117) than the median service price because ACOs would be unlikely to reduce higher-priced services in the absence of new payment incentives, whereas ACOs might restrict provision of lower-priced wasteful services even under fee-for-service incentives to improve quality without major reductions in revenue. Thus, reductions in the use of higher-priced, low-value services would provide stronger evidence of changes related specifically to ACO contract incentives.

Finally, to explore the possibility that patient preferences moderated providers’ responses to ACO contracts, we categorized services as less vs more sensitive to patient preferences (Table 1). For example, we considered testing for hypercoagulability following deep venous thrombosis as less sensitive to patient preferences because most patients would be unaware that such testing could be done. Table 1 presents each measure’s source and supporting literature, operational definition, and assigned categories of price and preference sensitivity.

For each beneficiary, the following demographic and clinical covariates were assessed from Medicare claims and enrollment files: age (<65, 65-69, 70-74, 75-79, 80-84, and ≥85 years), sex, race/ethnicity, disability as the original reason for Medicare entitlement, presence of end-stage renal disease, presence of 27 chronic conditions in the Chronic Condition Data Warehouse by the start of each study year (including indicators for each condition and indicators for having ≥2 to ≥9 conditions), and the patient’s hierarchical condition category risk score. Because most low-value service measures do not apply to all beneficiaries (eg, low-value prostate-specific antigen tests were considered those for men aged ≥75 years), we also created indicators for whether beneficiaries qualified for potential receipt of each low-value service (see eMethods in the Supplement for definitions of these qualifying indicators).

Because organizations providing more low-value services may have more opportunities to limit wasteful care, we assessed baseline use of low-value services for each ACO and tested whether changes in low-value service use associated with ACO contracts differed between ACOs with higher vs lower baseline use. We decomposed an ACO’s baseline level of low-value service use into 2 components. First, we assessed whether the ACO had a greater or lesser risk-adjusted count of low-value services per beneficiary than the control group in the ACO’s service area (eMethods in the Supplement). Second, we assessed whether the risk-adjusted rate of low-value service use among the control group in each ACO’s service area was greater or less than that of the median among ACO service areas.

This decomposition allowed us to examine whether an organization’s prior performance relative to its service area or service area performance relative to a national mean predicted changes under ACO contracts. This distinction bears on whether ACO contracts might be associated with convergence in provider practices within regions or across regions. Baseline levels of low-value care were assessed in 2008 to avoid bias from regression to the mean between the precontract period (2009-2011) and 2012; we found no evidence of regression to the mean in the precontract period (eMethods in the Supplement).

We conducted a difference-in-differences analysis to quantify changes in the annual per-beneficiary rate of low-value services in the ACO group that differed from concurrent changes in the control group from the precontract period (2009-2011) to the postcontract period (2012), while adjusting for geography and any coincident changes in the groups’ measured patient characteristics. Specifically, we fit the following linear regression model⁶⁴:

E(Y_i,t,k,h) = β₀ + β₁ACO_indicators_k + β₂(HRR_indicators_h × year_t) + β₃ACO_contract_kt + β₄covariates_it,

with E(Y_i,t,k,h) denoting the expected value of outcome Y (ie, count of low-value services) for beneficiary i during year t assigned to ACO or non-ACO taxpayer identification number k living in a hospital referral region (HRR) h. The ACO_indicators is a vector of indicators specifying each organization in the ACO group, omitting the control group as the reference group; HRR_indicators × year is a vector of indicators for each HRR in each year of the sample with a reference HRR-year combination omitted; ACO_contract is an indicator specifying a Pioneer ACO in 2012; and covariates include patient sociodemographic and clinical covariates described above. The ACO indicators adjust for an organization’s mean level of low-value services in the precontract period and for changes in the distribution of ACO-assigned beneficiaries across ACOs between the precontract and postcontract periods. The HRR indicators mean that estimates are based on comparisons of each beneficiary in the ACO group with control group beneficiaries in the same geographic area, and the interaction of HRR and year indicators adjusts for region-specific trends in the use of low-value services in the control group.

Thus, the quantity of interest (β₃) is the mean differential change in low-value services for ACO-attributed beneficiaries relative to local changes in low-value service use in the control group. To compare ACOs with higher vs lower baseline levels of low-value service use, we added to the model 2 interactions between the β₃ term and each of the 2 measures of ACOs’ baseline low-value service use.

A key assumption of this difference-in-differences analysis is that the difference in adjusted rates of low-value service use between the ACO group and the control group in the precontract period would have remained constant in the postcontract period in the absence of the Pioneer program.⁶⁵ We tested this assumption by comparing trends in low-value service use between the ACO group and control group during the 2009-2011 precontract period (eMethods in the Supplement).

We conducted several sensitivity analyses to test for potential sources of bias. First, we adjusted for any differences in trends in low-value service use between the ACO and control groups in the precontract period (eMethods and eFigure in the Supplement). Second, we excluded indicators of service qualification as covariates in case ACO contracts were associated with changes in the likelihood of patients satisfying qualifying conditions. Third, we tested for differential changes in sociodemographic and clinical characteristics from the precontract to postcontract periods between the ACO and control groups. If the composition of the ACO and control groups did not change differentially in these observed dimensions, it is less likely that there were differential changes in other unobserved dimensions. All analyses used robust variance estimators clustered at the level of ACOs (for the ACO group) or HRRs (for the control group).^66,67

The study sample included 693 218 person-years in the ACO group and 17 453 423 in the control group. In analyses adjusted for geographic area, beneficiary characteristics during the 2009-2011 precontract period were similar in the ACO and control groups, and differential changes in the ACO group were minimal (Table 2).

During the precontract period, the adjusted annual rate of low-value service use in the ACO group was 1.8 services per 100 beneficiaries lower (P = .02) than the control group (Table 3), but trends in the precontract period were similar (0.1 services per 100 beneficiaries per year greater for the ACO group; P = .74). Total spending on low-value services in the precontract period was similar for the ACO and control groups ($256 per 100 beneficiaries higher in the control group; P = .13), and trends were also similar ($20 per 100 beneficiaries per year greater for the control group; P = .88). The following results are reported as differential reductions (95% CIs). In year 1 of Pioneer contracts, there was a differential reduction in the use of low-value services for the ACO group (−0.8 services per 100 beneficiaries; −1.2 to −0.4; P < .001) or a reduction of 1.9% (−2.9% to −0.9%) relative to the expected 2012 mean for the ACO group of 41.0 services per 100 beneficiaries. This differential reduction in use corresponded to a 4.5% differential reduction in spending on low-value services (−7.5% to −1.4%; P = .004).

All clinical categories of low-value services except for preoperative services contributed to the overall differential reduction in the ACO group (Table 3). The differential reductions were statistically significant for 3 clinical categories (cancer screening, imaging, and cardiovascular testing and procedures). The greatest absolute reductions in service use occurred for the most frequently delivered services: cancer screening and imaging (Tables 1 and 3). Cardiovascular testing and procedures underwent the greatest differential reduction in relative terms (−6.3% for the ACO group; P = .05). In relative terms, differential reductions in low-value service use (differential relative reduction; 95% CI) were similar in magnitude for higher-priced services (−1.4%; −3.3% to 0.4%) and lower-priced services (−2.1%; −3.5% to −0.7%), as well as for services that were more and less sensitive to patient preferences (−1.7%; −3.2% to −0.3% vs −2.2%; −3.7% to −0.7%).

As shown in the Figure, ACOs with higher baseline levels of low-value service use than their service area exhibited a differential reduction of 1.2 services per 100 beneficiaries (95% CI, −1.7 to −0.7; P < .001 for test of estimate vs zero), and ACOs with lower baseline rates experienced a statistically insignificant differential reduction of 0.2 services per 100 beneficiaries (95% CI, −0.6 to 0.2; P = .41 for test of estimate vs zero; P = .003 for test of difference in differential reductions between ACO subgroups). Differential reductions in low-value service use were similar for ACOs serving areas with higher or lower baseline levels of low-value service use in the control group (P = .41).

Estimates were not substantially affected by adjusting for small differences in trends in low-value service use during the precontract period or by omitting service qualification indicators from regression models (eMethods and eFigure in the Supplement).

Although many studies have examined the effects of various provider payment reforms on health care spending and patient outcomes, the use of low-value services has not been a focus of this literature.⁶⁸ Use of such services has been shown to fall somewhat following the publication of clinical trials demonstrating their lack of effectiveness,^69,70 but whether payment reforms further discourage use of these services has not been assessed.

We found that the first year of the Medicare Pioneer ACO program was associated with a modest reduction in use of low-value services that could be measured directly with claims data. These results are consistent with the hypothesis that alternative payment models with global budgets can discourage overuse even while preserving broad provider discretion in determining which services are of low value. Notably, the first year of the Pioneer program was associated with a 4.5% differential reduction in spending on low-value services, substantially larger than the 1.2% reduction in overall spending previously estimated with the same methods.¹⁶ This finding suggests that Pioneer ACOs targeted low-value services in their efforts to reduce spending despite a lack of financial incentives or quality reporting requirements specifically concerning overused services.

Utilization changes occurred broadly across multiple clinical categories. Relative reductions were similar for higher-priced and lower-priced services, suggesting that overall reductions in low-value service use were not simply driven by restrictions on service use that could have occurred without causing significant losses in reimbursement under fee-for-service payment.⁷¹ Differential reductions in low-value service use were also similar for services that were more or less sensitive to patient preferences. This finding is consistent with providers in ACOs recommending fewer low-value services and with research^68,72,73 demonstrating that patient preferences may not be major obstacles to reducing low-value service use.

Reductions in low-value service use were concentrated among ACOs with higher baseline levels of use of these services relative to their service areas, whereas baseline performance of ACO geographic service areas did not predict reductions in low-value service use. First, these findings suggest that ACO initiatives may produce greater reductions in overuse if they encourage participation of provider organizations with more wasteful practices at baseline than other providers in their area. Second, these findings highlight the importance of practice variation within regional markets rather than across markets in predicting organizations’ prospects for improving efficiency.⁷⁴ In service areas where overuse is especially common, providers may face difficulties in markedly reducing low-value service use below local norms.

Several limitations of this study warrant discussion. First, organizations opting to participate in the volunteer Pioneer program may have been uniquely well positioned to identify and reduce wasteful practices. Consequently, similar results may not be achieved if the Pioneer program or similar programs are expanded to include a different set of provider organizations. Second, although our difference-in-differences study design controls for fixed differences between the ACO group and control group, and even though we detected no significant difference in temporal trends in low-value service use between these groups, it is nevertheless possible that an independent contemporaneous factor affecting ACOs produced a differential change in 2012. It is also possible that organizations entering the Pioneer program may have differentially reduced low-value service use even in the absence of the program. However, we found no evidence that these organizations were experiencing faster reductions in low-value service use before the ACO contracts. In addition, reductions in use of higher-priced low-value services would cause substantial losses in fee-for-service revenue in the absence of ACO contracts, and we found that reductions were unrelated to service price.

Third, given the limited number of organizations participating in the ACO program, we could not assess the many organizational characteristics that might modify reductions in the use of low-value services. Fourth, we only examined the first year of the Pioneer ACO program, which was the only year for which claims data were available. Although prior studies¹³ have shown increasing effects of commercial ACO contracts over time, the same pattern may not hold in Medicare. Finally, our results do not constitute conclusive evidence of value improvement among Pioneer ACOs. It is possible that important high-value services also experienced reductions in 2012.

The Pioneer ACO program was associated with a modest reduction in low-value services, with greater reductions within organizations providing more low-value care. Despite the limitations of the study, our findings, taken together with those of studies demonstrating spending reductions greater than Medicare bonus payments¹⁶ and improved or stable performance on measures of patient experiences and quality,¹² are consistent with the conclusion that the overall value of health care provided by Pioneer ACOs improved after their participation in an alternative payment model. Finally, our study demonstrates the utility of novel measures of low-value service use for evaluating the effects of health care policy initiatives.

Accepted for Publication: June 22, 2015.

Corresponding Author: J. Michael McWilliams, MD, PhD, Department of Health Care Policy, Harvard Medical School, 180 Longwood Ave, Boston, MA 02115 (mcwilliams@hcp.med.harvard.edu).

Published Online: September 21, 2015. doi:10.1001/jamainternmed.2015.4525.

Author Contributions: Drs Schwartz and McWilliams had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Schwartz.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Schwartz, McWilliams.

Obtained funding: Schwartz, McWilliams.

Administrative, technical, or material support: Chernew.

Study supervision: Chernew, Landon, McWilliams.

Conflict of Interest Disclosures: Drs Schwartz and McWilliams report consulting for the Medicare Payment Advisory Commission on the use of measures of low-value care. Dr Chernew reports that he is a partner in VBID Health, LLC, which has a contract with Milliman to develop and market a tool to help insurers and employers quantify spending on low-value services. No other disclosures are reported.

Funding/Support: This study was supported by grants from the National Institute on Aging (P01 AG032952-06A1 and F30 AG044106-01A1) and Laura and John Arnold Foundation. We also acknowledge funding from the National Institute of Mental Health (grant U01 MH103018) for work involving the development and operationalizing of measures of low-value care.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: Adam Elshaug, PhD (Menzies Centre for Health Policy, University of Sydney), contributed to the selection of low-value services for measurement, Pasha Hamed, MA, provided statistical programming consultation, and Jesse B. Dalton, MA (Department of Health Care Policy, Harvard Medical School), assisted with research. There was no financial compensation.