Association of Pioneer Accountable Care Organizations vs Traditional Medicare Fee for Service With Spending, Utilization, and Patient Experience

Importance  The Pioneer Accountable Care Organization (ACO) Model aims to drive health care organizations to reduce expenditures while improving quality for fee-for-service (FFS) Medicare beneficiaries.

Objective  To determine whether FFS beneficiaries aligned with Pioneer ACOs had smaller increases in spending and utilization than other FFS beneficiaries while retaining similar levels of care satisfaction in the first 2 years of the Pioneer ACO Model.

Design, Setting, and Participants  Participants were FFS Medicare beneficiaries aligned with 32 ACOs (n = 675 712 in 2012; n = 806 258 in 2013) and a comparison group of alignment-eligible beneficiaries in the same markets (n = 13 203 694 in 2012; n = 12 134 154 in 2013). Analyses comprised difference-in-differences multivariable regression with Oaxaca-Blinder reweighting to model expenditure and utilization outcomes over a 2-year performance period (2012-2013) and 2-year baseline period (2010-2011) as well as adjusted analyses of Consumer Assessment of Healthcare Providers & Systems (CAHPS) survey responses among random samples of beneficiaries in Pioneer ACOs (n = 13 097), FFS (n = 116 255), or Medicare Advantage (n = 203 736) for 2012 care.

Exposures  Beneficiary alignment with a Pioneer ACO in 2012 or 2013.

Main Outcomes and Measures  Medicare spending, utilization, and CAHPS domain scores.

Results  Total spending for beneficiaries aligned with Pioneer ACOs in 2012 or 2013 increased from baseline to a lesser degree relative to comparison populations. Differential changes in spending were approximately −$35.62 (95% CI, −$40.12 to −$31.12) per-beneficiary-per-month (PBPM) in 2012 and -$11.18 (95% CI, −$15.84 to −$6.51) PBPM in 2013, which amounted to aggregate reductions in increases of approximately −$280 (95% CI, −$315 to −$244) million in 2012 and −$105 (95% CI, −$148 to −$61) million in 2013. Inpatient spending showed the largest differential change of any spending category (−$14.40 [95% CI, −$17.31 to −$11.49] PBPM in 2012; −$6.46 [95% CI, −$9.26 to −$3.66] PBPM in 2013). Changes in utilization of physician services, emergency department, and postacute care followed a similar pattern. Compared with other Medicare beneficiaries, ACO-aligned beneficiaries reported higher mean scores for timely care (77.2 [ACO] vs 71.2 [FFS] vs 72.7 [MA]) and for clinician communication (91.9 [ACO] vs 88.3 [FFS] vs 88.7 [MA]).

Conclusions and Relevance  In the first 2 years of the Pioneer ACO Model, beneficiaries aligned with Pioneer ACOs, as compared with general Medicare FFS beneficiaries, exhibited smaller increases in total Medicare expenditures and differential reductions in utilization of different health services, with little difference in patient experience.

In 2012, the Centers for Medicare & Medicaid Services (CMS) launched the Pioneer Accountable Care Organization (ACO) Model and the Medicare Shared Savings Program (MSSP) as alternative payment approaches to engage physicians and health care organizations willing to assume collective responsibility for the cost and quality outcomes of a specified population of fee-for-service (FFS) Medicare beneficiaries.¹ FFS beneficiaries are attributed to ACOs based on their historical utilization patterns but are free to visit any physicians and health care facilities that participate in Medicare. Although MSSP is a permanent program, the Pioneer ACO Model is a more ambitious test of population health management that targets more experienced organizations with greater incentives for motivating the care transformation necessary to improve outcomes.

Quiz Ref IDAs part of the incentive to participate, Pioneer ACOs with an annual spending level lower than a projected spending level based on actuarial determinations can receive a portion of the difference between their spending and the projection as shared savings with CMS, conditional on their performance on a set of 33 quality measures. Beginning in 2013, Pioneer ACOs could conversely owe shared losses to CMS if spending for the beneficiaries exceeded the projected spending level. Between 2012 and 2013, Pioneer ACOs generated approximately $183 million in savings to the Medicare program relative to projected spending levels and improved their mean clinical quality scores from 70.8% to 84.0%, with the mean score increasing for 28 of 33 quality measures.² However, these results do not account for many factors that may confound the relationship between the model intervention and patient outcomes. In addition, the incentives to constrain spending to achieve shared savings could lead to reductions in utilization that beneficiaries might experience as barriers to care.³ Recent studies of spending among a sample of beneficiaries attributed to Pioneer ACOs found that total spending was a risk-adjusted −$29.20 per-beneficiary-per-quarter less than a control group in 2012 and that care experiences for beneficiaries were constant or slightly improved a year into being attributed to an ACO.^4,5

The Pioneer ACO Model is one of several attempts to test the viability of the ACO concept as means to improve quality of care and reduce spending in the US health care system.^1,6 This study assessed outcomes for beneficiaries cared for by physicians participating in Pioneer ACOs relative to likely outcomes among beneficiaries in the traditional Medicare program alone during the first 2 years of the model.

The beneficiaries used to calculate Pioneer ACOs’ shared savings for 2012 and 2013 were linked to their demographic, enrollment, and claims data in the Chronic Conditions Warehouse. Beneficiaries were aligned with all 32 Pioneer ACOs in 2012 or 2013, which included the 9 ACOs that participated through mid-year 2013 but were no longer participating by the end of that year.

The design of the analysis was predicated on the structure of the Pioneer ACO Model. ACOs update their participating clinician lists annually, which CMS uses to prospectively align beneficiaries at the start of each calendar year. Because the population of aligned beneficiaries changes each year, they were analyzed separately against a fixed baseline period to capture how ACOs’ year-to-year decisions around additions or subtractions to their participating clinician lists affect the care they deliver to their aligned patient population.

To be eligible for prospective alignment in a given year, a beneficiary must have had full Part A and B Medicare coverage; resided in the United States; and received at least half of his or her qualified evaluation and management (QEM) services from clinicians in the Pioneer’s market.⁷ A beneficiary was excluded if he or she were attributed to an MSSP ACO; enrolled at any time in a Medicare Advantage plan; or if Medicare was a secondary payer. The Pioneer alignment algorithm aligns beneficiaries to ACOs using combinations of tax identification numbers (TINs) and national provider identifiers (NPIs) for primary care clinicians and, in some cases, medical specialists participating in a given ACO. A beneficiary’s QEM allowed charges are weighted in each of the 3 alignment years preceding a performance year by 10% for the earliest year; 30% for the middle year; and 60% for the most recent year. If the plurality of a beneficiary’s weighted QEM allowed charges during the alignment years were with a Pioneer ACO’s TIN-NPI combinations, then the beneficiary was aligned with the ACO as long as the beneficiary remained alive at the start of the performance year (eFigure 1 in the Supplement). Details of the Pioneer ACO alignment methodology are available elsewhere.⁷

For the purposes of this analysis, each Pioneer ACO’s market was defined as the counties where Pioneer-participating clinicians were located and all contiguous counties in 2012. Among ACOs that participated through the end of 2013, a post hoc analysis found that an average of 97% of primary care clinicians participating in 2013 were located in counties comprising the originally defined market.

To differentiate the performance of ACOs against preexisting trends, the Pioneer alignment algorithm was used to identify decedent and nondecedent beneficiaries who would have been aligned to 2012 Pioneer clinicians in 2010 and 2011. Assuming they were active in the Medicare program, these clinicians treated Medicare beneficiaries prior to the inception of the Pioneer model, so the baseline population represents beneficiaries who were receiving much of their care from clinicians who would later join a Pioneer ACO. Only 2 alignment years with reweighted QEM allowed charges could be used to construct baseline populations of aligned beneficiaries in 2010 and 2011 because NPIs were required for alignment but not reliably used in billing prior to 2008 (eFigure 1 in the Supplement).

For each Pioneer ACO, a comparison beneficiary population was selected to approximate outcomes that would have occurred in the absence of the model. The comparison population included all alignment-eligible beneficiaries in the ACO’s market not attributed to any Medicare ACO during 2012-2013. Comparison populations in the baseline period were identified by retaining all alignment-eligible beneficiaries in ACO markets after removing beneficiaries attributed to any Medicare ACO in 2010-2011.

Quiz Ref IDThe Agency for Healthcare Research and Quality is responsible for overseeing the development of Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys, which are used by CMS to measure Medicare patient experiences with health care facilities, physicians, and managed care plans.⁸ The ACO CAHPS survey was developed from the Clinician & Group Survey version of CAHPS. A technical expert panel recommended survey items and domains, which were refined through input from Medicare beneficiary focus groups, cognitive interviews, and field testing.⁹

CMS selected a random target sample of 775 aligned beneficiaries per Pioneer ACO to survey about care they received in the prior 6 months. An approved survey vendor administered the ACO CAHPS survey from January to April 2013, and beneficiaries could respond through a mailed questionnaire or structured phone interview.

Pioneer ACO CAHPS data were compared with FFS CAHPS data and Medicare Advantage CAHPS data collected during the same period. The analysis was limited to 2012 to examine patient experience with respect to care delivered by clinicians in all 32 original Pioneer ACOs, since beneficiaries aligned with ACOs that ceased participating in the Pioneer ACO Model in 2013 were not surveyed about care they received that year.

Common CAHPS items among Pioneer ACO, FFS, and Medicare Advantage CAHPS surveys were grouped into 4 domains: (1) getting timely care, appointments, and information; (2) ease of getting care; (3) clinician communication; and (4) access to specialists (eTable 1 in the Supplement). In accordance with standard CAHPS reporting, responses to each item were collapsed to a scale (range, 1-3; 1 = “never” or “sometimes”; 2 = “usually”; 3 = “always”). Average scores per domain were constructed after risk adjusting and weighting individual responses by the number of respondents per ACO. These scores were then converted to a scale (range, 0-100) using recommended methods for analyzing CAHPS data.¹⁰

Federal common rule (section 45 CFR 46.101[b][5]) provides an exemption from the institutional review board requirements when the purpose of the research is to study, evaluate, or otherwise examine a public benefit or service program. Contractors signed a data use agreement to ensure all data were securely and solely used to study the Pioneer ACO model. CAHPS questionnaire data collection was subject to informed consent of beneficiaries, who were informed that their participation did not affect their future care in any way and were not offered compensation for their responses. CAHPS data analyzed in this study were deidentified at the patient level.

Outcome measures derived from claims data included total Medicare payments for hospital inpatient (Part A), physician services (Part B), hospital outpatient, skilled nursing facility, home health, hospice, and durable medical equipment. Since comparison populations were drawn from the same geographic areas as ACO populations, price differences in payments were expected to be a minimal concern. Utilization measures included acute inpatient days, evaluation and management office visits with primary care clinicians (general medicine, family medicine, internal medicine, geriatric medicine, nurse practitioner, or physician assistant), and major physician service categories as well as emergency department and postacute care. Other measures included all-cause 30-day acute hospital readmissions and 7-, 14-, and 30-day follow-up visits after hospital discharge as measures of care transitions to the community.

Spending and utilization outcomes for ACO and comparison populations were measured separately for each performance year relative to an averaged 2-year baseline period. Outcomes for each baseline year were averaged to generate a single baseline value against which the outcomes in each performance year were measured. Outcomes are reported in terms of beneficiary months, since not all beneficiaries remained eligible for alignment or were alive for the entirety of a given year.

Beneficiary-level models of spending and utilization were estimated using multivariable regression with Oaxaca-Blinder adjustment.^11,12 This adjustment reweights observations in the comparison population to more closely match the distribution of beneficiary characteristics in the ACO population.^10,11 It was used rather than propensity scores, since the beneficiary selection bias that propensity scores attempt to mitigate is less of a concern because beneficiaries do not directly choose to be aligned with a Pioneer ACO but are instead passively aligned by CMS through a claims-based algorithm. Moreover, the comparison population includes the universe of beneficiaries otherwise eligible for alignment within an ACO’s market, rather than a sample of such beneficiaries.

The multivariable model covariates included beneficiary indicator variables for age (<64 years, 65-74 years, 75-84 years, and ≥85 years); sex; race/ethnicity (white, black, Hispanic, Asian/Pacific Islander, and other); Medicaid dual eligibility; end-stage renal disease; death in the year of observation; separate indicators for hip fracture, colorectal cancer, stroke, acute myocardial infarction, or lung cancer in the year; and separate indicators for whether a beneficiary had any of these conditions in the 3 prior years. These conditions were chosen because they reflect illness severity but are unlikely to be influenced by diagnostic coding practices.¹³

All models also included indicator variables for beneficiary alignment with an ACO, performance year, and interaction terms for the year of observation and whether the beneficiary was aligned to an ACO in that year. The adjusted results for each outcome were generated for beneficiaries in Pioneer ACOs (intervention group) and other beneficiaries in Pioneer ACO markets (comparison group) for each baseline and performance year as difference-in-differences estimates of conditional means. The difference-in-differences approach calculates the change over time in the intervention group minus the change over time in the comparison group. The change over time for the intervention group was calculated as its performance-year conditional mean minus its baseline conditional mean; the change over time for the comparison group was calculated as its performance-year conditional mean minus its baseline conditional mean. Then, the difference-in-differences estimate was computed by subtracting the comparison group difference from the intervention group difference.

Standard errors were calculated through 400 bootstrapped samples and converted to 95% confidence intervals. Bootstrapping estimates the standard error of a mean by taking samples of observations to account for clustering among observations within an ACO, with 2-sided statistical significance at the P < .05 threshold. Standard errors were calculated for each ACO and for all ACOs combined, with weighting by the number of aligned beneficiaries per ACO.

CAHPS outcomes were modeled using stereotype logistic regression given the ordered nature of CAHPS responses and included several covariates: age; sex; education level; race/ethnicity; use of a proxy to read, write, or respond to the survey; self-reported general and mental health, presence of chronic conditions, and functional impairments.¹⁴ ACO CAHPS outcomes were adjusted for additional factors, since beneficiaries were sampled at the level of the ACO: number of ACO-aligned beneficiaries; percentage of ACO-aligned beneficiaries in urban counties; Medicare Advantage penetration; household income; and average total Medicare payments in the ACO’s market. Results were compared for statistical significance at the 2-sided P < .05 threshold by creating 95% confidence intervals from the standard errors of 200 bootstrap replications against FFS CAHPS as a reference group.

Sensitivity analyses of total spending were conducted with subgroups of ACOs as well as alternative risk adjustment strategies. We estimated total spending separately for the 23 Pioneer ACOs that continued with the model through 2013 and the 9 Pioneer ACOs that did not (additional outcomes in eTable 2 in the Supplement). To examine whether there might be any evidence of a spillover effect from the care Pioneer ACO physicians delivered to nonaligned beneficiaries in an ACO’s own market, we tested whether total spending for all 32 Pioneer ACOs differed relative to comparison groups from separate markets that resembled each ACO’s market characteristics (eTable 3 in the Supplement). To test the robustness of our risk adjustment approach, we compared alternative risk adjustment approaches—31 Elixhauser comorbidity indicators and the Hierarchical Condition Categories score—with intraclass correlation coefficients on 1 year of data (eTable 4 in the Supplement).

Beneficiaries aligned with Pioneer ACOs were similar to comparison populations across baseline and performance years (Table 1). The number of beneficiaries aligned with Pioneer ACOs and their comparison populations each year ranged from 647 371 to 806 258 and from 12 134 154 to 14 611 733, respectively. Unadjusted proportions of demographic and clinical characteristics of beneficiaries, such as race/ethnicity, Medicaid dual eligibility, and history of chronic conditions, were similar between ACO and comparison populations each year, with a slightly higher proportion of women beneficiaries in the ACO population in baseline years.

Table 2 shows the conditional means and differential changes in spending and utilization for Pioneer ACO–aligned beneficiaries relative to their comparison and baseline populations of FFS beneficiaries. In 2012, Pioneer ACOs had a significantly smaller increase in adjusted expenditures of −$35.62 (95% CI, −$40.12 to −$31.12) per-beneficiary-per-month (PBPM) and −$11.18 (95% CI, −$15.84 to −$6.51) PBPM in 2012 and 2013, respectively. These differences were largely driven by a decrease in spending by Pioneer ACOs in 2012 after parallel trends in spending between ACO and comparison populations in the baseline years (eFigure 2 in the Supplement). These differential changes translate into aggregate reductions in increases of almost −$280 (95% CI, −$315 to −$244) million in 2012 and approximately −$105 (95% CI, −$148 to −$61) million in aggregate in 2013. In both years, inpatient spending (Part A) accounted for the largest share of the smaller increase in spending (−$14.40 [95% CI, −$17.31 to −$11.49] PBPM in 2012 and −$6.46 [95% CI, −$9.26 to −$3.66] PBPM in 2013), followed by physician services (Part B) (−$8.29 [95% CI, −$9.32 to −$7.27] PBPM in 2012 and −$2.69 [95% CI, −$3.84 to −$1.54] PBPM in 2013). Spending in all other categories demonstrated significantly smaller increases in 2012, ranging from −$5.82 (95%CI, −$6.76 to −$4.88) PBPM for hospital outpatient to −$1.06 (95% CI, −$1.59 to −$0.54) PBPM for home health; only durable medical equipment was statistically significant in 2013 (−$0.92 [95% CI, −$1.13 to −$0.71] PBPM).

Differential changes in utilization were similar (Table 2). Acute inpatient days per 1000 beneficiary months decreased more for ACOs than for the comparison group in 2012 (−0.05 days [95% CI, −0.065 to −0.039]) and 2013 (−0.02 days [95% CI, −0.029 to −0.004]), as did evaluation and management office visits to primary care clinicians per 100 beneficiary months (−2.9 [95% CI, −3.0 to −2.8] visits in 2012 and −2.6 [95% CI, −2.7 to −2.5] visits in 2013). Major categories of physician services—procedures, imaging procedures, and tests—were associated with smaller increases for ACOs in both years, although the magnitude of the differences decreased in 2013. Differences in emergency department visits and inpatient admissions through the emergency department were statistically significant and either decreased more or increased less in both years, while differences in observation stays were not significant in 2012 but increased significantly more in 2013. Skilled nursing facility days and home health visits showed larger increases, while inpatient rehabilitation days and hospice days showed smaller increases for ACOs in 2012. Differences in skilled nursing facility days, home health visits, and hospice days were not significant in 2013.

Despite differential decreases in primary care office visits for evaluation and management, hospital discharge follow-up visits within 7 days had significant differential increases from 11.3 (95% CI, 4.6 to 18.0) visits per 1000 discharges in 2012 to 14.8 (95% CI, 8.5 to 21.0) visits per 1000 discharges in 2013 for beneficiaries aligned with ACOs. A significant differential increase was also seen in 2013 for follow-up visits within 14 days of discharge (10.7 [95% CI, 4.9 to 16.4] per 1000 discharges) but not within 30 days in either year. Differences in rates of all-cause readmissions were not significant in either year (−2.3 [95% CI, −5.6 to 0.9] per 1000 discharges in 2012 and 3.1 [95% CI, −0.1 to 6.4] per 1000 discharges in 2013).

Figure 1 displays the differential changes in PBPM spending for Pioneer ACOs relative to their comparison populations in 2012 and 2013 grouped by geographic areas with more than 1 ACO: Boston, California, and the Upper Midwest. Beneficiaries aligned with many Pioneer ACOs in these markets were associated with small increases in expenditures. Those aligned with all 5 Pioneer ACOs in the Boston market had significantly smaller increases in spending that averaged −$54.90 PBPM relative to their comparison groups in 2012, whereas in 2013, 3 Pioneer ACOs realized a smaller increase in spending, 1 had larger increases in spending, and another had statistically indistinguishable results at an overall average of −$31.42 PBPM. Beneficiaries aligned with all 5 California ACOs had significantly smaller increases in spending in 2012, averaging −$51.40 PBPM; 1 had significantly smaller increases and 1 had larger increases in spending, for a total average of −$9.32 PBPM in 2013. Beneficiaries aligned with 1 ACO in the Upper Midwest generated significantly smaller increases in spending both years and those in a second ACO in that region did the second year. In 2012, beneficiaries aligned with 8 of the 18 Pioneer ACOs in other geographic areas had significantly smaller increases in spending than their comparison populations in 2012, whereas 5 of the remaining 18 ACOs had relatively smaller increases in spending in 2013.

In sensitivity analyses, differences in total spending when restricting to the 23 ACOs remaining in the Pioneer ACO Model through 2013 remained statistically significant at an estimated smaller increase of −$36.73 (95% CI, −$41.14 to −$32.32) PBPM in 2012 and −$17.38 (95% CI, −$21.93 to −$12.82) PBPM in 2013 (eTable 2 in the Supplement). Using separate markets as a comparison population, beneficiaries aligned with all 32 Pioneer ACOs had statistically smaller increases in total spending in 2012 that were larger in magnitude than the comparison group within ACO markets (−$44.78 [95% CI, −$75.79 to −$13.77] PBPM) but were not statistically significant in 2013 (−$22.71 [−$55.49 to $10.07] PBPM). The multivariable model performed similarly with different risk adjustment specifications, with the correlation coefficient among combinations of models more than 0.7 in all cases (eTable 4 in the Supplement).

A total of 13 097 ACO CAHPS questionnaires were completed, with response rates ranging from 40.3% to 61.5% per ACO and an average response rate of 52.8%, which was higher than the FFS CAHPS response rate of 42.7% (n = 116 255 completed questionnaires) and the Medicare Advantage response rate of 46.0% (n = 203 736 completed questionnaires) for the same period.

Despite smaller increases in spending in 2012, beneficiaries aligned with Pioneer ACOs reported experiences with care that were clinically similar to the experiences reported by FFS Medicare beneficiaries as a whole and their Medicare Advantage counterparts, with higher ratings in the timely access to care and clinician communication domains (Figure 2). For example, compared with other Medicare beneficiaries, ACO-aligned beneficiaries reported higher mean scores for timely care (77.2 [ACO] vs 71.2 [FFS] vs 72.7 [MA]) and for clinician communication (91.9 [ACO] vs 88.3 [FFS] vs 88.7 [MA]).

Quiz Ref IDDuring Pioneer ACOs’ first 2 performance years, total spending for 1 481 970 aligned beneficiaries increased approximately $385 million ($280 million in year 1; $105 million in year 2) less than spending of similar FFS beneficiaries. A large portion of the smaller increase in spending was from decreases in inpatient utilization among ACO-aligned beneficiaries, although greater decreases in primary care evaluation and management office visits, and smaller increases in the use of tests, procedures, and imaging services, also were related to the observed differential changes in spending. There was no difference in all-cause readmissions within 30 days of discharge, but follow-up visits after hospital discharge increased more for ACO-aligned beneficiaries. Beneficiary reports of care experiences were similar to both the general FFS and Medicare Advantage populations in the first performance year and may have been better in terms of timely care and clinician communication.

These results are encouraging, given how historically challenging it has been for physicians to achieve spending reductions in Medicare demonstration projects.¹⁵ The relative reductions in both inpatient and physician service utilization are largely consistent with results from the Physician Group Practice Demonstration, a Medicare shared savings initiative widely considered to be the precursor of current Medicare ACO programs and models.¹⁶ In the first year of a commercial ACO-like arrangement, the Blue Cross Blue Shield of Massachusetts Alternative Quality Contract, decreases in spending were driven by referrals to care settings with lower prices.¹⁷ In contrast, decreases in Medicare spending for beneficiaries in Pioneer ACOs relative to their comparison populations were related to significant reductions in utilization in a range of care settings.

These findings are consistent with those from another study of the first year of the Pioneer model. McWilliams et al⁴ found spending reductions in several care settings and a total reduction in spending growth in 2012 of approximately −$10 PBPM, which is smaller in magnitude than our finding of about −$35 PBPM. Differences in the magnitude of smaller increases between the 2 studies may be primarily a function of how beneficiary populations were selected for both the ACO and comparison groups. We used the ACOs’ prospective beneficiary alignment lists and replicated the alignment algorithm as closely as feasible to construct baseline populations, mitigating the chance for regression to the mean in the outcomes. The comparison group included all otherwise alignment-eligible beneficiaries within the counties and contiguous counties of Pioneer ACO physicians. In contrast, McWilliams et al used a retrospective beneficiary assignment methodology that also assigned beneficiaries within hospital referral regions around ACOs to a control group of non-ACO TINs. Since we did not attribute the comparison population to TINs, it is possible that this step may have made the comparison group appear more like large organizations, thereby attenuating the differences between the ACO and comparison groups. Our comparison group was the general population of FFS Medicare beneficiaries since the aim of Pioneer ACOs is to improve outcomes for this population.

Quiz Ref IDWe found greater reductions in spending growth in the first performance year of the Pioneer ACO Model than in the second year, which may be explained by several reasons. First, ACOs may have achieved large decreases in the growth of spending in the first year by focusing on care management of high-cost patients, which may have been difficult for ACOs to sustain in the second year. Second, ACOs may have faced challenges sustaining their performance as a result of turnover in participating physicians and their aligned beneficiaries. Third, since beneficiaries in the comparison group could still receive care from ACO-affiliated physicians, the practice patterns of those physicians may have affected the care of other Medicare patients, making care of patients in the comparison group appear more similar to that of ACO-attributed patients over time. Such spillover effects of ACOs have been noted for Medicare beneficiaries treated by physicians under a commercial risk-based contract prior to the existence of Medicare ACOs.¹⁸ When we considered comparison populations in separate markets, we found larger reductions in spending growth each performance year with respect to comparison populations within the ACO markets, although they were no longer statistically significant in the second year. The larger reductions in spending growth in the separate markets suggest the possibility of spillover in ACO markets; however, these results were not adjusted for price differences between markets. Last, it may be challenging for ACOs to continually improve on baseline performance over time, particularly since growth in spending in the overall FFS Medicare program has declined in recent years.¹⁹

Despite decreases in spending growth, results from this study and previously reported data on Pioneer ACOs’ performance on clinical quality measures suggest it is possible to reduce expenditure growth while maintaining or improving quality in a FFS payment environment. In addition, beneficiaries aligned with Pioneer ACOs reported similar experiences of care compared with the general FFS and Medicare Advantage populations. These results are consistent with other research showing that patients in Medicare ACOs tend to report some improvements in the timeliness of their care and clinicians’ knowledge of a patient’s use of specialists, with otherwise no decrements in access.¹⁹

Pioneer ACOs demonstrated that growth in Medicare spending can be reduced in a variety of market contexts, at least in the early years of the intervention. The Boston metropolitan area is a leading market in risk-based commercial insurance contracting and has the largest concentration of Pioneer ACOs, most of which had smaller increases in spending than their comparison populations.¹⁶ Similarly, most Pioneer ACOs in California had relatively smaller increases in spending even though California has historically been a leader in managed care and organized medical groups, which means that both ACOs and physicians seeing patients in their comparison groups would have had experience with risk-based contracting.^20,21 The Upper Midwest has been characterized for years by relatively low levels of health care utilization, in which even modest reductions in spending growth may be more difficult to achieve for an ACO, yet some Pioneer ACOs were able to exhibit smaller increases in spending than their comparison populations.²²

Although such success may be replicable in regions with varying market characteristics, not all ACOs did well fiscally—one-third of Pioneer ACOs did not generate lower expenditure growth relative to their comparison populations in their first 2 years, and 2 generated significantly higher expenditure growth their second year. Multiple factors may contribute to these findings. It may take more time for some ACOs to redesign care delivery and learn how to effectively manage the care of a population of FFS Medicare beneficiaries to realize smaller increases in spending. CMS may also need to reexamine specific design elements to facilitate better performance, such as expenditure benchmarking methodologies that are more predictable to the ACO or enhanced benefits and other tools to engage beneficiaries.²³ Reducing Medicare spending through the Pioneer ACO model, then, likely depends on an array of market, organizational, programmatic, and physician-related factors that should be better understood in future implementation and research.²⁴

For individual Pioneer ACOs, the results of this difference-in-differences analysis differ from the results that determine whether they share in savings with CMS.²⁵ The latter results were derived from reconciling the ACO’s projected and actual spending levels as part of their financial incentive to participate in the model, whereas the primary goal of this analysis was to assess the performance of Pioneer ACOs on a set of key spending and utilization outcomes compared with such outcomes in the absence of the intervention. Quiz Ref IDThis study does not address other important concerns such as the degree to which the Pioneer ACO model can sustain small increases in spending and high quality performance over longer periods or whether Pioneer ACOs can achieve a meaningful return on investment for the resources they devote to improving care. These results are also limited to Pioneer ACOs and do not include other types of Medicare ACOs.

Our study is not without additional limitations. First, CMS selected these ACOs to participate in the Pioneer model because they demonstrated the capacity to manage the care of a patient population and many had experience in risk contracting arrangements; hence, by design they deliver care inherently different from the care received by the typical FFS beneficiary. Second, since it would not be operationally feasible to identify a control group of similarly structured and experienced organizations as Pioneer ACOs, neither the participating physicians nor their aligned beneficiaries were randomized, which means that despite efforts to control for differences in patient characteristics and disease burden, our analyses may not have accounted for unmeasured differences between ACO and comparison beneficiary populations. Third, because each ACO’s comparison group comprised similar populations of geographically bounded FFS beneficiaries, any spillover in practice patterns from physicians affiliated with ACOs to patients not aligned with ACOs would attenuate differences in outcomes between them. Fourth, total spending does not include Part D drug spending or cost-sharing payments by beneficiaries. Fifth, the response rate for the ACO CAHPS survey was only 52.8% and no information is available about nonresponders in any of the CAHPS surveys.

In the first 2 years of the Pioneer ACO Model, beneficiaries aligned with Pioneer ACOs, as compared with general Medicare FFS beneficiaries, exhibited smaller increases in total Medicare expenditures and differential reductions in utilization of different health services, with little difference in patient experience.

Corresponding Author: Rahul Rajkumar, MD, JD, Centers for Medicare & Medicaid Services, Center for Medicare and Medicaid Innovation, 7500 Security Blvd, Mailstop WB-06-05, Baltimore, MD 21244 (rahul.rajkumar@cms.hhs.gov).

Published Online: May 4, 2015. doi:10.1001/jama.2015.4930.

Author Contributions: Dr Nyweide had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: Nyweide, Lee, Cuerdon.

Drafting of the manuscript: All authors.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Nyweide, Lee, Cuerdon.

Obtained funding: Nyweide, Pham.

Administrative, technical, or material support: All authors.

Study supervision: Nyweide, Lee, Cuerdon, Rajkumar, Conway.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Disclaimer: This article represents the views of the authors and not necessarily the views or policies of CMS or the evaluation contractors.

Additional Information: CMS contracted L&M Policy Research LLC and its partners—Abt Associates, Avalere Health, Social & Scientific Systems, and Truven Health Analytics—to conduct the claims and CAHPS analyses. SAS EG (version 5.1) was used to create the claims files, and STATA (version 13.0) was used to analyze claims and CAHPS data. CMS provided guidance with performing and interpreting the analyses and designed and wrote the manuscript.

This article is based on the work of the evaluation contractors for the Evaluation of CMMI Accountable Care Organization Initiative performed under Contract HHSM-500-2011-00019i/HHSM-500-T0002 with the Centers for Medicare & Medicaid Services (CMS). CMS provided access to the data to the evaluation contractors to perform analyses under the guidance of Drs Nyweide, Lee, and Cuerdon. In conjunction with the contractors, Drs Nyweide, Lee, and Cuerdon designed the study and helped with interpretation of the data.

Additional Contributions: Special thanks to Renee Mentnech, MS, BSN, and Matthew Press, MD (Center for Medicare and Medicaid Innovation, Centers for Medicare & Medicaid Services), for their feedback on an earlier version of the manuscript. Neither received compensation for their contributions.