Pay for Performance Alone Cannot Drive Quality

Objective  To determine whether aligning design characteristics of a pay-for-performance program with objectives of an asthma improvement collaborative builds improvement capability and accelerates improvement.

Design  Interrupted time series analysis of the impact of pay for performance on results of an asthma improvement collaborative.

Setting  Forty-four pediatric practices within greater Cincinnati.

Participants  Forty-four pediatric practices with 13 380 children with asthma.

Interventions  The pay-for-performance program rewarded practices for participating in the collaborative, achieving network- and practice-level performance thresholds, and building improvement capability. Pay for performance was coupled with additional improvement interventions related to the collaborative.

Outcome Measures  Flu shot percentage, controller medication percentage for children with persistent asthma, and written self-management plan percentage.

Results  The pay-for-performance program provided each practice with the potential to earn a 7% fee schedule increase. Three practices earned a 2% increase, 13 earned a 4% increase, 2 earned a 5% increase, 14 earned a 6% increase, and 11 earned a 7% increase. Between October 1, 2003, and November 30, 2006, the percentage of the network asthma population receiving “perfect care” increased from 4% to 88%. The percentage of the network asthma population receiving the influenza vaccine increased from 22% to 41%, and then to 62% during the prior 3 flu seasons.

Conclusion  Linking design characteristics of a pay-for-performance program to a collaborative focused on improving care for a defined population, building improvement capability, and driving system changes at the provider level resulted in substantive and sustainable improvement.

Despite the rapid growth of pay-for-performance programs across the United States,^1-3 evidence regarding their effect on quality of care is limited.^4-11 Even in instances in which pay-for-performance programs have been linked to measurable improvement, attribution is problematic.^12-14 Although “guiding principles” exist,^8,9,15-20 the lack of evidence regarding effective design characteristics for pay-for-performance programs remains a significant concern.

Although ideal aspects of pay-for-performance programs remain elusive, we hypothesized that aligning pay-for-performance program design characteristics with the primary objectives of a large- scale asthma improvement collaborative and coupling pay for performance with other interventions would enhance improvement capability and accelerate improvement, within and across primary care practices. This approach was based on the contention that pay for performance should be viewed as a catalyst to accelerate sustainable transformation at the provider level and that an overdependence on pay for performance alone to drive quality should be avoided. Based on results achieved, key pay-for-performance program design principles will be reviewed to inform the national dialog among providers, payers, and employers.

The Physician-Hospital Organization (PHO) affiliated with Cincinnati Children's Hospital Medical Center launched an asthma improvement collaborative in October 2003, impacting more than 13 000 children with asthma across 44 primary care practices (165 physicians) within greater Cincinnati, representing approximately 35% of the region's pediatric asthma population. The primary care practices are organized as an independent practice association. The PHO elected to focus on asthma because the prevalence is high, care is usually managed by primary care practices, and extensive literature exists regarding the positive impact of improvement interventions on process and outcome measures. The aim of the asthma initiative is to improve evidence-based care, thus reducing asthma-related emergency department/urgent care visits, admissions, office visits because of acute symptoms, missed school days, missed workdays, and daytime and evening symptoms. The initiative is also designed to build improvement capability and redesign care delivery within primary care practices, thus supporting sustainable systems for future improvement.

The PHO approached Anthem Blue Cross and Blue Shield in Ohio (Anthem) in early 2004 to recruit support for an asthma pay-for-performance program. Anthem provides coverage to the highest percentage of the commercially insured population in greater Cincinnati.²¹ Anthem agreed to fund the pay-for-performance program and allowed the PHO to design the program. Primary objectives of the asthma pay-for-performance program were to reward measurable improvements in asthma care achieved at the network and practice level for the all-payer population, accelerate practice engagement in improvement work, support the business case for quality improvement, obtain experience designing and administering pay-for-performance programs, and influence the design of future pay-for-performance programs initiated by payers and employers.

Recognizing the importance of coupling pay for performance with a comprehensive approach to quality improvement, the asthma collaborative included the following strategies: multidisciplinary quality leadership teams at each practice (ie, physician, nurse or medical assistant, and office manager); concurrent data collection at the encounter through use of an asthma decision support tool; all-payer asthma population identification based on chart review confirmation of administrative data obtained from practices, Cincinnati Children's Hospital Medical Center, and Anthem; Web-based asthma registry with real-time reporting capabilities, including network-, practice-, and patient-level data for the process and outcome measures of focus; transparent comparative practice data using tabular, bar chart, and statistical process control formats (discussed at monthly independent practice association board meetings and shared with practices via the PHO Web site and direct mail); practice workflow redesign based on principles of high reliability²² (ie, process redesign to reduce missed opportunities to capture data on, and address, key aspects of care at the patient encounter); patient self-management collaborative; flu shot improvement collaborative; and multiple network meetings and conference calls to promote communication and collaboration among practices.

The asthma pay-for-performance program consisted of 3 reward levels (Figure), with practices having the potential to earn a 7% fee schedule increase. Single and all-payer data from the PHO asthma registry were used to calculate network and practice-specific performance, respectively. Because the improvement collaborative has maintained a significant focus on engaging all levels of practice personnel and redesigning practice workflow, we elected to frame the pay-for-performance initiative as a practice reward program; thus, a physician-level incentive was not included. In addition, because of challenges measuring performance at the individual physician level,²³ the primary focus of the improvement collaborative has been network- and practice-level performance.

The first-level reward (2% fee schedule increase) was designed to recognize practices for committing to the asthma improvement collaborative objectives and for devoting significant time and effort among physicians, nurses, office manager, and other staff. All practices received the first-level reward, regardless of practice-specific performance.

The second-level reward was a network-level incentive de signed to accelerate practice engagement and promote communication and collaboration among practices. Communication and collaboration were deemed highly important to accelerate the spread of successful interventions across practices, particularly those related to improving reliability. The flu shot percentage and controller medication percentage for children with asthma insured by Anthem were selected as the network-level process measures because of provider relevancy and the ability of Anthem to query claims data and compare results with the PHO asthma registry data. The network flu shot threshold of 30% represented a 36% increase relative to network performance for the 2003-2004 flu season (22%). The network controller medication threshold was set at 70%, the 2004 national average reported by the National Committee for Quality Assurance.²⁴ Although using an all-payer asthma population denominator was preferred, calculating network-level performance based on the Anthem asthma population denominator was deemed important to recruiting payer support for the pay-for-performance program and to documenting value to payer leadership. Failure of the network to meet both thresholds would have resulted in no further fee schedule increase to any practice, regardless of practice-specific performance. Network performance exceeded thresholds for both measures, with all practices receiving another 2% fee schedule increase, once again regardless of practice-specific performance.

The third level was designed to reward individual practices for outstanding performance for 3 process measures relative to the all-payer asthma population; however, each practice had to first meet designated eligibility criteria. The eligibility criteria were designed to address improvement capability and sustainability within each practice. Practices were required to develop their asthma registry by conducting chart reviews to confirm the asthma diagnosis and active status of patients identified via administrative data obtained from the practice, Cincinnati Children's Hospital Medical Center, and Anthem. Practices also had to incorporate concurrent data collection into workflow with a high level of reliability by capturing designated process and outcome data, per the standardized decision support tool, on at least 85% of the confirmed all-payer asthma population within 15 months from project inception. Requiring practices to achieve the 85% threshold was essential to ensuring that adequate data were available to measure clinical performance relative to the all-payer asthma population denominator. The third level included 3 process measures with high thresholds: flu shot percentage (50% threshold), controller medication percentage (75% threshold), and written self-management plan percentage (80% threshold). If eligibility criteria were met, practices earned a 1% fee schedule increase for each measure for which the threshold was achieved.

Network- and practice-specific performance were assessed as of December 31, 2004, with the fee schedule increases effective from May 1, 2004, through December 31, 2005, for the first-level reward, and from March 1, 2005, through December 31, 2005, for the second- and third-level rewards. The fee schedule increases applied to all services billed across all Anthem-covered lives (under both fully insured and self-insured products) receiving care at these practices (ie, not limited to patients with asthma or to asthma-related services). Although the asthma pay-for-performance program concluded on December 31, 2005, Anthem subsequently initiated a communitywide pediatric pay-for-performance program in early 2006, rewarding practice performance on 5 measures, 2 of which were asthma related (ie, flu shot percentage and controller medication percentage).

The distribution of rewards earned by 43 practices according to fee schedule increase was as follows: those with a fee schedule increase of 2%, 3 (7%); an increase of 4%, 13 (30%); an increase of 5%, 2 (5%); an increase of 6%, 14 (33%); and an increase of 7%, 11 (26%) (percentages do not total 100 because of rounding). One practice was ineligible for the asthma pay-for-performance program because of a separate contractual relationship with Anthem. The 3 practices with a 2% fee schedule increase were deemed ineligible for further rewards because of failure to meet independent practice association board–designated requirements regarding level of participation in the asthma improvement initiative.

Regarding the second-level reward, network performance exceeded thresholds for both measures: 54% for the flu shot measure (30% target) and 90% for the controller medication measure (70% target).

Among the 40 practices considered for the third-level reward, 40 (100%) completed requirements relative to establishing an asthma registry and 27 (68%) captured the key process and outcome data on at least 85% of the all-payer asthma population. Among the 27 practices meeting both eligibility criteria for the third-level reward, 26 (96%) achieved the 75% threshold for the controller medication measure, 19 (70%) achieved the 80% threshold for the written self-management plan measure, and 18 (67%) achieved the 50% threshold for the flu shot measure.

Although the asthma pay-for-performance program concluded on December 31, 2005, the improvement collaborative has continued. The following results reflect network and practice performance relative to the process measures of focus: between October 1, 2003, and November 30, 2006, the cumulative percentage of the network all-payer asthma population receiving “perfect care” increased from 4% to 88%, with 18 of 44 practices (41%) achieving a perfect care percentage of 95% or greater (perfect care is a composite measure reflecting patients with severity classified based on the National Heart, Lung, and Blood Institute guideline criteria,²⁵ a written self-management plan, and controller medications [if classified with persistent asthma]); and the percentage of the network all-payer asthma population receiving the influenza vaccine increased from 22% at baseline (2003-2004 season [September 1 through March 31]) to 41% for the 2004-2005 season, to 62% for the 2005-2006 season, with 7 of 44 practices (16%) achieving an influenza vaccination percentage of 80% or greater for the 2005-2006 season. Although not described in this article, improvement in outcome measures related to admissions, emergency department visits, urgent care visits, office visits because of acute symptoms, missed school days, missed workdays, and parent or patient confidence in managing the condition has also been documented.

Aligning pay for performance with the asthma improvement collaborative has resulted in high perfect care and influenza vaccination percentages for the network all-payer asthma population, a higher level of improvement capability among practices, and substantial progress toward system redesign. Based on this experience, we suggest that the following pay-for-performance design principles are highly effective in supporting provider efforts to improve care. Provider is used to denote a practice, hospital, or other site of care, not a physician. Group is used to denote performance across multiple practices, hospitals, or other sites of care. A physician-level incentive was not included in the pay-for-performance program, nor is it included in the following recommendations. Although the pay-for-performance design principles were applied in the context of process measures, they could also be linked to outcome measures (Table).

The first recommendation is to allocate a portion of pay-for-performance funds to reward all providers for committing to, and investing resources toward, improvement efforts, regardless of provider-specific performance. There is concern that pay-for-performance programs tend to reward preexisting high performers²⁶ and may be demotivating to providers with the greatest opportunity to improve, particularly if performance thresholds are viewed as unobtainable. Promoting provider engagement in quality improvement efforts, and sustaining commitment, by awarding a portion of pay-for-performance funds to all providers, regardless of provider-specific performance, may help address these concerns.²⁷

The second recommendation is to reward all providers for achieving group-level performance thresholds (regardless of provider-specific performance) before rewarding provider-specific performance. We found that the group-level incentive had a powerful effect in promoting shared learning and the spread of successful interventions across providers, pushing early adopters to even higher performance levels to increase the likelihood that the group-level thresholds would be achieved, accelerating engagement of providers in the improvement initiative, and maintaining focus on improving care to the aggregate population across practices. Transparency of comparative provider data was particularly helpful in maximizing the impact of the group-level incentive. Use of a group-level incentive does not require that providers be organized as an independent practice association or PHO (eg, a group-level incentive could be created among providers participating in an improvement collaborative or among providers grouped based on geographical boundaries or other factors). Recognizing the variable success of improvement collaboratives,^28,29 use of a group-level incentive may be an important factor to consider; the availability of a centralized Web-based registry or regional health information organization to assemble data across provider sites is also essential in this regard. Although extensively used to alter behavior at the physician or single provider site level, we are not aware of any pay-for-performance programs linked to group-level performance.

The third recommendation is to calculate group- and provider-level performance using an all-payer population denominator. Linking financial rewards to all-payer population performance is consistent with how providers collect and analyze data for quality improvement and how registries are established for populations with chronic conditions. An all-payer population focus also promotes efforts to address equity differences. Concerns regarding adverse payer mix at the group and provider level would need to be addressed; however, these concerns should not preclude use of the all-payer population denominator. One potential option would be to adjust group- and provider-level targets based on the Medicaid/uninsured payer mix percentage. Although desirable in terms of increasing the reward pool, use of an all-payer population denominator does not require pay-for-performance involvement by multiple payers; support from even a single large commercial or governmental payer can have a powerful effect in driving providers to focus on improving all-payer population-based measures. This approach would likely necessitate use of provider data to calculate performance. A major benefit of using provider data is that the dialogue is more likely to remain focused on improvement work that needs to be accomplished, vs extended debate over the validity of payer claims data³⁰; use of provider data also affords the opportunity to include reward measures that cannot be tracked via payer claims data (eg, written self-management plans and quality-of-life measures). This approach necessitates that providers have access to robust data tracking and reporting systems; providers will also need to closely monitor data quality and be prepared to address concerns raised by payers and employers.

The fourth recommendation is to require providers to pursue evidence-based interventions that build improvement capability and sustainability before rewarding provider-specific performance. In addition to rewarding clinical performance, pay-for-performance programs can be leveraged to enhance improvement capability and promote sustainability among providers. This objective was accomplished by requiring practices to meet “eligibility criteria” to qualify for third-level rewards. Using the quality framework of Donabedian,³¹ identifying the all-payer asthma population and creating an electronic registry represent “structural” aspects; redesigning practice workflow through the use of high-reliability change concepts represents “process” aspects. Although evidence is not yet available, it is reasonable to expect that the long-term return on investment to patients, payers, and employers will be enhanced by incorporating improvement capability and sustainability into the design of pay-for-performance programs.

The fifth recommendation is to allocate a portion of pay-for-performance funds to reward outstanding provider-specific performance. Although it may be argued that this portion of pay-for-performance funds will be predominantly earned by preexisting high performers,²⁶ rewarding providers for achieving high quality remains important. Aggressive performance thresholds should be established so that even preexisting high performers are encouraged to further improve care.

By aligning design characteristics of the pay-for-performance program with a collaborative focused on improving processes and outcomes of care for a condition-specific population, building improvement capability, and driving system changes at the provider level, we have established a framework for achieving more substantive and sustainable improvement. Overdependence on pay for performance to drive improved quality is likely a suboptimal approach with questionable long-term viability; rather, pay for performance, when coupled with robust approaches to quality improvement, can be a catalyst to accelerate sustainable transformation among providers.

Correspondence: Keith E. Mandel, MD, Physician-Hospital Organization, Cincinnati Children's Hospital Medical Center, 3333 Burnet Ave, Mail Location Code 9011, Cincinnati, OH 45229-3039 (keith.mandel@cchmc.org).

Accepted for Publication: January 4, 2007.

Author Contributions: Dr Mandel had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Mandel and Kotagal. Acquisition of data: Mandel. Analysis and interpretation of data: Mandel. Drafting of the manuscript: Mandel and Kotagal. Critical revision of the manuscript for important intellectual content: Mandel and Kotagal. Statistical analysis: Mandel. Administrative, technical, and material support: Mandel. Study supervision: Mandel and Kotagal.

Financial Disclosure: None reported.

Additional Contributions: We would like to acknowledge the primary care practices and leadership of Ohio Valley Primary Care Associates, LLC, and Tri State Child Health Services, Inc, the leadership of Cincinnati Children's Hospital Medical Center, and the PHO quality improvement team members, for all their contributions and support. We also thank Barry Malinowski, MD, and colleagues at Anthem Blue Cross and Blue Shield of Ohio for their support of the pay-for-performance program.