The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians

Background  Public disclosure of physician-specific performance data is becoming increasingly common. However, the influence that public reporting of outcome data has on the delivery of care by physicians who are being assessed is not well understood.

Methods  Since 1994, the New York State Department of Health has collected and periodically published observed and risk-adjusted patient mortality rates for all interventional cardiologists practicing coronary angioplasty in the state. To assess the influence that these reports exert on the physicians being monitored, a questionnaire was administered in an anonymous manner to all interventional cardiologists included in the most recent report.

Results  The vast majority (79%) of interventional cardiologists agreed or strongly agreed that the publication of mortality statistics has, in certain instances, influenced their decision regarding whether to perform angioplasty on individual patients. Physicians expressed an increased reluctance to intervene in critically ill patients with high expected mortality rates. Among the respondents, 83% agreed or strongly agreed that patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician-specific patients’ mortality rates. Additionally, 85% believed that the risk-adjustment model used in the Percutaneous Coronary Interventions (PCI) in New York State 1998-2000 report is not sufficient to avoid punishing physicians who perform higher-risk interventions.

Conclusions  Public reporting of physician-specific outcome data may influence physicians to withhold procedures from patients at higher risk, even when physicians believe that the procedure might be beneficial. This phenomenon should be recognized in the design and administration of physician performance profiles.

Public reporting of physician-specific outcome data are becoming more widespread. The New York State Department of Health, for example, currently collects and reports numbers of interventions and patient mortality statistics for all surgeons in the state who perform coronary artery bypass surgery and for all interventional cardiologists who perform coronary angioplasty.^1,2 The principal objectives of these types of reports, often referred to as physician “scorecards,” are to allow the public to make better-informed decisions when choosing a physician or hospital, and to provide physicians and health care organizations with comparative data that ultimately will serve to improve the quality of health care.³

Although not well studied, several concerns have been raised regarding the impact that physician scorecards may have on patient care.⁴ Of primary concern, it has been suggested that physicians, knowing that their procedural mortality rates will be published, may be less inclined to offer procedures to patients at higher risk who, nevertheless, may benefit from undergoing the procedure.^5-7 While most scorecards use risk-adjustment models in an attempt to account for differences in the severity of patients’ illnesses, physicians remain uncertain about the ability of these models to adequately credit practitioners who perform interventions on sicker patients. Thus, while scorecards provide the public with objective information, it remains uncertain whether these reports simultaneously alter the way that physicians care for patients.

In an effort to better understand the influence that public reporting of mortality statistics has on the medical decision-making process of the individual physicians who are being scrutinized, we undertook the present study to evaluate systematically the opinions and experiences of all physicians who were included in the most recent Percutaneous Coronary Interventions (PCI) in New York State 1998-2000 report² (available at: http://www.health.state.ny.us/nysdoh/pci_1998-2000.pdf).

The study population consisted of all interventional cardiologists in New York State who were included in the most recent PCI in New York State report, PCI in New York State1998-2000, published in January 2003. This report contains physician-specific and hospital-specific outcome data for all percutaneous coronary revascularization procedures performed in New York State from January 1997 to December 2000. The report includes data on numbers of interventions performed as well as actual and risk-adjusted patient mortality rates for each physician.

The study instrument consisted of a mail-based questionnaire designed to elicit the opinions of cardiologists regarding the PCI in New York State report and its potential impact on their clinical practice. Current mailing addresses for physicians were obtained from medical society membership directories, local telephone directories, and Internet-based directories.

The questionnaire consisted of a single-page document that first asked each physician to provide selected background information (physician’s age, number of years since cardiology training was completed, board certification status in interventional cardiology, and annual number of angioplasty procedures; type of hospital where most of the physician’s procedures were performed; and risk-adjusted patient mortality rate as reported in the PCI in New York State1998-2000 report). The questionnaire recipients were then presented with 9 statements or questions regarding the report and were asked to indicate whether they strongly agreed, agreed, disagreed, or strongly disagreed with each statement or question.

To foster accuracy, we asked that the questionnaires be returned to us in an anonymous manner. The questionnaire was sent to all cardiologists twice (at a 4-week interval) in an attempt to maximize the response rate. The cardiologists who returned the questionnaire after the first mailing were instructed to disregard the second mailing.

Based on an analysis of prior questionnaire-based studies published in the medical literature, our goal response rate for this study was 60%.⁸

Responses were entered into a central database for analysis. Evaluation for potential relationships between background variables and responses to the various queries presented in the questionnaire was carried out with the χ² test, with physician responses dichotomized (“disagree” and “strongly disagree” vs “agree” and “strongly agree”). P<.05 denoted statistical significance.

Questionnaires were mailed to the 186 interventional cardiologists included in the most recently published iteration of the PCI in New York State report, of whom 120 responded. This response rate of 65% met our preset goal. There were no “refusals,” ie, no individuals informed us that they refused to answer the questionnaire.

Self-reported background information of the respondents is presented in Table 1. Seventy-eight percent of the respondents were board certified in interventional cardiology and there was a wide distribution among the physicians with respect to the number of years since cardiology training was completed, annual number of angioplasty procedures performed, and hospital type where the physicians performed most of their angioplasty procedures.

Responses of the interventional cardiologists to the 9 specific queries related to the PCI in New York State1998-2000 report are presented in Table 2. Three statements focused on the respondents’ overall perception of the report. Specifically, 76% of respondents disagreed or strongly disagreed that it serves to improve patient care in New York State. Likewise, 85% of the respondents disagreed or strongly disagreed that mortality statistics provide an accurate measure of physician quality, and 82% disagreed or strongly disagreed that mortality statistics provide information that is useful in selecting a physician or hospital for angioplasty.

The overwhelming majority of respondents indicated that the presence of the PCI in New York State report influences their clinical decision-making process. Among the respondents, 83% agreed or strongly agreed that “patients who might benefit from angioplasty may not receive the procedure as a result of public reporting of physician specific mortality rates.” Similarly, 79% agreed or strongly agreed that the knowledge that mortality statistics will be made public has, in certain instances, influenced their decision on whether to perform angioplasty on individual patients. Additionally, 79% agreed or strongly agreed that their decision-making process regarding whether to intervene in critically ill patients with high expected mortality rates is influenced by the presence of the scorecard.

Physicians also indicated discomfort with the risk adjustment model used in the PCI in New York State1998-2000 report. Among respondents, 85% disagreed or strongly disagreed with the statement that the risk stratification model used in the report is sufficient so as not to penalize physicians who perform higher-risk interventions. The vast majority (88%) of respondents agreed or strongly agreed that when submitting patient data for the PCI in New York State report, physicians may report higher-risk conditions for their patients' features to improve their risk-adjusted mortality statistics.

Analysis was performed to determine whether significant associations existed between physician-specific variables (Table 1) and responses to the survey questions. Opinions regarding the PCI in New York State report were not associated with the respondent’s board certification status, annual numbers of angioplasty procedures, or patient risk-adjusted mortality rate on the most recent report. Younger physicians were significantly less likely than older physicians to agree that “the PCI in New York State report serves to improve patient care in New York State” (χ²₂ = 8.2, P<.02), and that “public reporting of mortality statistics for interventional cardiologists should be adopted by other states” (χ²₂ = 10.3, P<.01). Additionally, physicians who performed most of their angioplasty procedures at major university teaching hospitals were significantly more likely than other physicians to agree that “the publication of mortality statistics factors into their decision on whether to intervene in critically ill patients with high expected mortality rates” (χ²₂ = 8.2, P<.02).

To our knowledge, the present study represents the first systematic analysis of the attitudes and experiences of cardiologists with respect to the public reporting of physician-specific outcome data related to coronary angioplasty. The findings provide evidence that physician scorecards are not passive devices simply monitoring physician quality, but, rather, that they exert an active influence on the clinical decision-making process of the physicians who are being assessed. The overwhelming majority of New York State interventional cardiologists indicated that knowing that their patient mortality statistics were to be publicly reported has at times influenced their decision on whether to perform angioplasty on individual patients. Specifically, physicians indicated that the reporting of mortality statistics may prevent them from performing procedures on patients at higher risk even when they believe that these patients might benefit from the procedure. While the PCI in New York State report incorporates a risk-adjustment model, physicians did not believe that the methods of risk adjustment were sufficient to prevent penalizing cardiologists who perform higher-risk interventions. Opinions regarding the angioplasty scorecard were independent of most background variables, including board certification status, number of angioplasty procedures performed, or the respondent’s risk adjusted patient mortality rate.

Reporting of hospital- and physician-specific outcome data has been undertaken for a variety of medical and surgical therapies.⁹ One fundamental aim of such scorecards is to promote improvements in quality of care, but documenting quality of care on outcomes has been difficult. Following initiation of the report on coronary artery bypass surgery in New York State in 1989, Hannan et al¹⁰ reported that the risk-adjusted mortality rate for bypass surgery in the state of New York declined by 41%. Other authors, however, have speculated that this improvement may have been attributable to factors unrelated to better surgical results, including an increased refusal by surgeons to operate on sicker patients; increased reporting of characteristics of patients at high risk in an attempt to reduce risk-adjusted mortality; improvements in surgical techniques that have been occurring across the United States; and increased numbers of out-of-state referrals for bypass surgery.^11-14

As other states have begun to follow New York’s lead and report physician-specific outcome data for cardiac surgery, whether patients have been helped or harmed by these devices remains debated. While Hannan et al¹⁵ reported that in-hospital and 30-day risk-adjusted mortality rates following bypass surgery were significantly lower in regions with public dissemination programs than in the remainder of the United States, a provocative analysis by Dranove et al⁶ suggested that such reductions in surgical mortality merely reflect a shift by surgeons in scorecard states toward operating on patients with lower illness severity. In comparing trends in New York State and Pennsylvania before and after these states adopted reporting systems for cardiac surgery, the authors noted that sicker patients became less likely to receive bypass surgery, and that mortality rates among higher-risk coronary patients have consequently increased as these individuals were less likely to be treated surgically. Indeed, in a survey of all cardiac surgeons practicing in New York State in 1997, 67% of respondents indicated that they had refused to offer bypass surgery to at least 1 patient within the previous year, with the New York State cardiac surgery reporting system “being an integral part of the decision-making process.”⁷

The influence that published scorecard data has had on the behavior of health care consumers and referring physicians has likewise been a matter of debate. Among a cohort of 474 patients who underwent bypass surgery in Pennsylvania following the publication of that state’s Consumer Guide to Coronary Artery Bypass Surgery 1997-1999, less than 3% indicated that the scorecard had a significant impact on their choice of surgeon.¹⁶ While Hannan et al¹⁷ reported that 38% of surveyed cardiologists in New York State responded that the state’s bypass surgery report had affected their referrals to surgeons, another study found that 87% of cardiologists in Pennsylvania indicated that the Consumer Guide to Coronary Artery Bypass Surgery had minimal or no influence on their referral recommendations.⁵

Several investigators have raised the concern that practitioners may refuse to perform potentially beneficial procedures on sicker patients for fear that their reported mortality statistics be impacted adversely. In Pennsylvania, 59% of cardiologists reported increased difficulty in finding a surgeon willing to perform cardiac surgery in severely ill patients in need of such surgery after initiation of the state’s cardiac surgery reporting system, and 63% of cardiac surgeons reported that they were less or much less willing to operate on the most severely ill patients.⁵ Similarly, in our study, the vast majority of interventional cardiologists in New York State indicated that public reporting of mortality statistics influences their decision on whether to intervene in critically ill patients with high expected mortality rates. This is an important public health concern, as reluctance among physicians to perform coronary revascularization procedures on patients with appropriate indications may portend worse clinical outcomes. In 1 analysis, patients deemed to have an appropriate indication for coronary revascularization by an expert panel, but who did not undergo the procedure, were more likely to experience adverse clinical events than patients with similar indications who underwent revascularization.¹⁸

Scorecards typically incorporate risk-adjustment models designed to account for differences in baseline risk among patients. The choice of specific details used in the design of any risk-adjustment model, such as the clinical variables to be included and the weighting of these variables, exerts a fundamental impact on the ultimate conclusions about the data set to which it applied.^19,20 Some investigators have cautioned that, for coronary angioplasty, statistical modeling to discriminate between provider outcomes is limited by factors such as a low incidence of major procedural complications and the inclusion of surgeons who perform few procedures, whereas others have demonstrated excellent accuracy for such models.^21-23 The Coronary Artery Bypass in New York State 1997-1999 and the PCI in New York State 1998-2000 reports use sophisticated models that have been validated and are periodically updated.¹⁰ Despite the care taken in constructing these models, Green and Wintfeld¹² calculated that predicted patient mortality rates assigned to surgeons in an early version of the Coronary Artery Bypass in New York State report explained only a small portion (7.3%) of the variance in the observed patient mortality among surgeons, and the power of the model to predict outcomes for individual patients was also low at 8.0%.

Even if models adequately adjust risk for a population, physicians may remain skeptical about the ability of the model to adequately offset risk when confronted with a critically ill individual. Of the interventional cardiologists included in our survey, 85% believed that the risk stratification model used by New York State is insufficient to prevent penalizing physicians who perform higher-risk interventions. This is noteworthy, as patients with the greatest baseline risk often derive the greatest degree of benefit from more aggressive medical or surgical interventions. For example, in the Should We Emergently Revascularize Occluded Coronaries for Cardiogenic Shock (SHOCK) trial, which randomized patients with acute myocardial infarction complicated by cardiogenic shock to undergo early coronary revascularization (with bypass surgery or angioplasty) or initial medical stabilization, mortality rates were high (>50%) at 6 months in both treatment groups. Patients treated with a strategy of early revascularization, however, demonstrated a statistically significant 12.8% absolute reduction in death at 6 months compared with patients in whom early revascularization was not undertaken (50.3% vs 63.1%).²⁴

One method by which physicians can lower their risk-adjusted patient mortality rates is to report conditions with higher risks than their patients’ actual conditions, a technique that has been referred to as “gaming.” While the New York State Department of Health attempts to verify the validity of the data that it receives by a variety of methods, 88% of the interventional cardiologists who responded to our survey believed that physicians still may use “gaming” to improve their risk-adjusted patient mortality. And while some care needs to be taken in the interpretation of this statement, as it asks the respondents to speculate on the actions of other cardiologists, the widespread belief that “gaming” exists points to a general lack of confidence in the accuracy of the risk-stratification algorithm used. The use of mortality as the sole outcome measure to indicate physician performance was also viewed critically, as 85% of our respondents expressed disagreement that mortality statistics provide an accurate measure of physician quality.

While physician scorecards will likely continue to increase in number and scope, refinements should be considered to limit their potentially negative impact on physicians’ decisions. For example, the outcomes of patients with very high baseline risks (such as patients presenting with acute myocardial infarction and cardiogenic shock), who, nevertheless, have been shown to benefit from a particular intervention, could be considered separately from the “regular” outcomes on the report card. Additionally, physicians could benefit from better education regarding the risk-adjustment algorithms used in outcome reports.

This study attempts to gain insights into the clinical decision-making process of physicians, which is difficult to measure by objective means. Our conclusions, as is the nature of all survey-based studies, assume that the subjective responses of the answering physicians reflect their true clinical thoughts and actions. It remains possible that preexisting biases toward the concept of “scorecarding” may have influenced the responses of some physicians. The response rate for our study was 65%, which exceeds the average response rate for survey-based studies published in the medical literature.⁸ It is possible that the views of the nonrespondents were different from those of the physicians who responded to the survey; however, it is unlikely that the qualitative conclusions of the study would have changed if additional responses had been received. Because the questionnaire was administered in an anonymous manner, we are not able to compare demographic data among responders and nonresponders.

The results of this study indicate that public reporting of physician-specific outcomes influences physician behavior. At one extreme, physicians may withhold procedures from patients at higher risk for adverse outcomes, even when they believe that these procedures might be beneficial. In this respect, physician scorecards, designed as a method to improve health care outcomes for the general population, may paradoxically serve to adversely affect health care decisions for individual patients.

This study also underscores the current uncertainty of whether physician scorecards are an accurate measure of physician quality, and if they serve to improve or harm patient outcomes. The prevalence and impact of scorecards are expected to continue to increase, as third-party payers have expressed interest in using these statistical models to ration payment. It is critical, therefore, that the impact of scorecards on physician decisions and patient outcomes be continually reassessed as these instruments evolve.

Correspondence: Craig R. Narins, MD, Cardiology Unit, University of Rochester Medical Center, Box 679, Rochester, NY 14642 (craig_narins@urmc.rochester.edu).

Accepted for Publication: July 14, 2004.

Financial Disclosure: None.