Schneider EC, Epstein AM. Use of Public Performance ReportsA Survey of Patients Undergoing Cardiac Surgery. JAMA. 1998;279(20):1638-1642. doi:10.1001/jama.279.20.1638
From the Department of Health Policy and Management, Harvard School of Public Health, Division of General Internal Medicine, Section on Health Services and Policy Research, Brigham and Women's Hospital, Boston, Mass.
Context.— Publicly released performance reports ("report cards") are expected
to foster competition on the basis of quality. Proponents frequently cite
the need to inform patient choice of physicians and hospitals as a central
element of this strategy.
Objective.— To examine the awareness and use of a statewide consumer guide that
provides risk-adjusted, in-hospital mortality ratings of hospitals that provide
Design.— Telephone survey conducted in 1996.
Setting.— Pennsylvania, where since 1992, the Pennsylvania Consumer Guide to Coronary Artery Bypass Graft [CABG] Surgery has provided
risk-adjusted mortality ratings of all cardiac surgeons and hospitals in the
Participants.— A total of 474 (70%) of 673 eligible patients who had undergone CABG
surgery during the previous year at 1 of 4 hospitals listed in the Consumer Guide as having average mortality rates between 1% and 5%
were successfully contacted.
Main Outcome Measures.— Patients' awareness of the Consumer Guide,
their knowledge of its ratings, their degree of interest in the report, and
barriers to its use.
Results.— Ninety-three patients (20%) were aware of the Consumer
Guide, but only 56 (12%) knew about it before surgery. Among these
56 patients, 18 reported knowing the hospital rating and 7 reported knowing
the surgeon rating, 11 said hospital and/or surgeon ratings had a moderate
or major impact on their decision making, but only 4 were able to specify
either or both correctly. When the Consumer Guide
was described to all patients, 264 (56%) were "very" or "somewhat" interested
in seeing a copy, and 273 (58%) reported that they probably or definitely
would change surgeons if they learned that their surgeon had a higher than
expected mortality rate in the previous year. A short time window for decision
making and a limited awareness of alternative hospitals within a reasonable
distance of home were identified as important barriers to use.
Conclusions.— Only 12% of patients surveyed reported awareness of a prominent report
on cardiac surgery mortality before undergoing cardiac surgery. Fewer than
1% knew the correct rating of their surgeon or hospital and reported that
it had a moderate or major impact on their selection of provider. Efforts
to aid patient decision making with performance reports are unlikely to succeed
without a tailored and intensive program for dissemination and patient education.
INFORMATION on the quality of care provided by physicians, hospitals,
and health plans has traditionally been collected for internal quality assurance
and has almost always remained confidential.1
However, the last decade has seen explosive growth in the publication of reports
on the quality of care.2 Large-scale purchasers
of health care services have driven the process, but state health agencies
and traditional accrediting bodies are now demanding that health care providers
furnish performance data for public use.3,4
Public performance reports are intended to guide patients' selection
of providers, aid purchasers in contracting decisions, and stimulate quality
improvement among providers. Prior research suggests that providers change
their behavior in various ways in response to public reporting,5- 7
but much less is known about the ways consumers use performance data.8,9 In spite of this, national consumer
publications such as Consumers Digest,10U.S. News and World Report,11,12
and Newsweek13 now
publish rankings of health plans and hospitals on patient satisfaction and
quality of care. Both the Agency for Health Care Policy and Research14 and the Health Care Finance Administration15 have launched major programs to develop, evaluate,
and disseminate quality measures to inform consumers selecting health plans
and other medical care services.
The Pennsylvania Health Care Cost Containment Council has been at the
forefront of this trend in the collection, analysis, and reporting of hospital
and provider-specific data on cardiac surgery since 1992. The agency regularly
publishes and disseminates risk-adjusted mortality rates on every Pennsylvania
hospital, surgeon, and surgical group providing coronary artery bypass graft
(CABG) surgery in its Consumer Guide to Coronary Artery
Bypass Graft Surgery .16- 19
The agency distributed 15000 copies of the first and second volumes of the Consumer Guide to hospitals, surgeons, public libraries,
business groups, legislators, and the media.20
It is available free to any individual who requests it. Public release of
the Consumer Guide has received extensive media coverage.
Cardiac surgery is a dramatic event, frequently elective, with a significant
operative mortality rate. Previous studies have shown that mortality rate
variations are related to the quality of care.21- 26
Thus one might expect that patients or their advisors would be particularly
motivated to use the reported data. We examined use of the Pennsylvania Consumer Guide by patients who underwent CABG surgery at
Forty-one Pennsylvania hospitals provide CABG surgery. Volume 4 of the Consumer Guide reported that 3 hospitals had lower and
5 had higher than expected mortality rates. We selected 4 hospitals that performed
at least 400 operations within 1 year that are located in different regions
of the state and that were willing to participate in our study. At each hospital,
we asked individual surgeons or surgical groups to participate. Eighteen of
24 practicing surgeons agreed to participate. Participating surgeons performed
86% of all the CABG procedures in the 4 hospitals.
The Consumer Guide bases its rating on a hospital's
in-hospital mortality rate relative to its expected mortality rate.7 Expected mortality rates are derived from clinical
data describing the patients' preoperative severity of illness. During the
year immediately prior to our survey, the 4 study hospitals received 3 distinct Consumer Guide ratings: 2 had a lower than expected mortality,
1 had higher than expected mortality, and 1 was within the expected mortality
range. Similar to the range of hospitals statewide, the study hospitals' unadjusted
in-hospital mortality rates ranged from 1% to 5%.
Each participating cardiac surgeon or group provided a list of patients
who had undergone cardiac bypass surgery between July 1995 and March 1996
— the months following the June 1995 public release of volume 4 of the Consumer Guide. The overall sample included 1140 cardiac
surgery patients. After excluding patients known to have died, we randomly
selected 200 patients from each institution. To eliminate duplicate entries
at1 hospital, we adjusted each hospital's sample to 196 patients. Human research
review committees at each hospital granted permission to survey patients.
Using patient focus groups, expert advice, and formal pretesting, we
developed a telephone survey to assess patients' perception of their decision
making prior to surgery. The survey assessed 4 issues:
To what extent patients were aware of the Consumer Guide before or after they underwent cardiac surgery,
and whether characteristics of the patients or their hospitals were associated
with such awareness. Specifically, we described the Consumer
Guide and then asked, "Have you heard of this booklet?"; "Have you
ever seen a copy of this booklet?"; and "Did you become aware of it before
or after your operation?" We collected information on patients, including
age, sex, education, income, marital status, self-reported health status,
type of insurance coverage, length of time with heart disease, and number
of prior coronary catheterizations. We also asked respondents which of 3 possible
choices they considered most important: choice of hospital, choice of surgeon,
or choice of surgical group.
To what extent they used the Consumer Guide. We asked if they knew how the Consumer
Guide' s categorical mortality rating had ranked their hospital, surgical
group, or surgeon and whether they discussed the mortality rating with physicians
or other health professionals.
The level of general interest they had in performance
reports such as the Consumer Guide. We developed
3 measures of patient interest in performance reports. First, we described
the content of the Consumer Guide to all patients,
even those who had already seen it. We then ascertained their level of interest
in the Consumer Guide. We posed a scenario in which
patients needed another CABG operation and asked whether they would change
surgeons if the surgeon they had intended to use was reported to have had
more deaths than the average surgeon in the previous year. We also asked about
their willingness to pay ($0, $5, $10, $20, $50, $100) for a copy of the Consumer Guide.
Identify the constraints or barriers limiting patients'
opportunity to use performance reports. We inquired about 5 potentially important
constraints: time, distance to the hospital, opportunity to leave the hospital
between the decision to operate and the actual operation, cost, and restrictions
imposed by insurance companies or health plans. Specifically, we asked how
many days passed between the decision that they needed surgery and the actual
operation and whether this was enough time to learn about the surgeon and
hospital. We asked whether they knew of other hospitals that performed CABG
surgery within a "reasonable distance" of home as well as how important it
was to them to undergo cardiac surgery at a hospital near home. We asked patients
whether the decision to operate was made while they were in the hospital and
whether they had remained an inpatient during the time between the decision
and the operation. We asked, "Did the cost of the operation affect your choice?"
We also asked if restrictions by insurance influenced their key choices.
Telephone interviews with patients were conducted from June through
December 1996 by Datastat (Ann Arbor, Mich). The statistical significance
of differences in responses was assessed by a χ2test for binary
response items and by a Wilcoxon rank sum test for pairwise comparisons of
ordinal scaled responses. To evaluate the significance of associations between
sociodemographic characteristics and awareness of the Consumer
Guide, we calculated odds ratios (ORs) and 95% confidence intervals
(CIs). Two-tailed P values are reported for all comparisons.
More than 95% of respondents answered each of the items with the exception
of the query about income (80%). Nonrespondents to specific questions were
excluded from the analysis of those questions.
Of the 784 patients we attempted to contact, we completed interviews
with 474 (60%). Among the original cohort, 111 patients (14.2%) could not
complete the survey: 38 had died, 64 were too disabled, 7 had language incompatibilities,
and 2 failed to recall having had an operation. Another 137 otherwise eligible
patients (20.3%) refused participation, and 62 patients (9.2%) could not be
contacted. The response rate among eligible patients was 70.4% (range, 68.7%-74.0%
among the participating hospitals).
Characteristics of the respondents appear in Table 1. Comparing the frequency of each characteristic across the
4 hospitals, respondents differed in education levels (P<.01), in number of days between deciding an operation was needed
and undergoing the operation (P=.03), and in the
proportion reporting the following sources of payment for the operation: private
insurance (P=.02), Blue Cross/Blue Shield (P=.03), and health maintenance organization (P<.01). Respondents from different hospitals also varied with respect
to the factor most influencing their choice (hospital vs surgical group vs
surgeon) (P<.01). Respondents were similar with
respect to age, sex, marital status, self-reported health status prior to
surgery, income, number of prior catheterizations, and length of time with
heart disease. They were also similar in the proportion of those reporting
that Medicaid or Medicare paid in part for the operation.
Table 2 summarizes the number
and proportion of patients reporting awareness, knowledge, and use of the Consumer Guide. Ninety-three of the patients (20%) were
aware of the Consumer Guide, and 56 (12%) of those
said they knew of it prior to their operation. Two thirds of these patients
(n=37) had only heard of the guide, while one third (n=19) had actually seen
a copy. Eighteen (4%) reported knowing the hospital's categorical mortality
rating (higher than, lower than, or within the expected number of deaths).
Eleven (2%) reported that the information influenced the choice of hospital,
but only 4 of these knew the correct categorical rating, which amounted to
less than 1% of all respondents. Only 6 (1%) reported discussing the ratings
with a physician.
Similarly, very few patients reported knowing the Consumer Guide' s categorical rating of the surgeon or surgical group
(n=7). Four patients claimed that the Consumer Guide
was a major or moderate influence on the choice of surgeon or knew the correct
categorical rating of the surgeon or surgical group. Altogether, only these
4 patients reported that the Consumer Guide was a
major or moderate influence on the choice of hospital or surgeon and reported
the correct categorical mortality rating of the hospital, surgeon, or surgical
Table 3 displays patient
characteristics correlated with awareness of the Consumer
Guide prior to surgery. Patients were significantly more likely to
report awareness of the Consumer Guide prior to the
operation if they were younger than 65 years (OR, 2.00; CI, 1.14-3.51), had
attended college (OR, 2.10; CI, 1.19-3.70), reported poor or fair preoperative
health status (OR, 1.88; CI, 1.06-3.33), or reported having heart disease
for more than 1 year (OR, 1.91; CI, 1.05-3.50). Men were somewhat more likely
than women to be aware of the Consumer Guide prior
to surgery (OR, 2.03; CI, 0.96-4.27), and patients with incomes greater than
$30000 were also somewhat more likely to be aware (OR, 1.81; CI, 0.97-3.38).
Rates of awareness of the Consumer Guide did not
differ significantly among patients operated on in hospitals with categorical
ratings higher than, lower than, or within the expected mortality range, nor
were they related to whether the patient had previously been admitted to the
same hospital or to the number of days between the decision to operate and
the date of the operation. In a logistic regression analysis with "being aware
of the Consumer Guide prior to surgery" as the dependent
variable, younger age (P<.01), higher attained
education level (P<.01), and higher health status
(P=.02) were statistically significant predictors
of "being aware" in the final model.
Table 4 shows findings on
3 measures of patient interest in the Consumer Guide.
After the content of the Consumer Guide was described
to all patients, 264 (56%) reported being somewhat or very interested in seeing
a copy if they required another operation. Younger patients (P=.0002), those having some college education (P=.003), and those who were aware of the Consumer
Guide prior to surgery (P<.05) were most
likely to be somewhat or very interested in seeing a copy if they needed another
operation. There was no significant difference in level of interest between
patients who were and were not aware of the Consumer Guide at the time of the survey.
Most patients reported that they probably or definitely would change
surgeons if they learned that their surgeon had a higher than expected mortality
rate in the previous year. Nearly one third of patients said they would definitely
change surgeons under this scenario. Nevertheless, one third of the patients
reported that they would not be willing to pay any money to see a copy of
the Consumer Guide . Thirty-five percent reported
that they would be willing to pay at least $20 to see a copy. Only 8% said
they would be willing to pay $50 or more.
Table 5 provides data on
selected barriers to consumer choice for cardiac surgery patients. Thirty-eight
percent had fewer than 3 days to decide on a hospital or surgeon before their
operation. Only 12% of all the patients surveyed perceived that they had less
than enough time to learn about the surgeon and hospital. However, 19% of
the patients with fewer than 3 days to decide perceived that they had less
than enough time, while 7% of the patients who had more than 7 days perceived
that they had less than enough time (P<.01). Thirty-three
percent of patients reported that there was no alternative hospital within
a reasonable distance. Sixty-six percent of all the patients considered distance
somewhat or very important in determining their choice of hospital, and these
patients were more likely to report that there was no alternative CABG surgery
hospital within a reasonable distance of their home (38% vs 23%, P<.01).
Forty-three percent of patients remained in the same hospital from the
time it was decided that they would need an operation until the operation
was performed. Only 2% reported that cost played any role in the choice of
hospital, and only 4% perceived any restriction imposed by managed care insurance.
We are unaware of any previous studies of patient use of outcome data
to choose physicians and hospitals.20 Because
of the extensive publicity given to the Pennsylvania Consumer
Guide to Coronary Artery Bypass Graft Surgery, its 5-year track record,
the salience of a major heart operation, and the 5-fold variation in mortality
rates among hospitals, we expected that the Consumer Guide would be widely used by patients selecting providers for CABG surgery.
We found just the opposite. It is striking that even among those who were
aware of the Consumer Guide before surgery, almost
no one used it in decision making.
What could account for the lack of awareness and use of the Consumer Guide among cardiac patients? First, referring physicians
are a very important source of information about the quality of surgical specialists.
Our previous survey of cardiologists7 and a
similar study conducted in New York State27
showed that very few of these providers discussed the Consumer
Guide with patients, citing skepticism about the accuracy of its methods.
The present survey confirms that these discussions are indeed rare.
As in New York State, the process for dissemination relies primarily
on media, such as television and newspapers. Unlike a hospital quality reporting
program in Cleveland, Ohio, the Consumer Guide is
free. However, efforts to distribute it to patients appear to have been inadequate.
It is possible that budget constraints, criticism of technical aspects of
the reports, and political pressure from hospitals and physicians in Pennsylvania
have deterred more aggressive dissemination of the Consumer
Guide by the Pennsylvania Health Care Cost Containment Council. However,
poor distribution alone cannot explain our observation that very few patients
who were aware of the Consumer Guide ratings were
able to comprehend and make use of them accurately.
A significant number of patients face serious constraints in their ability
to seek and use the Consumer Guide. We found that
most patients have a limited amount of time for decision making. Many perceived
that there were no alternative cardiac surgery hospitals within a reasonable
distance despite the fact that the hospitals we studied were relatively near
other hospitals that provide CABG surgery. Finally, some patients may be skeptical
of the value of such data. A recent survey of Americans' use of quality data
on health plans found that the public values anecdotal reports from such trusted
sources as relatives and friends more than objective reports from such sources
as the government and the news media.28
Our study provides conflicting information about patients' interest
in the sorts of quality data that are frequently suggested to be useful to
consumers. Although few patients used the Consumer Guide, a much larger number expressed interest in seeing a copy when it
was described to them. One third of patients said they would definitely switch
surgeons if they found that their surgeon had a higher than expected mortality
rate. On the other hand, one third of them were unwilling to pay any amount
to see the Consumer Guide, and most were unwilling
to pay more than $20. Patients may view such information as a public good
that should be inexpensively available.
Of course, public reporting of performance data may help improve quality
of care even if patients do not use the data in selecting providers. Both
employers and insurers may use such data in contracting decisions. Hospitals
may use the reports to select physicians and curtail physician privileges.29,30 Health care providers may use the
reports to identify specific clinical areas for quality improvement efforts
and gauge their success.6 Nevertheless, providing
data on quality directly to consumers to inform them as they choose providers
is a notion with very wide political and popular appeal.15
Our study has several limitations. We surveyed patients from only 4
hospitals. These hospitals or the patients they serve may differ from other
hospitals or patients. However, if willingness to participate in our study
signals a more sympathetic attitude toward the Consumer
Guide, then estimates of awareness and use might be even lower in other
hospitals or patient groups. We surveyed patients after surgery. Some respondents
may have forgotten their exposure to the Consumer Guide or may have reported that they were aware before surgery when in fact
they only learned of the Consumer Guide afterward.
Although we surveyed patients relatively soon after surgery, we cannot exclude
the possibility that recall bias may have artificially lowered our estimate
of awareness and use of the Consumer Guide among
cardiac surgery patients. Another limitation is the inherent challenge of
interpreting the responses of consumers regarding their interest in a publication
that few have directly seen. We also had limited power to examine differences
among hospitals. Our study had a power of 0.80 to detect a 15% absolute difference
in rates of awareness (10% vs 25%) among patients at the 4 hospitals. Finally,
our design precluded an evaluation of patients who considered but did not
have surgery or who went to cardiac surgery centers outside of Pennsylvania
after reading the Consumer Guide.
Despite these limitations, we found formidable evidence that public
reporting of mortality outcomes in Pennsylvania has had virtually no direct
impact on patients' selection of hospitals or surgeons. Nevertheless, a substantial
number of patients expressed interest in data on mortality outcomes and claimed
that they would use such reports in their decision making. Clearly, measurement
and public reporting of physician and hospital performance is only a prelude
to serving this interest. Existing quality measurement efforts have been criticized
for methodological reasons.31,32
Although the methodological barriers to reliable and valid performance measurement
are substantial, delivering performance information to patients in an effective
and usable format could prove even more formidable. Further efforts to develop
quality information for general public use should explore the use of Internet-based
and other media for communicating quality information. Providers may also
play an important role. Without a tailored and intensive program for dissemination
and patient education, efforts to aid patient decision making with performance
reports are unlikely to succeed.