Relationship Between Hospital Performance on a Patient Satisfaction Survey and Surgical Quality

Importance  The Centers for Medicare and Medicaid Services include patient experience as a core component of its Value-Based Purchasing program, which ties financial incentives to hospital performance on a range of quality measures. However, it remains unclear whether patient satisfaction is an accurate marker of high-quality surgical care.

Objective  To determine whether hospital performance on a patient satisfaction survey is associated with objective measures of surgical quality.

Design, Setting, and Participants  Retrospective observational study of participating American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP) hospitals. We used data from a linked database of Medicare inpatient claims, ACS NSQIP, the American Hospital Association annual survey, and Hospital Compare from December 2, 2004, through December 31, 2008. A total of 103 866 patients older than 65 years undergoing inpatient surgery were included. Hospitals were grouped by quartile based on their performance on the Hospital Consumer Assessment of Healthcare Providers and Systems survey. Controlling for preoperative risk factors, we created hierarchical logistic regression models to predict the occurrence of adverse postoperative outcomes based on a hospital’s patient satisfaction scores.

Main Outcomes and Measures  Thirty-day postoperative mortality, major and minor complications, failure to rescue, and hospital readmission.

Results  Of the 180 hospitals, the overall mean patient satisfaction score was 68.0% (first quartile mean, 58.7%; fourth quartile mean, 76.7%). Compared with patients treated at hospitals in the lowest quartile, those at the highest quartile had significantly lower risk-adjusted odds of death (odds ratio = 0.85; 95% CI, 0.73-0.99), failure to rescue (odds ratio = 0.82; 95% CI, 0.70-0.96), and minor complication (odds ratio = 0.87; 95% CI, 0.75-0.99). This translated to relative risk reductions of 11.1% (P = .04), 12.6% (P = .02), and 11.5% (P = .04), respectively. No significant relationship was noted between patient satisfaction and either major complication or hospital readmission.

Conclusions and Relevance  Using a national sample of hospitals, we demonstrated a significant association between patient satisfaction scores and several objective measures of surgical quality. Our findings suggest that payment policies that incentivize better patient experience do not require hospitals to sacrifice performance on other quality measures.

Measures of patient experience are now widely accepted as core components of health care quality.^1,2 Accordingly, the Centers for Medicare and Medicaid Services (CMS) now measure and publicly report patient satisfaction using the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey.³ New CMS payment strategies likewise aim to encourage high-value care by attaching a percentage of hospital reimbursement to performance on various quality measures relating to clinical processes of care, certain risk-adjusted outcomes, efficiency, and patient experience of care.⁴ In 2015, 30% of a hospital’s weighted performance score was based on measures of patient experience, thereby creating a strong incentive for hospitals to deliver care that is patient centered.⁵

However, several studies that have explored the relationship between patient satisfaction and other, more objective measures of quality have produced contradictory findings. Some suggest that patient perception of health care quality correlates positively with certain process^6,7 and outcome^7-10 measures, while others demonstrate either the lack of a relationship or an inverse relationship.^11-13 One study even linked higher satisfaction scores to higher mortality rates.¹⁴

Within the field of surgery, results are equally inconclusive. Recent studies have shown no association between patient satisfaction and adherence to the Surgical Care Improvement Project process measures^10,15 or the occurrence of postoperative complications¹⁰ and mortality.¹⁶ In contrast, a study using a large sample of hospitals demonstrated that higher patient satisfaction was associated with lower postoperative mortality, readmissions, and length of stay.¹⁷ However, that study relied exclusively on administrative claims data, which limits the ability to compare rates of postoperative complications.^18-20

The relationship between hospital performance on patient satisfaction measures and objective measures of surgical quality is of great interest to policy makers. A strong, positive correlation between the two would suggest that measuring both would be redundant, adding little to what is already known about hospital quality. A negative correlation, on the other hand, would caution that incentivizing one of these measures could compromise efforts to improve the other. Finally, the absence of a relationship would suggest independent domains of quality, each warranting individual attention. To examine this relationship, we used data from a national sample of hospitals to determine whether there is an association between objective measures of surgical quality, as measured by risk-adjusted rates of postoperative complications, failure to rescue, readmissions, and mortality, and patient-centered care, as measured by hospital performance on the HCAHPS survey.

We linked data from 4 sources from December 2, 2004, through December 31, 2008: (1) Medicare inpatient claims, (2) the American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP), (3) the American Hospital Association annual survey, and (4) Hospital Compare. These years of data were made available by CMS as part of a collaborative research contract. The data merge between Medicare claims and ACS NSQIP is described in detail elsewhere.²¹ In brief, Medicare data were obtained from the 100% Medicare Provider Analysis and Review file and linked at the patient level with ACS NSQIP data using indirect patient identifiers and a deterministic linkage algorithm. The validity of the linkage procedure was supported by the excellent agreement on patient-level coding of mortality (κ = 0.969).²¹ The ACS NSQIP registry is an institution-based, multispecialty, clinical registry for patients undergoing surgery. Data collected include preoperative risk factors, type of operation performed, and details on more than a dozen postoperative complications, including mortality. Patient satisfaction data were obtained from the CMS Hospital Compare website for 2008, the first year these data were publicly reported.³

Using the linked data sets, our study population included Medicare patients older than 65 years who underwent an operation at a participating ACS NSQIP hospital during the study period. Patients were excluded if they underwent surgery at hospitals that had fewer than 20 cases represented in our data (86 patients treated at 8 hospitals) or if patient satisfaction data were unavailable (17 117 patients treated at 29 hospitals).

The study design and procedures were approved by the RAND Health institutional review board. Because the study was retrospective, informed consent was not required.

We analyzed 5 separate 30-day outcomes at the patient level: postoperative mortality, any major complication, any minor complication, failure to rescue (death after any complication), and readmission. Major complications included any of the following, as defined by ACS NSQIP: organ-space surgical site infection, wound dehiscence, pneumonia, respiratory failure, pulmonary embolism, acute renal failure, stroke, coma, myocardial infarction, cardiac arrest, bleeding requiring transfusion, sepsis or septic shock, and return to the operating room. Minor complications, also in accordance with ACS NSQIP definitions, included superficial or deep-space surgical site infection, deep vein thrombosis, progressive renal failure, or urinary tract infection. Hospital readmissions were analyzed only for patients who survived to discharge.

Our primary predictor of interest was hospital performance on the HCAHPS survey, which is the primary component of the Patient Experience of Care domain in CMS’s Value-Based Purchasing program.³ The HCAHPS survey consists of 27 questions asked to recently discharged patients about their hospital stay and encompasses various aspects of the hospital experience, such as physician and nurse communication, responsiveness of staff, and the cleanliness and quietness of the hospital environment. Details on the survey’s psychometrics have been published elsewhere.^22-24 Two questions pertaining to the patient’s global impression of care received served as our primary measure of patient satisfaction. Quiz Ref IDWe created a composite score for patient satisfaction by taking the average of the responses to 2 questions: (1) the number of patients reporting that they would recommend the hospital to family or friends, and (2) the number of patients giving the hospital a global rating of 9 or 10 out of 10. We chose these items for 2 reasons. First, this is consistent with much of the previous literature.^6,10,17 Second, these measure a global assessment of patient experience, one that is less susceptible to bias from a single negative experience. Our preliminary analyses demonstrated a very strong correlation between responses to these 2 questions (Spearman ρ = 0.91). We then grouped hospitals into quartiles based on their performance on this composite score.

To adjust for patient condition, we controlled for preoperative risk factors as recorded by ACS NSQIP: age, sex, admission source (home, acute care facility, other), American Society of Anesthesiologists class, functional status (independent, partially dependent, fully dependent), wound class (clean/clean-contaminated, contaminated, dirty), emergency case, and a number of comorbidities, including diabetes mellitus, dyspnea (at rest or on exertion), ventilator dependence, chronic obstructive pulmonary disease, myocardial infarction within 6 months, congestive heart failure, hypertension requiring medication, renal failure, presence of ascites, disseminated cancer, recent chemotherapy or radiation treatment, weight loss, steroid use, bleeding disorder, or preoperative sepsis. We controlled for procedure type by calculating linear risk probabilities for 23 Current Procedural Terminology buckets grouped by organ system; we calculated separate linear risks for each outcome modeled in accordance with ACS NSQIP methods.²⁵

We also controlled for hospital characteristics, as defined by the American Hospital Association annual survey, including hospital ownership (public, for profit, nonprofit), hospital size (>400 beds [large]; 100-400 beds [medium]; <100 beds [small]), teaching hospital status, and hospital census region (Midwest, Northeast, South, and West).

Data preparation and analyses were performed using SAS version 9.3 (SAS Institute, Inc) and Stata version 13.1 (StataCorp LP) statistical software. We compared patient and hospital characteristics across patient satisfaction quartiles using χ² tests for categorical variables and Wilcoxon tests for continuous variables. P < .05 was considered statistically significant.

We modeled the relationship between each measure of hospital quality and patient satisfaction quartiles using hierarchical logistic regression. Each model controlled for patient risk factors and procedure type as well as hospital structural characteristics. Marginal effects of patient satisfaction quartiles were also estimated.

Our study sample consisted of 103 866 patients treated at 180 unique hospitals. Table 1 summarizes the overall study population divided by hospital quartile of patient satisfaction scores. The mean age was 75.5 years, and the majority were female (51.5%). Quiz Ref IDMost patients were functionally independent (92.8%) and admitted from home (92.7%). In general, patient characteristics differed significantly across patient satisfaction quartiles. For example, there were significant differences across quartiles in the number of patients who smoked or had diabetes, chronic obstructive pulmonary disease, congestive heart failure, hypertension, ascites, and preoperative sepsis. In most cases, the prevalence of each comorbidity was highest among patients treated at hospitals in the lowest satisfaction quartile.

Overall patient satisfaction scores ranged from 44.8% to 82.8% (mean [SD], 68.0% [7.2%]). The mean overall patient satisfaction score was 58.7% for the first quartile, 66.2% for the second quartile, 70.8% for the third quartile, and 76.7% for the fourth quartile (Table 2). Of the 180 hospitals in our data, most were nonprofit (82.8%), large (57.8%), and teaching hospitals (52.2%). Hospitals were most commonly located in the Midwest (36.7%) followed by the Northeast (23.9%). We found no significant association between patient satisfaction quartile and hospital size or census region. We did, however, note significant differences with regard to teaching hospital status and hospital ownership (P = .01 and .03, respectively). Quiz Ref IDThe lowest patient satisfaction quartile had more nonteaching hospitals than teaching hospitals (65.2% vs 34.8%, respectively), while the highest quartile more commonly had teaching hospitals than nonteaching hospitals (61.4% vs 38.6%, respectively). Hospitals in the highest satisfaction quartile were most frequently nonprofit (93.2%).

In multivariate regression, we found that patient satisfaction quartile was significantly associated with 30-day mortality and failure to rescue (Table 3). Quiz Ref IDIn comparison with patients treated at the lowest quartile, those at the highest quartile had 15% lower odds of death within 30 days (odds ratio [OR] 0.85; 95% CI, 0.73-0.99) and 18% lower odds of failure to rescue (OR = 0.82; 95% CI, 0.70-0.96). Patients treated at the highest satisfaction quartile hospitals also experienced 13% lower odds of a minor complication (OR = 0.87; 95% CI, 0.75-0.99). Major complications did not reach statistical significance. We found a nonlinear relationship between patient satisfaction and hospital readmission. Patients treated at hospitals in the second quartile had significantly lower odds of readmission (OR = 0.90; 95% CI, 0.82-0.98), while those treated at hospitals in the third or highest quartile did not have significantly higher odds of readmission.

The Figure displays the risk-adjusted rate of each outcome across patient satisfaction quartiles. Quiz Ref IDCompared with patients treated in hospitals with the lowest satisfaction scores, those treated at hospitals with the highest satisfaction scores had a 0.6% lower rate of 30-day mortality (relative risk reduction = 11.1%; P = .04), a 2.2% lower rate of failure to rescue (relative risk reduction = 12.6%; P = .02), a 1.0% lower rate of major complication (P = .11), and a 1.2% lower rate of minor complication (relative risk reduction = 11.5%; P = .04). Patients in the third quartile of patient satisfaction had the lowest risk-adjusted rates of both mortality and failure to rescue, although these rates were not statistically significantly different from those in the highest quartile (P = .18 for mortality and P = .12 for failure to rescue). For readmission, patients in the second quartile had the lowest risk-adjusted 30-day readmission rate (11.0% vs 12.0% for the lowest quartile; P = .02), while patients treated at hospitals in the highest quartile had the highest rate (12.4%; P = .45 for comparison with the lowest quartile).

The extent to which measures of patient experience correlate with more objective markers of health care quality remains unclear. Using a national sample of patients, we found significant associations between both patient and hospital characteristics and hospital-level measures of patient satisfaction. After controlling for these differences, patients treated at hospitals with higher patient satisfaction scores experienced lower rates of 30-day mortality, failure to rescue, and minor complications. However, we were unable to demonstrate significant differences in major complications or hospital readmissions.

The relationship between patient satisfaction and hospital quality has long been controversial. More than 30 years ago, Donabedian argued, “to the extent that client satisfaction is a judgment on the quality of care, it is not part of the definition of quality.”²⁶ Over the years, this skepticism regarding the validity of patient satisfaction as a quality metric has been supported by several studies,^{11,12,16,27,28} including one notable study by Fenton et al¹⁴ that found high satisfaction scores were associated with higher rates of mortality. Within surgery, 2 recent articles also suggested the absence of any relationship between patient satisfaction and surgical quality^15,16; however, both were limited by a small sample of hospitals and studied a limited number of surgical outcomes. Combined, these studies have led some to question whether patient satisfaction should be used as a marker of hospital quality.¹⁵

In contrast, our study adds to a growing body of literature suggesting that providing a high-quality patient experience need not preclude the delivery of high-quality care.^6,9,29,30 One recent study used Medicare data to demonstrate that high patient satisfaction scores were associated with higher compliance on the Surgical Care Improvement Project measures, fewer readmissions, decreased length of stay, and lower mortality following surgery.¹⁷ Another study, using administrative data from the University HealthSystem Consortium, found no relationship between patient satisfaction scores and compliance with process measures, patient safety indicators, and length of stay but did find an association between high satisfaction scores and lower risk-adjusted mortality.¹⁰ Our study is the first, to our knowledge, to use high-quality clinical data from a national sample, allowing for robust risk adjustment and measurement of important clinical outcomes that are not measured reliably with administrative data, such as postoperative complications and failure to rescue. Considered together, these studies provide important evidence in support of CMS’s inclusion of patient experience as a core component of its Value-Based Purchasing program.³¹

However, in line with the conflicting results of previous literature, our findings draw attention to the complex relationship between patient satisfaction and certain measurements of surgical quality. While we demonstrated significant reductions in several postoperative adverse events in hospitals with high patient satisfaction scores, the relationships were not always linear. For example, for both mortality and failure to rescue, the lowest risk-adjusted rates were noted in the second highest quartile, with slightly higher rates (although not statistically significantly higher) in the highest quartile. Similarly, risk-adjusted readmissions were lowest in the second quartile and highest in the highest patient satisfaction quartile.

The nonlinear relationship between patient satisfaction and surgical quality has several possible explanations and implications. First, instead of reflecting a patient’s health care experience, patient satisfaction may actually be more closely related to patient-specific factors, such as expectations of care³² or their current state of health.⁹ Furthermore, in recollecting their experiences, patients tend to focus disproportionately on only a few key moments of their encounter, which may not be representative of their entire health care experience.^33,34 Without a perfect measurement of patient experience, its relationship with other measures of quality may be tenuous. Second, patient satisfaction may fall into a different domain of health care quality from other surgical quality metrics, which may partially explain the conflicting findings in the patient satisfaction literature as well as the inconsistent findings in our study.^1,35 Yet, the absence of a relationship and the occasional nonlinear relationship between these quality domains are not necessarily a critique of either’s value as a quality metric; each is simply measuring an independent component of a larger definition of quality. Third, patient satisfaction itself is composed of multiple domains and therefore varies depending on the measurement tool.⁹ The HCAHPS survey, which is completed by patients within 42 days of discharge, is the most widely used instrument but may not appropriately capture the domains of patient satisfaction most relevant to surgical patients.³⁶ A surgery-specific version of HCAHPS, the Consumer Assessment of Healthcare Providers and Systems Surgical Care Survey, was recently endorsed by the National Quality Forum and will likely provide a more accurate assessment of patient satisfaction among surgery patients.³⁷

There are limitations to our study. First, several years have passed since this unique data set was created as part of a contract with CMS, during which time temporal trends may have led to improvements in both patient satisfaction scores and surgical outcomes. Nevertheless, while the values of these 2 measures may have changed, there is no reason to suspect that the relationship between them would differ in a more recent sample. Furthermore, such an analysis would not be possible with more recent ACS NSQIP data owing to the lack of hospital identifiers in public use files. Our data are therefore uniquely able to test the relationship between patient satisfaction and surgical outcomes. Second, because participation in ACS NSQIP is voluntary, our data represent a convenience sample of hospitals and our results may not be generalizable to other hospitals in the United States. Third, because we found older, sicker patients being treated at hospitals with low satisfaction scores (although many of these differences may not be clinically significant), it is possible that our risk-adjustment strategy was inadequate to fully control for these differences. Nonetheless, we used high-quality clinical data from ACS NSQIP, which represents the most advanced risk-adjustment technique available. Fourth, HCAHPS data were collected from a sample of all discharged patients, not just those undergoing surgery. While no patient-level data are currently available to determine whether individual patient satisfaction is associated with the quality of care received, there is evidence to suggest that hospitals that perform well on 1 quality measure tend to do just as well on other measures.^38,39 Finally, our data set consists only of Medicare patients, which may limit the generalizability of our findings to nonelderly patients. As Medicare patients undergo a large proportion of surgical procedures, have higher rates of morbidity and mortality, and represent the precise target population for the CMS Value-Based Purchasing program, we believe our results can help inform future policy decisions.

Using a national sample of patients undergoing surgery, we demonstrated a significant association between hospital performance on a patient satisfaction survey and objective measures of surgical quality. Patients treated at hospitals with the highest satisfaction scores experienced lower rates of postoperative mortality, failure to rescue, and minor complications. Our findings suggest that payment policies that incentivize better patient experience do not require hospitals to sacrifice performance on other quality measures.

Corresponding Author: Greg D. Sacks, MD, MPH, Department of Surgery, David Geffen School of Medicine, University of California, Los Angeles, 10833 Le Conte Ave, 72-215 CHS, Los Angeles, CA 90095 (gsacks@mednet.ucla.edu).

Accepted for Publication: March 16, 2015.

Published Online: June 24, 2015. doi:10.1001/jamasurg.2015.1108.

Author Contributions: Dr Sacks had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Sacks, Lawson, Dawes, Zingmond, Ko.

Acquisition, analysis, or interpretation of data: Sacks, Lawson, Dawes, Russell, Maggard-Gibbons, Zingmond.

Drafting of the manuscript: Sacks.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Sacks, Lawson, Dawes, Russell, Zingmond, Ko.

Administrative, technical, or material support: Dawes.

Study supervision: Russell, Maggard-Gibbons, Zingmond, Ko.

Conflict of Interest Disclosures: None reported.

Funding/Support: Drs Sacks and Dawes were supported by the Robert Wood Johnson Foundation Clinical Scholars Program at the University of California, Los Angeles.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.