Evaluating Popular Media and Internet-Based Hospital Quality Ratings for Cancer Surgery

Objective  We sought to determine whether “best hospitals,” as defined by the US News & World Report or HealthGrades, have lower mortality rates than all other American hospitals for cancer surgery.

Design  Retrospective cross-sectional study.

Setting  Medicare database (2005-2006).

Patients  All patients with a diagnosis of malignancy who underwent pancreatectomy, esophagectomy, and colectomy (n = 82 724).

Main Outcomes Measures  Risk-adjusted mortality rates at best hospitals according to both the US News & World Report and HealthGrades, was compared with all other US hospitals, adjusting for differences in patient factors and surgical acuity. Risk-adjusted mortality rates between best hospitals and all other hospitals was compared after controlling for differences in hospital volume.

Results  Risk-adjusted mortality was significantly lower in US News & World Report best hospitals for all 3 procedures: pancreatectomy (odds ratio [OR], 0.42; 95% confidence interval [CI], 0.30-0.58), esophagectomy (OR, 0.48; 95% CI, 0.37-0.62), and colectomy (OR, 0.69; 95% CI, 0.55-0.86). Risk-adjusted mortality was significantly lower in HealthGrades best hospitals for colectomy (OR, 0.79; 95% CI, 0.65-0.95). However, after accounting for hospital volume, risk-adjusted mortality was only significantly lower at the US News & World Report best hospitals for colectomy (OR, 0.79; 95% CI, 0.62-0.99) and was not significantly lower at HealthGrades best hospitals for any of the 3 oncologic procedures.

Conclusions  Publicly marketed hospital rating systems of surgical quality such as the US News & World Report “America's Best Hospitals” and HealthGrades “Best Hospitals” may identify high-quality hospitals for some oncologic surgeries. However, these ratings fail to identify other high-volume hospitals of equal quality.

The US News & World Report “America's Best Hospitals” and the HealthGrades “Best Hospitals” are two of the most recognized publicly available hospital ratings. The US News & World Report “America's Best Hospitals” is an annual rating of the “fifty best” hospitals across 16 medical and surgical specialties including the 50 best cancer hospitals. HealthGrades markets hospital report cards to health care consumers on the internet, reporting the “quality” of health care of 5000 American hospitals.¹

Both the US News & World Report and HealthGrades hospital quality ratings receive significant publicity. Hospitals frequently use these ratings in advertising, increasing the influence of the ratings in public perception of hospital quality.² The US News & World Report is widely available, boasting a paper circulation of more than 2 million readers and an online readership with free access to all reports free of charge. Similarly, HealthGrades reports more than 5 million visits monthly to it's consumer Web site and markets itself as the nation's “leading healthcare ratings organization.”¹

Although these quality ratings are marketed as a consumer aid for choosing hospitals, it is unclear whether these ratings appropriately identify high-quality hospitals. Previous studies have shown that competing hospital quality ratings frequently provide inconsistent results.³ No previous studies have addressed the question of whether the “best hospitals” have better cancer surgery outcomes than other hospitals in the United States.

We sought to determine independently whether these two quality ratings truly identify the best hospitals for cancer surgery. We used the 2005-2006 MedPar data set to compare the 30-day mortality in the best hospitals and all other hospitals for 3 cancer surgery procedures.

The 50 top-ranked cancer hospitals were determined from the 2008 publication of the US News & World Report “America's Best Hospitals.”⁴ These hospitals are determined by a combination of 3 equally weighted measures: hospital infrastructure, hospital reputation among subspecialists, and 30-day mortality rate. Only hospitals that are either a teaching hospital or have a minimum number of advanced technologies are considered eligible for the ratings. In addition, hospitals must meet a minimum discharge and surgical volume threshold for 12 subspecialties (cancer; ear, nose, and throat; endocrinology; gastrointestinal disorders; geriatric care; gynecology; heart and heart surgery; kidney disease; neurology and neurosurgery; orthopedics; respiratory disorders; and urology). The US News & World Report evaluates hospital infrastructure using measures from the American Hospital Association annual survey, including hospital teaching status, Nurse Magnet status, staff-to-bed ratios, hospital volume, and the presence of advanced technologies. The most subjective portion of the rating system, reputation of the hospitals among subspecialists, is based on a survey sent out to fewer than 100 members of each subspecialty to rank the 5 best hospitals. Risk-adjusted mortality rates for each hospital are calculated using Medicare claims data from the previous year. Using a composite score of these 3 measures, hospitals are ranked and the top 50 hospitals for each specialty are published.

The HealthGrades “America's 50 Best Hospitals” were collected directly from the HealthGrades Web site in October 2008.¹In contrast to the US News & World Report ratings, the HealthGrades ratings are based solely on Medicare data. HealthGrades generates hospital ratings using a proprietary method, comparing predicted with observed mortality rates for all hospitals. The predicted 30-day mortality rate is derived from Medicare Part A billing data (MedPar 2005-2006). These predicted mortality rates were then compared with the observed mortality rates for each hospital, generating an observed to expected ratio. The hospitals are then rated based on a ratio of observed to expected mortality rate for 27 procedures and diagnoses. The 50 hospitals with the best observed to expected ratios are then included in the final list available on the HealthGrades Web site.

All patients who underwent the 3 cancer operations, pancreatectomy, esophagectomy, and colectomy, were identified from the MedPar database from 2005 through 2006 using International Statistical Classification of Diseases, 10th Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes (n = 82 724). A total of 6455 patients underwent surgery in the 50 US News & World Report “America's Best Cancer Hospitals,” compared with 76 369 patients in the remaining 4445 hospitals nationwide. A total of 3452 patients from the 50 HealthGrades “Best Hospitals” were compared with 79 112 patients from the remaining 4445 hospitals nationwide. A total of 639 hospitals that do not participate in the American Hospital Association Annual Survey were not included in the analysis. All of the best hospitals from the US News & World Report “America's Best Cancer Hospitals” and the HealthGrades “Best Hospitals” participate in the American Hospital Association Annual Survey.

Given the lack of agreement between the two quality ratings, the 50 best hospitals in each rating were compared with all other hospitals separately for each individual procedure. Membership in the Council of Teaching Hospitals was obtained from the 2003 American Hospital Association Annual Survey. Patient age, race, comorbidities, surgical acuity, and hospital characteristics were compared between best hospitals and all other hospitals using χ² statistics, t tests, and analyses of variance.

Using all patients who underwent each of these 3 operations, we generated expected mortality rates using logistic regression models, adjusted for comorbidity burden, patient demographics, and emergent operation. Our independent variable was whether patients received care in a best hospital or all other hospitals. Random-effects logistic regression was used to provide robust adjustment for the effect of hospital clustering.

Next, to assess the relative effect of procedural volume on mortality in best hospitals, we created quintiles of procedure-specific hospital volume. Using random-effects logistic regression, we compared mortality following each of the 3 operations between best hospitals and all other hospitals, adjusting for patient covariates and hospital volume quintile. Using the relative attenuation of the odds ratio (OR), we estimated the relative effect of hospital volume on mortality in the best hospitals compared with all other hospitals.⁵

Comparing the US News & World Report “America's Best Cancer Hospitals” and the HealthGrades “Best Hospitals,” only 2 hospitals are considered best hospitals in both ratings.

Patient characteristics were compared between the 50 best hospitals in each rating and all other hospitals for each individual operation and each rating system (Table 1). The patients treated in the best hospitals were significantly different from the patients in all other hospitals. Across all procedures, US News & World Report “America's Best Hospitals” were less likely to operate emergently or urgently. HealthGrades “Best Hospitals” treated fewer black patients and were less likely to operate emergently or urgently. Patient comorbidity burden did not significantly differ between best hospitals from either rating system and all other hospitals.

The characteristics of hospitals in each rating differed significantly from all other hospitals (Table2). US News & World Report “America's Best Cancer Hospitals” were significantly less likely to be very-low-volume hospitals for all 3 operations. “America's Best Cancer Hospitals” were also more likely to be teaching hospitals as compared with all other hospitals. Similarly, HealthGrades “Best Hospitals” were less likely to be very-low-volume for any of the cancer operations. HealthGrades “Best Hospitals” were also more likely to be teaching hospitals as compared with all other hospitals.

Adjusting for patient characteristics and surgical acuity, 30-day mortality rates were lower in US News & World Report “America's Best Cancer Hospitals” for all 3 operations: pancreatectomy (OR, 0.42; 95% confidence interval [CI], 0.30-0.58), esophagectomy (OR, 0.48; 95% CI, 0.37-0.62), and colectomy (OR, 0.69; 95% CI, 0.55-0.86). In contrast, risk-adjusted mortality was significantly lower in HealthGrades “Best Hospitals” following 1 of 3 procedures: colectomy (OR, 0.79; 95% CI, 0.65-0.95). The risk-adjusted mortality rates of the US News & World Report and HealthGrades rated hospitals and all other hospitals are shown in Figure 1 and Figure 2, respectively.

After accounting for hospital volume, US News & World Report “America's Best Cancer Hospitals” 30-day mortality rates were significantly lower for only 1 of 3 procedures: colectomy (OR, 0.69; 95% CI, 0.62-0.99). In comparison, accounting for hospital volume, mortality rates did not differ significantly between HealthGrades “Best Hospitals” and all other hospitals for any of the 3 procedures (Table2).

Using the relative attenuation of the OR to determine how much of the differences in mortality can be explained by volume, a significant proportion of the differences in mortality between “Best Hospitals” and all other hospitals in both rating systems can be attributed to hospital volume (Table 3). As much as 71% of the observed differences in mortality following pancreatectomy between US News & World Report “Best Hospitals” and other hospitals is owing to differences in hospital volume.

Current consumer-marketed ratings of hospital quality may identify hospitals with lower surgical mortality rates for 3 cancer surgery procedures. “America's Best Cancer Hospitals” and HealthGrades “Best Hospitals” had significantly lower risk-adjusted mortality rates for all 3 and 1 of 3 operations, respectively. Previous studies of hospital ratings have shown that both HealthGrades and US News & World Report best hospitals have lower mortality rates for patients admitted with acute myocardial infarction and some cardiovascular procedures.^6-8 Interestingly, these differences in mortality rates did not exist for all operations in both rating systems. Of note, HealthGrades rating system did not identify high-quality hospitals for patients who underwent high-risk cancer surgery, pancreatectomy, and esophagectomy.

Although these ratings systems may identify hospitals with lower-than-average mortality, a significant amount of the differences in surgical mortality are attributable to differences in procedural volume. In this analysis, procedural volume completely explained the differences in mortality for pancreatectomy and esophagectomy. These differences in procedural volume have been previously reported as important predictors of risk-adjusted mortality, especially for less-common, high-morbidity surgical procedures such as esophagectomy and pancreatectomy.⁹ Similarly, when comparing hospitals performing cardiovascular procedures, volume has a significant effect on quality ratings.⁸ Although both the US News & World Report and the HealthGrades quality ratings rely on risk-adjusted mortality rates, neither rating controls for differences in procedural volume. Our findings show that both ratings fail to identify equally well-performing hospitals of similar volume. Mortality rates between best hospitals and hospitals of similar volume were not significantly different.

This study has important implications for patients trying to choose safe hospitals for oncologic surgery. While we found a mortality rate benefit to choosing these highly rated hospitals, patients may equally benefit from going to a similarly high-volume hospital closer to their home. Pervious study by Birkmeyer and colleagues has shown that most patients live close to a high-volume hospital. In an analysis of patients who underwent pancreatectomy or esophagectomy, more than one-quarter of patients lived closest to a high-volume hospital and nearly 75% of patients would travel fewer than 30 minutes to reach a high-volume hospital.¹⁰ Because these ratings list only a selected number of best hospitals, these consumer aids may dissuade patients from seeking care at closer high-volume, equal-quality hospitals.

This study has several limitations. First, the study assessed the validity of these hospital ratings using the outcome of 30-day mortality. Mortality represents just one domain of surgical quality; other outcomes are equally important such as readmission rates, complications, and cost. In addition, in the Donabedian structure-process-outcomes model of quality,¹¹ structural and process measures may play important roles in measurement of surgical quality. Unfortunately, current databases do not include robust measures of these domains. Although this analysis was limited to mortality rate, this is a key measure used in the methodologies of both hospital ratings. Second, our study relies on Medicare claims data, limiting our study population to patients older than 65 years. However, of elderly persons represent most patients undergoing these procedures, and elderly patients have the highest risk associated with these surgeries. In addition, administrative data lack robust clinical detail to entirely account for differences in case-mix and patient illness severity that may confound the differences in mortality rate between hospitals. We have controlled for available patient risk factors including demographics, comorbidities, and acuity of admission/surgery. Although differences in patient factors may exist between hospitals, there is a lack of clinical data to adequately account for such differences. More robust patient and hospital data will likely improve our ability to accurately rate the quality of hospitals.

This study has significant policy implications. As politicians, patients, and payers continue to demand accessible quality measurements of medical care, accurate hospital quality ratings must be developed. Reliable and accessible methods of rating hospital quality will require a collaboration of not only independent ratings organizations, but hospitals and payers as well. Unfortunately, current quality ratings may confuse patients and dissuade them from receiving care in high-quality hospitals close to home. Future methods of rating hospital quality should consider not only quality but access to those quality hospitals.

Both US News & World Report and HealthGrades may identify high-quality hospitals for receiving cancer surgery. However, these ratings fail to identify other high-volume hospitals with equivalent outcomes. Further research into the measurement of surgical quality will be necessary to help aid patient decision making.

Correspondence: Nicholas H. Osborne, MD, 6312 Medical Science Building I, 1150 W Medical Center Dr, SPC5604, Ann Arbor, MI 48109-5604 (nichosbo@umich.edu).

Accepted for Publication: April 9, 2010.

Author Contributions:Study concept and design: Osborne and Dimick. Acquisition of data: Osborne, Ghaferi, Nicholas, and Dimick. Analysis and interpretation of data: Osborne, Ghaferi, Nicholas, and Dimick. Drafting of the manuscript: Osborne, Ghaferi, Nicholas, and Dimick. Critical revision of the manuscript for important intellectual content: Osborne, Ghaferi, Nicholas, and Dimick. Statistical analysis: Osborne, Ghaferi, Nicholas, and Dimick. Obtained funding: Osborne. Study supervision: Dimick.

Financial Disclosure: None reported.