International Validity of the HOSPITAL Score to Predict 30-Day Potentially Avoidable Hospital Readmissions

Importance  Identification of patients at a high risk of potentially avoidable readmission allows hospitals to efficiently direct additional care transitions services to the patients most likely to benefit.

Objective  To externally validate the HOSPITAL score in an international multicenter study to assess its generalizability.

Design, Setting, and Participants  International retrospective cohort study of 117 065 adult patients consecutively discharged alive from the medical department of 9 large hospitals across 4 different countries between January 2011 and December 2011. Patients transferred to another acute care facility were excluded.

Exposures  The HOSPITAL score includes the following predictors at discharge: hemoglobin, discharge from an oncology service, sodium level, procedure during the index admission, index type of admission (urgent), number of admissions during the last 12 months, and length of stay.

Main Outcomes and Measures  30-day potentially avoidable readmission to the index hospital using the SQLape algorithm.

Results  Overall, 117 065 adults consecutively discharged alive from a medical department between January 2011 and December 2011 were studied. Of all medical discharges, 16 992 of 117 065 (14.5%) were followed by a 30-day readmission, and 11 307 (9.7%) were followed by a 30-day potentially avoidable readmission. The discriminatory power of the HOSPITAL score to predict potentially avoidable readmission was good, with a C statistic of 0.72 (95% CI, 0.72-0.72). As in the derivation study, patients were classified into 3 risk categories: low (n = 73 031 [62.4%]), intermediate (n = 27 612 [23.6%]), and high risk (n = 16 422 [14.0%]). The estimated proportions of potentially avoidable readmission for each risk category matched the observed proportion, resulting in an excellent calibration (Pearson χ² test P = .89).

Conclusions and Relevance  The HOSPITAL score identified patients at high risk of 30-day potentially avoidable readmission with moderately high discrimination and excellent calibration when applied to a large international multicenter cohort of medical patients. This score has the potential to easily identify patients in need of more intensive transitional care interventions to prevent avoidable hospital readmissions.

Hospital readmissions are frequent, harmful and costly—18% of Medicare patients can expect to be readmitted within 30 days in the United States at a cost of more than $17 billion.¹ Recent changes in health care policy aimed at reducing readmission have substantially increased attention to this major health care issue.

Although some readmissions are unavoidable and the proportion of readmissions that are truly avoidable remains controversial, at least some appear to be preventable, therefore leaving room for improvement by hospitals and health care systems.^2,3 A recent systematic review and meta-analysis⁴ showed a relative readmission risk reduction of 0.82 from transitions interventions (P < .01); however, the most effective interventions were of high complexity. To most efficiently reduce hospital readmissions, hospitals need to target complex and intensive discharge interventions at those patients at high risk of potentially avoidable readmission who are more likely to benefit. However, prior research indicates that clinical health care providers are not able to accurately identify which patients are at high risk for readmission.⁵

Prediction models may provide a more efficient and accurate means to identify those patients at highest risk. Unfortunately, most existing predictions models have yielded only fair discriminative ability, do not focus on readmissions that are avoidable, are not externally validated, or are not easy to use in practice.⁶ In response to these shortcomings, we previously derived and internally validated the HOSPITAL score consisting of 7 readily available clinical predictors (Table 1 lists components forming the acronym). This simple prediction model accurately identified patients at high risk of potentially avoidable readmissions among more than 10 000 medical inpatients⁸ and is innovative in several ways: it focuses on 30-day readmissions that are potentially avoidable, shows good discrimination power to accurately differentiate low-risk from high-risk patients, includes only readily available predictors, and can be calculated before discharge.

Before wider application of the HOSPITAL score can be recommended, it is important to understand how the score performs in populations outside those from which the score was developed. To assess its generalizability, we aimed to externally validate the HOSPITAL score in a large international multicenter cohort of medical patients.

Box Section Ref ID

This multicenter multinational retrospective cohort study included all consecutive adult patients discharged alive between January 1, 2011, and December 31, 2011, from the medical services of 9 hospitals in 4 different countries: the United States, Canada, Israel, and Switzerland (ICARE: International Cohort of Avoidable REadmissions). To minimize inclusion of observation stays, only hospitalizations with a length of stay of more than 1 day were included. We then excluded patients who were transferred to another acute health care facility (hospital or psychiatric hospital) or who left against medical advice.

The participating centers were: (1) University of California San Francisco Medical Center, San Francisco, California; (2) Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania; (3) Harborview Hospital Medicine, University of Washington, Seattle, Washington; (4) Vanderbilt University Medical Center, Nashville, Tennessee; (5) Northwestern Memorial Hospital, Chicago, Illinois; (6) Christiana Hospital, Wilmington, Delaware; (7) Sheba Medical Center, Tel Hashomer, Israel; (8) William Osler Health System, Ontario, Canada; (9) Bern University Hospital, Bern, Switzerland. Of these, 7 are university hospitals and 2 are community hospitals (Christiana Hospital and William Osler Health System). All are not-for-profit hospitals.

The trial protocol was approved by the institutional review board of each site and also of the managing site at Brigham and Women’s Hospital/Partners Healthcare, Boston, Massachusetts.

The study follows the criteria from the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) Initiative (checklist available upon request).⁹

First, all 30-day readmissions to any division of the same medical center were identified. The primary outcome of 30-day potentially avoidable readmission was then identified through the same process as in the derivation study, using the SQLape algorithm.⁸ This algorithm, based on administrative data and diagnosis codes, was initially developed in Switzerland and has been used there for now for more than 5 years for hospital benchmarking purposes.^10-12 Readmissions which are foreseeable are classified as unavoidable, including readmissions for transplantation, labor and delivery, chemotherapy or radiotherapy, follow-up or rehabilitation treatment, specific surgical procedures, or some specific diseases deemed difficult to cure (eg, multiple sclerosis and idiopathic thrombocytopenia). Then, the SQLape algorithm classifies readmissions as unavoidable if they involve a new organ system unknown to be affected during the preceding hospitalization. Conversely, complications of treatment (eg, deep vein thrombosis, catheter associated urinary tract infection, drug-induced disorders) were considered potentially avoidable.

The unit of analysis was the hospital discharge, and each admission could be considered as both an index hospitalization and a rehospitalization when several hospital stays occurred within an interval of less than 30 days.

Information on the 7 predictor variables included in the HOSPITAL score was collected at each site using the same definition as the derivation study (Table 1).⁸ HOSPITAL is the acronym for the 7 variables included in the score: last available Hemoglobin before discharge (positive if <12 g/dl [to convert to g/L, multiply by 10]), discharge from an Oncology service, last available Sodium level before discharge (positive if <135 mEq/L [to convert to mmol/L, multiply by 1]), any Procedure performed during the hospitalization (any International Classification of Disease [ICD-9⁷ or ICD-10¹³] coded procedure), Index admission Type (emergent or urgent as opposed to elective), number of Admissions in the previous 12 months, and Length of stay (positive if ≥5 days). International Classification of Disease procedures in medical patients included, for example, endoscopy, hemodialysis, heart cardiac catheter, paracentesis or thoracocenthesis, transfusion, computed tomography scan or magnetic resonance imaging, or percutaneous transluminal coronary angioplasty. The scoring system ranges from 0 to a maximum of 13 points, with a risk of 30-day potentially avoidable readmission that increases with the number of points.

Owing to the absence of patients with 0 previous admissions in the last 12 months at 2 sites, patients with only 1 admission in the previous year were attributed 0 points instead of 1 as in the original score. Overall, 6907 of 117 065 of hemoglobin values (5.9%) and 3980 of 117 065 of sodium values (3.4%) were missing. The values were imputed as normal when missing (ie, no points attributed for the missing variable). Other data collected included demographic information in order to describe the population.

Patients’ baseline characteristics were presented as proportions, means with standard deviation (SD), and as medians with interquartile ranges (IQR) as appropriate.

First, the HOSPITAL score was calculated for each hospital discharge. Then the risk for an admission to be followed by a 30-day potentially avoidable readmission was categorized into 3 groups according to the total number of points: low risk (0 to 4 points), intermediate risk (5 to 6 points), and high risk (≥7 points). These categories were created in the derivation study for ease of interpretation, roughly corresponding to 5%, 10%, and 20% risk of potentially avoidable readmission.

The score was evaluated according to its overall performance, its discriminatory power, and its calibration. We used the Brier score to quantify how close predictions are to the actual outcome (overall performance).^14,15 A useful prediction rule ought to have a Brier score smaller than 0.25 (the lower the better). The discriminatory power reflects the ability of the score to discriminate the cases from the noncases based on both sensitivity and specificity (C statistic).^16,17 The calibration shows the ability of the model to generate probabilities that match the observed rates.¹⁵ We used the Pearson χ² test to summarize this measure and presented both observed and predicted rates for each risk group of the score (P ≥ 0.05 implies a good fit, with higher numbers signifying better fit).

Discharges followed by an unavoidable 30-day readmission (according to SQLape classification) were grouped with the discharges not followed by any 30-day readmission (group control) for the main analysis. In a sensitivity analysis, we also evaluated the performance of the HOSPITAL score when these unavoidable readmissions were completely excluded from the analysis.

All tests were conducted as 2-sided at a .05 level of significance. In the logistic regression, we used a fixed effect for hospital to account for possible differences in outcomes across the 9 hospitals. We also used a robust sandwich variance estimator to account for repeated admissions from the same patient within hospital.¹⁸ Analyses were performed with SAS Software, version 9.3 (SAS Institute Inc).

During the study period, a total of 121 136 adults were discharged alive from the medical department of the 9 participating centers, and 117 065 (96.6%) discharges were included in the analysis (Figure 1). Within 30 days after discharge, 15.0% (n = 17 516) of the medical patients had a readmission, and 9.7% (n = 11 307) had a potentially avoidable readmission. The median (IQR) time to readmission was 12 (5-19) days. Baseline characteristics of patients are summarized in Table 2. Patients had a mean (SD) age of 60.8 (18.2) years and had a median (IQR) length of stay of 4 (3-7) days. Overall, patients with a potentially avoidable readmission more often had an urgent or emergent index admission, were more frequently discharged from an oncology service, had a length of stay greater than 5 days, had more hospitalizations in the past year, more often had a procedure, and more often had a low hemoglobin or low sodium level at discharge. All these differences were highly statistically significant (P < .001).

The score ranged from 0 to 13, with a median (IQR) HOSPITAL score of 4 (2-5). Using the HOSPITAL score, 73 612 patients (62.4%) were categorized as low risk for a potentially avoidable readmission; 27 612 [23.6%], intermediate risk; and 16 422 [14.0%], high risk. Risk of 30-day potentially avoidable readmission for patients with a HOSPITAL score in the low, intermediate, and high risk groups were 5.8%, 12.0%, and 22.8%, respectively.

We found that the HOSPITAL score had a C statistic of 0.72 (95% CI, 0.72-0.72), showing good discrimination (Figure 2). The Brier score was 0.08, signifying very good overall performance. In US hospitals, the C statistic was 0.72 (95% CI, 0.71-0.72); Canada, 0.78 (95% CI, 0.76-0.80); Israel, 0.68 (95% CI, 0.67-0.69); and Switzerland, 0.68 (95% CI, 0.66-0.71).

In a sensitivity analysis, excluding patients with unavoidable readmissions rather than combining them with patients without any readmissions did not significantly change the C statistic (C statistic, 0.73; 95% CI, 0.73-0.74).

In terms of calibration, the estimated risk of potentially avoidable readmission calculated with the HOSPITAL score matched the observed proportion of potentially avoidable readmissions in each risk group: 5.8% for the low risk group; 11.9%, intermediate; and 22.8%, high risk (Table 3). This is also reflected by an excellent Pearson χ² test with a P value of .89. When calibration is analyzed for each individual point score, calibration remains excellent except at the extreme ends of the range (low [0 points] and high [≥11 points]), where sample size is limited (eTable 1 in the Supplement). The positive likelihood ratio for each point score showed a similar pattern (eTable 2 in the Supplement).

In a large international multicenter external validation study, we found that the HOSPITAL score had good discriminative ability and excellent calibration for predicting the risk of 30-day potentially avoidable readmission. The HOSPITAL score is easy to use and can be calculated before discharge, which makes it a practical tool for identification of patients at high risk for preventable readmission and the timely administration of high-intensity interventions designed to improve transitions of care.

A recent systematic review⁶ comparing existing readmission scores pointed out the lack of validated, reliable, and generalizable score stratification models for readmission. Most scores developed to date have exhibited poor performance, lacked geographical validation, were developed using small sample size, or are complex. Among the most frequently cited score is the LACE score.¹⁹ Derived and validated in large populations in Ontario, Canada, the score showed a C statistic of 0.68 in Ontario and 0.70 in an external validation study in Singapore.²⁰ However, it has never been validated in other countries, including the United States. Also, the LACE score requires users to calculate another score, the Charlson comorbidity index, that estimates the 1-year mortality of the patient based on billing coded comorbidities, practically available only after discharge.

In comparison, the HOSPITAL score applies 7 readily available predictors. Importantly, it can be calculated before the discharge of the patient, in time for interventions to be started in the hospital. It showed good performance in this large international multicenter study, which is, to our knowledge, the first prediction model for readmission to be so widely validated across an international cohort of patients and one of the few that follows current methodological recommendations.²¹ The size of the validation cohort is also one of the largest. Moreover, the HOSPITAL score was designed specifically to focus on readmissions that are potentially avoidable (ie, those that are theoretically most likely to benefit from interventions). This is of critical importance because hospitals and health systems face decisions about how to best deploy limited resources, yet there has been a dearth of tools to help with resource disbursement.²² In a recent study²³ among nearly 20 000 medical patients in Denmark, the HOSPITAL score was significantly better than the LACE score for identifying patients at risk of 30-day readmission (P < .001).

With a C statistic of 0.72, the HOSPITAL score performs similarly to other commonly used prediction models. As a comparison, the well-known CHADS2 and CHADS2-VASC scores for stroke risk stratification in atrial fibrillation and the HAS-BLED score for bleeding risk on anticoagulation all have a C statistic of 0.68 or less in external validation studies.^24-27

This study has several limitations. First, by design, the model focuses on a medical population, and it may not be generalizable to other patient populations such as surgical patients. Second, additional predictors could have been used to improve the model’s discrimination (eg, functional status, socioeconomic status); however, we purposefully chose to use a smaller number of readily available predictors to increase clinical applicability and ease of use. Third, we should note that 30 days may not be the most appropriate time frame to judge preventable readmissions. However, we chose it because it has become the standard for research studies and for hospitals given its use by the Centers for Medicare and Medicaid Services in their Readmission Reduction Program. Fourth, the variables cannot and should be not used to guide readmission reduction interventions. The variables included are predictors and not necessarily modifiable risk factors. For example, we would not presume that transfusing a patient with a hemoglobin level less than 12 g/dL would be indicated to lower their readmission risk. Fifth, we acknowledge that transitional care interventions are most effective when implemented as early in the hospitalization as possible and that the score cannot always identify high-risk patients early because it includes length of stay and laboratory values at discharge. However, some patients can be identified as high risk at the time of admission (eg, based on admission type, service, and number of prior hospitalizations), while initially lower-risk patients may be monitored and then identified as high risk prior to discharge once they have had a procedure or their length of stay reaches 5 days. Intensive interventions could then be started at that time. Further research would be needed to determine the effects of interim low sodium and hemoglobin levels on readmission risk given their ability to normalize (or not) by the time of discharge. Last, we acknowledge that the outcome of interest in this study is “potentially avoidable readmissions” and not “avoidable readmissions.” The SQLape algorithm identifies a population with unplanned readmissions that are likely related to the index admission (eg, same organ system, possible complications of therapy). This population is enriched with patients whose readmissions are truly preventable but is neither 100% sensitive nor specific for identifying such patients, assuming such patients could ever be identified with certainty. The extent to which readmissions are preventable is controversial.

The advantage of tools such as the HOSPITAL score is that we can use it to begin to answer the more relevant question of which patients are most likely to benefit from interventions designed to prevent readmissions. Predicting potentially avoidable readmissions (as opposed to all readmissions) is an important step along this progression. This work is particularly important given recent incentives to reduce readmissions and several recent unsuccessful efforts to reduce them.²⁸ Future intervention studies should therefore be adequately powered to identify subgroups most likely to benefit, such as those identified by the HOSPITAL score.

The HOSPITAL score had good to excellent ability to identify patients at high risk of 30-day potentially avoidable readmission when applied to a large international cohort of medical patients. The HOSPITAL score is the first risk prediction score to focus on potentially avoidable readmissions as opposed to all-cause readmissions, using readily available predictors at the time of discharge. This study externally validated it in a large cohort in 4 countries. This score has the potential to reliably identify patients in need of more intensive transitional care interventions to prevent hospital readmissions.

Corresponding Author: Jacques Donzé, MD, MSc, Universitätsklinik für Allgemeine Innere Medizin, INSELSPITAL, Bern University Hospital, 3010 Bern, Switzerland (donze.author@gmail.com).

Published Online: March 7, 2016. doi:10.1001/jamainternmed.2015.8462.

Author Contributions: Dr Donzé had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Donzé, Schnipper.

Acquisition, analysis, or interpretation of data: Donzé, Williams, Robinson, Zimlichman, Aujesky, Vasilevskis, Kripalani, Metlay, Wallington, Fletcher, Auerbach, Schnipper.

Drafting of the manuscript: Donzé.

Critical revision of the manuscript for important intellectual content: Donzé, Williams, Robinson, Zimlichman, Aujesky, Vasilevskis, Kripalani, Metlay, Wallington, Fletcher, Auerbach, Schnipper.

Statistical analysis: Donzé.

Obtained funding: Donzé, Robinson.

Administrative, technical, or material support: Donzé, Robinson, Zimlichman, Aujesky, Vasilevskis, Metlay, Wallington, Fletcher, Auerbach.

Study supervision: Donzé, Kripalani, Schnipper.

Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Donzé worked as consultant at Profility LLC, and Homeward Health Inc. Dr Zimlichman is a consultant and advisory board member at Earlysense LTD, which makes monitors for patients on hospital wards, a consultant and advisory board member at Hello Doctor, which develops patient record applications, a consultant to Profility, which develops big data applications for improvement of the management of elderly populations, a founder and advisory board member at ValueScope Health, which creates financial management systems for healthcare organizations, and a founder and advisory board member at Ethos, which develops mobile health applications for patient engagement. Dr Wallington has been a consultant for Novartis to provide advice on screening for cardiovascular disease. Dr Schnipper has received grant funding from Sanofi Aventis for an investigator-initiated study to design and evaluate an intensive discharge and follow-up intervention in patients with diabetes. The funder had no role in the design of the study. No other conflicts are reported.

Funding/Support: Dr Donzé was supported by the Swiss National Science Foundation and the Swiss Foundation for Medical-Biological Scholarships (grant No. PASMP3-142734). Dr Vasilevskis was supported by the National Institute on Aging of the National Institutes of Health (award No. K23AG040157) and the Veterans Affairs Clinical Research Center of Excellence, the Geriatric Research, Education, and Clinical Center (GRECC).

Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Previous Presentation: A portion of the study findings were presented at the Annual Meeting of the Society of General Internal Medicine; Toronto, Ontario, Canada; May 24, 2015.

Additional Contributions: We thank Yves Eggli, MD, for having screened the database for potentially avoidable readmissions using the SQLape algorithm. Dr Yves Eggli is associate physician at the Institute of Health Economics and Management of the University of Lausanne, Switzerland. He did not have a role in the design and conduct of this study, the analysis or interpretation of the data, or the preparation of this manuscript, and did not receive any compensation for his help.