[Skip to Content]
[Skip to Content Landing]
Figure 1.
Study Flow Diagram
Study Flow Diagram

During the study period, a total of 121 136 adults were discharged alive from the medical department of the 9 participating centers, and 117 065 (96.6%) discharges were included in the analysis.

Figure 2.
Receiving Operating Characteristic Curve of the HOSPITAL Score
Receiving Operating Characteristic Curve of the HOSPITAL Score

The HOSPITAL score had a C statistic of 0.72 (95% CI, 0.72-0.72), showing good discrimination.

Table 1.  
HOSPITAL Scorea for 30-Day Potentially Avoidable Readmissions
HOSPITAL Scorea for 30-Day Potentially Avoidable Readmissions
Table 2.  
Baseline Characteristics of the International Study Population Stratified According to 30-Day Potentially Avoidable Readmission
Baseline Characteristics of the International Study Population Stratified According to 30-Day Potentially Avoidable Readmission
Table 3.  
Observed vs Predicted 30-Day Potentially Avoidable Readmissions
Observed vs Predicted 30-Day Potentially Avoidable Readmissions
1.
Jencks  SF, Williams  MV, Coleman  EA.  Rehospitalizations among patients in the Medicare fee-for-service program.  N Engl J Med. 2009;360(14):1418-1428.PubMedGoogle ScholarCrossref
2.
van Walraven  C, Bennett  C, Jennings  A, Austin  PC, Forster  AJ.  Proportion of hospital readmissions deemed avoidable: a systematic review.  CMAJ. 2011;183(7):E391-E402.PubMedGoogle ScholarCrossref
3.
Auerbach  AD, Kripalani  S, Vasilevskis  EE,  et al.  Preventability and causes of readmissions in a national cohort of general medicine patients  [published online March 7, 2016].  JAMA Intern Med. doi:10.1001/jamainternmed.2015.7863.Google Scholar
4.
Leppin  AL, Gionfriddo  MR, Kessler  M,  et al.  Preventing 30-day hospital readmissions: a systematic review and meta-analysis of randomized trials.  JAMA Intern Med. 2014;174(7):1095-1107.PubMedGoogle ScholarCrossref
5.
Allaudeen  N, Schnipper  JL, Orav  EJ, Wachter  RM, Vidyarthi  AR.  Inability of providers to predict unplanned readmissions.  J Gen Intern Med. 2011;26(7):771-776.PubMedGoogle ScholarCrossref
6.
Kansagara  D, Englander  H, Salanitro  A,  et al.  Risk prediction models for hospital readmission: a systematic review.  JAMA. 2011;306(15):1688-1698.PubMedGoogle ScholarCrossref
7.
World Health Organization.  International Classification of Diseases, Ninth Revision (ICD-9). Geneva, Switzerland: World Health Organization; 1977.
8.
Donzé  J, Aujesky  D, Williams  D, Schnipper  JL.  Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model.  JAMA Intern Med. 2013;173(8):632-638.PubMedGoogle ScholarCrossref
9.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.  BMJ. 2015;350:g7594.PubMedGoogle ScholarCrossref
10.
Halfon  P, Eggli  Y, Prêtre-Rohrbach  I, Meylan  D, Marazzi  A, Burnand  B.  Validation of the potentially avoidable hospital readmission rate as a routine indicator of the quality of hospital care.  Med Care. 2006;44(11):972-981.PubMedGoogle ScholarCrossref
11.
Halfon  P, Eggli  Y, van Melle  G, Chevalier  J, Wasserfallen  JB, Burnand  B.  Measuring potentially avoidable hospital readmissions.  J Clin Epidemiol. 2002;55(6):573-587.PubMedGoogle ScholarCrossref
12.
SQlape. Potentially avoidable readmissions. http://www.sqlape.com/READMISSIONS.htm. 2013. Accessed July 18, 2013.
13.
World Health Organization.  International Statistical Classification of Diseases, Tenth Revision (ICD-10). Geneva, Switzerland: World Health Organization; 1992.
14.
Khudyakov  P, Gorfine  M, Zucker  D, Spiegelman  D.  The impact of covariate measurement error on risk prediction.  Stat Med. 2015;34(15):2353-2367.PubMedGoogle ScholarCrossref
15.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
16.
Hanley  JA, McNeil  BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve.  Radiology. 1982;143(1):29-36.PubMedGoogle ScholarCrossref
17.
Pencina  MJ, D’Agostino  RB  Sr.  Evaluating Discrimination of Risk Prediction Models: The C Statistic.  JAMA. 2015;314(10):1063-1064.PubMedGoogle ScholarCrossref
18.
Zeger  SL, Liang  KY.  Longitudinal data analysis for discrete and continuous outcomes.  Biometrics. 1986;42(1):121-130.PubMedGoogle ScholarCrossref
19.
van Walraven  C, Dhalla  IA, Bell  C,  et al.  Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community.  CMAJ. 2010;182(6):551-557.PubMedGoogle ScholarCrossref
20.
Tan  SY, Low  LL, Yang  Y, Lee  KH.  Applicability of a previously validated readmission predictive index in medical patients in Singapore: a retrospective study.  BMC Health Serv Res. 2013;13:366.PubMedGoogle ScholarCrossref
21.
Bouwmeester  W, Zuithoff  NP, Mallett  S,  et al.  Reporting and methods in clinical prediction research: a systematic review.  PLoS Med. 2012;9(5):1-12.PubMedGoogle ScholarCrossref
22.
Lavenberg  JG, Leas  B, Umscheid  CA, Williams  K, Goldmann  DR, Kripalani  S.  Assessing preventability in the quest to reduce hospital readmissions.  J Hosp Med. 2014;9(9):598-603.PubMedGoogle ScholarCrossref
23.
Cooksley  T, Nanayakkara  PW, Nickel  CH,  et al; safer@home consortium.  Readmissions of medical patients: an external validation of two existing prediction scores.  QJM. doi:10.1093/qjmed/hcv130.PubMedGoogle Scholar
24.
Barnes  GD, Gu  X, Haymart  B,  et al.  The predictive ability of the CHADS2 and CHA2DS2-VASc scores for bleeding risk in atrial fibrillation: the MAQI(2) experience.  Thromb Res. 2014;134(2):294-299.PubMedGoogle ScholarCrossref
25.
Lip  GY, Frison  L, Halperin  JL, Lane  DA.  Comparative validation of a novel risk score for predicting bleeding risk in anticoagulated patients with atrial fibrillation: the HAS-BLED (Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile INR, Elderly, Drugs/Alcohol Concomitantly) score.  J Am Coll Cardiol. 2011;57(2):173-180.PubMedGoogle ScholarCrossref
26.
Roldán  V, Marín  F, Fernández  H,  et al.  Predictive value of the HAS-BLED and ATRIA bleeding scores for the risk of serious bleeding in a “real-world” population with atrial fibrillation receiving anticoagulant therapy.  Chest. 2013;143(1):179-184.PubMedGoogle ScholarCrossref
27.
Apostolakis  S, Lane  DA, Buller  H, Lip  GY.  Comparison of the CHADS2, CHA2DS2-VASc and HAS-BLED scores for the prediction of clinically relevant bleeding in anticoagulated patients with atrial fibrillation: the AMADEUS trial.  Thromb Haemost. 2013;110(5):1074-1079.PubMedGoogle ScholarCrossref
28.
Dhalla  IA, O’Brien  T, Morra  D,  et al.  Effect of a postdischarge virtual ward on readmission or death for high-risk patients: a randomized clinical trial.  JAMA. 2014;312(13):1305-1312.PubMedGoogle ScholarCrossref
Original Investigation
April 2016

International Validity of the HOSPITAL Score to Predict 30-Day Potentially Avoidable Hospital Readmissions

Author Affiliations
  • 1Division of General Internal Medicine, Bern University Hospital, Bern, Switzerland
  • 2Division of General Medicine, Brigham and Women’s Hospital, Boston, Massachusetts
  • 3Harvard Medical School, Boston, Massachusetts
  • 4Center for Health Services Research, University of Kentucky, Lexington
  • 5Value Institute, Christiana Care Health System, Wilmington, Delaware
  • 6Sheba Medical Centre, Tel Hashomer, Israel
  • 7Section of Hospital Medicine, Vanderbilt University Medical Center, Nashville, Tennessee
  • 8Center for Clinical Quality and Implementation Research, Vanderbilt University Medical Center, Nashville, Tennessee
  • 9Veterans Affairs Tennessee Valley – Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee
  • 10Division of General Internal Medicine, Massachusetts General Hospital, Boston
  • 11William Osler Health System, Ontario, Canada
  • 12Department of Medicine, Harborview Medical Center, University of Washington, Seattle
  • 13Division of Hospital Medicine, University of California-San Francisco, San Francisco
 

Copyright 2016 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.

JAMA Intern Med. 2016;176(4):496-502. doi:10.1001/jamainternmed.2015.8462
Abstract

Importance  Identification of patients at a high risk of potentially avoidable readmission allows hospitals to efficiently direct additional care transitions services to the patients most likely to benefit.

Objective  To externally validate the HOSPITAL score in an international multicenter study to assess its generalizability.

Design, Setting, and Participants  International retrospective cohort study of 117 065 adult patients consecutively discharged alive from the medical department of 9 large hospitals across 4 different countries between January 2011 and December 2011. Patients transferred to another acute care facility were excluded.

Exposures  The HOSPITAL score includes the following predictors at discharge: hemoglobin, discharge from an oncology service, sodium level, procedure during the index admission, index type of admission (urgent), number of admissions during the last 12 months, and length of stay.

Main Outcomes and Measures  30-day potentially avoidable readmission to the index hospital using the SQLape algorithm.

Results  Overall, 117 065 adults consecutively discharged alive from a medical department between January 2011 and December 2011 were studied. Of all medical discharges, 16 992 of 117 065 (14.5%) were followed by a 30-day readmission, and 11 307 (9.7%) were followed by a 30-day potentially avoidable readmission. The discriminatory power of the HOSPITAL score to predict potentially avoidable readmission was good, with a C statistic of 0.72 (95% CI, 0.72-0.72). As in the derivation study, patients were classified into 3 risk categories: low (n = 73 031 [62.4%]), intermediate (n = 27 612 [23.6%]), and high risk (n = 16 422 [14.0%]). The estimated proportions of potentially avoidable readmission for each risk category matched the observed proportion, resulting in an excellent calibration (Pearson χ2 test P = .89).

Conclusions and Relevance  The HOSPITAL score identified patients at high risk of 30-day potentially avoidable readmission with moderately high discrimination and excellent calibration when applied to a large international multicenter cohort of medical patients. This score has the potential to easily identify patients in need of more intensive transitional care interventions to prevent avoidable hospital readmissions.

Introduction

Hospital readmissions are frequent, harmful and costly—18% of Medicare patients can expect to be readmitted within 30 days in the United States at a cost of more than $17 billion.1 Recent changes in health care policy aimed at reducing readmission have substantially increased attention to this major health care issue.

Although some readmissions are unavoidable and the proportion of readmissions that are truly avoidable remains controversial, at least some appear to be preventable, therefore leaving room for improvement by hospitals and health care systems.2,3 A recent systematic review and meta-analysis4 showed a relative readmission risk reduction of 0.82 from transitions interventions (P < .01); however, the most effective interventions were of high complexity. To most efficiently reduce hospital readmissions, hospitals need to target complex and intensive discharge interventions at those patients at high risk of potentially avoidable readmission who are more likely to benefit. However, prior research indicates that clinical health care providers are not able to accurately identify which patients are at high risk for readmission.5

Prediction models may provide a more efficient and accurate means to identify those patients at highest risk. Unfortunately, most existing predictions models have yielded only fair discriminative ability, do not focus on readmissions that are avoidable, are not externally validated, or are not easy to use in practice.6 In response to these shortcomings, we previously derived and internally validated the HOSPITAL score consisting of 7 readily available clinical predictors (Table 1 lists components forming the acronym). This simple prediction model accurately identified patients at high risk of potentially avoidable readmissions among more than 10 000 medical inpatients8 and is innovative in several ways: it focuses on 30-day readmissions that are potentially avoidable, shows good discrimination power to accurately differentiate low-risk from high-risk patients, includes only readily available predictors, and can be calculated before discharge.

Before wider application of the HOSPITAL score can be recommended, it is important to understand how the score performs in populations outside those from which the score was developed. To assess its generalizability, we aimed to externally validate the HOSPITAL score in a large international multicenter cohort of medical patients.

Box Section Ref ID

Key Points

  • Question: What is the performance of the HOSPITAL score to predict readmission in a large multicenter multinational external validation study?

  • Findings: In this retrospective cohort study that included 117 065 adults from 9 large hospitals across 4 different countries, the HOSPITAL score identified patients at high risk of 30-day potentially avoidable readmission with high accuracy, moderately high discrimination, and excellent calibration.

  • Meaning: This score has the potential to easily identify patients in need of more intensive transitional care interventions to prevent avoidable hospital readmissions.

Methods
Study Design and Participants

This multicenter multinational retrospective cohort study included all consecutive adult patients discharged alive between January 1, 2011, and December 31, 2011, from the medical services of 9 hospitals in 4 different countries: the United States, Canada, Israel, and Switzerland (ICARE: International Cohort of Avoidable REadmissions). To minimize inclusion of observation stays, only hospitalizations with a length of stay of more than 1 day were included. We then excluded patients who were transferred to another acute health care facility (hospital or psychiatric hospital) or who left against medical advice.

The participating centers were: (1) University of California San Francisco Medical Center, San Francisco, California; (2) Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania; (3) Harborview Hospital Medicine, University of Washington, Seattle, Washington; (4) Vanderbilt University Medical Center, Nashville, Tennessee; (5) Northwestern Memorial Hospital, Chicago, Illinois; (6) Christiana Hospital, Wilmington, Delaware; (7) Sheba Medical Center, Tel Hashomer, Israel; (8) William Osler Health System, Ontario, Canada; (9) Bern University Hospital, Bern, Switzerland. Of these, 7 are university hospitals and 2 are community hospitals (Christiana Hospital and William Osler Health System). All are not-for-profit hospitals.

The trial protocol was approved by the institutional review board of each site and also of the managing site at Brigham and Women’s Hospital/Partners Healthcare, Boston, Massachusetts.

The study follows the criteria from the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) Initiative (checklist available upon request).9

Study Outcome

First, all 30-day readmissions to any division of the same medical center were identified. The primary outcome of 30-day potentially avoidable readmission was then identified through the same process as in the derivation study, using the SQLape algorithm.8 This algorithm, based on administrative data and diagnosis codes, was initially developed in Switzerland and has been used there for now for more than 5 years for hospital benchmarking purposes.1012 Readmissions which are foreseeable are classified as unavoidable, including readmissions for transplantation, labor and delivery, chemotherapy or radiotherapy, follow-up or rehabilitation treatment, specific surgical procedures, or some specific diseases deemed difficult to cure (eg, multiple sclerosis and idiopathic thrombocytopenia). Then, the SQLape algorithm classifies readmissions as unavoidable if they involve a new organ system unknown to be affected during the preceding hospitalization. Conversely, complications of treatment (eg, deep vein thrombosis, catheter associated urinary tract infection, drug-induced disorders) were considered potentially avoidable.

The unit of analysis was the hospital discharge, and each admission could be considered as both an index hospitalization and a rehospitalization when several hospital stays occurred within an interval of less than 30 days.

Predictor Variables

Information on the 7 predictor variables included in the HOSPITAL score was collected at each site using the same definition as the derivation study (Table 1).8 HOSPITAL is the acronym for the 7 variables included in the score: last available Hemoglobin before discharge (positive if <12 g/dl [to convert to g/L, multiply by 10]), discharge from an Oncology service, last available Sodium level before discharge (positive if <135 mEq/L [to convert to mmol/L, multiply by 1]), any Procedure performed during the hospitalization (any International Classification of Disease [ICD-97 or ICD-1013] coded procedure), Index admission Type (emergent or urgent as opposed to elective), number of Admissions in the previous 12 months, and Length of stay (positive if ≥5 days). International Classification of Disease procedures in medical patients included, for example, endoscopy, hemodialysis, heart cardiac catheter, paracentesis or thoracocenthesis, transfusion, computed tomography scan or magnetic resonance imaging, or percutaneous transluminal coronary angioplasty. The scoring system ranges from 0 to a maximum of 13 points, with a risk of 30-day potentially avoidable readmission that increases with the number of points.

Owing to the absence of patients with 0 previous admissions in the last 12 months at 2 sites, patients with only 1 admission in the previous year were attributed 0 points instead of 1 as in the original score. Overall, 6907 of 117 065 of hemoglobin values (5.9%) and 3980 of 117 065 of sodium values (3.4%) were missing. The values were imputed as normal when missing (ie, no points attributed for the missing variable). Other data collected included demographic information in order to describe the population.

Statistical Analysis

Patients’ baseline characteristics were presented as proportions, means with standard deviation (SD), and as medians with interquartile ranges (IQR) as appropriate.

First, the HOSPITAL score was calculated for each hospital discharge. Then the risk for an admission to be followed by a 30-day potentially avoidable readmission was categorized into 3 groups according to the total number of points: low risk (0 to 4 points), intermediate risk (5 to 6 points), and high risk (≥7 points). These categories were created in the derivation study for ease of interpretation, roughly corresponding to 5%, 10%, and 20% risk of potentially avoidable readmission.

The score was evaluated according to its overall performance, its discriminatory power, and its calibration. We used the Brier score to quantify how close predictions are to the actual outcome (overall performance).14,15 A useful prediction rule ought to have a Brier score smaller than 0.25 (the lower the better). The discriminatory power reflects the ability of the score to discriminate the cases from the noncases based on both sensitivity and specificity (C statistic).16,17 The calibration shows the ability of the model to generate probabilities that match the observed rates.15 We used the Pearson χ2 test to summarize this measure and presented both observed and predicted rates for each risk group of the score (P ≥ 0.05 implies a good fit, with higher numbers signifying better fit).

Discharges followed by an unavoidable 30-day readmission (according to SQLape classification) were grouped with the discharges not followed by any 30-day readmission (group control) for the main analysis. In a sensitivity analysis, we also evaluated the performance of the HOSPITAL score when these unavoidable readmissions were completely excluded from the analysis.

All tests were conducted as 2-sided at a .05 level of significance. In the logistic regression, we used a fixed effect for hospital to account for possible differences in outcomes across the 9 hospitals. We also used a robust sandwich variance estimator to account for repeated admissions from the same patient within hospital.18 Analyses were performed with SAS Software, version 9.3 (SAS Institute Inc).

Results

During the study period, a total of 121 136 adults were discharged alive from the medical department of the 9 participating centers, and 117 065 (96.6%) discharges were included in the analysis (Figure 1). Within 30 days after discharge, 15.0% (n = 17 516) of the medical patients had a readmission, and 9.7% (n = 11 307) had a potentially avoidable readmission. The median (IQR) time to readmission was 12 (5-19) days. Baseline characteristics of patients are summarized in Table 2. Patients had a mean (SD) age of 60.8 (18.2) years and had a median (IQR) length of stay of 4 (3-7) days. Overall, patients with a potentially avoidable readmission more often had an urgent or emergent index admission, were more frequently discharged from an oncology service, had a length of stay greater than 5 days, had more hospitalizations in the past year, more often had a procedure, and more often had a low hemoglobin or low sodium level at discharge. All these differences were highly statistically significant (P < .001).

HOSPITAL Score

The score ranged from 0 to 13, with a median (IQR) HOSPITAL score of 4 (2-5). Using the HOSPITAL score, 73 612 patients (62.4%) were categorized as low risk for a potentially avoidable readmission; 27 612 [23.6%], intermediate risk; and 16 422 [14.0%], high risk. Risk of 30-day potentially avoidable readmission for patients with a HOSPITAL score in the low, intermediate, and high risk groups were 5.8%, 12.0%, and 22.8%, respectively.

Discriminatory Power

We found that the HOSPITAL score had a C statistic of 0.72 (95% CI, 0.72-0.72), showing good discrimination (Figure 2). The Brier score was 0.08, signifying very good overall performance. In US hospitals, the C statistic was 0.72 (95% CI, 0.71-0.72); Canada, 0.78 (95% CI, 0.76-0.80); Israel, 0.68 (95% CI, 0.67-0.69); and Switzerland, 0.68 (95% CI, 0.66-0.71).

In a sensitivity analysis, excluding patients with unavoidable readmissions rather than combining them with patients without any readmissions did not significantly change the C statistic (C statistic, 0.73; 95% CI, 0.73-0.74).

Calibration

In terms of calibration, the estimated risk of potentially avoidable readmission calculated with the HOSPITAL score matched the observed proportion of potentially avoidable readmissions in each risk group: 5.8% for the low risk group; 11.9%, intermediate; and 22.8%, high risk (Table 3). This is also reflected by an excellent Pearson χ2 test with a P value of .89. When calibration is analyzed for each individual point score, calibration remains excellent except at the extreme ends of the range (low [0 points] and high [≥11 points]), where sample size is limited (eTable 1 in the Supplement). The positive likelihood ratio for each point score showed a similar pattern (eTable 2 in the Supplement).

Discussion

In a large international multicenter external validation study, we found that the HOSPITAL score had good discriminative ability and excellent calibration for predicting the risk of 30-day potentially avoidable readmission. The HOSPITAL score is easy to use and can be calculated before discharge, which makes it a practical tool for identification of patients at high risk for preventable readmission and the timely administration of high-intensity interventions designed to improve transitions of care.

A recent systematic review6 comparing existing readmission scores pointed out the lack of validated, reliable, and generalizable score stratification models for readmission. Most scores developed to date have exhibited poor performance, lacked geographical validation, were developed using small sample size, or are complex. Among the most frequently cited score is the LACE score.19 Derived and validated in large populations in Ontario, Canada, the score showed a C statistic of 0.68 in Ontario and 0.70 in an external validation study in Singapore.20 However, it has never been validated in other countries, including the United States. Also, the LACE score requires users to calculate another score, the Charlson comorbidity index, that estimates the 1-year mortality of the patient based on billing coded comorbidities, practically available only after discharge.

In comparison, the HOSPITAL score applies 7 readily available predictors. Importantly, it can be calculated before the discharge of the patient, in time for interventions to be started in the hospital. It showed good performance in this large international multicenter study, which is, to our knowledge, the first prediction model for readmission to be so widely validated across an international cohort of patients and one of the few that follows current methodological recommendations.21 The size of the validation cohort is also one of the largest. Moreover, the HOSPITAL score was designed specifically to focus on readmissions that are potentially avoidable (ie, those that are theoretically most likely to benefit from interventions). This is of critical importance because hospitals and health systems face decisions about how to best deploy limited resources, yet there has been a dearth of tools to help with resource disbursement.22 In a recent study23 among nearly 20 000 medical patients in Denmark, the HOSPITAL score was significantly better than the LACE score for identifying patients at risk of 30-day readmission (P < .001).

With a C statistic of 0.72, the HOSPITAL score performs similarly to other commonly used prediction models. As a comparison, the well-known CHADS2 and CHADS2-VASC scores for stroke risk stratification in atrial fibrillation and the HAS-BLED score for bleeding risk on anticoagulation all have a C statistic of 0.68 or less in external validation studies.2427

This study has several limitations. First, by design, the model focuses on a medical population, and it may not be generalizable to other patient populations such as surgical patients. Second, additional predictors could have been used to improve the model’s discrimination (eg, functional status, socioeconomic status); however, we purposefully chose to use a smaller number of readily available predictors to increase clinical applicability and ease of use. Third, we should note that 30 days may not be the most appropriate time frame to judge preventable readmissions. However, we chose it because it has become the standard for research studies and for hospitals given its use by the Centers for Medicare and Medicaid Services in their Readmission Reduction Program. Fourth, the variables cannot and should be not used to guide readmission reduction interventions. The variables included are predictors and not necessarily modifiable risk factors. For example, we would not presume that transfusing a patient with a hemoglobin level less than 12 g/dL would be indicated to lower their readmission risk. Fifth, we acknowledge that transitional care interventions are most effective when implemented as early in the hospitalization as possible and that the score cannot always identify high-risk patients early because it includes length of stay and laboratory values at discharge. However, some patients can be identified as high risk at the time of admission (eg, based on admission type, service, and number of prior hospitalizations), while initially lower-risk patients may be monitored and then identified as high risk prior to discharge once they have had a procedure or their length of stay reaches 5 days. Intensive interventions could then be started at that time. Further research would be needed to determine the effects of interim low sodium and hemoglobin levels on readmission risk given their ability to normalize (or not) by the time of discharge. Last, we acknowledge that the outcome of interest in this study is “potentially avoidable readmissions” and not “avoidable readmissions.” The SQLape algorithm identifies a population with unplanned readmissions that are likely related to the index admission (eg, same organ system, possible complications of therapy). This population is enriched with patients whose readmissions are truly preventable but is neither 100% sensitive nor specific for identifying such patients, assuming such patients could ever be identified with certainty. The extent to which readmissions are preventable is controversial.

The advantage of tools such as the HOSPITAL score is that we can use it to begin to answer the more relevant question of which patients are most likely to benefit from interventions designed to prevent readmissions. Predicting potentially avoidable readmissions (as opposed to all readmissions) is an important step along this progression. This work is particularly important given recent incentives to reduce readmissions and several recent unsuccessful efforts to reduce them.28 Future intervention studies should therefore be adequately powered to identify subgroups most likely to benefit, such as those identified by the HOSPITAL score.

Conclusions

The HOSPITAL score had good to excellent ability to identify patients at high risk of 30-day potentially avoidable readmission when applied to a large international cohort of medical patients. The HOSPITAL score is the first risk prediction score to focus on potentially avoidable readmissions as opposed to all-cause readmissions, using readily available predictors at the time of discharge. This study externally validated it in a large cohort in 4 countries. This score has the potential to reliably identify patients in need of more intensive transitional care interventions to prevent hospital readmissions.

Back to top
Article Information

Corresponding Author: Jacques Donzé, MD, MSc, Universitätsklinik für Allgemeine Innere Medizin, INSELSPITAL, Bern University Hospital, 3010 Bern, Switzerland (donze.author@gmail.com).

Published Online: March 7, 2016. doi:10.1001/jamainternmed.2015.8462.

Author Contributions: Dr Donzé had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Donzé, Schnipper.

Acquisition, analysis, or interpretation of data: Donzé, Williams, Robinson, Zimlichman, Aujesky, Vasilevskis, Kripalani, Metlay, Wallington, Fletcher, Auerbach, Schnipper.

Drafting of the manuscript: Donzé.

Critical revision of the manuscript for important intellectual content: Donzé, Williams, Robinson, Zimlichman, Aujesky, Vasilevskis, Kripalani, Metlay, Wallington, Fletcher, Auerbach, Schnipper.

Statistical analysis: Donzé.

Obtained funding: Donzé, Robinson.

Administrative, technical, or material support: Donzé, Robinson, Zimlichman, Aujesky, Vasilevskis, Metlay, Wallington, Fletcher, Auerbach.

Study supervision: Donzé, Kripalani, Schnipper.

Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Donzé worked as consultant at Profility LLC, and Homeward Health Inc. Dr Zimlichman is a consultant and advisory board member at Earlysense LTD, which makes monitors for patients on hospital wards, a consultant and advisory board member at Hello Doctor, which develops patient record applications, a consultant to Profility, which develops big data applications for improvement of the management of elderly populations, a founder and advisory board member at ValueScope Health, which creates financial management systems for healthcare organizations, and a founder and advisory board member at Ethos, which develops mobile health applications for patient engagement. Dr Wallington has been a consultant for Novartis to provide advice on screening for cardiovascular disease. Dr Schnipper has received grant funding from Sanofi Aventis for an investigator-initiated study to design and evaluate an intensive discharge and follow-up intervention in patients with diabetes. The funder had no role in the design of the study. No other conflicts are reported.

Funding/Support: Dr Donzé was supported by the Swiss National Science Foundation and the Swiss Foundation for Medical-Biological Scholarships (grant No. PASMP3-142734). Dr Vasilevskis was supported by the National Institute on Aging of the National Institutes of Health (award No. K23AG040157) and the Veterans Affairs Clinical Research Center of Excellence, the Geriatric Research, Education, and Clinical Center (GRECC).

Role of the Funder/Sponsor: The funders/sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Previous Presentation: A portion of the study findings were presented at the Annual Meeting of the Society of General Internal Medicine; Toronto, Ontario, Canada; May 24, 2015.

Additional Contributions: We thank Yves Eggli, MD, for having screened the database for potentially avoidable readmissions using the SQLape algorithm. Dr Yves Eggli is associate physician at the Institute of Health Economics and Management of the University of Lausanne, Switzerland. He did not have a role in the design and conduct of this study, the analysis or interpretation of the data, or the preparation of this manuscript, and did not receive any compensation for his help.

References
1.
Jencks  SF, Williams  MV, Coleman  EA.  Rehospitalizations among patients in the Medicare fee-for-service program.  N Engl J Med. 2009;360(14):1418-1428.PubMedGoogle ScholarCrossref
2.
van Walraven  C, Bennett  C, Jennings  A, Austin  PC, Forster  AJ.  Proportion of hospital readmissions deemed avoidable: a systematic review.  CMAJ. 2011;183(7):E391-E402.PubMedGoogle ScholarCrossref
3.
Auerbach  AD, Kripalani  S, Vasilevskis  EE,  et al.  Preventability and causes of readmissions in a national cohort of general medicine patients  [published online March 7, 2016].  JAMA Intern Med. doi:10.1001/jamainternmed.2015.7863.Google Scholar
4.
Leppin  AL, Gionfriddo  MR, Kessler  M,  et al.  Preventing 30-day hospital readmissions: a systematic review and meta-analysis of randomized trials.  JAMA Intern Med. 2014;174(7):1095-1107.PubMedGoogle ScholarCrossref
5.
Allaudeen  N, Schnipper  JL, Orav  EJ, Wachter  RM, Vidyarthi  AR.  Inability of providers to predict unplanned readmissions.  J Gen Intern Med. 2011;26(7):771-776.PubMedGoogle ScholarCrossref
6.
Kansagara  D, Englander  H, Salanitro  A,  et al.  Risk prediction models for hospital readmission: a systematic review.  JAMA. 2011;306(15):1688-1698.PubMedGoogle ScholarCrossref
7.
World Health Organization.  International Classification of Diseases, Ninth Revision (ICD-9). Geneva, Switzerland: World Health Organization; 1977.
8.
Donzé  J, Aujesky  D, Williams  D, Schnipper  JL.  Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model.  JAMA Intern Med. 2013;173(8):632-638.PubMedGoogle ScholarCrossref
9.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.  BMJ. 2015;350:g7594.PubMedGoogle ScholarCrossref
10.
Halfon  P, Eggli  Y, Prêtre-Rohrbach  I, Meylan  D, Marazzi  A, Burnand  B.  Validation of the potentially avoidable hospital readmission rate as a routine indicator of the quality of hospital care.  Med Care. 2006;44(11):972-981.PubMedGoogle ScholarCrossref
11.
Halfon  P, Eggli  Y, van Melle  G, Chevalier  J, Wasserfallen  JB, Burnand  B.  Measuring potentially avoidable hospital readmissions.  J Clin Epidemiol. 2002;55(6):573-587.PubMedGoogle ScholarCrossref
12.
SQlape. Potentially avoidable readmissions. http://www.sqlape.com/READMISSIONS.htm. 2013. Accessed July 18, 2013.
13.
World Health Organization.  International Statistical Classification of Diseases, Tenth Revision (ICD-10). Geneva, Switzerland: World Health Organization; 1992.
14.
Khudyakov  P, Gorfine  M, Zucker  D, Spiegelman  D.  The impact of covariate measurement error on risk prediction.  Stat Med. 2015;34(15):2353-2367.PubMedGoogle ScholarCrossref
15.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
16.
Hanley  JA, McNeil  BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve.  Radiology. 1982;143(1):29-36.PubMedGoogle ScholarCrossref
17.
Pencina  MJ, D’Agostino  RB  Sr.  Evaluating Discrimination of Risk Prediction Models: The C Statistic.  JAMA. 2015;314(10):1063-1064.PubMedGoogle ScholarCrossref
18.
Zeger  SL, Liang  KY.  Longitudinal data analysis for discrete and continuous outcomes.  Biometrics. 1986;42(1):121-130.PubMedGoogle ScholarCrossref
19.
van Walraven  C, Dhalla  IA, Bell  C,  et al.  Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community.  CMAJ. 2010;182(6):551-557.PubMedGoogle ScholarCrossref
20.
Tan  SY, Low  LL, Yang  Y, Lee  KH.  Applicability of a previously validated readmission predictive index in medical patients in Singapore: a retrospective study.  BMC Health Serv Res. 2013;13:366.PubMedGoogle ScholarCrossref
21.
Bouwmeester  W, Zuithoff  NP, Mallett  S,  et al.  Reporting and methods in clinical prediction research: a systematic review.  PLoS Med. 2012;9(5):1-12.PubMedGoogle ScholarCrossref
22.
Lavenberg  JG, Leas  B, Umscheid  CA, Williams  K, Goldmann  DR, Kripalani  S.  Assessing preventability in the quest to reduce hospital readmissions.  J Hosp Med. 2014;9(9):598-603.PubMedGoogle ScholarCrossref
23.
Cooksley  T, Nanayakkara  PW, Nickel  CH,  et al; safer@home consortium.  Readmissions of medical patients: an external validation of two existing prediction scores.  QJM. doi:10.1093/qjmed/hcv130.PubMedGoogle Scholar
24.
Barnes  GD, Gu  X, Haymart  B,  et al.  The predictive ability of the CHADS2 and CHA2DS2-VASc scores for bleeding risk in atrial fibrillation: the MAQI(2) experience.  Thromb Res. 2014;134(2):294-299.PubMedGoogle ScholarCrossref
25.
Lip  GY, Frison  L, Halperin  JL, Lane  DA.  Comparative validation of a novel risk score for predicting bleeding risk in anticoagulated patients with atrial fibrillation: the HAS-BLED (Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile INR, Elderly, Drugs/Alcohol Concomitantly) score.  J Am Coll Cardiol. 2011;57(2):173-180.PubMedGoogle ScholarCrossref
26.
Roldán  V, Marín  F, Fernández  H,  et al.  Predictive value of the HAS-BLED and ATRIA bleeding scores for the risk of serious bleeding in a “real-world” population with atrial fibrillation receiving anticoagulant therapy.  Chest. 2013;143(1):179-184.PubMedGoogle ScholarCrossref
27.
Apostolakis  S, Lane  DA, Buller  H, Lip  GY.  Comparison of the CHADS2, CHA2DS2-VASc and HAS-BLED scores for the prediction of clinically relevant bleeding in anticoagulated patients with atrial fibrillation: the AMADEUS trial.  Thromb Haemost. 2013;110(5):1074-1079.PubMedGoogle ScholarCrossref
28.
Dhalla  IA, O’Brien  T, Morra  D,  et al.  Effect of a postdischarge virtual ward on readmission or death for high-risk patients: a randomized clinical trial.  JAMA. 2014;312(13):1305-1312.PubMedGoogle ScholarCrossref
×