Figure. Study flow diagram. SQLape indicates Striving for Quality Level and Analyzing of Patient Expenses.
Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med. Published online March 25, 2013. doi:10.1001/jamainternmed.2013.3023.
eTable 1. Multivariable model with all predictors
eTable 2. Final multivariable model
eTable 3. Scoring system details in the derivation (A) and validation (B) sets
eTable 4. Observed and expected likelihood of readmission by decile in the derivation (A) andvalidation (B) sets
eTable 5. Observed vs. predicted 30-day potentially avoidable readmissions in the entire cohort This
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially Avoidable 30-Day Hospital Readmissions in Medical Patients: Derivation and Validation of a Prediction Model. JAMA Intern Med. 2013;173(8):632–638. doi:10.1001/jamainternmed.2013.3023
Author Affiliations: Division of General Medicine and Primary Care (Drs Donzé and Schnipper and Ms Williams) and Hospitalist Service (Dr Schnipper), Brigham and Women's Hospital, Boston, Massachusetts; Harvard Medical School, Boston (Drs Donzé and Schnipper); and Division of General Internal Medicine, Bern University Hospital, Bern, Switzerland (Dr Aujesky).
Importance Because effective interventions to reduce hospital readmissions are often expensive to implement, a score to predict potentially avoidable readmissions may help target the patients most likely to benefit.
Objective To derive and internally validate a prediction model for potentially avoidable 30-day hospital readmissions in medical patients using administrative and clinical data readily available prior to discharge.
Design Retrospective cohort study.
Setting Academic medical center in Boston, Massachusetts.
Participants All patient discharges from any medical services between July 1, 2009, and June 30, 2010.
Main Outcome Measures Potentially avoidable 30-day readmissions to 3 hospitals of the Partners HealthCare network were identified using a validated computerized algorithm based on administrative data (SQLape). A simple score was developed using multivariable logistic regression, with two-thirds of the sample randomly selected as the derivation cohort and one-third as the validation cohort.
Results Among 10 731 eligible discharges, 2398 discharges (22.3%) were followed by a 30-day readmission, of which 879 (8.5% of all discharges) were identified as potentially avoidable. The prediction score identified 7 independent factors, referred to as the HOSPITAL score: h emoglobin at discharge, discharge from an o ncology service, s odium level at discharge, p rocedure during the index admission, i ndex t ype of admission, number of a dmissions during the last 12 months, and l ength of stay. In the validation set, 26.7% of the patients were classified as high risk, with an estimated potentially avoidable readmission risk of 18.0% (observed, 18.2%). The HOSPITAL score had fair discriminatory power (C statistic, 0.71) and had good calibration.
Conclusions and Relevance This simple prediction model identifies before discharge the risk of potentially avoidable 30-day readmission in medical patients. This score has potential to easily identify patients who may need more intensive transitional care interventions.
Hospital readmissions are common and costly. Recent studies estimated a 30-day readmission rate in the United States of 18%; among Medicare beneficiaries, readmissions are estimated to cost $17 billion annually.1Quiz Ref IDBecause at least some hospital readmissions may be avoidable, readmission rates are now used for benchmarking across hospitals, with financial penalties for hospitals with high risk-adjusted rates.2
Several interventions have been shown3-6 to be effective in reducing the rate of readmission. To improve efficiency, the highest intensity interventions should be targeted to patients who are most likely to benefit. Quiz Ref IDFew models exist to predict 30-day readmission risk in general medical patients.7-10 These models do not distinguish between avoidable and unavoidable readmissions, often have poor discriminatory power or calibration, and/or use complex scores not calculable before hospital discharge.
To help clinicians target transitional care interventions most efficiently, we derived and internally validated a prediction model for potentially avoidable 30-day hospital readmissions in medical patients using readily available administrative and clinical data.
This retrospective cohort study included consecutive adult patient discharges from all medical services of the Brigham and Women's Hospital with a discharge date between July 1, 2009, and June 30, 2010. Brigham and Women's Hospital is a 750-bed academic medical center in Boston, Massachusetts, with 44 000 inpatient admissions per year. Medical services include general medicine, cardiology, oncology, bone marrow transplant, endocrinology, gastroenterology, hematology, infectious diseases, rheumatology, and nephrology. To avoid inclusion of observation stays, only hospitalizations with a length of stay of more than 24 hours were included. We also excluded hospitalizations when the patient died before discharge, was transferred to another acute health care facility, or left against medical advice. The hospitalizations were randomized into a derivation set (two-thirds of admissions) and a validation set (one-third of admissions). The protocol was approved by the institutional review board of Brigham and Women's Hospital/Partners HealthCare.
Among the included patients, there were 3 possible outcomes: admissions not followed by any 30-day readmission, admissions followed by a 30-day potentially avoidable readmission, and admissions followed by a 30-day unavoidable readmission. Because we were interested in the identification of predictors specific to avoidable readmissions, and to give a clear contrast, we chose to compare the admissions followed by a potentially avoidable readmission with those not followed by any 30-day readmission. We excluded the unavoidable readmissions because they correspond to an intermediate and heterogeneous population of patients (eg, some of these patients might have subsequently developed an avoidable readmission had they not already been readmitted). Hospital readmissions were considered to be unavoidable if any of the following characteristics were present: (1) planned readmission (eg, scheduled at the time of the index admission, planned treatment follow-up, and planned chemotherapy) or (2) unforeseen readmission for newly developed conditions not related to known diseases during the index hospitalization.
Quiz Ref IDTo exclude the unavoidable readmissions meeting these criteria, we used a validated computerized algorithm (Striving for Quality Level and Analyzing of Patient Expenses [SQLape], developed by Yves Eggli, MD) commonly used in Switzerland to benchmark and compare hospitals.11 This algorithm is based on administrative data and International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes of both the index admission and readmission to identify unavoidable readmissions as described in the previous paragraph. Conversely, the algorithm identifies as potentially avoidable readmissions related to a previously coded medical condition or resulting from a complication of treatment (eg, deep vein thrombosis). The sensitivity as well as specificity of the screening algorithm reached 96% when compared with medical record review (using the same criteria) in a random sample of admission-readmission pairs drawn from the university hospital from which the tool was originally derived.12 Using medical record review, the original SQLape investigators estimated that 23% of potentially avoidable readmissions (1.2% of all readmissions) were clearly avoidable had different action been taken in the hospital (eg, premature discharge, erroneous diagnosis, inappropriate treatment, or inadequate patient education).
Because administrative data may not always accurately identify elective from nonelective readmissions, all potentially avoidable readmissions identified with SQLape were further reviewed by 1 of 9 trained senior medical residents from Brigham and Women's Hospital to exclude patients with a planned readmission.
Because potentially avoidable readmissions to departments and hospitals other than the Department of Medicine of Brigham and Women's Hospital could occur, we looked at readmissions to any service of Brigham and Women's Hospital, the Massachusetts General Hospital, or the Faulkner Hospital, all of which are affiliated with the Partners HealthCare network. Each admission could be considered as both an index admission and a readmission when several readmissions occurred within an interval of less than 30 days.
We collected data on several types of variables from easily obtainable sources (Table 1), including demographic information, previous health care utilization, primary care provider information, and index admission characteristics from administrative data sources; procedures and chronic medical conditions from billing data; and last known laboratory values before discharge. Variables were chosen a priori and according to the medical literature.8,13,14
The presence of any difference in baseline characteristics between the groups with a 30-day potentially avoidable readmission and those not readmitted was tested by univariable logistic regression. Using the derivation set, all variables noted in Table 1 were included in an initial multivariable model. When laboratory values were missing (<1.1% of discharges), these variables were considered as within normal limits. Because the same patient could have had several admissions and readmissions during the study period, data were clustered at the patient level using general estimating equations. The less strongly linked variables were removed from the model one at a time by backward elimination until all predictors were significant at P < .05.
The result of the multivariable regression model was then used to develop a prediction score by using a regression coefficient–based scoring method.15 Integer scores were assigned by dividing risk-factor coefficients by the smallest coefficient and rounded up to the nearest integer. After model derivation, the risk for an admission to be followed by a 30-day potentially avoidable readmission was categorized into 3 groups (low, intermediate, and high) for ease of interpretation.
The discriminatory power of the resulting score was assessed in both the derivation and validation set by calculating the cross-validated C statistic,16 which refers to the ability to differentiate between admissions followed and those not followed by a 30-day potentially avoidable readmission. How closely the predicted probabilities reflected actual risk was assessed by calibration and with the Hosmer-Lemeshow goodness-of-fit test (nonsignificant P values on this test imply good fit). Details of calibration are shown by comparing the estimated risk obtained with the score for patients in a specific category with the observed probability, ie, the actual proportion of 30-day potentially avoidable readmissions for patients in that category. Finally, because in practice unavoidable readmissions cannot be identified and excluded before risk estimation, we reevaluated the performance of the score when applied to the complete cohort, including both potentially avoidable and unavoidable readmissions. Analyses were performed with commercial software (SAS, version 9.2; SAS Institute, Inc).
During the study period, a total of 12 383 patients were discharged from the medical services of the Brigham and Women's Hospital (Figure). Of these patients, 1652 admissions (13.3%) were excluded because of death before discharge, because of transfer to another acute health care facility, or because the patient left against medical advice. Among the 10 731 remaining discharges, 2398 (22.3%) were followed by a 30-day readmission, of which 879 readmissions (8.5% of all index discharges, 36.7% of readmissions) were identified as potentially avoidable. The admissions not followed by a 30-day readmission (n = 8333) and the potentially avoidable readmissions (n = 879) were together randomly divided into a derivation set (two-thirds [n = 6141]) and a validation set (one-third [n = 3071]). Overall, 7123 unique patients accounted for all 9212 index discharges.
Baseline characteristics are reported in Table 1. The patients' mean age at inclusion was 61.3 years, and approximately half were male. At least 1 procedure was performed in 60.5% of index admissions, with a mean of 1.8 procedures per admission (Table 2 includes a list of commonly performed procedures).
The backward multivariable logistic regression analysis identified 8 covariates that were significant independent predictors (eTable 1 reports on the model with all a priori identified predictors). To simplify the score, we further excluded the covariate congestive heart failure from the model for 2 reasons: (1) not using any comorbidities would greatly simplify the calculation of the score and, in some cases, the diagnosis might be available only after discharge; (2) the covariate congestive heart failure had one of the higher P values (.03) in the multivariable logistic regression. We ended up with a 7-factor predictor score, which we refer to as the HOSPITAL score: h emoglobin at discharge, discharge from an o ncology service, s odium level at discharge, p rocedure during the index admission (any ICD-9-CM –coded procedure), i ndex t ype of admission (nonelective vs elective), number of a dmissions during the past 12 months, and l ength of stay (Table 3; eTable 2 lists the β coefficients and odds ratios for each predictor). Using the score, the risk of potentially avoidable readmission was stratified into 3 categories: low, intermediate, and high. Low-risk patients having 0 to 4 points (49.3% of patients) had a 5.2% estimated risk of potentially avoidable readmission and anobserved proportion of 5.4% in the derivation set; high-risk patients having 7 or more points (24.4% of patients) had an 18.3% estimated probability of potentially avoidable readmission and an observed probability of 18.7% (Table 4; eTable 3 lists the risk level for each total point score). The results were very similar in the validation set. The Hosmer-Lemeshow goodness-of-fit statistics were P = .28 and P = .15 in the derivation and validation sets, respectively, indicating good calibration (eTable 4 reports observed and expected results by decile of risk). The discriminatory power of the score was fair, with a cross-validated C statistic of 0.69 in the derivation set and 0.71 in the validation set. When the HOSPITAL score was applied to the complete cohort before exclusion of unavoidable readmissions (n = 10 731), discrimination remained fair, with a cross-validated C statistic of 0.67 and very good calibration (eTable 5).
In this study of 9212 eligible adult medical discharges from a teaching hospital, we developed and internally validated the HOSPITAL score that predicted the risk of 30-day potentially avoidable readmission in medical patients with fair discriminatory power and good calibration. This easy-to-use model enables physicians to prospectively identify approximately 27% of the patients as high-risk of having a potentially avoidable readmission and would allow targeting intensive transitional care interventions to patients who might benefit the most.
The 22.3% overall readmission rate and 8.5% avoidable rate in our cohort are consistent with the findings of previous studies.1,2Quiz Ref IDAs expected and shown in prior studies of all-cause or unplanned readmissions,8,9,17 the number of prior hospitalizations and the length of stay of the index admission were important predictors of potentially avoidable readmission. A plausible explanation is that previous hospitalization may account for the total burden of illness, illness severity, functional status, and/or social environment. Length of stay also represents the severity of illness.
Even though all planned readmissions for chemotherapy were excluded from the outcome, being discharged from an oncology service was still associated with a high risk of potentially avoidable readmission. Patients discharged from an oncology division may be more fragile than most other medical service patients, and many readmissions may result from infectious complications, fluid and electrolyte abnormalities, a decrease in functional status, or end-of-life issues that might be avoidable. The Charlson Comorbidity Index or Elixhauser Comorbidity Index used in many prediction models is driven in large part by a diagnosis of cancer. Allaudeen et al10 and Silverstein et al14 found that the specific Elixhauser Comorbidity Index diagnosis of cancer was significantly associated with a 30-day readmission. It remains to be seen whether this finding is unique to large academic medical centers and/or institutions affiliated with a major cancer center, where the proportion of patients with cancer and the severity of disease are high.
Nonelective admissions seem to be followed by a higher risk for potentially avoidable readmission when compared with elective admissions. This may reflect the stability of the patient and is consistent with the findings of van Walraven et al.9
To our knowledge, no previous studies included the number of procedures performed during the index stay in their models. Depending on the procedure, this predictor may reflect disease severity as much as or more than avoidable complications of the procedure.
Finally, sodium and hemoglobin levels are markers of general prognosis (eg, in patients with heart failure, pulmonary embolism, and pneumonia).18-20 To our knowledge, no other studies have included the sodium level as a predictor of readmission. Anemia was shown to be associated with hospital readmission within 90 days of discharge in one study.21
Surprisingly, none of the most frequent comorbidities in these patients (based on index admission ICD-9-CM codes) was retained in the final model. Only congestive heart failure was significantly associated with a higher risk of 30-day potentially avoidable readmission, but this variable was not highly significant compared with most other covariates. This finding contrasts with some previous studies.7,10 It is possible that laboratory values, the number of previous admissions, the index length of stay, the need for inpatient procedures, nonelective admission, and oncology service are better predictors for the severity of illness and instability of the patient and reflect better the risk of readmission than a comorbidity measure does. The hypothesis that comorbidities or causes of admission do not matter as much as illness severity or clinical instability is attractive and has intuitive appeal. Another explanation could be the use of potentially avoidable readmissions as the outcome as opposed to the outcomes of previous studies.
Of the 26 unique prediction models identified in a recent systematic review,22 only 14 used the standard 30-day outcome, which is more relevant for targeting interventions and for public reporting. Of these models, 6 are for patients with specific diseases, such as heart failure or pneumonia, which makes them difficult to apply, especially in patients with multiple conditions. Among the remaining 8 models, 6 include both medical and surgical patients, who may have different predictors of potentially avoidable readmissions. Also, of these 8 models, only 3 have a C statistic value greater than 0.65,7,9,11 and they might be good for risk adjustment but are less useful for prospectively targeting interventions because they do not focus on potentially avoidable readmissions and/or include predictors not available before discharge (eg, billing-based comorbidity scores). These limitations make these models less useful as a tool to target transitional care.
In comparison, the HOSPITAL score has fair discriminatory power and good calibration in identifying the risk of potentially avoidable readmission. It was developed using readmissions that might be prevented and are therefore potentially actionable. It can be used for all medical patients regardless of their main cause of admission. Finally, and most important, the HOSPITAL score is able to indicate the risk before a patient is discharged to allow targeting a timely transitional care intervention. We believe, of course, that all patients should receive high-quality transitional care that meets certain standards. We are not recommending that low- and average-risk patients be deprived of effective transitional interventions. However, certain interventions that have been shown to be successful are resource intensive.3,4,23 One way to make best use of those limited resources is to reserve them for patients most likely to benefit.
These data must be interpreted in the context of this study's limitations. First, we were unable to identify readmissions that occurred outside our hospital network. However, previous studies24,25 using 30-day follow-up telephone calls and access to all available medical records have shown that more than 80% of Brigham and Women's Hospital medical patients are readmitted to 1 of the 3 hospitals in the Partners HealthCare network evaluated in this study. Moreover, there is no obvious reason to believe that the potentially avoidable readmissions outside the network might have such different predictors to change the current model.
Second, although we excluded inpatient deaths, postdischarge deaths were not identified or included in the outcome. Death within 30 days of discharge is a rare event (0.7% in one study),9 so a model predictive of potentially avoidable death or readmission would be unlikely to differ from a model predicting only potentially avoidable readmission.
Third, although SQLape has good performance characteristics in Switzerland, it has not been validated in the United States. We partially compensated for this by manually excluding planned readmissions (eg, because of coding errors). Other limitations in coding (eg, failure to document a complication of treatment) and in the algorithm itself mean that SQLape cannot perfectly distinguish avoidable from unavoidable readmissions, but it did allow us to create a cohort enriched with avoidable readmissions and develop a model better able to identify predictors of avoidable readmission than would otherwise be the case.
Quiz Ref IDFourth, it is likely that our model excluded important predictors of potentially avoidable readmission, such as functional status, health literacy, degree of social support, and previous medication adherence.7 However, we chose not to include these predictors, which are infrequently measured and often difficult to obtain, because the study's goal was to derive a model that could be easily and widely used.
Fifth, hemoglobin and sodium values are available only near the time of discharge and might limit the time available to effectively deploy an intensive discharge intervention. However, we calculated that 60% of patients defined as high risk could be identified 2 or more days before discharge, based on information available at that time, allowing sufficient time for intervention deployment. Similarly, procedures were technically defined by postdischarge ICD-9-CM codes (Table 2), but these could be recognized easily by clinicians or administrators during the hospitalization as any procedure requiring patient consent.
Sixth, this study performed only an internal validation of the model. An external validation of the HOSPITAL score is likely required, particularly in smaller hospitals, before widespread implementation.
Last, predicting potentially avoidable readmissions is only a proxy for identifying who might benefit from specific interventions. Intervention studies targeting this patient population need to be done to definitively prove its usefulness.
In conclusion, we propose a prediction model, HOSPITAL, that provides a practical tool to assess 30-day potentially avoidable readmission risk in medical patients. The use of this simple score before discharge may help target transitional care for patients who might benefit the most and consequently reduce the rate of avoidable readmission.
Correspondence: Jacques Donzé, MD, MSc, Division of General Medicine and Primary Care, Brigham and Women's Hospital, 1620 Tremont St, OBC-3-2U, Boston, MA 02120 (firstname.lastname@example.org).
Accepted for Publication: October 26, 2012.
Published Online: March 25, 2013. doi:10.1001/jamainternmed.2013.3023
Author Contributions:Study concept and design: Donzé and Schnipper. Acquisition of data: Donzé and Williams. Analysis and interpretation of data: All authors. Drafting of the manuscript: Donzé. Critical revision of the manuscript for important intellectual content: All authors. Statistical analysis: Donzé. Obtained funding: Donzé. Administrative, technical, and material support: Williams. Study supervision: Aujesky and Schnipper.
Conflict of Interest Disclosures: Dr Schnipper is a consultant to QuantiaMD, for whom he has helped create online educational materials for providers and patients regarding patient safety, including medication safety during transitions in care. The findings of this study are not a part of those materials. Dr Schnipper has received grant funding from sanofi-aventis for an investigator-initiated study to design and evaluate an intensive discharge and follow-up intervention for patients with diabetes mellitus.
Funding/Support: Dr Donzé was supported by grants PBLAP3-131814 and PBLAP3-136815 from the Swiss Science National Foundation and by the SICPA Foundation.
Role of the Sponsors: The Swiss Science National Foundation and the SICPA Foundation had no role in the design and conduct of this study, the analysis or interpretation of the data, or the preparation of the manuscript.
Previous Presentation: A portion of the study's findings were presented at the Annual Meeting of the Society of General Internal Medicine; May 12, 2012; Orlando, Florida.
Additional Contributions: Yves Eggli, MD, from the Institute of Health Economics and Management of the University of Lausanne, Switzerland, screened the database for potentially avoidable readmissions using the SQLape algorithm.