The figure shows the original patient population, excluded patients, and the resulting patient population. ACS NSQIP indicates American College of Surgeons National Surgical Quality Improvement Program; UW, University of Wisconsin School of Medicine and Public Health billing data.
Points are assigned for each variable by drawing a line upward from the corresponding variable to the points line. The sum of the points plotted on the total points line corresponds with the prediction of 30-day readmission. ASA indicates American Society of Anesthesiologists; CR, colorectal; GS, general surgery; HPB, hepatopancreaticobiliary; MIS, minimally invasive surgery; N, no; and Y, yes.
aProlonged operative time per surgical subspecialty: MIS, >133 minutes; GS, >155 minutes; HPB, >356 minutes; and CR, >259 minutes.
eTable. Variables Associated With Readmission on Univariate Analysis.
Tevis SE, Weber SM, Kent KC, Kennedy GD. Nomogram to Predict Postoperative Readmission in Patients Who Undergo General Surgery. JAMA Surg. 2015;150(6):505-510. doi:10.1001/jamasurg.2014.4043
The Centers for Medicare and Medicaid Services have implemented penalties for hospitals with above-average readmission rates under the Hospital Readmissions Reductions Program. These changes will likely be extended to affect postoperative readmissions in the future.
To identify variables that place patients at risk for readmission, develop a predictive nomogram, and validate this nomogram.
Design, Setting, and Participants
Retrospective review and prospective validation of a predictive nomogram. A predictive nomogram was developed with the linear predictor method using the American College of Surgeons National Surgical Quality Improvement Program database paired with institutional billing data for patients who underwent nonemergent inpatient general surgery procedures. The nomogram was developed from August 1, 2006, through December 31, 2011, in 2799 patients and prospectively validated from November 1, 2013, through December 19, 2013, in 255 patients at a single academic institution. Area under the curve and positive and negative predictive values were calculated.
Main Outcomes and Measures
The outcome of interest was readmission within 30 days of discharge following an index hospitalization for a surgical procedure.
Bleeding disorder (odds ratio, 2.549; 95% CI, 1.464-4.440), long operative time (odds ratio, 1.601; 95% CI, 1.186-2.160), in-hospital complications (odds ratio, 16.273; 95% CI, 12.028-22.016), dependent functional status, and the need for a higher level of care at discharge (odds ratio, 1.937; 95% CI, 1.176-3.190) were independently associated with readmission. The nomogram accurately predicted readmission (C statistic = 0.756) in a prospective evaluation. The negative predictive value was 97.9% in the prospective validation, while the positive predictive value was 11.1%.
Conclusions and Relevance
Development of an online calculator using this predictive model will allow us to identify patients who are at high risk for readmission at the time of discharge. Patients with increased risk may benefit from more intensive postoperative follow-up in the outpatient setting.
Preventing hospital readmissions has become a national priority given the prevalence of readmissions and new legislation penalizing hospitals with high risk-adjusted rates of readmission. Postoperative readmissions are common and have been found to range from 4% to 25% in patients who undergo general surgery.1- 9 A review of medical and surgical publications found that preventable readmissions account for 9% to 50% of all readmissions.8,10 Although the use of hospital readmission rates as a measure of quality remains controversial,9- 11 the Center for Medicare and Medicaid Services has tied hospital reimbursement with readmissions. As of 2010, the Center for Medicare and Medicaid Services has reduced reimbursement to hospitals with higher-than-expected readmission rates.12- 14 Although these changes do not currently affect surgical patients, it is only a matter of time before postoperative readmissions face the same reimbursement penalties. As a result, identifying patients at high risk for readmission and implementing quality-improvement projects aimed at decreasing readmissions has become a significant priority.
Algorithms for calculating readmission risk are common in medical publications15; however, surgical publications have focused more on general scoring systems for postoperative morbidity and mortality. The Physiologic and Operative Severity Score for the enUmeration of Mortality and Morbidity scoring system has been found to predict the risk of morbidity and mortality in a variety of surgical patients.16- 19 Unfortunately, the scoring system cannot be used to evaluate hospital readmission separate from other morbidity. In patients who undergo vascular, thoracic, and general surgery, a model using length of stay and American Society of Anesthesiologists (ASA) classification was found to be predictive of readmission within 30 days of surgery.20 However, the authors did not independently evaluate patients who underwent general surgery.
In this study, we sought to develop and validate a nomogram to predict readmission within 30 days after hospital discharge. Our study aims were to (1) identify risk factors for postoperative readmission in patients who underwent general surgery, (2) create a predictive nomogram for postoperative readmission, and (3) prospectively validate the readmission nomogram in an independent group of patients.
Patients who underwent general surgery procedures at a single institution were identified retrospectively from the prospectively maintained American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) database. Deidentified ACS NSQIP data were paired with hospital billing data to identify readmissions that occurred within 30 days of discharge because ACS NSQIP only reports readmissions within 30 days of surgery. This study was deemed of minimal risk and was therefore declared exempt from approval by the University of Wisconsin Institutional Review Board. Patients were included if they underwent an elective general surgery operation from August 1, 2006, through December 31, 2011. Quiz Ref IDExclusion criteria included death within 30 days of surgery and urgent or emergent operations. Emergent operations were defined by the ACS NSQIP as patients whose condition was deemed emergent by the attending surgeon and/or anesthesiologist. Patients with an ASA classification of 5, preoperative sepsis, dependence on a ventilator at the time of surgery, and/or a preoperative open wound were also excluded.
Explanatory variables included the following patient characteristics: age, sex, body mass index (calculated as weight in kilograms divided by height in meters squared), ASA classification, functional status, and 14 NSQIP-defined preoperative comorbidities. Patients aged 65 years or older were compared with patients who were younger than 65 years. High-risk body mass index was defined as 30 or greater, while ASA classification was categorized as 1 or 2 vs 3 or 4. Functional status was evaluated as independent and dependent (partially and completely dependent) based on NSQIP definitions. The NSQIP-defined comorbidities included diabetes mellitus, current smoker within 1 year of surgery, chronic obstructive pulmonary disease, ascites within 30 days before surgery, renal insufficiency, dialysis dependence, disseminated cancer, corticosteroid use within 30 days of surgery, weight loss of more than 10% in 6 months, preoperative transfusion within 72 hours before surgery, bleeding disorder, hypertension, dyspnea, and congestive heart failure within 30 days of surgery. Intraoperative variables included procedure length, surgical specialty, and wound classification. Anatomic procedure types were assessed, and, because the study was not powered to analyze procedure-specific risk of readmission, procedures were grouped into surgical specialties by study authors (S.E.T. and G.D.K.). Surgical specialties included advanced, minimally invasive, colorectal, hepatopancreaticobiliary, soft-tissue oncology, and general surgery. Wound classifications included clean, clean-contaminated, contaminated, and dirty. Procedure length was evaluated as a dichotomous variable, with prolonged procedure length defined as greater than the 4th quartile in each specialty (minimally invasive, >133 minutes; colorectal, >259 minutes; general, >155 minutes; and hepatopancreaticobiliary, >356 minutes). Postoperative variables included in-hospital complications and discharge to a higher level of care. In-hospital complications included any NSQIP-defined complication that was diagnosed before discharge from the hospital. Patients were classified as being discharged to the same or higher level of care compared with the preadmission level of care. The primary outcome of interest was readmission to the hospital within 30 days of discharge from the index hospitalization. Thirty-day readmission data were extracted from the hospital billing database and linked with the institutional NSQIP database.
The study group consisted of patients who underwent an operation from August 1, 2006, through December 31, 2011. This group was used to develop a nomogram that was predictive of readmission. Then, for 6 weeks (November 1, 2013, through December 19, 2013), patients who received general surgery were identified from the daily operative schedule for a validation group, which was evaluated prospectively.
Descriptive statistics were conducted to characterize the study populations. The χ2 test was used to evaluate for associations between explanatory variables and 30-day readmission in the study group. A correlation matrix was used to evaluate all explanatory variables for collinearity. Functional status and discharge to a higher level of care were found to be highly correlated and were combined for multivariable analysis. A logistic regression analysis was then performed to identify variables that were predictive of readmission. In an effort to maximize the predictive ability of the model, all variables in the multivariable model were used to develop a prognostic nomogram using the linear predictor method.
We applied the nomogram to the study and validation populations to determine the nomogram-predicted probability of readmission. A logistic regression analysis compared predicted (nomogram) probability of readmission with actual 30-day readmission in the validation group. The area under the curve was also calculated to quantify the accuracy of the nomogram. High risk of readmission was defined based on sensitivity and specificity from the area under the curve graph. The positive and negative predictive values were calculated for this readmission rate. All statistical analyses were conducted from October 8, 2013, through November 27, 2013, in SPSS, version 20 (IBM). Significance was defined as P < .05.
From our institutional NSQIP database, we identified 3186 patients who underwent a general surgery procedure from August 1, 2006, through December 31, 2011. As demonstrated in Figure 1, after exclusions, the study population consisted of 2799 patients with a 10.2% readmission rate. The prospective validation population consisted of 255 patients, with 24 patients (9.4%) readmitted. The characteristics of both patient populations are listed in Table 1.
We evaluated preoperative, operative, and postoperative variables in association with 30-day readmission (eTable in the Supplement). Quiz Ref IDIndicators of worse overall health, including higher ASA classification, dependent functional status, and recent weight loss, were associated with readmission.Quiz Ref ID Colorectal and hepatopancreaticobiliary procedures were associated with readmission, as was longer operative time. In-hospital complications also correlated with readmission within 30 days of discharge.
Predictors of readmission are listed in Table 2. Bleeding disorder (odds ratio [OR], 2.549; 95% CI, 1.464-4.440), long operative time (OR, 1.601; 95% CI, 1.186-2.160), in-hospital complications (OR, 16.273; 95% CI, 12.028-22.016), and dependent functional status or discharge to a higher level of care (OR, 1.937; 95% CI, 1.176-3.190) all independently predicted 30-day readmission. The area under the curve was calculated with a C statistic of 0.797. The nomogram predicting readmission based on this linear regression model is shown in Figure 2. Risk of readmission can be determined by assigning points for each variable by drawing a line upward from the corresponding variable to the points line, summing the points, and identifying the prediction of 30-day readmission associated with the total points line.
The nomogram was used to calculate the predicted readmission risk for all patients. Readmission risk ranged from 3.6% to 53.9% (median risk, 10.0%) in the study population and 3.6% to 32.7% (median risk, 8.89%) in the prospective validation population. Logistic regression and area under the curve were used to compare nomogram predictions with actual readmission rates. We found the nomogram to be quite predictive of risk for readmission in the prospective validation arm of the study with a C statistic of 0.756 (OR, 1.219; 95% CI, 1.112-1.336).
Finally, in an attempt to assign a meaningful risk to patients by using this new predictive tool, we investigated the negative and positive predictive values of a number of different risk estimates generated by the nomogram. We found that, at a risk of readmission of 6.0%, the nomogram had a sensitivity and specificity for readmission prediction of 95.1% and 22.3%, respectively. Quiz Ref IDUsing 6.0% as our cutoff for high risk, we calculated the positive and negative predictive values. The positive predictive values were 12.2% (study population) and 11.1% (validation population), and the negative predictive values were 97.6% (study population) and 97.9% (validation population).
The aim of this study was to develop and validate a nomogram to predict postoperative readmission in patients who undergo general surgery. Quiz Ref IDBleeding disorder, long operating time, in-hospital complications, and dependent functional status or the need for a higher level of care at discharge independently predicted readmission within 30 days of discharge. Risk factors for readmission were used to produce a nomogram that was able to predict postoperative readmissions, as validated with prospective analysis.
Other authors have identified similar risk factors for postoperative readmissions. Prolonged length of stay and complications have been shown to be major drivers of readmission in patients who undergo general surgery.3,5,6 Preoperative comorbidity has previously been associated with surgical readmissions.3,5,7 In a study of readmissions following major gastrointestinal resection, Kelly and colleagues21 also found prolonged operative time (>4 hours) to independently predict readmissions. To evaluate the risk of readmission for individual patients, a model is needed to incorporate these risk factors into a risk calculator.
A previous evaluation of multiple surgical subspecialties using national NSQIP data20 sought to develop a predictive model for postoperative readmissions. The authors identified prolonged length of stay, in-hospital complications, and comorbidity as strong predictors of readmission. However, Lucas et al20 found patients who underwent hepatopancreatobiliary surgery to be at highest risk for readmission, while we identified patients who underwent colorectal surgery to also be at very high risk. This discrepancy is likely owing to differing definitions of surgical subspecialties between the 2 studies. Other differences between that study and ours included the study populations; our study was limited to patients who underwent general surgery, while Lucas and colleagues included patients who received vascular and thoracic surgery. Although national NSQIP readmission data are limited to 30 days after surgery, we were able to evaluate readmissions within 30 days of discharge by pairing institutional NSQIP data with hospital billing data.
The predictive ability of our model was acceptable (C statistic = 0.756 in prospective validation) and was in line with the model created by Lucas et al20 as well as many predictive models in medical publications.15 The high-risk cutoff of 6% maximizes the sensitivity of the model in an effort to identify as many high-risk patients as possible. Validation of our readmission nomogram yielded a high negative predictive value (97.6%). However, the positive predictive value was less impressive (<15.2%). Given that proposed interventions for high-risk patients are low cost and low risk (eg, postdischarge telephone calls, early postoperative follow-up, or local lodging for those traveling from a distance), we used a cutoff that would identify most high-risk patients. This model will allow physicians to estimate the risk of readmission using readily available patient information at the time of discharge.
Currently, the discharge process is not standardized at our institution. Time to postoperative follow-up, postdischarge telephone calls, and other services are variable within and across surgical services. Although medical patients who are at high risk of readmission are flagged in the electronic medical record, no such tool exists for surgical patients. This predictive nomogram will be used to implement quality-improvement programs aimed at decreasing postoperative readmissions at our institution. We are using this nomogram to develop an online calculator and “smart phone” application so physicians can easily calculate the risk of readmission at the time of discharge. We plan to target patients who are at high risk of readmission for close outpatient follow-up after discharge from the hospital, including nursing telephone calls and earlier postoperative clinic visits. A systematic review of hospital readmissions for both medical and surgical patients demonstrated a 12% to 75% reduction in readmissions with the implementation of interventions aimed at decreasing readmissions in most studies (14 of 19 studies). Furthermore, of the 7 studies that evaluated mortality, 3 identified improved mortality after interventions were applied.10 Naylor et al22 demonstrated an improvement in readmission rates with advanced-practice nurse-centered discharge planning and home follow-up in elderly medical and cardiac surgical patients. Patients were given individual discharge plans, nursing telephone follow-up, and home visits, if needed. Patients in the intervention group incurred less costs and had longer time to readmission if they were readmitted.
This study has limited generalizability owing to its inclusion of patients from a single institution. We were also unable to account for potential risk factors for readmission, such as social support, which were not recorded in our institutional NSQIP database. Ideally, we would have evaluated the procedure-specific readmission risk, but our study was not powered for this detailed an analysis. To obtain adequate power for the study, we were unable to assess procedure-specific risk of readmission. In the future, we plan to apply the nomogram to specific groups of patients in an effort to refine the risk calculator for various surgical subspecialties.
While the use of institutional NSQIP data may limit the applicability of our results to other institutions, in other ways, it strengthened our study. We were able to pair hospital billing data with our NSQIP database to evaluate readmissions within 30 days of discharge as opposed to 30 days after surgery. This approach eliminates the immortal person-time bias that is inherent in previously published readmission studies using NSQIP data. We also were able to perform a medical record review for all readmitted patients, which allowed us to exclude planned readmissions from our analysis. Another strength of this study is the prospective validation of the nomogram. A prospective validation more likely represents the population of patients who will be included in our future quality-improvement projects using this tool; therefore, we are confident that our model will predict 30-day readmissions.
We have developed a nomogram to predict readmission 30 days after discharge in patients who undergo general surgery. Prospective validation of the nomogram at a single institution demonstrated reasonable predictive ability. Application of this nomogram in the form of an online calculator at the time of discharge will better inform patients and health care professionals of the risk of readmission. It will also allow for tailored discharge planning based on readmission risk.
Accepted for Publication: October 31, 2014.
Corresponding Author: Gregory D. Kennedy, MD, PhD, Department of Surgery, University of Wisconsin, 650 Highland Ave, Madison, WI 53792 (firstname.lastname@example.org).
Published Online: April 22, 2015. doi:10.1001/jamasurg.2014.4043.
Author Contributions: Dr Tevis had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: All authors.
Acquisition, analysis, or interpretation of data: Tevis, Weber, Kent.
Drafting of the manuscript: Tevis, Weber, Kennedy.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Tevis.
Administrative, technical, or material support: All authors.
Study supervision: Weber, Kent, Kennedy.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported by T32 training grant CA090217 from the National Institutes of Health (Dr Tevis), the Clinical and Translational Science Award program grant UL1TR000427 through the National Institutes of Health National Center for Advancing Translational Sciences (Dr Weber), and the University of Wisconsin School of Medicine and Public Health’s Wisconsin Partnership Program.
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Wisconsin Partnership Program.
Previous Presentation: This study was presented at the 2014 Academic Surgical Congress; February 4, 2014; San Diego, California.
Additional Contributions: Glen Leverson, PhD, and Chee Paul Lin, PhD, Department of Surgery, University of Wisconsin, provided statistical advice for this study. They were not financially compensated.