Plots of observed and universal model–predicted rates of postacute care discharge for Current Procedural Terminology codes. The dashed limit lines on either side of the diagonal representing perfect agreement are set at ±25%. A, Scatterplot. B, Interquartile ranges around the predicted values for each Current Procedural Terminology code. This represents the theoretical differences in patient factors within each Current Procedural Terminology code.
Mohanty S, Liu Y, Paruch JL, Kmiecik TE, Cohen ME, Ko CY, Bilimoria KY. Risk of Discharge to Postacute CareA Patient-Centered Outcome for the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator. JAMA Surg. 2015;150(5):480-484. doi:10.1001/jamasurg.2014.3176
Individualized risk prediction tools have an important role as decision aids for use by patients and surgeons before surgery. Patient-centered outcomes should be incorporated into such tools to widen their appeal and improve their usability.
To develop a patient-centered outcome for the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) Surgical Risk Calculator, a web-based, individualized risk prediction tool.
Design, Setting, and Participants
Retrospective cohort study using data from the ACS NSQIP, a national clinical data registry. A total of 973 211 patients from July 2010 to June 2012, encompassing 392 hospitals, were used in this analysis.
Main Outcomes and Measures
Risk of discharge to a postacute care setting.
The overall rate of discharge to postacute care was 8.8%. Significant predictors of discharge to postacute care included being 85 years or older (odds ratio [OR] = 9.17; 95% CI, 8.84-9.50), the presence of septic shock (OR = 2.43; 95% CI, 2.20-2.69) or ventilator dependence (OR = 2.81; 95% CI, 2.56-3.09) preoperatively, American Society of Anesthesiologists class of 4 or 5 (OR = 3.59; 95% CI, 3.46-3.71), and totally dependent functional status (OR = 2.27; 95% CI, 2.11-2.44). The final model predicted risk of discharge to postacute care with excellent accuracy (C statistic = 0.924) and calibration (Brier score = 0.05).
Conclusions and Relevance
Individualized risk of discharge to postacute care can be predicted with excellent accuracy. This outcome will be incorporated into the ACS NSQIP Surgical Risk Calculator.
Individualized surgical risk prediction tools have become an important opportunity for shared decision making in surgical patients. These are tools that allow input of the procedure type as well as individual patient factors, such as demographic characteristics and comorbidities, after which an underlying prediction algorithm outputs predicted risk for a number of postoperative outcomes.1 Currently, most of these tools offer traditional outcomes such as postoperative cardiac events, urinary tract infections, or surgical site infections. Although understanding the risk of these outcomes is important for decision making, there has been recent focus on complementary outcomes that represent the concerns and preferences of patients and are perhaps easier to conceptualize. These patient-centered outcomes allow more informed decision making, particularly when it comes to choosing one of several treatment or intervention options. Their incorporation into prediction tools is essential for widening their appeal and use by surgeons and patients. Discharge to postacute care settings such as acute rehabilitation facilities is an example of one such outcome. It is patient centered, reflecting in part a functional outcome following surgery, and predicting patients at high risk for such a discharge can be important for discharge planning, health system resource allocation, and cost containment.
The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) Surgical Risk Calculator currently allows for entry of 21 preoperative patient factors in addition to the type of procedure and uses regression models to output the individualized risk of 8 surgical outcomes. This universal risk prediction tool leverages a large amount of clinical data and has been shown to provide predictions that are nearly as accurate as those from specialized calculators.1 Since 2011, patient discharge destination has been collected from all ACS NSQIP hospitals. This encompasses both home and postacute care destinations such as skilled nursing facilities, acute rehabilitation facilities, and unskilled nursing homes. We sought to use these data to develop a new outcome for the Surgical Risk Calculator: risk of discharge to a postacute care facility.
Data were obtained from ACS NSQIP Semiannual Report data sets from July 2010 to June 2012. This program, including its history, purpose, and sampling method, has been detailed extensively in the literature.2- 4 Briefly, ACS NSQIP is a nationally validated, risk-adjusted, outcomes-based program designed to measure and improve the quality of surgical care. It has collected data from more than 500 hospitals in a standardized, reliable fashion on a variety of patient factors, including demographic characteristics and comorbidities, operations, and postoperative outcomes. These data are collected by trained and audited surgical clinical reviewers at each participating hospital. Following the index operation, patients are followed up for 30-day postoperative outcomes. In this study, all patients undergoing operations between July 2010 and June 2012 were included. Because this study did not meet the definition of human research, institutional review board approval and informed consent were not required.
Twenty-one preoperative risk factors were used in predicting patient-specific risk of postacute care discharge. These preoperative factors were selected based on predictive value, availability, and face validity. All preoperative factors were made categorical before inclusion into regression models. Risk of postacute care discharge coming purely from the procedure was determined from preliminary models in which Current Procedural Terminology (CPT) code was used as a random effect in a hierarchical model. This generated a continuous variable—a logit-transformed predicted probability of the outcome—unique for each procedure. An additional variable for work relative value units was included to account for procedure complexity.1
The outcome of interest was discharge to postacute care. All nonhome destinations, which included skilled nursing facilities, unskilled nursing facilities, and all acute rehabilitation facilities, were grouped together. Patients who died were excluded, and those who were discharged from the hospital to facilities at which they were living prior to their operation were grouped with patients discharged home.
For this study, two-thirds of the total cohort, randomly sampled without replacement, was used to develop random-intercept, fixed-slope hierarchical models. This accounts for clustering of cases in hospitals and uses an empirical Bayes-type shrinkage adjustment.5 Twenty-one patient factors as well as the operation CPT linear risk score and a work relative value units variable were used for risk prediction.5,6 The final model was validated on the remaining one-third of the cohort. Model performance was evaluated using the C statistic and the Brier score. The C statistic, or area under the receiver operating characteristic curve, is a measure of discrimination, with 0.5 corresponding to discrimination that is no better than chance and 1.0 corresponding to perfect prediction. Finally, the Brier score is both a measure of discrimination and calibration and is defined as the means squared difference between predicted and actual outcomes, with 0.0 indicating perfect model performance.5 After model validation, a final model was developed using the entire cohort for incorporation into the risk calculator. All data manipulation and statistical analysis were performed using SAS version 9.4 statistical software (SAS Institute, Inc).
There were 973 211 cases from 392 hospitals in the final analysis. These encompassed all surgical specialties except transplant and trauma. The overall event rate of discharge to postacute care was 8.8%. This varied by specialty, with the highest rates in patients undergoing orthopedic (24.5%), cardiac (20.3%), and vascular (16.3%) procedures (Table 1).
In aggregate, patients discharged home were younger and healthier, with fewer comorbidities and better functional status. The derivation (n = 643 211) and validation (n = 330 000) cohorts were not significantly different in terms of their demographic characteristics or comorbidities (Table 2).
The most significant predictors associated with continuing care discharge, adjusted for procedure type and complexity, included being 85 years or older (odds ratio [OR] = 9.17; 95% CI, 8.84-9.50), the presence of septic shock (OR = 2.43; 95% CI, 2.20-2.69) or ventilator dependence (OR = 2.81; 95% CI, 2.56-3.09) preoperatively, American Society of Anesthesiologists class of 4 or 5 (OR = 3.59; 95% CI, 3.46-3.71), and totally dependent functional status (OR = 2.27; 95% CI, 2.11-2.44). All predictors are shown in Table 3. In addition to these patient factors, the final model included emergent case status, the presence of dyspnea, renal failure, chronic obstructive pulmonary disease, congestive heart failure, body mass index, steroid use, smoking status, hypertension, diabetes mellitus, recent cardiac event, wound class, the presence of locally invasive cancer, and whether the patient was on dialysis preoperatively. Model fit and calibration, as measured with the C statistic and Brier score, were excellent (C statistic = 0.924; Brier score = 0.05).
The relationship between the observed and expected (predicted) rates of postacute care discharge for 498 CPT codes for the continuing care discharge outcome in the validation cohort is shown in the Figure both as a scatterplot and with interquartile ranges. This illustrates that the model for predicting continuing care discharge is reasonably accurate, with each point representing a CPT clustering around the diagonal representing perfect agreement. In theory, the interquartile ranges represent the perceived differences among patients undergoing the same procedure.
In this study, we developed a tool that focuses on an outcome easily conceptualized by patients—the risk of discharge to postacute care—and incorporated it into the universal, patient-specific ACS NSQIP Surgical Risk Calculator. This calculator allows the entry of 21 patient factors as well as the procedure and outputs the risk of several adverse postoperative outcomes, allowing a frank discussion between surgeon and patient about the risks of surgery. Importantly, it may also allow hospitals to predict resource utilization more accurately.
Recent work by Hyder et al7 has demonstrated the feasibility of an individual risk calculator designed to predict home discharge after surgery. Their calculator focused on usability and questions that can be answered relatively quickly and easily by the patient; excluded all joint replacements; and showed good discrimination and calibration. In contrast, the ACS NSQIP Surgical Risk Calculator is designed to be used by the surgeon and has a preexisting user interface that facilitates the consideration and use of numerous patient factors. In addition, we also incorporated information about the procedure into our risk prediction, and we included all joint replacements, which account for the largest proportion of nonhome discharges in the ACS NSQIP.
Predicting discharge to postacute care settings is particularly important in orthopedic surgery. Several investigators have examined patient factors that predict discharge to either a rehabilitation facility or skilled nursing facility. Most studies are single-institution or small multicenter studies using claims data that have identified age, increased American Society of Anesthesiologists class, poor functional status and/or use of a gait aid, obesity, and increased comorbidity burden as important predictors.8- 11 Our study similarly found that total joint replacements and hip fracture operations had the highest risk of discharge to postacute care and that the same patient factors were the most significant risk factors.
Discharge to a nursing or acute rehabilitation facility is easily understood as a kind of functional outcome. Some authors have suggested that, in the absence of a medical reason, patient preferences play an important role in discharge to rehabilitation following joint replacement.9 Ideally, such preferences are guided by objective risk information, which this tool provides, based on the procedure and patient comorbidity burden. This allows an honest framing of expectations for recovery and return to independence. In addition, this information could give confidence to decision making during a preoperative appointment or during the informed consent process.
At the hospital or health system level, knowledge of this information would allow for a number of strategies that could potentially improve the quality and efficiency of recovery pathways and discharge planning in elective surgery. For example, preoperative risk stratification could identify a high-risk cohort for whom early, evidence-based interventions designed to prevent functional decline could be implemented.12- 14 This cohort might warrant earlier or more aggressive involvement of physical and occupational therapy. Case managers and social workers might also be able to complete earlier needs assessment and skilled nursing or acute rehabilitation eligibility evaluation.
The results of this study should be considered in light of several limitations. First, our study was limited to ACS NSQIP hospitals, which are typically larger and have a focus on quality improvement. Thus, these hospitals may be different from the rest of the hospitals in the United States regarding use of postacute care facilities. Second, this risk prediction does not incorporate information about postoperative adverse outcomes that likely affect the decision to send a patient to postacute care. This information is not available prospectively, however, and it would be important to communicate this in any discussions with patients. Finally, this outcome only takes into account patient factors and the procedure and does not account for practice variation, socioeconomic factors, home support, and hospital characteristics with regard to use of postacute care. Recent work has shown that inpatient surgical volume or other proxies for hospital size and resource availability affect the use of postacute care.15 With that said, when using the ACS NSQIP Surgical Risk Calculator, the surgeon is free to modify risk estimates in accordance with local conditions through use of the surgeon adjustment score.1
Using a large clinical data set, we developed a model with excellent discriminatory ability and calibration that allows determination of the risk of discharge to postacute care as a new outcome for the ACS NSQIP Surgical Risk Calculator. This will undoubtedly prove useful to patients and hospitals when planning for surgery and postoperative care.
Corresponding Author: Sanjay Mohanty, MD, Division of Research and Optimal Patient Care, American College of Surgeons, 633 N St Clair, 22nd Floor, Chicago, IL 60611 (firstname.lastname@example.org).
Accepted for Publication: October 1, 2014.
Published Online: March 25, 2015. doi:10.1001/jamasurg.2014.3176.
Author Contributions: Dr Mohanty had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Mohanty, Paruch, Ko, Bilimoria.
Acquisition, analysis, or interpretation of data: Mohanty, Liu, Kmiecik, Cohen, Bilimoria.
Drafting of the manuscript: Mohanty, Bilimoria.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Mohanty, Liu, Paruch, Cohen.
Administrative, technical, or material support: Mohanty, Kmiecik.
Study supervision: Ko, Bilimoria.
Conflict of Interest Disclosures: None reported.
Funding/Support: Dr Mohanty is supported by the Clinical Scholar in Residence Program from the American College of Surgeons and by the American Geriatric Society and the Hartford Foundation.
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.