Development and Validation of a Machine Learning Model to Aid Discharge Processes for Inpatient Surgical Care

This prognostic study develops and validates the performance of a neural network machine learning model compared with a model based on median length of stay for predicting which patients are likely to be discharged within 24 hours from inpatient surgical care and their barriers to discharge.


Introduction
As hospitals continue to experience high demand for their services, efficient capacity management is critical to fulfilling their mission to serve patients. [1][2][3][4] Surgical teams' ability to deliver timely operative care is reliant on the availability of postoperative floor beds. Furthermore, when inpatient beds are not available, patients needing transfer from outside hospitals to receive procedural care at large medical centers experience delays.
Timely patient discharge requires a coordinated effort among surgeons, nurses, case managers, physical therapists, and others. They must identify candidates for discharge early and the barriers that prevent the patient's transition out of the hospital. [5][6][7][8] Some hospitals request that the clinical teams identify and prioritize patients who will be discharged each day, although this task is manual, complex, and inconsistently performed. 9 Without clearly identifying patients and their discharge barriers, successful coordination of discharge-related activities is difficult.
Few tools exist that are capable of automatically and accurately predicting which patients will be discharged and their barriers to discharge. Machine learning algorithms able to process substantial electronic clinical and operational data may offer an opportunity to create timely, systematic, and accurate predictions of which patients will be discharged. [10][11][12] However, these algorithms must be combined with an interpretable clinical model to create an output that helps guide the clinical discharge processes. In doing so, such models can augment clinical teams by helping to focus their efforts on patients nearing discharge, explicitly identifying barriers to discharge that they can address, and creating transparency among multidisciplinary team members critical to the discharge process by communicating these elements automatically.
Using the clinical and operational data available in our hospital's electronic medical record (EMR), we sought to develop a machine learning algorithm based on a clinical model that could successfully predict inpatient surgical care discharges. Our primary aim was to validate the performance of such a model in predicting inpatient surgical care discharges and barriers to discharge. Our secondary aim was to analyze the cause of cases in which the model predicted that the patient would be discharged but the patient remained in the hospital.

Methods
The Partners Healthcare Human Research Committee reviewed and approved the study and access to EMR data for the purposes of this study. The need for oral or written consent from study participants was waived by the committee because data were deidentified and stored in secure, encrypted computers.

Study Setting and Population
The study was conducted at a 1034-bed quaternary care teaching hospital where more than 36 000 surgical procedures are performed each year. During the study period, there was a mean of 151.8 patients discharged per day from the hospital, 31.2 of whom were patients who underwent inpatient surgical care.

Data Sources
We used clinical and administrative data from an institutional database linked to patients' EMRs. The data for the daily prediction incorporated all data until 23:59 on the day prior to prediction. This time was used because the clinical and administrative database was updated every 24 hours, starting at 23:59. We included demographic, environmental, administrative, and clinical data in the model ( Table 1). Most data elements were structured, including either numerical values (eg, laboratory results) or multiple choice of predetermined text phrases (eg, diet descriptions within diet orders).
Other data elements were in free-text form in case manager notes and required text processing to transform them into data elements usable by the model. We used a method of key word search in which we searched for the existence of a predefined list of words in the note such as, "home today," "discharge today," "is medically ready," and "referral placed."

Model Development Clinical Milestones of Recovery and Barriers to Discharge
Clinical data were included in the model based on whether they represented a clinical milestone of recovery or a clinical or administrative barrier to discharge. A clinical milestone of recovery was defined as an event that marked the patient's progress toward recovery and discharge. A barrier to discharge was defined as an event that may have postponed the patient's discharge. Clinical milestones and barriers to discharge were identified by a multidisciplinary team that included physicians, nurses, case managers, physical therapists, and nutritionists. Milestones and barriers were recorded by a member of the study team and were ultimately included in the model based on whether data related to these events were available in the database linked to the patient's EMR.
Clinical milestones could be specific to a type of surgical procedure or generalizable across multiple types of surgical procedures. Milestones were tracked by the model for changes in value that could be clinically interpreted as a milestone having been achieved. Milestones were defined in the following categories: transition from intravenous to oral medications; laboratory results within reference ranges; vital signs within reference ranges; completion of consult assessments; completion of imaging studies; removal of surgical drains, catheters, and wound devices; resolution of barriers to postacute care transitions; improvement in functional or mobility challenges; and nursing bedside assessments within reference ranges (eg, no cognitive abnormalities).
Potential barriers to discharge were defined in the following categories: medication administration; laboratory results outside of reference ranges; vital signs outside of reference ranges; awaiting specialist consult assessments; new scheduled surgical procedures; presence of surgical drains, catheters, and wound devices; postacute care transition barriers; functional or mobility challenges; and nursing bedside assessments outside of reference ranges (eg, cognitive abnormalities). Barriers to discharge were tracked by the model for changes in value that could be clinically interpreted as resolution.

Prediction Algorithm, Model Selection, and Barrier Identification
We elected to use a multilayer perceptron feedforward neural network model that predicts a continuous score between 0 and 1 associated with the likelihood of a patient to be discharged within 24 hours. The decision to use a neural network model was motivated by our assumption that the final decision to discharge a patient is associated with underlying clinical and nonclinical variables that interact in a complex, highly nonlinear fashion. 13 Further, the neural network model supported online learning, so that when new data became available, they could be used to update the best prediction for future data in an adaptive manner. This allowed us to continue improving the prediction accuracy A set of candidate barriers to discharge was identified by manually selecting from the model's variables those that pertain to a clinical or operational indicator related to discharge (eg, "Not taking a regular oral diet," "Home visiting nurse services not available"). A candidate barrier was determined to be an actual barrier to discharge by systematically toggling the value of the variable and recalculating the prediction score by the model. If the new score was higher than the initial score, the candidate barrier was marked as an actual barrier for the patient.

Statistical Analysis Model Performance and Comparisons
We computed an estimated out-of-sample AUC by conducting 10 random experiments. In each, we randomly distributed the cohort into training (80%) and validation (20%) sets. 14 We measured the testing accuracy of the model using the fixed parameters and randomly sampled training data.
To simplify the model output for clinical end-users into a binary classification, we determined an optimal threshold value above which we designated that a patient was predicted to be discharged and below which a patient was predicted not to be discharged. We defined the optimum threshold as that which would equally balance model sensitivity and positive predictive value (PPV). We calculated the threshold using experiments with the random validation samples in which we continuously adjusted the threshold score from 0 to 1 and identified the threshold at which the sensitivity and PPV were equal. Finally, we rounded the threshold to the nearest tenth digit. Using the binary output, we calculated the sensitivity, specificity, PPV, and negative predictive value (NPV).

JAMA Network Open | Health Informatics
We compared the neural network model relative to a baseline model that captures a simple discharge planning approach aligned with current practice at our institution. The baseline model used the historical median length of stay (LOS) of a given surgical procedure type to predict when the patient would be discharged. We calculated the sensitivity, specificity, PPV, and NPV of the median LOS model.

Model Performance and Comparisons
Patient characteristics for the cohort used to train and validate the feedforward neural network model are presented in  The top 20 barriers in terms of relative weight on the model's predictions are listed in Table 3.
A complete list of the barriers and their weights are presented in eTable 2 in the Supplement.

Prospective Observational Analysis of Discharge Delays
The prospective observational analysis was performed on 605 patients. Overall, 136 patients were discharged later than the model predicted. Figure 2 presents the reasons why patients were not discharged on the day they were predicted to by the model and the associated number of bed-days

Discussion
This cohort study examined the performance of a feedforward neural network model to predict which patients who had undergone inpatient surgical care would be discharged 24 hours in advance and their respective clinical and nonclinical barriers. We found that the model demonstrated strong discrimination, with an AUC of 0.84, and strictly dominated a comparison model that used historical median LOS by surgical procedure type to predict discharges. In a prospective analysis of patients who were discharged later than the date the model predicted, we found that in most cases, patients remained in the hospital owing to nonclinical reasons or variation in clinical practice. These results were used to estimate potentially avoidable bed-days.
To our knowledge, this is the first study to demonstrate a model that can predict daily discharges at the individual patient level and automatically provide feedback on discharge barriers.
Based on our findings, we believe that this model could aid clinicians in overcoming several challenges to timely discharges. First, it provided transparency into which patients were candidates for discharge within the next 24 hours, which can help clinical teams focus their efforts on appropriate patients each day. Second, it identified actionable clinical and nonclinical barriers. These functions can aid the coordination of discharge activities across the multidisciplinary team required to transition the patient out of the hospital. We intended for the model to facilitate targeted discussions regarding discharge and create alignment among the multidisciplinary clinical team tasked with discharging the patient. Importantly, the model was developed for the purpose of improving the discharge process rather than replacing the decisions of the clinical team. The model's output is intended to interact with the knowledge of clinical team members not captured in the tool and, in combination, augment the discharge process.
Previous models have included variables that, while important to understanding the context of the patient regarding discharge, most often represented nonactionable or nonmodifiable factors, such as age, chief concern, day of the week, insurance status, and race/ethnicity. 15,16 During this model's development, we gathered the input of physicians, nurses, case managers, physical   therapists, nutritionists, and others responsible for the care of the patient. These discussions led to the generation of additional features in the model and provided critical guidance as to which barriers were most actionable by the team and helpful to display.
In addition, the model identified patients who were not discharged when predicted, and we analyzed the causes of these nondischarges. This demonstrated the use of the model as a tool that could inform hospital process improvement and system redesign efforts regarding discharge.
Typically, these efforts are challenged by the lack of a designated expected day of discharge. Thus, there is no mechanism to identify scenarios in which a patient was discharged later than expected.
The current practice at our hospital is to ask nurses, case managers, or physicians to input expected days of discharge into the EMR; however, this is a manual, labor-intensive process and can vary across clinicians.
We found that the model could be used to focus the review and analysis of discharge delays on a subset of patients who were not discharged on the day predicted by the model. We found that these patients remained in the hospital for reasons that were likely addressable by the clinical teams.
Most of these patients did not have substantially different or new medical treatment provided to them. Future research is needed to study the use of this tool by clinical teams in daily practice to determine whether this tool could help clinical teams decrease avoidable bed-days.
One challenge with this model is determining the threshold above which the patient is predicted to be discharged. We selected a threshold that balanced true-positives and false-positives.
We found that the model had high specificity, although its sensitivity was relatively lower. Depending on what the clinical team is attempting to accomplish, the appropriate threshold may vary. For example, clinician users may want to choose a lower threshold that is more sensitive to ensure that they capture as many discharges as possible while tolerating some patients who may not yet be ready for discharge. Others in the hospital, such as bed managers, may prefer an approach that is highly specific as they may only want to know about beds that are certain to become available as they look to accommodate incoming patient demand.
Our findings add to the limited and emerging body of literature on machine learning applications designed to improve clinical operations. Previous studies have examined whether neural networks could be used to predict LOS. These models have typically been developed to predict LOS prior to admission to aid capacity planning by bed managers. [14][15][16][17] By predicting LOS at a time remote from discharge, these models have typically not identified barriers to discharge or actions that the team could take in the moment to ensure timely discharge. In addition, to our knowledge, previous models have neither predicted discharges at a patient-level nor provided information regarding barriers to discharge that could guide clinical teams. Furthermore, some models that have predicted daily discharges have done so across all patients in the hospital, which is potentially useful at the hospital level but not actionable for clinical teams caring for the individual patient. 18,19

Limitations
Our study has some limitations. The model was trained and validated on patients within a single institution; thus, its generalizability to other hospitals is uncertain. We included demographic and clinical variables that are common to patients regardless of geographic location or health system; however, it is plausible that the model would behave differently in hospitals outside of our institution.
In addition, we validated the model's performance by comparing it with a model that used median LOS of the patient's type of surgical procedure to predict the day of discharge. We chose to use this model as a comparison because we believed it to be a reasonable approximation of how clinicians estimate the discharge date in their typical practice. This is particularly true of surgical procedures with standard pathways of recovery. For example, a patient who receives a laparoscopic sleeve gastrectomy in our institution is generally known to have an LOS of 1 day, and teams expect to discharge patients on this day. However, for patients who do not have standard pathways of recovery, historical median LOS may not be a good approximation of the expected discharge date.
The inputs for the model were limited to those discoverable on the EMR. If the data were not included in the EMR, such as verbal discussions about a patient's status with regard to discharge, they could not be captured by the model. As a result, the model may underestimate or overestimate the likelihood of discharge and underreport important barriers to discharge.

Conclusions
This study found that a clinically interpretable neural network model could predict daily inpatient surgical care discharges and their barriers. We used the model to identify causes of discharge delay.
Such a model could potentially be used in the future to guide process improvement and increase the timeliness of discharges through daily prioritization of patients and barriers identification. These functions could aid hospitals in addressing capacity challenges and thereby reduce delays in patient care.