Receiver operating characteristic curves. DVT indicates deep vein thrombosis; PE, pulmonary embolism; SSI, surgical site infection; and UTI, urinary tract infection.
eTable 1. Postoperative Order Utilization by Complication
eTable 2. Multivariable Logistic Regression Models and Feature Coefficients for Each Complication
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Bucher BT, Ferraro JP, Finlayson SRG, Chapman WW, Gundlapalli AV. Use of Computerized Provider Order Entry Events for Postoperative Complication Surveillance. JAMA Surg. 2019;154(4):311–318. doi:10.1001/jamasurg.2018.4874
Can computerized provider order entry events be used to identify patients who would need manual medical record review to detect postoperative complications?
In a cohort study of 21 775 patients who had undergone surgical procedures, the use of computerized provider order entry to screen for postoperative complications appeared to decrease the burden of manual medical record review by 55.4% to 90.3%.
Monitoring computerized provider order entry events may augment manual medical record review for selected postoperative complications by excluding from review those patients with low likelihood of complications.
Conventional approaches for tracking postoperative adverse events requires manual medical record review, thus limiting the scalability of such efforts.
To determine if a surveillance system using computerized provider order entry (CPOE) events for selected medications as well as laboratory, microbiologic, and radiologic orders can decrease the manual medical record review burden for surveillance of postoperative complications.
Design, Setting, and Participants
This cohort study reviewed the medical records of 21 775 patients who underwent surgical procedures at a university-based tertiary referral center (University of Utah, Salt Lake City) from July 1, 2007, to August 31, 2017. Patients were included if their case was selected for review by a surgical clinical reviewer as part of the National Surgical Quality Improvement Program. Patients were excluded if they had incomplete follow-up data.
Main Outcomes and Measures
Thirty-day postoperative occurrences of superficial surgical site infection, deep surgical site infection, organ space surgical site infection, urinary tract infection, pneumonia, sepsis, septic shock, deep vein thrombosis requiring therapy, and pulmonary embolism, as defined by the National Surgical Quality Improvement Program. A logistic regression model was developed for each postoperative complication using CPOE features as predictors on a development set, and performance was measured on a holdout internal validation set. The models were internally validated using bootstrapping with 10 000 replications to determine the sensitivity, specificity, positive predictive value, and negative predictive value of CPOE-based surveillance system.
The study included 21 775 patients who underwent surgical procedures. Among these patients, 11 855 (54.4%) were women and 9920 (45.6%) were men, with a mean (SD) age of 51.7 (16.8) years. Overall, the prevalence of postoperative complications was low, ranging from 0.2% (pulmonary embolism) to 2.6% (superficial surgical site infection). Use of CPOE events to detect patients who experienced at least 1 complication had a sensitivity of 74.8% (95% CI, 71.1%-78.4%), specificity of 86.8% (95% CI, 85.5%-88.3%), positive predictive value of 33.8% (95% CI, 31.2%-36.4%), negative predictive value of 97.5% (95% CI, 97.1%-97.8%), and area under the curve of 0.808 (95% CI, 0.791-0.824). The negative predictive value for individual complications ranged from 98.7% to 100%. Use of CPOE events to screen for adverse events was estimated to diminish the burden of manual medical record review by 55.4% to 90.3%. A CPOE-based surveillance system performed well for both inpatient and outpatient procedures.
Conclusions and Relevance
A CPOE-based surveillance of postoperative complications has high negative predictive value, which demonstrates that this approach can augment the currently used, resource-intensive manual medical record review process.
Postoperative complication surveillance is a cornerstone of local and national surgical quality improvement efforts in the United States.1 Locally, surveillance systems are an important tool that hospitals use to track incidence of complications and the response to targeted quality improvement interventions. Nationally, surveillance systems aid payers and decision makers in developing health care policy to improve surgical care.2 One such surveillance system is the American College of Surgeons National Surgical Quality Improvement Program (NSQIP).3 The program provides hospitals the tools, training, and resources to collect 30-day risk-adjusted postoperative outcomes, with the goal of providing hospitals the methods and data to drive local quality improvement efforts. The advantage of these systems is the reliability of obtaining clinical data through manual medical record abstraction.4,5 The high-quality abstracted data allow for risk adjustment and accurate surveillance over time. However, manual data abstraction can be costly, typically requiring 1 to 2 full-time data abstractors.6 Because of the cost associated with manual data abstraction, most surveillance systems rely on either a case-based or a time-based sampling method. In addition, manual data abstraction introduces the possibility of inconsistency and variability in abstraction methods. The cost and complexity of abstraction, therefore, limit the scalability and dissemination of these systems across hospitals.
Over the past decade, hospitals and other health care organizations have widely adopted electronic health records (EHRs).7 Most US hospitals, primarily in response to the Health Information Technology for Economic and Clinical Health Act, have adopted comprehensive EHRs that have electronic physician documentation, clinical decision support, and computerized provider order entry (CPOE) capabilities. Comprehensive EHR systems have been shown to decrease adverse events and improve compliance with antibiotic prescribing guidelines and venous thromboembolism prophylaxis protocol.8-11
The dissemination of EHRs has created an opportunity for developing automated surveillance systems for a variety of conditions.12,13 Surveillance systems based on both structured EHR data and unstructured EHR data using natural language processing (NLP) have been shown to achieve reasonable performance in detecting postoperative complications.14,15 However, the complexity of these systems and lack of interoperability currently make their generalizability across health care systems difficult.16 In addition, these systems have not performed sufficiently well to eliminate the need for manual medical record review.17 However, a hybrid model is possible in which EHR data are used to improve the manual review process.
The goal of the present study is to determine if a surveillance system using common CPOE features derived from routine clinical care can assist surgical clinical reviewers (SCRs) with identifying appropriate medical records to manually review for postoperative complications. We hypothesize that a CPOE-based surveillance system can be used to exclude patients with a low likelihood of complications and thus substantially decrease the medical record review burden of monitoring postoperative complications without compromising accuracy.
The institutional review board of the University of Utah approved this study. Because of the study design to use existing data, a waiver of informed consent was granted by the institutional review board of the University of Utah.
We performed a retrospective cohort study of patients who underwent surgical procedures at a university-based tertiary referral center (University of Utah, Salt Lake City) from July 1, 2007, to August 31, 2017. Patients were included if their case was selected for review by an NSQIP-trained SCR. Briefly, as part of the NSQIP (which has been described previously18), the SCRs are trained in the NSQIP methods and definitions to examine the medical records of a random selection of patients who had surgical procedures. The NSQIP sampling is based on an 8-day cycle, which is designed to prevent bias in choosing cases for assessment. The SCRs analyze medical records for perioperative complications that occur within 30 days of the operation. Patients with incomplete follow-up information in the EHR are contacted to complete a 30-day follow-up. Patients were excluded from the present study if they had incomplete follow-up data.
The University of Utah has maintained an EHR with CPOE capability for both inpatient and outpatient encounters since 2007, and the data are stored in the university enterprise data warehouse. For this study, we obtained all electronic orders for patients in the enterprise data warehouse, including orders for selected medications (eg, antibiotics, anticoagulants, neuromuscular relaxants, and vasopressors), laboratory tests (eg, basic metabolic panel, complete blood cell count, blood glucose, prothrombin time, arterial blood gas, and vancomycin level), microbiologic orders (eg, urine culture, wound culture, respiratory culture, and blood culture), and radiologic orders (eg, chest, abdomen, and pelvis computed tomography; duplex venous ultrasonography; abscess drainage; and image-guided procedures); see eTable 1 in the Supplement for CPOE elements included in the final models. Orders that were temporary, canceled, or not completed were excluded. We elected to include only the orders placed 2 to 30 days after the date of operation. This limitation was aimed at excluding common postoperative order sets used as part of routine care, such as perioperative antibiotics. Individual orders were treated as binary variables, either present if at least 1 order was placed during the postoperative period or absent if no specific order was placed.
Guided by clinical experience and the literature, we elected a priori to examine 9 common postoperative complications: superficial surgical site infection (SSI), deep SSI, organ space SSI, urinary tract infection, pneumonia, sepsis, septic shock, deep vein thrombosis (DVT) requiring therapy, and pulmonary embolism (PE). In addition, we created a composite outcome of any complication if at least 1 of these compilations was present. These complications were defined according to standard NSQIP definitions (which differ from National Healthcare Safety Network and Agency for Healthcare Research and Quality definitions) and were chosen because of the complexity of screening for them, in contrast to complications associated with easily identifiable elements in the EHR, such as acute myocardial infarction with troponin elevation. Outcomes were modeled as binary variables (present or absent).
Analysis was performed with Python, version 3.1 (Python Software Foundation) and R Statistical software package (R Foundation for Statistical Computing). Univariate analysis was performed using χ2 tests for discrete variables. Model comparisons were performed using the test for equality of proportions to calculate 2-sided P values. P < .05 was used to indicate statistical significance.
We modeled the association of orders with each individual binary outcome using multivariable logistic regression. For CPOE feature selection, we used recursive feature elimination with 10-fold cross-validation.19 We used bootstrapping with downsampling for a model selection technique.20 Features associated with each individual outcome in more than 70% of 500 random samples were retained in the model. During each iteration, model fit was assessed on the basis of area under the curve (AUC), and the top-performing features were retained in the model.
We created classification models for each outcome of interest (the 9 postoperative complications) using multivariable logistic regression. We categorized CPOE features as binary variables (present or absent) and used these features as independent variables in each multivariable logistic regression model. We did not assess any interaction terms in model development. The features for each outcome are listed in eTable 2 in the Supplement. We trained logistic regression models for each outcome using bootstrapping with downsampling.20,21 During each iteration, the data set was randomly split into a development set (67%) and a validation set (33%). The development set was then randomly downsampled to create a 1:1 ratio of no complication to complication. The model was trained on the downsampled data set to obtain the feature coefficients. The trained model was then tested on the full internal validation set to obtain classification performance for these metrics: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NVP), area under the receiver operating characteristic curve, number needed to screen, and percent reduction in medical record review.22 We performed 10 000 bootstrap iterations to generate the mean feature coefficients, mean classification performance, and 95% CI for each metric.
The number needed to screen was defined as the number of medical records needed to be manually reviewed to identify 1 complication and was calculated as 1 per PPV. We defined the percent reduction in medical record review as the percentage of decrease in the number of records to analyze if only the records flagged by the CPOE were evaluated. We started by calculating the number of CPOE–flagged records to review, which was equal to (True Positives + False Positives), which in turn was equal to (Prevalence × Total Number)/PPV. The percent reduction in medical record review was then calculated as (Total Medical Records – Flagged Medical Records)/Total Medical Records.
We performed 2 sensitivity analyses using our predictive models on the full data set. In the first sensitivity analysis, we varied the probability threshold for the classification of each complication from 10% to 90% to generate area under the receiver operating characteristic curves for the full data set.22 In the second sensitivity analysis, we split the data set into inpatient and outpatient surgical procedures and measured the classification performance (sensitivity, specificity, PPV, NPV, and AUC) for each subset. Using the test of equality of proportion, we determined whether the sensitivity and specificity of the models were the same between inpatient and outpatient procedures.
We identified 21 775 patients who underwent surgical procedures from July 1, 2007, to August 31, 2017. Of these patients, 11 855 (54.4%) were women and 9920 (45.6%) were men, with a mean (SD) age of 51.7 (16.8) years. Patient and operative characteristics are shown in Table 1. The patients were predominantly female (11 855 [54.4%]), were white (16 695 [76.%]), and had American Society of Anesthesiologists Physical Status Classification 2 (11 005 [50.5%]). Gastrointestinal (15 492 [71.1%]), genitourinary (56 [0.3%]), and gynecologic (29 [0.1%]) procedures as well as skin and soft tissue procedures (5368 [24.7%]) were the most common. Most procedures were nonemergent (19 183 [88.1%]) and resulted in a postoperative inpatient admission (11 257 [51.7%]). Overall, the incidence of postoperative complications ranged from 0.2% (PE) to 2.6% (superficial SSI).
The use of postoperative orders is shown in eTable 1 in the Supplement. Patients who experienced at least 1 included complication were more likely to have at least 1 laboratory order (odds ratio [OR], 12.2; 95% CI, 10.6-14.0), microbiologic order (OR, 13.6; 95% CI, 12.2-15.1), radiologic order (OR, 8.54; 95% CI, 7.7-9.5), or medication order (OR, 12.3; 95% CI, 10.5-14.5).
We developed a multivariable logistic regression classification model for each individual complication using postoperative orders as the model covariates. The final model for each outcome is shown in eTable 2 in the Supplement. The number of features in each model ranged from 2 (PE and deep SSI) to 14 (any complication). The features in each model correlated clinically with each complication. For example, to classify a deep SSI, an order for wound culture (adjusted OR [aOR], 10.1; 95% CI, 3.4-33.6) and an order for glycopeptide antibiotic (aOR, 9.7; 95% CI, 3.7-.37.7) were independently associated with the outcome. To classify a PE, a chest computed tomography order (aOR, 7.1; 95% CI, 2.1-24.7) and an anticoagulation order (aOR, 7.5; 95% CI, 2.9-21.2) were independently associated with the outcome.
The performance of the classification models on 10 000 bootstrap iterations is shown in Table 2. The model to predict at least 1 complication had a sensitivity of 74.8% (95% CI, 71.1%-78.4%), specificity of 86.8% (95% CI, 85.5%-88.3%), and AUC of 0.808 (95% CI, 0.791-0.824), PPV of 33.8% (95% CI, 31.2%-36.4%), and NPV of 97.5% (95% CI, 97.1%-97.8%). Among the individual complications, the sensitivity was highest for DVT (89.0%; 95% CI, 79.3%-97.6%) and specificity was highest for organ space SSI (92.9%; 95% CI, 91.4%-94.5%). The classification model for organ space SSI performed best (AUC, 0.907; 95% CI, 0.884-0.928), and the classification model for superficial SSI performed worst (AUC, 0.671; 95% CI, 0.651-0.715).
The value in using CPOE events for surveillance of postoperative complications is demonstrated in the NPV for each model. The NPV ranged from 98.7% (superficial SSI) to 100% (PE), which resulted in a false omission rate (1 – NPV) of 0% to 1.4%. If willing to accept this low false omission rate, SCRs would need to evaluate only those patient records flagged by each model, and the medical record review burden would decrease by 55.4% to 90.3%. The number of medical records needed to be screened to identify 1 complication would be reduced from between 39 and 403 to between 5 and 90 with the use of a predictive model that incorporates postoperative orders (Table 2).
We performed 2 sensitivity analyses to demonstrate the robustness of the method. First, we varied the probability threshold for each model to determine its association with classification performance. The resulting receiver operating characteristic curves are shown in the Figure. Changing the probability threshold for each model can be used to maximize sensitivity (minimize false-negatives) or maximize specificity (decrease manual medical record review). For example, decreasing the classification probability threshold for organ space SSI to 10% would result in a sensitivity of 96.5% and NPV of 99.9%, thus minimizing the false-negative classification. Increasing the probability threshold to 90% would result in a specificity of 97.5% and PPV of 38.9%, thus decreasing the medical record review burden.
One potential disadvantage of using CPOE events from the EHR is the uncertainty of capturing orders from outpatient visits outside the initial episode of care. To assess and compare the performance of the models in the outpatient setting, we performed a sensitivity analysis by applying each model to the entire data set stratified by inpatient and outpatient operations (Table 3). In general, the specificity for each model statistically significantly increased for outpatient procedures compared with inpatient procedures. For most models, no substantial change in sensitivity was found between inpatient and outpatient operations.
We have shown that the CPOE events from the EHR can be used as a tool to aid medical record review for surveillance of postoperative complications. The novel and important findings of the present study include the (1) simplicity in using clinically relevant CPOE events for postoperative complication surveillance, (2) high NPV for all complications examined, and (3) generalizability for both inpatient and outpatient procedures. Each detection model incorporated 2 to 14 common orders; however, substantial variation in performance was found within the 9 complications examined. Superficial SSI detection had the worst performance with 64% sensitivity (AUC, 0.671) and 73% specificity. Meanwhile, DVT had the highest sensitivity (89.0%), and organ space SSI had the highest specificity (92.9%). Overall, the use of CPOE features to detect patients experiencing at least 1 complication had a sensitivity of 74.8%, specificity of 86.8%, and AUC of 0.808. The method had a similar performance between inpatient and outpatient procedures.
Previous studies have demonstrated the value of using EHR data for complication surveillance. A study by Branch-Elliman et al13 developed a prediction model for SSI using several clinical variables. The AUC of the model was 0.93, with an NPV of 98% for patients with very low probability of complications. Pindyck et al17 subsequently validated the Branch-Elliman et al13 model in a second independent data set and found similar performance (AUC, 0.86; NPV, 98.2%-99.6%).
We separated individual SSI subtypes because of the differences in clinical presentation. Our model for organ space SSI included similar clinical variables with similar performance (AUC, 0.907). Our model for superficial SSI included similar variables but had lower classification performance (AUC, 0.671). Both models for organ space SSI and superficial SSI had excellent ability to exclude patients with low risk for complications (NPV, 99.7% vs 98.6%; Table 2). Our study did not include the use of International Classification of Diseases, Ninth Revision or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision codes because of the delay in obtaining these elements and the variable accuracy of these codes for surveillance. We have expanded on this work to include other postoperative complications such as urinary tract infection (UTI), pneumonia, DVT, and PE.
Alternative approaches to postoperative complication surveillance involve the use of NLP. In 2 studies by Murff et al15 and FitzHenry et al,23 an NLP-based system for detecting postoperative complications was developed in the Veterans Health Administration system. These complications included acute renal failure, cardiac arrest, DVT, acute myocardial infarction, pneumonia, PE, sepsis, UTI, and SSI. Overall, the sensitivity of this NLP-based surveillance system ranged from 56% to 95%, and its specificity ranged from 63% to 97%. The system performed best for cardiac arrest and worst for wound infections, and it compared inpatient and outpatient surgical procedures with similar performance in each group. Potential disadvantages of an NLP-based system include the difficulty of portability or transferring such a system between EHRs. Current efforts to port NLP algorithms involve the clinical document architecture–based ontologies; however, most EHR vendors have not adopted these paradigms.24 The CPOE-based surveillance approach we used does not require any substantial preprocessing and could be integrated with common clinical decision support systems.
An important advantage of a CPOE-based surveillance approach is the ability to detect complications that occur after hospital discharge. In a study by Merkow et al,25 most readmissions after surgical procedures were related to new-onset complications after discharge. This study demonstrates that a substantial number of complications occur after hospital discharge, which represent important reasons for readmission. These complications are an ideal target for hospital quality improvement initiatives. The advantage of using EHR-based surveillance systems is the ability to include data from both inpatient and outpatient sources. Structured data, such as International Classification of Diseases, Ninth Revision or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, codes are not suited for identifying these complications because these codes are collected after the episode of care.5 Our approach performs well for both inpatient and outpatient procedures and improves specificity for outpatient procedures. This improved specificity likely reflects the decreased prevalence of complications in this population. However, our results demonstrate the possibility that a missing order in the outpatient setting would not negatively affect the usefulness of the approach.
Our goal was to identify approaches that aid in detecting postoperative complications; thus, we elected not to include risk factors for complications into our models. Several risk factors for postoperative complications have been identified, including procedure-specific complication risks, American Society of Anesthesiologists Physical Status Classification, and patient comorbidities.12 The purpose of demonstrating the use of CPOE events was to show the ability to perform not risk prediction, which still requires accurate medical record review,18 but case detection, regardless of the underlying patient risk. Integrating both patient and procedure risk factors into the prediction model will likely improve the performance at a modest cost of increasing the complexity of the underlying model. In addition, it will increase the technical complexity of disseminating this approach across EHR systems.
The implication of this study is the ability to considerably decrease the medical record review burden for postoperative complications surveillance. Decreasing the effort required to perform manual review allows hospitals and other health care organizations to perform close to 100% complications surveillance compared with the currently used sampling method. By combining our approach with NSQIP, hospitals can identify targets for quality improvement efforts using NSQIP hospital comparison reports. Complications identified as high outliers at a given hospital system can then be collected using CPOE-based surveillance. This process will allow hospitals to identify priorities for quality improvement efforts and implement evidence-based strategies to improve clinical care. In addition, after quality improvement efforts have been implemented, the CPOE-based approach can be used to monitor progress after implementation.
This study is limited by being conducted at a single academic health care system and lacking external validation in a more expansive secondary data set. We attempted to demonstrate the generalizability of our approach by splitting the data into development and validation sets with 10 000 bootstrap iterations. We also defined complications according to standard NSQIP definitions. In addition, no standardized terminologies exist for most nonmedication orders; thus, developing a surveillance system that could easily be used with commercial EHR vendors may be difficult.
We have demonstrated the utility of CPOE events for surveillance of postoperative complications. This approach minimizes manual medical record review and allows targeted surveillance strategies for postoperative complications. In addition, it performs well for inpatient and outpatient surgical procedures. Future efforts will focus on further standardizing this approach and validating it on a larger scale.
Accepted for Publication: September 23, 2018.
Corresponding Author: Brian T. Bucher, MD, Division of Pediatric Surgery, Department of Surgery, University of Utah School of Medicine, 100 N Mario Capecchi Dr, Ste 3800, Salt Lake City, UT 84113 (firstname.lastname@example.org).
Published Online: December 26, 2018. doi:10.1001/jamasurg.2018.4874
Author Contributions: Dr Bucher had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: All authors.
Acquisition, analysis, or interpretation of data: Bucher, Ferraro, Chapman, Gundlapalli.
Drafting of the manuscript: Bucher, Gundlapalli.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Bucher, Ferraro, Gundlapalli.
Obtained funding: Bucher, Finlayson.
Administrative, technical, or material support: Bucher, Finlayson, Gundlapalli.
Supervision: Ferraro, Finlayson, Chapman.
Conflict of Interest Disclosures: Dr Chapman reported consulting for IBM and Health Fidelity. No other disclosures were reported.
Funding/Support: This study was funded by grant 1K08HS025776 from the Agency for Healthcare Research and Quality (Dr Bucher) and by VA Center of Innovation Award I50HX001240 from the Health Services Research and Development of the Office of Research and Development of the US Department of Veterans Affairs (Dr Gundlapalli).
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Create a personal account or sign in to: