Forest plot shows adjusted odds ratios (AORs) for the single characteristics of the patients and type of admission, preintervention vs postintervention, with weighting of the data by inverse probability of treatment.
eFigure. Checklists Used at the Bolzano Central Hospital and Sheet for Quality Control
eTable 1. Mapping of the Surgical Procedures Analyzed
eTable 2. Propensity Score by Percentile Classes Before and After Checklist
eTable 3. 90-Day Mortality Data After Univariate Analysis
eTable 4. 30-Day Mortality Data After Univariate Analysis
eTable 5. 30-Day Readmission Rate Data After Univariate Analysis
eTable 6. LOS Data After Univariate Analysis
eTable 7. Adjusted OR for 90-Day Mortality in Preimplementation and Postimplementation Groups After Multivariate Analysis
eTable 8. Adjusted OR for 30-Day Mortality in Preimplentation and Postimplementation Groups After Multivariate Analysis
eTable 9. Adjusted OR for 30-Day Readmission in Preimplentation and Postimplementation Groups After Multivariate Analysis
eTable 10. LOS in Preimplentation and PostImplementation Groups After Multivariate Analysis
eTable 11. OR for 30-Day Readmission (Adjusted)
eTable 12. LOS (Adjusted)
Bock M, Fanolla A, Segur-Cabanac I, Auricchio F, Melani C, Girardi F, Meier H, Pycha A. A Comparative Effectiveness Analysis of the Implementation of Surgical Safety Checklists in a Tertiary Care Hospital. JAMA Surg. 2016;151(7):639-646. doi:10.1001/jamasurg.2015.5490
The appropriately coached implementation of surgical safety checklists (SSCs) reduces the incidence of perioperative complications and 30-day mortality of patients undergoing surgery. The association of the introduction of SSCs with 90-day mortality remains unclear.
To assess the association between the implementation of SSCs and all-cause 90- and 30-day mortality rates.
Design, Setting, and Participants
Evaluation of the outcomes of surgical procedures performed during the 6 months before (January 1 to June 30, 2010) and after (January 1 to June 30, 2013) the introduction of SSCs by retrospective analysis of administrative databases. The study was conducted in a public, regional, university-affiliated hospital in Italy. Data were collected from October 23, 2013, to November 12, 2014, including 90-day all-cause mortality, 30-day all-cause mortality, length of hospital stay, and 30-day readmission rate among patients undergoing noncardiac surgery. Patients undergoing surgery during the 6-month periods before and after the implementation of SSCs were compared. Data were analyzed from September 17, 2014, to July 31, 2015.
Main Outcomes and Measures
Risk-adjusted rates of 90- and 30-day mortality, readmission rate, and length of stay.
The total study sample of 10 741 patients included 5444 preintervention and 5297 postintervention patients (5093 [47.4%] male and 5648 [52.6%] female patients; mean [SD] age, 53.0 [23.0] years). Ninety-day all-cause mortality was 2.4% (129 patients) before compared with 2.2% (118 patients) after the SSC implementation, for an adjusted odds ratio (AOR) of 0.73 (95% CI, 0.56-0.96; P = .02). Thirty-day all-cause mortality was 1.36% (74 patients) before compared with 1.32% (70 patients) after the SSC implementation, for an AOR of 0.79 (95% CI, 0.56-1.11; P = .17). Thirty-day readmission occurred in 797 patients (14.6%) in the preimplementation group vs 766 patients (14.5%) in the postimplementation group, for an AOR of 0.90 (95% CI, 0.81-1.01; P = .79). The adjusted length of stay was 10.4 (95% CI, 10.3-10.6) days in the preimplementation group compared with 9.6 (95% CI, 9.4-9.7) days in the postimplementation group (P < .001).
Conclusions and Relevance
The data cannot prove causality owing to the study design. The implementation of SSCs was associated with a 27% reduction of the adjusted risk for all-cause death within 90 days but not within 30 days. The adjusted length of stay was reduced after implementation of SSCs.
At present, inpatients worldwide may expect a 30-day-mortality of 1.5% after noncardiac surgery,1 depending on the region where surgery is performed,2 the surgical procedure,3 and the patients’ comorbidities. Data from the United States revealed a high variability of the hospital mortality for inpatients after oncologic and high-risk surgery,4,5 whereas the incidence of overall complications was less skewed and remained constant among US Medicare patients from 2005 to 2011.6 Consequently, effective risk management for patients undergoing surgery is crucial. This risk management requires an interdisciplinary, process-oriented approach to overcome barriers between single disciplines. Checklists aim to reduce risk and prevent patient harm by recognizing high-risk situations and optimizing communication, by minimizing the incidence of errors, and by improving latent conditions. The first experience of a checklist elaborated by the World Health Organization (WHO)1 and a more comprehensive perioperative checklist (the Surgical Patient Safety System [SURPASS])7 reported significant reduction in mortality and morbidity, contrary to a retrospective study on the implementation of WHO checklists within Ontario.8 These studies analyzed in-hospital mortality or 30-day mortality but not intermediate-term outcome variables. Almost one-quarter (23.6%) of the deaths within 30 days after surgery occurred after discharge, and 39.7% of patients undergoing surgery experienced only postdischarge complications.9 Ninety-day mortality often doubles 30-day mortality.10,11 In-hospital mortality and 30-day mortality might therefore underreport the real risk to these patients, especially after tumor surgery or among the elderly.12 Studies of the effect or the association of the implementation of surgical safety checklists (SSCs) on 90-day mortality are lacking.
As our primary aim, we assessed the association of the introduction of the adapted WHO SSC and 30-day or 90-day all-cause mortality among patients undergoing surgery at the Central Hospital of Bolzano (CHB) in Italy. We defined the association of the implementation of SSCs and the hospital length of stay (LOS) and 30-day readmission rate as secondary study aims.
We designed a retrospective comparative effectiveness analysis of administrative databases to investigate the association of the implementation of SSCs and outcome data. Study design and reporting were performed according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement.13 The study was approved by the local institutional review board of the public health care system in South Tyrol, Italy. The data in the deidentified analysis were exempt from informed consent.
The study took place at the CHB, a 715-bed inpatient facility that serves as a referral hospital for the public health care system within the Autonomous Province of South Tyrol, and, in case of major complications, for private hospitals in South Tyrol. The Italian government highly recommended the implementation of the WHO checklist in a ministerial decree released in 2009.14 Each Italian hospital must implement an SSC containing the 3 steps proposed by the WHO and a final discussion of the management of the prophylaxis for thromboembolism.
Until April 2010, a modified version of the checklist was designed for the surgical departments of the CHB (general, pediatric, vascular, thoracic, gynecologic and obstetric, neurosurgery, ophthalmology, otolaryngology, traumatology and orthopedic, and urology) by an interdisciplinary committee. In particular, we anticipated the items antibiotics and preoperative imaging in the first section of the SSC before the induction of anesthesia. In September 2010, we introduced a comprehensive SSC containing 24 items for standard surgical procedures and a modified checklist containing 17 items for short-term or same-day surgical procedures in the single surgical departments using a step-by-step regime. Cesarean sections were accompanied by a dedicated 22-item SSC that involved the obstetric, anesthesia, and neonatology teams (the 3 SSCs are described in the eFigure in the Supplement). As proposed by the WHO, all the SSCs follow the 3 sign-in, time-out, and sign-out phases and are completed in the operating room by physicians and nurses caring for the patient. The implementation of the SSCs was accompanied by guided supervision and assistance. During the introduction period, we analyzed the performance of execution of the checklists by evaluating compilation of the items and clinical process flow. Each SSC contained a table on the back (eFigure in the Supplement) where the staff in the operating room reported on performance errors. We judged the SSC to be performed correctly when the following criteria were fulfilled: each item was correctly filled out, no performance errors were noted on the back sheet, and the analysis of the SSC provided clinical evidence of its correct performance. The performance of the SSC in each operating room was presented officially to all the personnel working in the operating room in monthly intervals. We required an 80% rate of correctly executed SSCs for the official declaration of successful implementation in the single disciplines (achieved in October 2012).
We performed a retrospective analysis of administrative outcome data before and after the implementation of the SSCs. A 3-month period before the implementation and another after the successful introduction of the SSC constituted consolidation periods. We chose a 6-month period for data collection to enroll a minimum sample size of 9750 patients (1.5%-0.8% reduction of 30-day mortality as reported by Haynes et al1; α = .05; 1 − β = 0.90). Consequently, we evaluated the outcome of surgical procedures performed from January 1 to June 30, 2010, as the preintervention period and the outcome data of surgical procedures performed from January 1 to June 30, 2013, as the postintervention period. We randomly selected 500 general SSCs and 10 SSCs for cesarean section performed during the postintroduction period for quality control. These SSCs were chosen by one of us (I. S.-C.) blinded to the date and type of surgery.
We extracted data from the official database of the local government and classified the surgical procedures according to the 2007 file of the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM).15 Patients undergoing elective and emergency surgical procedures with ICD-9-CM codes 01 to 34 and 40 to 86 at the CHB were eligible. Cardiac (ICD-9-CM codes 35-37) and diagnostic (ICD-9-CM codes 87-99) procedures constituted exclusion criteria. We further eliminated procedures performed outside the operating room (ie, in the radiology and gastroenterology departments) because the SSC was not used in these departments during the study period. The procedures were coded as proposed by the National Healthcare Safety Network16 and grouped into various categories of surgical procedures (eTable 1 in the Supplement). Data were collected from October 23, 2013, to November 12, 2014.
We assessed 30- and 90-day all-cause mortality rates by matching the date of the surgical intervention with the local register of mortality. We performed 2 different analyses of mortality that considered the first or the last surgical procedure as reference in the case of multiple procedures. For the evaluation of secondary study aims, we analyzed the first operation in the case of multiple procedures.
We defined LOS as the days from the date of admission to the date of discharge. The readmission rate was judged by calculating the admissions for short-term treatment in a public health care facility of each included patient within 30 days after the original discharge from a public health care facility. For this analysis, we excluded transfers within a facility and admissions for rehabilitation, psychiatric therapy, radiotherapy or chemotherapy, dialysis, or pregnancy and/or labor.
We defined a priori elective (planned admission) and emergent (urgent admission) surgery, patient age class, and patient sex as covariates. We classified the surgical procedures for their risks and confounding factors similarly to classification in previous research.8 We further considered the clinical risk factors oncologic and cardiovascular comorbidity and diabetes mellitus, whose criteria had to be fulfilled by data obtained from official South Tyrolean registries and databases.
Data were analyzed from September 17, 2014, to July 31, 2015. Statistical evaluations were performed with SAS software (Enterprise Guide 4.3; SAS Institute, Inc). We used multiple logistic regressions and generalized linear models for the analysis of the impact of the introduction of the SSC. Poisson generalized estimating equation models served to estimate LOS; logistic regression was used for mortality and readmission analysis. The final effect of the introduction of the SSC in the subgroups of age, sex, type of admission, type of procedure, and comorbidity was calculated using a planned contrasts analysis.
For estimation of the propensity score, we used a logistic regression model with preintervention and postintervention as the outcome variables and the baseline characteristics (admission category, procedure type, age, sex, and comorbidity) as explanatory variables. We weighted the entire study sample by inverse probability of treatment derived from the propensity score (eTable 2 in the Supplement). We estimated the reduction in the probability of mortality and LOS using the method described by Lunceford and Davidian.17
Patient characteristics and the details of admission were depicted using descriptive statistics (frequencies and percentages, means [SDs], and 95% CIs) as appropriate. We used the 2-tailed t test, simple logistic regression, or analysis of variance to determine whether the means were significantly different between groups. For all analyses, significance was set at P < .05 using 2-sided tests.
The study sample included 10 741 patients (5093 [47.4%] male and 5648 [52.6%] female patients; mean [SD] age, 53.0 [23.0] years), with 5444 before and 5297 after the intervention. Patients in the postintervention group were older, were more frequently male, and had a higher prevalence of oncologic and cardiac diseases (Table 1). We found no significant difference with regard to the type of recovery (elective or urgent) or prevalence of diabetes mellitus. The surgical procedures differed significantly between groups, mainly owing to an increase in vascular, thoracic, and cranial procedures (Table 1).
A total of 340 patients in the preintervention group and 348 patients in the postintervention group underwent multiple procedures. A total of 447 of the 500 SSCs (89.4%) selected for quality control were filled out completely and free of errors in process flow. Ten SSCs for cesarean section chosen for quality control were correct.
Raw data revealed a 90-day all-cause mortality of 2.37% (129 of 5444 patients) before implementation of the SSC compared with 2.23% (118 of 5297 patients) after introduction of the SSC, respectively (P = .62). The 30-day all-cause mortality rate was 1.36% (74 of 5444 patients) before and 1.32% (70 of 5297 patients) after implementation of the SSC (P = .87). We found no significant differences in the 30-day readmission rate or in LOS (Table 2) between preintervention and postintervention groups. Data after univariate analysis are depicted in eTables 3 to 6 in the Supplement, whereas eTables 7 to 10 in the Supplement depict the data after multivariate analysis.
The introduction of the SSC was associated with a significant reduction of 90-day all-cause mortality (adjusted odds ratio [AOR], 0.73; 95% CI, 0.56-0.96; P = .02) (Table 3 and Figure 1). In addition, the implementation of the SSC was not associated with a reduction of 30-day all-cause mortality (AOR, 0.79; 95% CI, 0.56-1.11; P = .17) (Table 3 and Figure 2).
These data on mortality did not change when a second procedure was considered determinant in case of repeated surgical procedures. We found no significant difference in risk for death after 30 days (AOR, 0.75; 95% CI, 0.53-1.05; P = .09) after the introduction of the SSC, whereas the risk for death after 90 days was significantly lower in the postimplementation period (AOR, 0.69; 95% CI, 0.53-0.90; P < .01).
The 30-day readmission rate (AOR, 0.90; 95% CI, 0.81-1.01; P = .08) did not change after implementation of the SSC (eTable 11 in the Supplement). The mean adjusted LOS was significantly lower after the intervention at 9.6 (95% CI, 9.4-9.7) days compared with 10.4 (95% CI, 10.3-10.6) days before the intervention (P < .001) (eTable 12 in the Supplement).
We observed a 27% reduction of risk-adjusted all-cause 90-day mortality after implementation of SSCs. Adjusted all-cause 30-day mortality remained unchanged. The introduction of SSCs was associated with a reduction in LOS but not of 30-day readmission rate.
To our knowledge, this report is the first on the association of SSCs and 90-day all-cause mortality, which might be even more important than 30-day all-cause mortality. Thirty-day all-cause mortality might fail to capture intermediate-term complications, such as anastomosis leakage or pulmonary embolism, which occur despite prophylaxis late after trauma18 or genitourinary and general surgery.19 Ninety-day all-cause mortality included predominantly surgical complications after colectomy, whereas deaths related to the underlying oncologic disease occurred later.10 Patients 85 years or older had an elevated 90-day but not 30-day all-cause mortality after radical cystectomy.12,20 Ninety-day all-cause mortality almost doubled 30-day all-cause mortality after pancreatectomy,11 cancer surgery,21 lung resection,22 and renal mass resection.23 Moreover, 30-day all-cause mortality or in-hospital mortality might depend on discharge practice,21 an important aspect because 23.6% of the deaths within 30 days occurred after hospital discharge in the United States.9
Reduction of mortality, the most important outcome, is difficult to achieve in areas with a high standard of care. In-hospital mortality dropped from 1.5% to 0.8% (P = .003) in the original trial on the effectiveness of the WHO SSC,1 but no significant reduction of mortality in high-income countries was noted (0.9% to 0.6%; P = .18). The investigators of the Keystone Surgery program,24 an intensively coached quality improvement program focusing on infectious complications, might have failed to demonstrate a reduction of mortality because they analyzed only 30-day and not 90-day all-cause mortality. A stepped wedge cluster-randomized clinical trial reported a reduction of the complication rate and LOS of surgical patients in the 2 participating hospitals after the introduction of an SSC, but in-hospital mortality decreased in only 1 facility.25 Like other investigators,24,25 we failed to observe reduced 30-day all-cause mortality, which is mainly caused by cardiopulmonary or renal complications. The successful implementation of interdisciplinary guidelines, especially on the perioperative management of cardiopulmonary risks, in countries with a high standard of care might explain this aspect.
The SURPASS checklist7 addresses more aspects of perioperative safety and might have therefore produced a substantial improvement of in-hospital mortality in the Netherlands. Baseline characteristics differed between various trials. We included patients with same-day surgery, whereas the SURPASS trial analyzed only inpatients, who constituted a higher-risk class. An analysis on SSC implementation among cases enrolled in the European Surgical Outcome Study2 reported an association of SSC use and a reduced risk for 60-day in-hospital mortality26 without further specifying the checklists and their implementation. However, that study was limited to patients older than 16 years and excluded patients undergoing same-day and obstetric procedures, cardiac surgery, or neurosurgery. Two systematic reviews27,28 published before the Ontario study8 concluded that the introduction of SSCs was associated with a reduced incidence of complications. Only 1 study28 reported a significant decline of in-hospital mortality, but notably both studies27,28 excluded the SURPASS trial7 from analysis. Like other investigators,1,7,25,29 we accompanied the introduction of the checklist with a formal training of the staff, followed by a reporting of clinical performance. These aspects might explain the lack of decline in 30-day all-cause mortality and the risk for perioperative complications after the introduction of SSCs in Ontario,8 where a guided implementation was not obligatory. Compliance in the execution of SSCs, often requiring a strategic multifactorial approach,27,30 is difficult to achieve,29,31,32 but it is crucial for the success of the measure.33 In the trial by van Klei et al,33 adjusted 30-day all-cause mortality decreased after implementation of the SSC among 25 513 adult patients scheduled for inpatient surgery (AOR, 0.85; 95% CI, 0.73-0.98), an effect that strongly depended on SSC compliance. Kwok et al34 observed a reduced complication rate after the introduction of the WHO SSC and pulse oximetry in a Moldovan hospital. The high completion rate during the study period increased 2 years after implementation, whereas the complication rate and incidence of surgical site infections continued to decrease.35 Thirty-day all-cause mortality remained constant after long-term follow-up compared with the period immediately after the implementation.35 Surgical safely checklists can prevent serious harm by identifying possible pitfalls during the process flow. The correct completion of the SURPASS system might have prevented 40% of deaths and 29% of the incidence in an analysis of 294 malpractice claims.36
The single-center design and the lack of a control group are limitations of our study. The study design highly reduces the risk for observation bias (Hawthorne effect). Furthermore, we did not inform the staff about the purpose of our study. We analyzed only objective outcome data to reduce reporting bias as much as possible. Contrary to most other studies, ours took into consideration the first or the last surgery as reference in case of multiple procedures. Our database does not distinguish between programmed repeated surgery (diagnostic procedure preceding the curative procedure) and reoperations for the treatment of complications.
Our database is more comprehensive than those analyzing in-hospital mortality because it includes the total number of deaths within our region (ie, includes those occurring after discharge or transfer to another hospital). We cannot exclude dropouts in the mortality rate if patients were not residents of South Tyrol, but this lack would have affected both groups. Moreover, the percentage of these patients would be considerably low owing to the local public health care system. Staff changes appeared only in case of natural fluctuation during the study period, and no changes in health care policies or hospital management have occurred. Secular trends in improvement of care might have interfered between groups. Owing to its design, our study demonstrates an association between the implementation of the SSCs and intermediate-term outcome variables without proof of causality.
The observed decline in LOS suggests potential cost savings after the implementation of SSCs. Further trials should address this hypothesis and the effect on quality of care owing to a reduction of the costs of complications37,38 or unplanned reoperations.38
The implementation of SSCs was associated with a 27% reduction of the adjusted risk for all-cause mortality within 90 days. The risk for all-cause mortality within 30 days remained unchanged. Adjusted LOS was reduced after implementation of SSCs.
Corresponding Author: Matthias Bock, MD, Priv Doz, Department of Anesthesiology and Intensive Care Medicine, Merano Hospital “Franz Tappeiner,” Via Rossini 5, 39012 Merano, Italy (email@example.com).
Accepted for Publication: November 19, 2015.
Published Online: February 3, 2016. doi:10.1001/jamasurg.2015.5490.
Author Contributions: Dr Bock and Mr Fanolla had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Bock, Pycha.
Acquisition, analysis, or interpretation of data: Fanolla, Segur-Cabanac, Auricchio, Melani, Girardi, Meier, Pycha.
Drafting of the manuscript: Bock, Segur-Cabanac, Pycha.
Critical revision of the manuscript for important intellectual content: Fanolla, Auricchio, Melani, Girardi, Meier, Pycha.
Statistical analysis: Bock, Fanolla, Melani, Girardi, Pycha.
Administrative, technical, or material support: Segur-Cabanac, Auricchio, Meier.
Study supervision: Bock, Auricchio, Girardi, Pycha.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported by the Public Health Care Company of South Tyrol, Italy, and by the Autonomous Province of Bolzano, Italy.
Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.