P values are shown when statistically significant compared with the previous time period after adjusting for propensity weighting; all other comparisons are P > .05. SSI indicates surgical site infection.
Kim RY, Kwakye G, Kwok AC, Baltaga R, Ciobanu G, Merry AF, Funk LM, Lipsitz SR, Gawande AA, Berry WR, Haynes AB. Sustainability and Long-term Effectiveness of the WHO Surgical Safety Checklist Combined With Pulse Oximetry in a Resource-Limited SettingTwo-Year Update From Moldova. JAMA Surg. 2015;150(5):473-479. doi:10.1001/jamasurg.2014.3848
Little is known about the sustainability and long-term effect of surgical safety checklists when implemented in resource-limited settings. A previous study demonstrated the marked, short-term effect of a structured hospital-wide implementation of a surgical safety checklist in Moldova, a lower–middle-income country, as have studies in other low-resource settings.
To assess the long-term reduction in perioperative harm following the introduction of a checklist-based surgical quality improvement program in a resource-limited setting and to understand the long-term effects of such programs.
Design, Setting, and Participants
Twenty months after the initial implementation of a surgical safety checklist and the provision of pulse oximetry at a referral hospital in Moldova, a lower–middle-income, resource-limited country in Eastern Europe, we conducted a prospective study of perioperative care and outcomes of 637 consecutive patients undergoing noncardiac surgery (the long-term follow-up group), and we compared the findings with those from 2106 patients who underwent surgery shortly after implementation (the short-term follow-up group). Preintervention data were collected from March to July 2010. Data collection during the short-term follow-up period was performed from October 2010 to January 2011, beginning 1 month after the implementation of the launch period. Data collection during the long-term follow-up period took place from May 25 to July 6, 2012, beginning 20 months after the initial intervention.
Main Outcomes and Measures
The primary end points of interest were surgical morbidity (ie, the complication rate), adherence to safety process measures, and frequency of hypoxemia.
Between the short- and long-term follow-up groups, the complication rate decreased 30.7% (P = .03). Surgical site infections decreased 40.4% (P = .05). The mean (SD) rate of completion of the checklist items increased from 88% (14%) in the short-term follow-up group to 92% (11%) in the long-term follow-up group (P < .001). The rate of hypoxemic events continued to decrease (from 8.1 events per 100 hours of oximetry for the short-term follow-up group to 6.8 events per 100 hours of oximetry for the long-term follow-up group; P = .10).
Conclusions and Relevance
Sustained use of the checklist was observed with continued improvements in process measures and reductions in 30-day surgical complications almost 2 years after a structured implementation effort that demonstrated marked, short-term reductions in harm. The sustained effect occurred despite the absence of continued oversight by the research team, indicating the important role that local leadership and local champions play in the success of quality improvement initiatives, especially in resource-limited settings.
Team-based surgical quality improvement innovations such as the World Health Organization (WHO) Surgical Safety Checklist1 have been shown to be effective in decreasing surgical complication and mortality rates in diverse settings around the world.2- 5 A relatively cost-effective tool, checklists are structured to facilitate communication and teamwork while ensuring the consistent application of internationally accepted standards of perioperative care, enabling a systems-based approach to the safe delivery of surgical services. This structure is likely beneficial for the provision of care in resource-limited settings when implemented effectively. Although multiple studies2,3,5 have shown a short-term reduction in patient harm following the implementation of a checklist, little is known regarding the most effective and sustainable method of implementing such interventions, especially in resource-limited settings. A recent study6 from 2 hospitals in Liberia suggests that the WHO checklist can be implemented with demonstrable reductions in surgical site infections and overall complication rates. Concurrently, a growing body of literature suggests that haphazard implementation can hamper the meaningful and durable use of a checklist and ultimately lessen the clinical effect on patient safety.4,7- 9 However, to our knowledge, very little is known about how these checklists are used beyond the limited short-term implementation period, leaving the sustainability of this team-based intervention unclear.
In many environments, the safe delivery of surgical care can be affected by a lack of basic resources, such as pulse oximeters or antibiotics, both of which are required as part of the international standards of surgical safety and are included in the WHO checklist.10 In 2010, our group participated in the implementation of the WHO Surgical Safety Checklist in a Moldovan hospital to help reduce patient harm following surgery.11 In structuring this program, we attempted to ameliorate the deficiencies found in this resource-limited setting by providing pulse oximeters as part of our efforts. In the short-term follow-up, we found a 60.0% decrease in postsurgical complications after the checklist was implemented, while the number of hypoxemic events per l00 hours of oximetry decreased by 35.7%. This follow-up analysis seeks to understand the degree to which the use of the checklist and the continuous monitoring of oxygen saturation have been incorporated into routine perioperative practice, while assessing whether the short-term effects on patient outcomes have been sustained.
We conducted a follow-up study of mortality and complication rates 20 months after the implementation of the WHO Surgical Safety Checklist with the provision of pulse oximetry at the National Scientific and Practical Centre of Emergency Medicine Hospital, a state-owned, university-affiliated, general and trauma referral hospital in Chişinău, Moldova. According to the World Bank, Moldova is classified as a lower–middle-income country.11,12 This 630-bed hospital has the capacity to perform approximately 600 to 700 surgical procedures monthly using 22 operating stations located in 12 operating rooms (several rooms have multiple operating stations). Surgical subspecialties include general, gynecologic, neurologic, ophthalmologic, oral-maxillofacial, and orthopedic surgery.
The short-term results of the initial implementation of the hospital-wide WHO checklist with the provision of pulse oximetry have been described elsewhere.11 In brief, this was a before-after intervention study consisting of a 1-month preintervention period during which a local multidisciplinary implementation team was formed and trained to use a checklist and to perform oximetry. Only 3 of 22 operating stations (13.6%) had a functioning pulse oximeter prior to the intervention; accordingly, a pulse oximeter (model 7500; Nonin Medical Inc) was installed at each operating station as part of the implementation effort. We collected time-stamped oxygen saturation data every 30 seconds when the oximeters were turned on and when the sensors detected an adequate plethysmographic signal.
Ethics committee approval was sought from and granted by the Harvard School of Public Health in Boston, Massachusetts, and by the State University of Medicine and Pharmacy “Nicolae Testemitanu” in Chişinău, Moldova. The members of the surgical teams working at the hospital were enrolled in our study after giving oral informed consent.
The local team introduced the checklist and pulse oximeters to the operating room staff over a week-long launch period, with their gradual introduction into all operating rooms based on a rolling schedule. A data collection team was created and trained to conduct medical record reviews in order to accurately identify 30-day postoperative complications. Data collectors were assigned randomly to observe 30% of the cases and to collect process adherence measures. Preintervention data were collected from March to July 2010. Data collection during the short-term follow-up period was performed from October 2010 to January 2011, beginning 1 month after the implementation of the launch period.
Data collection during the long-term follow-up period was performed by the same personnel trained during the initial implementation and took place from May 25 to July 6, 2012, beginning 20 months after the initial intervention. Data collectors were able to collect intraoperative data and data on 30-day postoperative inpatient complications by reviewing the medical records of the patients. The standardized intraoperative data forms contained measures of process adherence that were completed by health care professionals trained to administer local anesthesia. All consecutive patients who underwent noncardiac surgery at the hospital during this period were enrolled. Both deidentified clinical data and downloaded oximetry data were electronically transferred to the Boston-based research team every 2 to 4 weeks.
The primary outcomes of interest were (1) intraoperative adherence to surgical safety process measures, (2) rate of postoperative inpatient complication or mortality within 30 days of the procedure, and (3) frequency of intraoperative hypoxemic events. Surgical safety process measures included (1) performance and documentation of an objective airway examination, (2) presence of a pulse oximeter for the duration of the surgical procedure, (3) administration of prophylactic antibiotics within 60 minutes prior to the surgical incision in cases without preexisting infection, (4) verbal confirmation of the surgical site by all members of the surgical team, and (5) completion of a sponge count at the end of the procedure. In addition, there were 4 indicators of good communication: a discussion regarding (1) the patient’s airway, (2) the readiness of the surgical team to proceed, (3) the anticipated blood loss and duration of the procedure, and (4) any patient-specific medical concerns by the surgical team. Completion of all 9 process measures was considered a proxy for adherence to the entirety of the checklist.
Data on 30-day postoperative complications, as defined by the American College of Surgeons National Surgical Quality Improvement Project, were collected.13 Pneumonia, sepsis, and surgical site infections were grouped under infectious complications. Other outcomes, such as unplanned intubation, cardiac arrest, cerebral vascular accident, and bleeding that required the transfusion of 4 or more units of red blood cells within the first 72 hours after surgery, were grouped under noninfectious complications. Any unanticipated surgical procedure occurring within the first 30 days of the hospital stay was considered an unplanned return to the operating room. Inpatient deaths within 30 days of the procedure were used to calculate mortality. Because some patients underwent multiple surgical procedures over the course of their hospital stay, we included only the initial procedure in our analysis and excluded subsequent procedures in order to avoid double counting of outcomes.
The oxygen saturation data were organized into “events” that were defined as groups of recordings for which the gap between 2 recordings was less than 5 minutes and for which the duration of the event was greater than 2 minutes. The cumulative hours of oximetry were the sum of the duration of the events. Low oxygen saturation levels were defined as groups of consecutive recordings for which the saturation level was less than 90% (hypoxemia) or less than 85% (severe hypoxemia) for 2 minutes or longer, and these definitions are commonly used in the existing literature.14,15 The numbers of hypoxemic and severe hypoxemic events lasting 2 minutes or longer per 100 hours of oximetry were then calculated.
Final data analysis was performed using SAS version 9.2 (SAS Institute Inc). With 500 surgical procedures, using an unadjusted χ2 test, we had 80% power (α = .05) to detect a 30% increase in compliance to all 5 process measures. A critical P value of .05 was used to determine statistical significance for all analyses. In addition, to detect a decrease in the number of hypoxemic events from 6.4 (as observed in the original study) to 3.4 per 100 hours of oximetry (as described in the literature from US hospitals), 871 hours of oxygen saturation data were required, assuming a Poisson distribution with a power of 80% and α = .05, using a 2-sample Wilcoxon rank sum test.
Unadjusted comparisons of process measures and outcomes were conducted using the Pearson χ2 test or the Fischer exact test. Adjusted comparisons were generated using propensity score weighting and a multivariate logistic regression model based on demographic covariates of age, sex, urgency of case, observation status, case mix, and type of anesthesia.16,17 Case mix was defined by surgical subspecialty (general surgery, obstetrics and gynecology, neurosurgery, ophthalmology and oral-maxillofacial surgery, and orthopedics) and assigned after review of the diagnosis and procedure. To avoid double counting of patients, only index cases were included in our analysis.
A total of 637 patients undergoing noncardiac surgery were included in the long-term follow-up group (during the period from May to July 2012). These patients were compared with 2106 patients who underwent surgery shortly after implementation in the short-term follow-up group (from October 2010 to January 2011). Compared with the short-term follow-up group, the long-term follow-up group consisted of a lower percentage of females (37.7% in the long-term follow-up group vs 42.1% in the short-term follow-up group; P = .05) and underwent more general surgical procedures (44.7% in the long-term follow-up group vs 38.9% in the short-term follow-up group; P = .01) and fewer orthopedic procedures (32.0% in the long-term follow-up group vs 36.7% in the short-term follow-up group; P = .03). The 2 groups were otherwise similar in other characteristics, such as mean age, mean number of urgent cases, and use of general anesthesia as listed in Table 1.
The overall rate of completion of the items on the checklist increased from 88% in the short-term follow-up group to 92% in the long-term follow-up group (P < .001) (Table 2). An improvement in both communication and safety indicators was observed in almost all intraoperative process measures collected. Patient-specific medical concerns were discussed 4.1% more often in the long-term follow-up group than in the short-term follow-up group (94.7% in the short-term follow-up vs 98.6% in the long-term follow-up group; P < .001). The rate of airway examination increased from 69.6% up to 92.6%, representing a 33% significant improvement in practice since the checklist was introduced (P < .001). Similarly, there was a 12.7% increase in the appropriate use of prophylactic antibiotics between the 2 groups (74.2% in the short-term follow-up group vs 83.6% in the long-term follow-up group; P < .001). However, the rate of communication regarding the patient’s airway decreased from 82.9% in the short-term follow-up group to 77.7% in the long-term follow-up group (P = .003). Adjusting for propensity weighting did not change the results obtained.
A further decrease in surgical complication rates was observed between the short- and long-term follow-up groups, with the rate of any surgical complication decreasing from 8.8% to 6.1% (P = .03) (Figure). Infectious complications decreased by 34.3% from the short-term to the long-term follow-up period (6.7% in the short-term follow-up group vs 4.4% in the long-term follow-up group; P = .03). This included a 40.4% decrease in surgical site infections (4.7% in the short-term follow-up group vs 2.8% in the long-term follow-up group; P = .05). However, the rate of other infectious complications such as pneumonia (2.6% in the short-term follow-up group vs 2.2% in the long-term follow-up group; P = .600) and sepsis (1.0% in the short-term follow-up group vs 0.5% in the long-term follow-up group; P = .33) remained stable. No significant change was found in the rate of noninfectious complications (1.5% in the short-term follow-up group vs 1.1% in the long-term follow-up group; P = .43) or in the rate of unexpected reoperations (1.5% in the short-term follow-up group vs 2.5% in the long-term follow-up group; P = .08). The rate of 30-day inpatient mortality also remained stable between both time periods (3.1% in short-term follow-up group vs 3.1% in the long-term follow-up group P = .99).
A total of 1391 cumulative hours of pulse oximetry recordings obtained during the 6-week long-term follow-up period were included in our analysis (231.8 hours per week). This was compared with the 4950 cumulative hours of pulse oximetry recordings obtained during the 24-week short-term follow-up period (206.3 hours per week). There was a 17% decrease observed in the rate of hypoxemic events (ie, having a peripheral capillary oxygen saturation level of <90%) per 100 hours lasting 2 minutes or longer between the 2 groups, but this did not reach statistical significance (P = .10) (Table 3).
The effect of a checklist-based intervention coupled with the introduction of pulse oximetry to monitor oxygen saturation persisted almost 2 years after implementation efforts were completed in a resource-limited setting. Changes in outcomes were not only sustained but continued to improve, with a 30.7% reduction in surgical complications between the short- and long-term follow-up periods. This had decreased by 59.1% between the preintervention and short-term follow-up in the study by Kwok et al.11 Surgical site infections, in particular, decreased by a total of 81.2% between the preintervention and long-term follow-up period, with a 68.5% significant decrease in the preintervention to short-term follow-up period11 and an additional 40.4% decrease in the long-term follow-up period. This correlated with sustained improvement in both intraoperative safety and communication process measures embedded within the checklist.
The continued improvement observed in both processes and outcomes can be attributed to several factors, including the effective implementation strategy.11 This plan was designed specifically for a low-income, resource-limited setting and included a month-long rollout of a hospital-wide checklist, the provision of functioning pulse oximeters for every operating room, and intraoperative coaching to ensure the effective use of the checklist. Success, as seen in other studies,6,18- 22 relied heavily on engaging local collaborators early in the implementation process and enabling local ownership of the work. This was crucial for gaining buy-in from the hospital leadership and clinical personnel. These local champions ultimately ensured the continued use of the checklist long after the initial implementation was completed.
To achieve local ownership of the process, steps were taken to incorporate checklists into everyday workflow. The Department of Anesthesia, for instance, used the checklist as a quality monitoring tool for the timely deliverance of prophylactic antibiotics and intraoperative anesthesia care and to provide a structured template for preanesthesia examinations. The medical records of all the patients were audited at hospital discharge to confirm the use of both an oral and written checklist for every operation. Staff members were also encouraged to raise issues of checklist use during periodic hospital meetings and to assist in developing solutions and modifications to the process. Because of the noticeable improvements in workflow and patient outcomes, the adoption of the checklist by the staff was universal and required minimal reinforcement from the hospital leadership.
The surgical teams in the most recent cohort completed a significantly higher number of items on the checklist compared with the short-term follow-up group (mean [SD] completion rate, 92% [11%] vs 88% [14%], respectively), despite having no oversight or active input from the research team. Lower rates of checklist completeness have been found in other studies, even in settings where checklist use was mandatory and a high compliance rate was reported.9,21 Fourcade et al,9 for instance, found a mean compliance rate of 90.2% but a mean completeness rate of only 61% in an analysis of 1440 surgical procedures at 18 hospitals in France. Our findings, as such, emphasize the important role trained local collaborators, leadership, and institutional support play in the successful implementation and continued sustainability of an intervention, especially in a resource-limited setting. They also indicate the significant change in culture that can occur when checklists are incorporated effectively into the everyday workflow and the sustained improvement in patient outcomes that can result.
The improvement in processes of care was not uniform; there was backsliding in some domains, such as the 6.3% decrease in the rate of communication regarding the patient’s airway. However, the rate of 30-day mortality remained at a lower level than baseline, which was similar between the 2 follow-up periods. This suggests that the significant reduction in mortality observed in other studies of checklist use is a sustained phenomenon.3,23 We saw a reduction in the frequency of hypoxemic events (ie, having a peripheral capillary oxygen saturation level of <90%) between the short- and long-term follow-up groups, but this did not reach statistical significance. A lack of adequate power in our study could account for the absence of detectable improvement, but the reduction from 11.5 to 6.4 hypoxemic events per 100 hours seen in the original study11 was clearly maintained.
Our study is limited by its single-center design and lack of a control hospital for comparison. We believe that the observations, although from a single institution, demonstrate the feasibility of integrating this intervention into the clinical routine, with sustained improvements in patient outcomes and safety process adherence. The study site is a major referral center, and there are no comparable Moldovan hospitals with equivalent data collection systems. As such, any changes observed in relation to the intervention could have been influenced by variables such as other interventions, changes in staff, or resource availability, which might have occurred over the same time. Routine measures of patient comorbidity, such as the American Society of Anesthesiologists class or other scores, are not routinely used in Moldova, and thus it is not possible to adjust for these factors in the analysis. We attempted to control for possible confounders by using propensity weights, but this technique is unable to account for all unmeasured variables. We believe, however, that the continued marked improvements observed, even in the absence of a control hospital, could not have been due to any other changes in practice instituted around the same period. There were no large-scale surgical quality improvement programs or significant staff turnovers at the hospital during the follow-up period. The reintroduction of observers into the operating rooms during the long-term follow-up study could also have resulted in the Hawthorne effect. However, the observers were the same individuals from the original study period, and we failed to observe an across-the-board improvement in outcomes or performance of all 9 proxy process measures on the checklist, which suggests that this effect was limited.
Another concern is that the provision of pulse oximeters could have resulted in a change that was independent of the checklist. It is impossible to distinguish the specific contributions of the checklist from that of the pulse oximeter owing to the inherent design of the data-recording oximeters used. This design allowed for analysis of overall cross sections or episodes of consecutive recordings but failed to provide any patient- or case-level data. We still believe that, given the integral role pulse oximeters play in the checklist and their recognition internationally as a surgical safety standard,10 studying them together with the checklist was ethically necessary. In addition, our data show that both interventions were used in a sustainable fashion, as evidenced by the hypoxemia data and the maintenance of safety-oriented processes of care, including measures of team communication unlikely to be affected by pulse oximeter provision alone.
Our study demonstrates that sustained use of the WHO Surgical Safety Checklist is possible in lower–middle-income, resource-limited countries when properly implemented in collaboration with local leadership. Continued use is associated with sustained or additional reduction in complication rates and improvement in process measures. Further studies, however, need to be conducted to understand the precise factors necessary to promote the checklist’s integration into care. This should be incorporated into implementation strategies of other quality improvement initiatives.
Accepted for Publication: September 30, 2014.
Corresponding Author: Alex B. Haynes, MD, MPH, Ariadne Labs, Codman Center for Clinical Effectiveness in Surgery, Department of Surgery, Massachusetts General Hospital, 401 Park Dr, 3rd Floor E, Boston, MA 02215 (firstname.lastname@example.org).
Published Online: March 25, 2015. doi:10.1001/jamasurg.2014.3848.
Author Contributions: Dr Haynes had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs Kim and Kwakye contributed equally to this article and should be regarded as joint first authors.
Study concept and design: Kim, Kwok, Ciobanu, Merry, Funk, Lipsitz, Gawande, Berry.
Acquisition, analysis, or interpretation of data: Kim, Kwakye, Baltaga, Ciobanu, Lipsitz, Gawande, Berry, Haynes.
Drafting of the manuscript: Kim, Kwakye, Ciobanu, Lipsitz, Berry, Haynes.
Critical revision of the manuscript for important intellectual content: Kwakye, Kwok, Baltaga, Merry, Funk, Lipsitz, Gawande, Berry, Haynes.
Statistical analysis: Kim, Kwakye, Lipsitz, Haynes.
Obtained funding: Gawande.
Administrative, technical, or material support: Kim, Baltaga, Ciobanu, Funk, Berry, Haynes.
Study supervision: Ciobanu, Gawande, Berry, Haynes.
Conflict of Interest Disclosures: Dr Gawande is on the board of Lifebox, a nonprofit organization that aims to provide pulse oximetry to operating rooms around the world that do not have them. He also receives royalties from publishers for books and essays on improvement of health care systems, including a book on the use of health care checklists. No other disclosures are reported.
Additional Information: Dr Baltaga was the principal investigator in Moldova.