Operating room checklist “whiteboard.” When the patient enters the room, white slider bars cover the green, exposing the red. As each step is completed, the slider is moved to the left, exposing the green. DVT indicates deep venous thrombosis.
Checklist use over time at Strong Memorial Hospital. Beginning in 2004, the scrub nurse was instructed not to hand up the knife until the checklist was completed.
Reports of incidents at Strong Memorial Hospital over time. This includes all levels of severity.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Sax HC, Browne P, Mayewski RJ, et al. Can Aviation-Based Team Training Elicit Sustainable Behavioral Change? Arch Surg. 2009;144(12):1133–1137. doi:10.1001/archsurg.2009.207
To quantify effects of aviation-based crew resource management training on patient safety–related behaviors and perceived personal empowerment.
Prospective study of checklist use, error self-reporting, and a 10-point safety empowerment survey after participation in a crew resource management training intervention.
Seven hundred twenty-two–bed university hospital; 247-bed affiliated community hospital.
There were 857 participants, the majority of whom were nurses (50%), followed by ancillary personnel (28%) and physicians (22%).
Main Outcome Measures
Preoperative checklist use over time; number and type of entries on a Web-based incident reporting system; and measurement of degree of empowerment (1-5 scale) on a 10-point survey of safety attitudes and actions given prior to, immediately after, and a minimum of 2 months after training.
Since 2003, 10 courses trained 857 participants in multiple disciplines. Preoperative checklist use rose (75% in 2003, 86% in 2004, 94% in 2005, 98% in 2006, and 100% in 2007). Self-initiated reports increased from 709 per quarter in 2002 to 1481 per quarter in 2008. The percentage of reports related to environment as opposed to actual events increased from 15.9% prior to training to 20.3% subsequently (P < .01). Perceived self-empowerment, creating a culture of safety, rose by an average of 0.5 point in all 10 realms immediately posttraining (mean [SD] rating, 3.0 [0.07] vs 3.5 [0.05]; P < .05). This was maintained after a minimum of 2 months. There was a trend toward a hierarchical effect with participants less comfortable confronting incompetence in a physician (mean [SD] rating, 3.1 [0.8]) than in nurses or technicians (mean [SD] rating, 3.4 [0.7] for both) (P>.05).
Crew resource management programs can influence personal behaviors and empowerment. Effects may take years to be ingrained into the culture.
In flying I have learned that carelessness and overconfidence are usually far more dangerous than deliberately accepted risks.—Wilbur Wright in a letter to his father, September 1900Aviation in itself is not inherently dangerous. But to an even greater degree than the sea, it is terribly unforgiving of any carelessness, incapacity or neglect.—Captain A. G. Lamplugh, British Aviation Insurance Group, London, circa early 1930s.Josie's death was not the fault of one doctor, or one nurse, or one misplaced decimal point; it was the result of a total breakdown in the system.—Sorelle King, mother of 18-month-old Josie King, who died at Johns Hopkins Hospital from medical error while recovering from burns
In flying I have learned that carelessness and overconfidence are usually far more dangerous than deliberately accepted risks.—Wilbur Wright in a letter to his father, September 1900
Aviation in itself is not inherently dangerous. But to an even greater degree than the sea, it is terribly unforgiving of any carelessness, incapacity or neglect.—Captain A. G. Lamplugh, British Aviation Insurance Group, London, circa early 1930s.
Josie's death was not the fault of one doctor, or one nurse, or one misplaced decimal point; it was the result of a total breakdown in the system.—Sorelle King, mother of 18-month-old Josie King, who died at Johns Hopkins Hospital from medical error while recovering from burns
Patient safety is now a central theme in American medicine. Brought to light by the Institute of Medicine report To Err Is Human: Building a Safer Health System,1 health care providers, hospitals, the government, and insurers have searched for answers to address issues leading to an unsafe environment for patients. Drawing on many of the analogies between medicine and aviation, programs have been developed based on aviation crew resource management (CRM) interventions.2-4 Developed in the late 1970s after the collision of two 747 airliners on a foggy runway in Tenerife, CRM focuses on both human and systems issues, improving communication, error management, and work culture.5 Although aviation accidents continue to occur, the overall rate of incidents has declined and commercial aviation is now the safest form of transportation on a per-mile basis. Over the last decade, medical institutions have contracted for CRM training for their staff, usually beginning in high-risk areas such as the emergency department, obstetrics/gynecology, and surgery.6,7 Because a successful program means that incidents do not happen, it is difficult to quantify objectively the effect of this training. We report on the results of a multidisciplinary CRM program instituted at both a tertiary care academic medical center and a medical school–affiliated community hospital. End points include checklist adoption and use, self-reporting of errors and unsafe conditions, and perception of personal and institutional empowerment to create a culture of safety.
Beginning in 2001, Strong Memorial Hospital, the University of Rochester's 722-bed teaching hospital, initiated multiple patient safety initiatives with the support of its malpractice carrier, MCIC Vermont. Since 2003, these included a regularly scheduled 6-hour course “Lessons from the Cockpit,” developed in consort by the chief medical officer, chief safety officer, nurses, anesthesiologists, a surgeon/general aviation pilot, and Indelta Learning Systems, LLC, an educational training company familiar with applying CRM concepts to nonaviation industries, such as refining and shipping. The course was multidisciplinary and highly interactive, using videos, team-building exercises, and open forums. To encourage and support attendance, there was no cost to the participant; physicians received Continuing Medical Education credits and a 5% discount on their malpractice premiums, and nurses and ancillary personnel received Certified Nurse Educator units as well as compensatory time. The course was held on a Saturday to avoid work conflicts.
A perioperative checklist was developed by the head nurse for cardiac surgery (Figure 1). It was modeled on preflight aviation checklists and designed to encourage team participation by listing all participants, as well as trapping “killer items,” including site and side, perioperative antibiotics, deep venous thrombosis prophylaxis, and β-blockade. Checklists were posted in each operating room and compliance monitored by the circulating nurse. During the initial rollout, there were no specific mandates for use, to assess whether training and awareness alone would change behavior.
Prior to institution of CRM training, Strong Memorial Hospital participated in a Web-based incident reporting system. Using a series of pull-down menus, any employee could report patient-related incidents as well as rate their severity from 1 (unsafe condition) to 9 (patient death). Reports were sent to an outside agency to encourage self-disclosure. Events of severity rated 7 or higher triggered an alert to hospital administration to institute prompt follow-up. Administration emphasized that self-reporting was to be used in a nonpunitive manner and reporting was viewed favorably during any root cause analysis discussions. This program was reviewed during the CRM course and comparisons drawn with the Aviation Safety Reporting System administered by NASA (http://asrs.arc.nasa.gov) to encourage participation in the hospital program. The number of reports was monitored over time, including the percentage of level 1 or 2 reports (unsafe condition/near miss), suggesting an awareness of the safety culture.
In 2005, one of us (H.C.S.) relocated to Providence, Rhode Island, and continued the program at The Miriam Hospital, a 247-bed community-based teaching hospital associated with The Warren Alpert Medical School of Brown University. Courses were open to any practitioner associated with Lifespan (the parent enterprise). A 10-point survey of empowerment (Table) was developed and administered immediately before (n = 276), immediately after (n = 242), and a minimum of 2 months postcourse (n = 140).
Checklist use and self-reporting numbers are presented over time. Comparison of report distribution before and after intervention was calculated via χ2 with Yates correction. For the empowerment survey, results are given as mean (SD) as determined by the Tukey Honestly Significant Difference procedure, maintaining α = .05.
In Rochester, New York, from 2003 through 2006, the course was held 6 times, training a total of 509 participants. In Providence, 349 people were trained during 4 courses. Some participants repeated the course on a yearly basis as a component of recurrency training, but this was not required. A small group of businessmen from local industries (Kodak, Bausch & Lomb, Xerox) also attended at the behest of the Greater Rochester Quality Council. In Providence, members of several hospital boards and major safety committees attended. The majority of participants were nurses (50%), followed by ancillary personnel (28%) and physicians (22%).
Checklists were placed in all operating rooms and their use was monitored by the circulating nurse. Not surprisingly, there was initial resistance because surgeons saw the checklists as speed bumps that hindered flow. The circulating nurse was then empowered to start the checklist process and the scrub nurse was instructed not to hand up the knife until the checklist was completed. Hospital administration and all clinical chiefs were broadly supportive of this process and any physician who was unwilling to participate was counseled. Use increased over time and many physicians became more supportive when the checklist caught an error, such as antibiotics not given or a specific piece of equipment not being available prior to the start of the case. Consistent checklist use rose from 75% in 2002 to 100% in 2007 and beyond (Figure 2).
Self-reporting of incidents rose from 709 per quarter in 2002 to 1481 per quarter in 2008 (Figure 3). There was an increase in willingness to report unsafe conditions or near misses (15.9% in 2002 and 2003 vs 20.3% in 2004 through 2008; P < .01). Nurses filed the majority of reports, followed by physicians and pharmacists.
During the period of 2005 to 2007, the course engendered an immediate training effect with significant increases in all 10 surveyed areas. The ease of confronting incompetence in others (questions 5-7) was lowest on entry into the course. Although significant improvement in empowerment occurred in all areas, a clear hierarchy was seen in perceived comfort in confronting incompetence in various specialties (mean [SD] rating, confronting a technician, 3.7 [0.8]; nurse, 3.3 [0.7]; and physician, 3.1 [0.7]). At a minimum of 2 months, all categories were maintained, with further empowerment in the realm of leadership (mean rating, precourse, 3.0; immediately postcourse, 3.4; and 2 months postcourse, 3.6; P < .05).
The institution of aviation-based CRM training altered behaviors, with acceptance of perioperative checklists and an increase in self-reporting, including reporting more unsafe conditions and near misses. The changes in personal empowerment toward patient safety actions were maintained, although there was less confidence in confronting those in higher positions.
The science of human error acknowledges the complex interaction of personal performance with systems factors and it is difficult to isolate the effect of a CRM program on overall behavior and safety. Strong Memorial Hospital and The Miriam Hospital/Lifespan, like many institutions that have emphasized patient safety, have multiple ongoing programs. These include executive safety walk rounds, risk management symposia, signage throughout the hospital addressing safety and communication issues, and buy-in from chairs of all the clinical departments. The commitment of the institution in providing these courses, as well as their multidisciplinary mix, helps break down barriers and encourage the elusive “culture of safety.” A successful institution holds individuals responsible for recklessness but encourages counseling and redesign for errors related to systems.8 Institution of the “Just Culture” into medicine will require a new level of trust and openness that has not been encouraged or supported under the current training and practice patterns.9
Like aviation, improvements in outcomes due to CRM training take time. Various end points can be followed, but some of the most obvious (lawsuits, unexpected death) are of such low frequency that statistical significance for an individual institution may not be met. It may well be that reduction in the more frequent, but costly, incidents, such as wound infection, deep venous thrombosis, or perioperative cardiac event will be the most easily measured. Along this line, a new Institute for Healthcare Improvement campaign “5 Million Lives”10 has been promulgated to recognize the magnitude of patients injured, but not killed, by medical error. The Centers for Medicare and Medicaid Services and other carriers have announced they will not pay for treatment of certain preventable complications. Many of these (wrong side surgery, deep venous thrombosis due to lack of prophylaxis) can be prevented by the adherence to checklists and protocols. Another less tangible measure would be employee turnover. Although economics always play an issue in staff leaving an institution, exit interviews suggest that the work environment plays a large role. Employees who feel empowered and valued as a part of the team are more likely to remain actively engaged.11 Both Strong Memorial Hospital and The Miriam Hospital have achieved Magnet Designation for nursing, suggesting a workforce committed to professional development and management emphasis on multidisciplinary communication with a willingness to report and act on unsafe conditions.12
Motivating physicians to engage in self-examination and personal growth can be challenging. Medicine favors individuals who have succeeded in a competitive, “solo” environment, not being rewarded for good communication skills and team dynamics.13 In a study of pilots and physicians, Sexton et al14 showed that pilots were least likely to deny the effects of fatigue on performance (26% for pilots vs 70% of a group of consultant surgeons in Europe). Even greater disparities were seen in perception of teamwork, with 73% of surgeons, but only 25% of nurses or anesthesiologists, reporting high levels of teamwork in the operating room.
This lack of awareness and a long-standing individualist attitude make institution of new programs difficult. Initial introduction of CRM into the aviation environment met with similar resistance. Over the last 3 decades, however, CRM has been integrated into the culture and personnel are evaluated and rewarded on their ability to participate as a team. Although pilots and medical providers are similar in many respects, the more rapid assimilation into aviation is one of self-preservation (“The pilot is the first to arrive at the scene of a fatal aviation accident”).15 With the greater diversification of medical student demographics, and the addition of patient safety to medical and nursing school curriculums, future generations of providers will likely view the skills taught by CRM training as an expected component of their professional lives.
The use of checklists has been helpful in compliance with the Medicare Surgical Care Improvement Project criteria including documenting prophylactic antibiotic dosing and timing, deep venous thrombosis prophylaxis, and β-blockade use. In addition to public reporting comparing hospitals, compliance will have a direct effect on pay for performance calculations and will provide financial incentive for adoption. Checklist use has gained additional support with the World Health Organization initiative Safe Surgery Saves Lives (www.who.int/patientsafety/safesurgery/en/index.html). A basic perioperative checklist has been developed with 3 components: “sign in”; “time out”; and “sign out.” The use encourages communications and collaboration among all team members and aids compliance with best practices for prophylactic antibiotics and site and side confirmation. Use of this checklist has caught errors before they reached the patient.16
Other indirect measures of successful systems change would be reductions in the observed to expected ratio for morbidity and mortality in standardized databases such as the National Surgical Quality Improvement Project, Society of Thoracic Surgeons Cardiac Database, or trauma registries. We also view an increase in self-reporting as a positive response and noted an increase in the slope of numbers of reports after institution of CRM training. Although this may be a normal response to the institution of a self-reporting program, the initial growth prior to training was slow.
Our study has several weaknesses. The measure of outcomes, such as self-reporting, could be influenced by other safety initiatives. The survey was locally developed and was not validated. Survey participation, even though done at the course, was only 80%, with 40% at 2 months posttraining. Individuals were not tracked longitudinally and the groups were heterogeneous. Any study with immediate surveys before and after will show a training effect. Grogan et al17 also confirmed an immediate favorable response to CRM training in 489 participants; however, there was no long-term follow-up. Our long-term follow-up was only a minimum of 2 months, and we are currently collecting 1-year data. The majority of participants were nurses, making broad application to the effect on physicians less conclusive. Subgroup analysis was not possible because of this fact.
It would also be naive to believe that surgeons will blindly accept dictums from management to follow specific protocols, although there may be a Hawthorne effect if they are aware of monitoring. Buy-in and support from management is critical, yet surgeons do not become believers until the checklist catches an unsafe condition or makes available equipment that expedites the procedure. In successful organizations, there needs to be a cheerleader, usually a well-respected physician who both “talks the talk” and “walks the walk.” Safety must be foremost during meetings of the hospital board,18 and a systems approach to morbidity and mortality conferences, including multiple disciplines, sets the tone for collaboration. The importance of support by malpractice carriers in the form of premium rebates and a buy-in from top management cannot be understated.19
In summary, the introduction of CRM training, combined with other initiatives, enhances personal commitment to patient safety and appears to alter behaviors relative to checklist use and self-reporting. Participants become aware of, and empowered, by these tools. Leadership of institutions must strive to foster the elusive “culture of safety” by creating an environment that focuses on systems issues as opposed to individual blame, maintains personal accountability, and encourages open communication in a supportive environment across all disciplines.8,14 Tragedies like that of Josie King do not have to occur.
Correspondence: Harry C. Sax, MD, The Warren Alpert Medical School of Brown University, The Miriam Hospital, 164 Summit Ave, Providence, RI 02906 (firstname.lastname@example.org).
Accepted for Publication: December 18, 2008.
Author Contributions:Study concept and design: Sax, Panzer, Hittner, and Coletta. Acquisition of data: Browne, Panzer, and Burke. Analysis and interpretation of data: Sax and Mayewski. Drafting of the manuscript: Sax and Panzer. Critical revision of the manuscript for important intellectual content: Sax, Browne, Mayewski, Hittner, Burke, and Coletta. Obtained funding: Burke. Administrative, technical, and material support: Sax, Browne, Mayewski, Panzer, Hittner, and Coletta. Study supervision: Sax.
Financial Disclosure: Dr Sax serves as a medical consultant to Indelta Learning Systems, LLC. Dr Sax has received compensation for course instruction outside of University of Rochester Medical Center and Lifespan. Mr Browne is an employee of Indelta Learning Systems, LLC.
Previous Presentation: This paper was presented in part at the Second Annual Academic Surgical Congress of the Society of University Surgeons; February 8, 2007, Phoenix, Arizona; and the 89th meeting of the New England Surgical Society; September 27, 2008; Boston, Massachusetts.
Additional Contributions: Nancy J. Coelho assisted in preparing the manuscript.