A Novel Method for Reproducibly Measuring the Effects of Interventions to Improve Emotional Climate, Indices of Team Skills and Communication, and Threat to Patient Outcome in a High-Volume Thoracic Surgery Center

Objective  To create and test a reproducible method for measuring emotional climate, surgical team skills, and threats to patient outcome by conducting an observational study to assess the impact of a surgical team skills and communication improvement intervention on these measurements.

Design  Observational study.

Setting  Operating rooms in a high-volume thoracic surgery center from September 5, 2007, through June 30, 2008.

Participants  Thoracic surgery operating room teams.

Interventions  Two 90-minute team skills training sessions focused on findings from a standardized safety culture survey administered to all participants and highlighting positive and problematic aspects of team skills, communication, and leadership. The sessions created an interactive forum to educate team members on the importance of communication and to role-play optimal interactive and communication strategies.

Main Outcome Measures  Calculated indices of emotional climate, team skills, and threat to patient outcome.

Results  The calculated communication and team skills score improved from the preintervention to postintervention periods, but the improvement extinguished during the 3 months after the intervention (P < .001). The calculated threat-to-outcome score improved following the team training intervention and remained statistically improved 3 months later (P < .001).

Conclusions  Using a new method for measuring emotional climate, teamwork, and threats to patient outcome, we were able to determine that a teamwork training intervention can improve a calculated score of team skills and communication and decrease a calculated score of threats to patient outcome. However, the effect is only durable for threats to patient outcome.

Strategies for improving team skills and communication are thought to increase safety in highly reliable organizations, including operating room teams. Previous work suggests that the safety climate may affect the ability of high-performance teams to function optimally.^1,2 Like the safety climate, the “emotional climate” of work may have consequences for team performance. There are no objective data using reproducible methods to support this assertion.

Thoracic surgical procedures are complex and usually undertaken by a dedicated group of anesthesiologists, nurses, and surgeons. Thoracic operations cause extreme abnormalities in ventilation and perfusion relationships, and these predictable perturbations provide an opportunity to study team behavior in the setting of stressful and changing patient conditions.

We created a reproducible, objective method of measuring emotional climate, indices of communication and team skills, and threats to patient outcome. In addition, we conducted an observational study to assess the impact of a defined team skills and communication improvement intervention on these measurements in a pod of thoracic surgery operating rooms in a high-volume thoracic surgery center.

Our intervention consisted of two 90-minute team skills training sessions focused on findings from a standardized safety culture survey administered to all participants and highlighting positive and problematic aspects of team skills, communication, and leadership.³ The sessions created an interactive forum to educate team members on the importance of communication and to role-play interactive and communication strategies designed to optimize safety. This format has been used by patient safety specialists previously, but the effect of the intervention on emotional climate, behavior, and threat to outcomes was unknown and, to our knowledge, had never been reproducibly measured. However, team member degree of engagement and levels of team tension are known to affect operational function in a variety of settings from nuclear power to aviation.⁴ Our hypothesis was that improved team skills, communication, and emotional climate would correlate with fewer threats to patient outcome.

This study was conducted in 4 thoracic surgery operating rooms at the Brigham and Women's Hospital, Harvard Medical School, from September 5, 2007, through June 30, 2008.

Approval for the study was obtained from the Partners Healthcare Human Research Committee. Information sessions were held to update thoracic surgery anesthesiologists, nurses, and surgeons concerning the study objectives, risks, and benefits. The study was authorized to begin after 70% of eligible nursing staff, surgical technicians, attending thoracic surgeons, and anesthesiologists voluntarily opted in by completing a deidentified patient safety climate survey during organized information sessions. From that point forward, anyone not wishing to participate had the ability to opt out by informing the principal investigator (M.N.). No one opted out.

The primary aim of this study was to create a reproducible method for measuring emotional climate, communication and team skills, and threats to patient outcome. The secondary aim was to evaluate the impact of a standardized intervention aimed at improving team skills and communication.

This study was developed in consultation with a steering group consisting of 2 thoracic surgeons, 2 anesthesiologists, 2 nurses, and 3 safety specialists. Observers included 2 anesthesiologists (M.N. and P.S.), 2 nurses (including A.K.), and 3 safety specialists (including A.F.). Two surgeons were approved investigators and observers but did not participate in observations because of scheduling conflicts. A standardized observation form was used, with definitions of the specific emotional climate and behaviors printed on the back of each form (Figure 1 and Figure 2). Observations focused on behavioral interactions in 3 environments of the operating room: the surgical field including surgeons and scrub nurses, the anesthesia environment, and the circulating environment.

Before initiation of the project, pilot observations were performed to desensitize subjects to being observed and to develop and test the operability of behavioral definitions to be used in the study. Definitions were refined in consultation with observers and the project steering group. Analyzed communications and team skills were refined from a standardized observational template that the patient safety group had used for previous studies. Definitions of threats to outcome were based on information from a group at Kaiser Permanente involved in an ongoing study analyzing threats to outcome.⁵ Once definitions were finalized, 30 paired observations of thoracic surgical procedures were performed to ascertain whether interrater reliability and agreement were acceptable. One new observer (P.S.) joined the team midway through the study, and he was required to perform paired observations until his interrater reliability matched that of the core group.

A randomization list of operating rooms was used to decide where to observe. After randomization, a single observer entered the room and stood adjacent to the doorway in a position where he or she could observe the operating field, anesthesia environment, and circulating environment. Patient vital signs present on entry into the operating room were recorded from the overhead display along with other demographic data. After this, observers assessed the emotional climate as disengaged, engaged, appropriately tense, or inappropriately tense in each of the 3 environments.

Observers then rated 6 behavioral team skills elements across all 3 environments as adequate, mediocre, or expected but not observed. Each instance of the behavior was recorded as a tick mark. Observation periods lasted 10 minutes. At the end of an observation period, observers identified any threats to outcome seen during the previous 10 minutes. These included adequacy of staffing, physical environment supportive of staff, readiness of equipment, availability of relevant information, interruptions, supervision, fatigue, communication with other services, and handoffs.

Data sheets were changed at the end of every 10-minute observation period or if the emotional climate changed and remained changed for 1 minute. A new set of patient vital signs was obtained within the first minute of using a new data sheet. All deidentified data were entered into the project database.

Data were collected in the following 3 periods: preintervention (11 consecutive weeks after the beginning of the study), postintervention (12 consecutive weeks after a training intervention), and a sustaining period. The intervention consisted of two 90-minute multidisciplinary team skills training sessions for surgeons, anesthesiologists, technicians, and nurses, during which a team training expert (A.F.) created an interactive forum to educate team members on the importance of communication and to role-play interactive and communication strategies designed to optimize safety. The session included a discussion of findings from a standardized safety culture survey administered to all participants and highlighting positive and problematic aspects of team skills, communication, and leadership.³ This standardized safety culture survey was administered before the implementation of the study and again at the study completion.

Scoring of different elements was as follows. Patient vital signs were categorized as abnormal if the mean arterial pressure was less than 55 mm Hg, the heart rate was more than 100 or less than 49 beats/min, or the oxygen saturation level was less than 88%. Communication and team skills were scored for each data sheet by weighting elements that were expected but not observed as 0, mediocre as 0.5, and adequate as 1; a weighted average for each skill was calculated, and these skill scores were then averaged together to calculate communication and team skills scores. A threat-to-outcome score was calculated as the total number of threats marked on each data sheet.

Interrater reliability and agreement for the 30 initial paired observations are reported in Table 1 and Table 2, respectively. The κ coefficients were greater than 0.70 for all but 3 of the communication and team skills observations. Agreement on threats to outcome exceeded 90% with the exception of 1 element for which there was 75% agreement. A total of 305 data sheets (91 preintervention, 165 postintervention, and 49 sustaining) were collected during the study period, representing approximately 50 hours of observation.

There were statistically significant differences in patient age, sex, abnormal vital signs, and whether the attending anesthesiologist was in the room between the preintervention, postintervention, and sustaining observation periods (Table 3). There were no statistically significant differences in the presence of the attending surgeon in the room during observations (Table 3). Because of these differences, we calculated values adjusted for these variables and found similar results unless specified. Herein we report the unadjusted values. All patients observed were classified as having American Society of Anesthesiologists scores of 2 or 3.

There was a trend toward the surgical environment being more frequently engaged and less frequently disengaged or appropriately tense from the preintervention to postintervention to sustaining periods, but this did not reach statistical significance (P = .06) (Table 4).

The communication and team skills score improved from the preintervention period to the postintervention period but then returned to a level not statistically different from the preintervention period (Table 5) (P < .001). The threat-to-outcome score improved following the team training intervention and remained statistically improved from the preintervention score, although the score indicated a decrease in improvement from the immediate postintervention period (Table 6) (P < .001).

Not all elements of team behavior improved after the intervention (Table 5). Specifically, the intervention had no effect on the adequacy of observed briefings, debriefings, or structured problem solving. There was a trend toward increased adequacy of verbal knowledge sharing after the intervention, but this did not reach statistical significance.

Not all elements of threats to outcome improved following the intervention (Table 6). Specifically, there was no improvement in the physical environment supporting staff, handoffs being comprehensive, or an event manager being established in crises. There was a trend toward improved staffing levels supporting safe care.

To our knowledge, this is the first study to develop, define, and use reproducible objective methods to measure emotional climate, team skills and communication, and threats to patient outcome. We demonstrated that an intervention focused on teamwork and communication improved a calculated score of team skills and communication and decreased a calculated score of threats to patient outcome. The durability of this effect was only observed for the calculated score of threats to patient outcome. In addition, we observed that the intervention was associated with a trend toward a more appropriately engaged surgical environment. This is the first study of which we are aware that has demonstrated a relationship between observational measurements of emotional climate and team skills behaviors known to be manifest in high-reliability organizations and an assessment of a calculated score of threats to patient outcome.

The team skills training intervention improved the team skills and communication score, and then this effect extinguished as the period of training became more remote. The threat-to-outcome score remained statistically improved during the sustaining period of the study but at a lower level, suggesting that this effect may have also extinguished had the study continued longer. These data suggest that ongoing team skills training might sustain improvements in communication indices.

We observed a trend toward the surgical environment being classified as more engaged after the team training intervention; however, this did not reach statistical significance. Previous data have shown relationships between emotional environment and a team's ability to perform. Bowles et al⁶ demonstrated that perceived stress reported by flight crews in simulated emergencies was lower in high-performance crews than in low- and medium-performance crews. The study suggested that the personality style of the captain affected crew member performance, with captains demonstrating an active, warm, confident, competitive personality commanding the most highly performing crews. Lingard et al⁷ demonstrated the negative impact of tension on other operating room team members.

We used threats to outcome as a presumptive surrogate for patient outcome. Other authors have looked directly at the relationship between operating room behavior and outcome. Carthey et al⁸ demonstrated that in a group of 16 pediatric cardiac surgeons, those with the best outcomes were more likely to demonstrate behavioral markers of excellence seen commonly in high-reliability organizations. Mazzocco et al⁵ demonstrated that less frequent briefings during handoff periods and less frequent information sharing during intraoperative and handoff periods increased the odds ratio for complication or death.

Another observational study by Christian et al⁹ in general surgery operating rooms demonstrated that communication breakdown had a negative influence on patient safety. The same group more recently showed that communication breakdown was a frequent feature of malpractice claims.¹⁰ Our data are consistent with this body of literature.

This study has multiple limitations. Definitions of emotional climate, communication and teamwork skills, threats to outcome, communication and teamwork skills score, and threat-to-outcome score were developed during this study, and the adequacy of these definitions has not been tested. The extent to which emotional climate or threats to patient outcome are directly observable has not been tested. Although surgeons did help refine the observation template during the pilot portion of the study in which they participated, there were no surgeon-observers during the formal portion of this investigation. Surgeons may have assessed the emotional climate and behaviors differently to nonsurgeon observers. We cannot exclude that individual behavior improved when observers were in the operating room. The study took place from the beginning to the end of an academic year, and we cannot exclude that observed improvements resulted from an increase in team skills by trainees. The indices of interrater reliability were not identical, and this may have affected our results. Although interrater reliability was assessed before the observations, it was never rechecked, and it is possible that during the course of the study interobserver variation became greater. Some of the behaviors taught during the team training were observed at a lower frequency, albeit often with greater adequacy when they did occur. This finding deserves further research to determine which behaviors in particular, and whether the adequacy or frequency of a behavior specifically, is most important for improving team performance and decreasing threats to patient outcome.

This study uses a new method for measuring emotional climate, indices of teamwork, and threats to patient outcome. A teamwork training intervention improved calculated indices of team skills and communication and decreased calculated indices of threats to outcome; however, the effect was only durable for calculated indices of threats to patient outcome. These data also suggest that team behaviors extinguish over time unless appropriately supported; this might include continuous highlighting during department meetings, open support by leadership, practice, role-play, and simulation. These and other methods to improve team skills and decrease threats to outcome should be prospectively tested using the methods described in the present study.

Correspondence: Michael Nurok, MD, PhD, Divisions of Thoracic Anesthesia and Surgical Critical Care, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115 (mnurok@partners.org).

Submitted for Publication: March 17, 2009; final revision received June 9, 2009; accepted June 14, 2009.

Author Contributions:Study concept and design: Nurok, Lipsitz, Kelly, and Frankel. Acquisition of data: Nurok, Satwicz, Kelly, and Frankel. Analysis and interpretation of data: Nurok, Lipsitz, Satwicz, and Frankel. Drafting of the manuscript: Nurok, Kelly, and Frankel. Critical revision of the manuscript for important intellectual content: Nurok, Lipsitz, Satwicz, and Frankel. Statistical analysis: Lipsitz. Obtained funding: Nurok and Frankel. Administrative, technical, and material support: Nurok, Lipsitz, Satwicz, and Kelly. Study supervision: Nurok, Kelly, and Frankel.

Financial Disclosure: None reported.

Funding/Support: This study was supported by Partners Healthcare.

Disclaimer: During this study, Dr Frankel was director of Patient Safety at Partners Healthcare. He is currently a principal at Pascal Metrics Inc; however, the methods used in this study are unlike any that are currently being used and have no bearing on the activity of Pascal Metrics Inc.

Additional Contributions: Raphael Bueno, MD, Margaret Doyle, RN, Philip Hartigan, MD, David Sugarbaker, MD, and their respective nursing, anesthesia, and surgery teams facilitated and participated in this study as subjects. Richard Adamski, RRT, MBA, designed the data sheets. Richard Adamski, RRT, MBA, Erin Graydon-Baker, RRT, and Linda A. Evans, RN, MSN, CNOR, participated in observations with Drs Nurok, Frankel, and Satwicz and Ms Kelly. Michael D’Ambra, MD, provided helpful comments on the manuscript.