Figure 1. Timeline of the Team Strategies and Tools to Enhance Performance and Patient Safety curriculum.
Figure 2. Self-evaluation scores by competency.
Figure 3. Aggregate performance ratings before and after intervention.
Figure 4. Score increase by competency for Team Strategies and Tools to Enhance Performance and Patient Safety.
Figure 5. Score increase by competency for the nontechnical skills evaluation instrument.
Meier AH, Boehler ML, McDowell CM, Schwind C, Markwell S, Roberts NK, Sanfey H. A Surgical Simulation Curriculum for Senior Medical Students Based on TeamSTEPPS. Arch Surg. 2012;147(8):761-766. doi:10.1001/archsurg.2012.1340
Author Affiliations: Departments of Surgery (Drs Meier, McDowell, and Sanfey, Mss Boehler and Schwind, and Mr Markwell) and Medical Education (Dr Roberts), Southern Illinois University School of Medicine, Springfield. Dr Meier is now with the Division of Pediatric Surgery, Department of Surgery, SUNY Upstate Medical University, Syracuse, New York.
Objective To investigate whether the existing Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) curriculum can effectively teach senior medical students team skills.
Design Single-group preintervention and postintervention study.
Setting and Intervention We integrated a TeamSTEPPS module into our existing resident readiness elective. The curriculum included interactive didactic sessions, discussion groups, role-plays, and videotaped immersive simulation scenarios.
Main Outcome Measures Improvement of self-assessment scores, multiple-choice examination scores, and performance ratings of videotaped simulation scenarios before and after intervention. The videos were rated by masked reviewers on the basis of a global rating instrument (TeamSTEPPS) and a more detailed nontechnical skills evaluation tool(NOTECHS).
Participants Seventeen students participated and completed the study.
Results The self-evaluation scores improved from 12.76 to 16.06 (P < .001). The increase was significant for all of the TeamSTEPPS competencies and highest for leadership skills (from 2.2 to 3.2; P < .001). The multiple-choice score rose from 84.9% to 94.1% (P < .01). The postintervention video ratings were significantly higher for both instruments (TeamSTEPPS, from 2.99 to 3.56; P < .01; and NOTECHS, from 4.07 to 4.59; P < .001).
Conclusions The curriculum led to improved self-evaluation and multiple-choice scores as well as improved team skills during simulated immersive patient encounters. The TeamSTEPPS framework may be suitable for teaching medical students teamwork concepts and improving their competencies. Larger studies using this framework should be considered to further evaluate the generalizability of our results and the effectiveness of TeamSTEPPS for medical students.
The delivery of health care has changed dramatically during the past 2 decades. The individualistic physician-nurse-patient relationship has in many instances been replaced by a multidisciplinary and multiprofessional team approach to patient care in an effort to deal with the increasingly complex health care environment.1 There is good evidence that well-functioning teamwork improves patient safety.2- 4 It has also been demonstrated that underdeveloped team skills of health care providers introduce risk for error and poor patient outcome.5,6 Therefore, the teaching of team-based competencies for surgical trainees has recently received increasing attention, as demonstrated by the development of the American College of Surgeons/Association of Program Directors in Surgery skills curriculum phase III.7 Surgical residents are expected to function as effective team members as soon as they start their postgraduate training. Most medical schools, however, currently do not provide any formal training in this domain. It is therefore important to advance strategies that provide students with these necessary team skills before graduation.
Some effort has been made to develop standardized curricula for health care providers working in teams. One example is the Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) program.8 It was developed by the US Department of Defense in collaboration with the Agency for Healthcare Research and Quality.9 TeamSTEPPS is based on 20 years' experience and lessons learned from high-reliability organizations, including military operations, aviation, emergency response services, and nuclear power industries. It focuses on knowledge, skills, and attitudes that have been shown to be important for highly functional teams.10 TeamSTEPPS provides a framework for organizations to implement structured processes in an effort to promote effective teamwork among health care workers and to improve patient safety. TeamSTEPPS interventions have been implemented by entire health care organizations in an effort to change organizational culture and by smaller work groups to improve team collaboration.11- 13 TeamSTEPPS also provides an “Essentials Course” that highlights key principles and concepts of teamwork in an abbreviated form and is suitable for smaller groups of learners. Educational materials for this course are available in print and on DVD at no cost.14 TeamSTEPPS defines team-based competencies in the following domains15:
competency to lead the team (leadership).
competency to remain aware of the environment (situation awareness).
competency to support all team members (mutual support).
competency to communicate clearly with team members (communication).
Despite solid evidence that TeamSTEPPS improves teamwork skills in the clinical environment, few data exist about whether this framework can be applied for medical students.
For many years, the Department of Surgery at the Southern Illinois University School of Medicine has been offering “residency readiness” electives for graduating medical students with interest in surgery and emergency medicine. The main purpose of these electives is to ease the transition between medical school and residency.16 In an effort to test the usefulness of TeamSTEPPS for medical students, we used these already existing electives and added an educational intervention based on the TeamSTEPPS Essentials Course.
The aim of our research protocol was to test the following hypotheses: (1) The TeamSTEPPS Essentials Course curriculum improves the knowledge of the participating medical students regarding the principals of teamwork. (2) The TeamSTEPPS Essentials Course curriculum improves the students' self assessment of their team-related competencies. (3) The TeamSTEPPS Essentials Course curriculum alters the behavior and improves the performance of the participating medical students to make them more effective team members.
All senior medical students taking the resident readiness electives for surgery or emergency medicine were eligible to participate in the study. Participation was voluntary, but no student opted out. The team skills curriculum used the published TeamSTEPPS Essentials Course materials that focus on leadership, situation awareness, mutual support, and communication. This educational content was delivered in multiple sessions throughout the elective. Interactive multimedia–enhanced lectures demonstrated good and bad team behaviors and their potential effect on patient outcomes. These lectures also introduced the students to specific TeamSTEPPS tools (eg, Situation-Background-Assessment-Recommendation [SBAR] and the 2-challenge rule) that they could subsequently practice in scheduled role-play activities and discussion groups. Because our goal was to assess the effectiveness of the existing TeamSTEPPS Essentials Course, we used these materials without any modification. The curriculum was structured identically for both electives and facilitated by the same faculty members (Figure 1).
To test our hypotheses, we implemented a single-group preintervention-postintervention study using the students as their own control subjects. We evaluated the educational impact of the intervention on the basis of the Kirkpatrick framework17,18 (Table) and focused on learning and performance change.
The students' team skills knowledge was tested using a multiple-choice examination before and after receiving the instructional module (Figure 1). This test is part of the TeamSTEPPS Essentials Course materials and has been validated previously for other learners but not for medical students.
To assess the students' self-evaluation of competency and their behavior in team situations, we used immersive simulator sessions. Each student participated in 1 simulation at the beginning of the readiness elective and 1 after completion of the intervention during the last week of the elective. Before the first simulation the students were oriented to the patient simulator and its functionality. We created 2 scenarios with similar but not identical content. Subsequently, each participant was scheduled individually to come to the simulation room. There, they were briefed that they were the resident covering the general surgery service. Shortly after this briefing they were told to come to the simulation room to see a patient with a postoperative problem. Each participant functioned as the team leader during the simulation and was joined by 2 embedded educators playing the scripted roles of team members. To simulate the patient we used iStan (Medical Education Technologies, Inc), with physiological parameters set to mimic the underlying clinical condition. A simulation staff member in the control room provided the patient's voice. All scenarios were reviewed and pilot tested before the study to make sure that the clinical content was appropriate for senior medical students and that the TeamSTEPPS principles could be applied during the simulation. All simulation sessions were videotaped and archived (Figure 1).
To minimize bias based on content, the scenarios were assigned randomly and no student went through the same scenario twice. The content was based on already existing modules from the American College of Surgeons e-learning website7 and a previously published curriculum by one of the educators (A.H.M.).19
To evaluate the impact of the curriculum, we compared the student's performance during the preintervention and postintervention simulations using 2 methods. First, immediately after participating in the simulations, the learners completed a 5-item, 5-point Likert self-assessment form rating their own competence regarding leadership, situation monitoring, mutual support, and communication as well as their overall performance. Second, 2 masked faculty members reviewed the videotaped and archived simulation encounters independently. One rater was a surgeon with a master's degree in health professions education (H.S.), the other a PhD in medical education (N.K.R.). Neither of them had participated in the development of the curriculum.
The raters used 2 instruments: a global rating scale provided by TeamSTEPPS that was similar to the self-assessment tool and a more detailed, modified, nontechnical skills evaluation instrument (NOTECHS).20,21 To train these raters, we had previously created 4 standard scenarios using the same type of simulation setting and had asked both faculty members to review and rate them using the described instruments. Subsequently, the raters proceeded to rate the simulation videos obtained during the study. For data analysis, the verbal anchors for both instruments were recoded numerically (TeamSTEPPS, 1 to indicate major problems and 5, exceptional; and NOTECHS, 1 to indicate not done; 2, done very poorly; and 6, done very well).
The study received approval by the Springfield Committee for Research Involving Human Subjects. We used SPSS, version 18 (SPSS, Inc), to perform the statistical analysis for quantitative data. We used paired sample t tests to compare means. Statistical significance was defined as P < .05.
Eighteen students participated in the curriculum. One student was unavailable for the initial multiple-choice test and simulation scenario and was therefore excluded from the study. Of the remaining 17 students, 14 demonstrated improved multiple-choice examination scores, 1 had a lower score, and the remaining 2 had the same score twice. The average score for the group improved from 84.9% to 94.1% after the intervention (P <.01).
The self-evaluation scores increased in 16 students and remained unchanged in 1 student. The average score rose from 12.76 to 16.06 (P < .001). This effect could be seen consistently across all tested competencies, and all the increases were statistically significant (Figure 2). The largest increase in confidence was seen for the team leadership competency.
To assess the students' performance during the simulation scenarios, we first analyzed the raters' aggregate scores across all competencies and compared the preintervention and postintervention averages. The observed performance scores increased significantly for both instruments. The global rating TeamSTEPPS score rose from 2.99 to 3.56 (P < .01), and the more detailed NOTECHS score increased from 4.07 to 4.59 (P < .001).
The NOTECHS aggregate scores were approximately 1 point higher due to the slightly different scoring matrix (Figure 3). Further analysis of the TeamSTEPPS instrument revealed that all tested competencies showed a significant increase when comparing preintervention with postintervention performance (Figure 4). When analyzing the competencies of the NOTECHS scale individually, we once again found an increase across all competencies; however, only 3 of the 5 competencies showed increases that were statistically significant (Figure 5).
Multidisciplinary teams play an increasing and important role in the delivery of modern health care. Gaba and DeAnda22 recognized this important development almost 20 years ago and began to use crisis resource management for team training, a concept based on the similar “crew resource management” curricula in the aviation industry.23 This approach was adopted in anesthesia training programs24 but did not immediately receive much recognition within other medical disciplines. Since then, a large body of evidence has been developed to support the importance of competence in the domains of leadership, mutual support, situation awareness, and communication for well-functioning teams.25- 34 Curricular frameworks like TeamSTEPPS were developed to focus specifically on such team-based competencies in an effort to optimize team performance in health care.
With the publication of the Institute of Medicine report “To Err Is Human”35 and the resulting increased focus on patient safety, these concepts quickly reached a much larger audience. Specialties such as emergency medicine, obstetrics, and intensive care medicine that rely heavily on well-functioning team performance have begun to integrate team training into their postgraduate curriculum.36- 40 In 2008, the American College of Surgeons and the Association of Program Directors in Surgery jointly developed and published phase III of the national curriculum, which provides a collection of team-training modules directed toward surgical care.41
The aim of our investigation was to analyze whether components of TeamSTEPPS could be used to teach medical students team skills, even though they were not the original target group. Despite TeamSTEPPS’ background and its widespread application in hospitals, there is so far only limited scientific evidence regarding its effectiveness for trainees, especially in regard to medical students.42 In general, literature on formal team-based training for medical students is scarce. Two recent studies from Germany43 and Australia44 describe a crisis resource management curriculum provided for medical students. Another study45 describes a simulation curriculum for lifesaving techniques that also includes a team training–related session. In all cases, learner reaction and self-assessment provided the only measurement of effectiveness. To our knowledge, there are currently no studies evaluating TeamSTEPPS as an educational tool for medical students.
Our evaluation of the intervention showed good acceptance by the participants as well as measurable and statistically significant increases on Kirkpatrick levels 2 and 3. It was especially encouraging to see that the masked review of the videotapes demonstrated improved performance of the group in the examined competencies. To our knowledge, this is the first study that shows statistically significant improvement in team performance of medical students after they undergo a TeamSTEPPS educational module. Creating team-training curricula from scratch is a daunting and time-consuming task; an already existing framework, such as TeamSTEPPS, may therefore be an attractive alternative for surgical and medical educators.
We noted that the multiple-choice test provided by TeamSTEPPS might not be ideal for medical students. Even though we saw a significant improvement, the high preintervention test scores may reflect that the questions are not challenging enough for this learner group. Modifying this test in the future and adjusting it to the targeted learner group should be considered.
Almost all students evaluated their performance as improved, but this was not always confirmed on the video ratings. Three TeamSTEPPS and 4 NOTECHS ratings showed worsening performance in the simulated scenarios. This confirms the concern that self-evaluation does not always predict better performance when rated by others objectively. Even though self-assessment of competency, also known as self-efficacy,46,47 has been used as a proxy for performance improvement in the past, recent studies48,49 have questioned the validity of such an approach. In our opinion, performance needs to be measured directly to provide reasonably valid assessment data.
Because of its design and integration into our resident readiness curricula and its proximity to graduation, our study did not include data on postintervention workplace performance. One advantage of simulation is the possibility to create realistic scenarios outside of the real working environment, and this was done for our educational intervention. The performance in simulated scenarios may be considered a proxy for real-life performance and thereby give insight into the curriculum's effect on the learner's performance in the real workplace. However, this still remains an approximation. Further efforts should focus on evaluating the students in their native work environment after completing this curriculum. This will be a challenge because the senior students match in different specialties at a number of institutions and will be exposed to a variety of other curricular interventions during their training. Despite these issues, we believe that, overall, the curriculum was clearly effective. Because our study involved a fairly small group, it may be problematic to generalize from our results to a larger population of learners. Further studies will be necessary to accomplish this goal. One possible way to improve generalizability and to investigate how this curriculum might affect real-world performance would be the earlier integration of a TeamSTEPPS module into the core medical school instruction schedule. This would significantly increase the study group size and expose the students to these concepts even earlier. In general, earlier formal exposure to team skills during medical school would familiarize the students with these concepts and prepare them better for their future, increasingly team-oriented work environment. Moreover, educators could identify students with significant deficiencies in team competencies early enough to successfully remediate them before graduation.
In conclusion, the integration of a TeamSTEPPS curriculum into our residency readiness elective led to measurable knowledge and team performance improvement for senior medical students. Further studies should be considered to investigate the generalizability of these findings. The integration of team skills curricula into medical schools using the TeamSTEPPS framework may be feasible and help prepare graduating medical students for their postgraduate activities.
Correspondence: Andreas H. Meier, MD, MEd, Division of Pediatric Surgery, Department of Surgery, SUNY Upstate Medical University, 725 Irving Ave, Ste 401, Syracuse, NY 13210-2306 (email@example.com).
Accepted for Publication: February 28, 2012.
Author Contributions:Study concept and design: Meier, Boehler, Schwind, and Sanfey. Acquisition of data: Meier, Boehler, McDowell, Schwind, Roberts, and Sanfey. Analysis and interpretation of data: Meier, Boehler, Markwell, and Sanfey. Drafting of the manuscript: Meier and Sanfey. Critical revision of the manuscript for important intellectual content: Meier, Boehler, McDowell, Schwind, Markwell, Roberts, and Sanfey. Statistical analysis: Meier and Markwell. Obtained funding: Meier. Administrative, technical, and material support: Boehler, McDowell, Schwind, and Sanfey. Study supervision: McDowell and Schwind.
Financial Disclosure: None reported.
Funding/Support: This study was supported by a surgical team-based training grant from the American College of Surgeons Division of Education and Medical Education Technologies, Inc.
Previous Presentation: This study was presented at the 97th Annual Clinical Congress of the American College of Surgeons; October 24, 2011; San Francisco, California.