The scores of postgraduate year (PGY) 2 residents significantly increased between the first and second 6 months of the year. The error bars indicate mean (standard error of the mean).
aP = .001.
Nicksa GA, Anderson C, Fidler R, Stewart L. Innovative Approach Using Interprofessional Simulation to Educate Surgical Residents in Technical and Nontechnical Skills in High-Risk Clinical Scenarios. JAMA Surg. 2015;150(3):201-207. doi:10.1001/jamasurg.2014.2235
The Accreditation Council for Graduate Medical Education core competencies stress nontechnical skills that can be difficult to evaluate and teach to surgical residents. During emergencies, surgeons work in interprofessional teams and are required to perform certain procedures. To obtain proficiency in these skills, residents must be trained.
To educate surgical residents in leadership, teamwork, effective communication, and infrequently performed emergency surgical procedures with the use of interprofessional simulations.
Design, Setting, and Participants
SimMan 3GS was used to simulate high-risk clinical scenarios (15-20 minutes), followed by debriefings with real-time feedback (30 minutes). A modified Oxford Non-Technical Skills scale (score range, 1-4) was used to assess surgical resident performance during the first half of the academic year (July-December 2012) and the second half of the academic year (January-June 2013). Anonymous online surveys were used to solicit participant feedback. Simulations were conducted in the operating room, intensive care unit, emergency department, ward, and simulation center. A total of 43 surgical residents (postgraduate years [PGYs] 1 and 2) participated in interdisciplinary clinical scenarios, with other health care professionals (nursing, anesthesia, critical care, medicine, respiratory therapy, and pharmacy; mean number of nonsurgical participants/session: 4, range 0-9). Thirty seven surgical residents responded to the survey.
Simulation of high-risk clinical scenarios: postoperative pulmonary embolus, pneumothorax, myocardial infarction, gastrointestinal bleeding, anaphylaxis with a difficult airway, and pulseless electrical activity arrest.
Main Outcomes and Measures
Evaluation of resident skills: communication, leadership, teamwork, problem solving, situation awareness, and confidence in performing emergency procedures (eg, cricothyroidotomy).
A total of 31 of 35 (89%) of the residents responding found the sessions useful. Additionally, 28 of 33 (85%) reported improved confidence doing procedures and 29 of 37 (78%) reported knowing when the procedure should be applied. Oxford Non-Technical Skills evaluation demonstrated significant improvement in PGY 2 resident performance assessed during the 2 study periods: communication score increased from 3 to 3.71 (P = .01), leadership score increased from 2.77 to 3.86 (P < .001), teamwork score increased from 3.15 to 3.86 (P = .007), and procedural ability score increased from 2.23 to 3.43 (P = .03). There were no statistically significant improved scores in PGY 2 decision making or situation awareness. No improvements in skills were seen among PGY 1 participants.
Conclusions and Relevance
The PGY 2 residents improved their skills, but the PGY 1 residents did not. Participants found interprofessional simulations to be realistic and a valuable educational tool. Interprofessional simulation provides a valuable means of educating surgical residents and evaluating their skills in real-life clinical scenarios.
During the past 24 years, the Accreditation Council for Graduate Medical Education (ACGME) has changed medical education1; changes have focused on the key aspects of resident education and resident fatigue, with limiting work hours to 80 hours per week in 2003 and, in 2011, further limiting interns to 16-hour maximum duty shifts with stringent oversight rules.2- 6 In 1999, the ACGME identified 6 core competencies residency programs are required to teach: interpersonal and communication skills, professionalism, patient care, systems-based practice, practice-based learning and improvement, and medical knowledge.7 The Joint Commission in 2011 and 2012 identified failures in communication, human factors, and leadership as the most common issues as root causes in sentinel events.8
Work-hour limitations significantly affect surgery because it is an experience-driven specialty. Thus, the limitations created many unique challenges in meeting surgical educational goals by the ACGME and surgical programs.9 During emergencies, surgeons work in interprofessional teams and are required to not only perform procedures, but effectively communicate, lead, and manage teams. To obtain proficiency in these skills, residents often take years to gain the confidence and expertise to master the nontechnical and technical skills needed to address high-risk clinical emergencies. We used interdisciplinary emergency clinical simulations to educate surgical residents in leadership, teamwork, effective communication, and infrequently performed emergency surgical procedures. We had observed many junior surgical residents (in the first 2 years of training) were unfamiliar with procedures they would be required to do in high-risk clinical scenarios while on call. We sought to see whether participating in high-fidelity simulations of high-risk clinical scenarios made a difference in junior residents’ procedural and nontechnical skills when done in the first half of the academic year versus the second half when they had more experience, as well as whether postgraduate year (PGY) of training made a difference.
Junior surgical residents (PGYs 1 and 2) on general surgery, vascular surgery, and cardiothoracic surgery at the San Francisco VA Medical Center participated in interdisciplinary clinical scenarios with other health care professionals (nursing, anesthesia, critical care, medicine, respiratory therapy, and pharmacy). Health care professionals from nursing, respiratory therapy, and pharmacy were full-time staff or students. Health care professionals from anesthesia, critical care, and medicine were residents or fellows.
This study was reviewed and approved by the University of California, San Francisco and San Francisco VA institutional review boards (Human Research Protection Program Committee on Human Research, 12-10475); patient consent was waived.
SimMan 3GS was used to simulate high-risk clinical scenarios lasting 15 to 20 minutes. A total of 43 surgical residents, participating in 28 simulation sessions, were observed and their performance analyzed between the first half of the academic year (July-December 2012) and second half of the academic year (January-June 2013). Each individual resident’s performance was analyzed as a separate session entry. The scenarios were conducted in various locations including the intensive care unit, emergency department, postanesthesia care unit, operating room, and simulation center. Session locations were selected to maximize the reality of the scenario and number of participants.
Simulations involved multidisciplinary teams as would occur during high-risk clinical scenarios. Examples of these scenarios included a postoperative pulmonary embolism, pneumothorax, treatment of an acute myocardial infarction, gastrointestinal bleeding, anaphylaxis with a difficult airway, fires in the operating room, and pulseless electrical activity arrest. Sessions with a surgical role required the surgical residents to perform a procedure during the session (eg, cricothyrotomy, chest tube, and central venous catheter or access). Sessions were of varied types, with the most common scenarios being cardiac arrest and cricothyrotomy (Table 1).
All simulation sessions were followed by 30-minute debriefings with real-time feedback. During the first 10 minutes, the participants reflected on the experience and their thoughts on the simulation; in the remaining 20 minutes, videos were viewed (if available) and the simulation team provided feedback, facilitated discussion, instructed on areas of knowledge as needed, provided insight, and encouraged participant self-reflection.
A modified Oxford Non-Technical Skills (NOTECHS)10 scale (score range, 1-4) was used to assess surgical resident nontechnical performance. The elements of the modified NOTECHS scale include the following.
Participants were rated on closed loop communication, interactions with other health care professionals, and how they communicated their findings and gave/took instruction.
Participants were rated on their observation of the standards of care, team approval, authority, and workload management.
Participants were rated on team building, maintaining support of others, and conflict solving.
Participants were rated on making appropriate diagnoses, assessing risk, monitoring outcomes, and adjusting treatment strategies as needed.
Participants were rated on their awareness of surroundings, anticipating surgical and medical needs, and adapting to changes in the environment if/when they occurred.
The NOTECHS scale ranges from a score of 1 to 4. A score of 1 (below standard) indicated that a participant’s behavior directly compromised patient safety and effective teamwork; 2 (basic standard) indicated that a participant’s behavior is acceptable but not ideal and might compromise care and teamwork; 3 (standard) indicated a participant’s behavior maintained an effective level of patient safety and teamwork; and 4 (excellent) indicated a participant’s behavior enhanced patient safety and teamwork and served as a model for other team members to follow.
After each session, an anonymous online survey was sent out by SurveyMonkey.com to solicit participant feedback. These surveys were used to gauge participants’ views of the value of the sessions, the aspects of the simulation they felt were most beneficial, qualities taught or enhanced their learning, what they would like to see more of in future sessions, and whether they felt confident in treating a patient in a similar scenario in the future. We also used these surveys to fine tune and adjust our simulation scenarios throughout the year as we received feedback from participants.
Statistical analysis was performed using analysis of variance or t test for interval data; χ2 tests (or Fisher exact test) were used for variables on a nominal scale (rates and proportions). SPSS version 21 was used for statistical analyses; P < .05 was considered significant.
A total of 43 individual surgical resident simulation sessions were observed and analyzed. Participating residents were evenly distributed between PGY 1 and PGY 2 levels (23 sessions with PGY 1 residents and 20 sessions with PGY 2 residents) and between the academic year (21 sessions in the first half of the academic year and 22 sessions in the second half). Residents had a surgical procedural role in 31 sessions, while residents had no surgical procedural role in 12 sessions.
Most sessions were interdisciplinary (77%). Participation by other health care professionals was as follows: anesthesia (74%), medicine (67%), nursing (63%), pharmacy (44%), critical care/pulmonary (21%), respiratory therapy (14%), other surgical residents (28%), and the following disciplines participated in a simulation session (2%) each: nurse circulator, scrub technician, surgeon, cardiologist, certified registered nurse anesthetist, and emergency medicine physician. The average number of additional health care professionals per session was 4.1 (range, 0-9).
The simulation sessions identified differences in ability between resident level of training and time of the year. Overall, PGY 2 residents demonstrated significantly greater proficiency in nontechnical skills compared with PGY 1 residents (mean [SD] NOTECHS overall score, 16 [2.4] vs 14 [1.8] for PGY 2 vs PGY 1 residents, respectively; P = .003). During the course of the academic year, the PGY 2 residents’ overall NOTECHS score significantly improved (mean [SD], 14.8 [2.1] for the first 6 months vs 18.1 [1.2] for the second 6 months; P = .001), while PGY 1 NOTECHS scores were not significantly different (Figure; Table 2).
Simulation sessions were also able to differentiate resident level on the individual dimensions of communication, leadership, teamwork, decision making/problem solving, and situational awareness (Table 2). Overall communication scores were similar for PGY 1 and PGY 2 residents (mean [SD], 3.25 [0.64] vs 3.04 [0.77] for PGY 2 vs PGY 1 residents, respectively; P = not significant); however, PGY 2 residents’ communication scores improved significantly during the course of the academic year. Postgraduate year 2 residents scored higher in leadership skills (mean [SD], 3.15 [0.75] vs 2.70 [0.76] for PGY 2 vs PGY 1 residents, respectively; P = .06), and PGY 2 residents demonstrated a significant improvement in leadership skills over the course of the academic year (Table 2). Overall teamwork scores were similar (mean [SD], 3.40 [0.60] vs 3.17 [0.65] for PGY 2 vs PGY 1, respectively; P = not significant); however, PGY 2 residents demonstrated a significant improvement in this dimension over the course of the academic year (Table 2). Sessions also demonstrated a difference in problem solving and decision making between resident levels; PGY 2 residents performed at a significantly more advanced level (mean [SD], 3.1 [0.72] vs 2.40 [0.5] for PGY 2 vs PGY 1 residents, respectively; P < .001). Finally, PGY 2 residents showed a greater degree of situational awareness (mean [SD], 3.05 [0.76] vs 2.65 [0.57] for PGY 2 vs PGY 1 residents, respectively; P = .05) and this also improved during the course of the PGY 2 academic year. In contrast, PGY 1 residents demonstrated no significant change in any of these dimensions.
A total of 37 (86%) of the residents responded to the anonymous surveys. There was an overwhelming positive response to simulation by the residents, with 89% of them finding the sessions very useful. They felt sessions were most useful for improving their diagnostic abilities and teamwork (51%) (Table 3). Most residents (87%) thought the opportunity to work with other services in the simulations was valuable. They found the simulation sessions particularly valuable for improving their own communication skills (57%) and problem diagnosis (55%). Residents felt that their procedural comfort was improved after simulation sessions (85%). Most residents (78%) after the simulation session felt confident in knowing when procedures stressed during the session should be done in an emergent situation. Residents recommended future sessions emphasizing procedures (75%), and they put the highest value on teamwork when differentiating a good versus excellent resident (75%) (Table 3).
Surgical residents found the interprofessional simulations to be realistic and valuable. Historically, graduate medical education and continuing medical education activities have focused on didactic sessions, with limited impact on physician behavior and patient outcomes.11,12 Simulation-based education has been shown to directly impact physicians’ clinical behavior and change outcomes.13 Simulation is learner-centered education in which real-time feedback can be given.14 We sought to create a high-fidelity environment both functionally and psychologically to emulate the environments in which residents find themselves when faced with a critical emergent situation. This brought a reality that is often lacking in simulation to our sessions and buy-in by the residents and other learners.
High-risk patient care situations occur in a dynamic, high-pressured environment, with interprofessional teams working together to ensure safe, successful patient outcomes.15 The interprofessional team is susceptible to adverse events occurring from both technical and nontechnical issues. High-risk organizations in industries such as aviation, nuclear, and the military have applied human factors research to develop safety training programs through simulation, ensuring safe practices by each team member. In aviation, simulations include training in technical and nontechnical skills known as the Crew Resource Management program.15,16 In addition, there is interest in using simulation to train surgeons, maintain certification, and evaluate those re-entering practice after a reprieve.15 In the past, clinicians and administrators have viewed simulation with some skepticism; however, with increasing demand for patient safety and the need for novel ways to train residents in the nebulous areas of professionalism, communication, and teamwork, simulation has become a valuable tool.
A total of 89% of the residents felt the sessions were of great benefit in their education. They felt they were better prepared to handle similar clinical situations. We conducted simulation sessions every week at a defined time and day of the week when other services were also available. The PGY 1 and PGY 2 residents who rotated for 1 month at a time on the general, vascular, and cardiovascular surgery services at the VA were asked to participate in 1 to 2 sessions per month that lasted an hour. The time taken away from clinical care or the operating room was minimal and allowed them to choose any week(s) they had less going on to participate in the simulations. Surgical residents appreciated the opportunity simulation gave them for a safe environment to practice key skills such as teamwork, communication, and professionalism, while receiving immediate feedback that could be instantly incorporated into their daily practice. A total of 32 (87%) of the surgical residents felt working in interdisciplinary teams was helpful and imparted a more realistic atmosphere of a high-risk clinical scenario. They felt the other health care professionals’ knowledge, communication, leadership, teamwork, and team management skills were the most useful aspects of working with them during the simulation.
Traditionally, surgical education has not formally taught leadership skills, effective communication, professionalism, or team management; instead, these skills are learned on the job during surgical residency. When To Err Is Human was published, it brought public awareness and attention to safer health care practices including improved teamwork and using simulation training.17,18 Simulation allows us to put residents in uncomfortable, high-risk, or difficult environments they may find themselves in and gives them a safe environment to act out what they might do in that situation. They are able to get direct feedback, learn directly and reenact some of the situations if needed, and be prepared for dealing with similar situations in the future. It is a dynamic, safe environment for teaching and learning that allows errors to be made and corrected while not harming a patient. Residents felt that many of the skills they learned from participating in the simulations were transferable to high-risk clinical scenarios and allowed the residents to handle them in a safer manner with improved communication skills, leadership, procedural expertise, and team management.
The Institute of Medicine has called for the incorporation of interprofessional education into the curricula of health care professionals to facilitate different professional disciplines working together in environments where shared educational goals can be achieved.19 The American College of Surgeons (ACS) has also taken an initiative at looking at interprofessional education models and ways to transform our traditional uniprofessional and multidisciplinary (parallel instead of integrated) medical education models in educating trainees and staff into an interprofessional educational model. Historical ways of educating residents give individuals less insight into the roles, scope of practice, or value that other professionals bring to the health care team.20 Interprofessional education has been recognized for its collaborative patient-centered focus, patient safety, addressing of health care costs, and more efficient care for patients with complex health care needs.20 In conducting the interprofessional simulation sessions, we sought to have residents work with other disciplines as one team, not just as a respective specialty consultant for their own discipline. Postgraduate year 2 residents did significantly better with their communication, leadership, teamwork, and team management skills in the second half of the year. Many of the residents felt that teamwork and leadership skills were particularly important, as well as mastering steps in emergency procedures. Most residents felt that the difference between a good and excellent resident was his or her teamwork and team management skills (75%) followed by clinical acuity and management of a problem (67%). This highlights the strength of conducting interprofessional simulation sessions with surgical residents and other health care professionals in diverse environments.
Our PGY 2 residents showed improvements on the NOTECHS surveys between the first and second halves of the academic year, correlating to more simulation sessions and more experience going into the simulations. They were able to take more away from it because their focus was on the whole experience rather than just their lone experience within the simulation. In contrast, PGY 1 residents did not demonstrate a significant improvement in their simulation scores over the year. While the scores seemed to decrease numerically, there was no significant difference between the scores in the first and second portions of the year. It is not clear why PGY 1 residents did not show the robust improvement seen by the PGY 2 residents. This may reflect their overall steep learning curve during the first clinical year or their general role in the residency. We plan to continue to monitor this to determine whether there are changes we could address in our educational curricula for PGY 1 residents.
Limitations of this study included the timeline of 1 academic year in which we conducted this study. This was the first time surgical residents routinely participated in simulation sessions but simulation for other specialties have been conducted weekly for more than 5 years. We do not have a longitudinal study to track the true effects of simulation on surgical residents’ clinical performance or the data to track an individual resident’s progress. We may have overestimated our results especially when considering the positive results from the residents’ surveys. Another limitation was that a single person rated the surgical resident performances. We did not have any formal interrater evaluations but there was general consensus during the sessions and minimal differences in instructor’s feedback. It must also be noted that residents may have had a heightened motivation to perform during formal assessment.21 The responses to the surveys may have reflected their desire to please the instructors and replies to surveys are quite subjective with room for interpretation. Despite this, it was felt that their anonymous responses were genuine, with written accounts to back up many responses.
Understanding and evaluating considerations around human factors and design of simulation-based training will continue to present challenges to planning, experiencing, and evaluating the best way to educate future generations in the delivery of safe health care.22 Nontechnical skills have become a prominent feature of the surgical literature and, to a lesser extent, surgical curricula. A systematic review of the literature on this topic showed a very interesting finding that the operating room team members’ nontechnical skills (teamwork, nursing leadership and management, and team communication) also influences a surgeon’s performance as well.22 The ACS Program for the Accreditation of Education Institutes was created to identify, develop, and promote stands for quality education and training.23 One of the goals of the ACS program is to enhance quality and promote patient safety through simulation-based education and training. There is a list of simulation centers on the ACS website for programs looking to use a simulation center or create their own program.24
The literature is very heterogeneous in the use of simulation in surgical graduate medical education and what research questions should be asked and how to go forward with the use of simulation in educating future surgeons and maintenance of certification.25 More research needs to be done on using interprofessional models for teaching where we are working with other specialties in complex and challenging environments that require both technical expertise and fine-tuned nontechnical skills in leadership, effective communication, and teamwork.
Corresponding Author: Lygia Stewart, MD, Department of Surgery, San Francisco VA Medical Center and University of California, San Francisco, 4150 Clement St, San Francisco, CA 94121 (email@example.com).
Accepted for Publication: May 9, 2014.
Published Online: January 7, 2015. doi:10.1001/jamasurg.2014.2235.
Author Contributions: Dr Stewart had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Nicksa, Fidler, Stewart.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: All authors.
Critical revision of the manuscript for important intellectual content: Nicksa, Stewart.
Statistical analysis: Stewart.
Administrative, technical, or material support: Nicksa, Fidler, Stewart.
Study supervision: Nicksa, Fidler, Stewart.
Conflict of Interest Disclosures: None reported.
Previous Presentation: This study was presented at the 85th Annual Meeting of the Pacific Coast Surgical Association; February 17, 2014; Dana Point, California, and is published after peer review and revision.