A, Day 1 is a formative experience during which participants perform trauma procedures guided by experienced trauma surgery faculty in a 1:1 participant:faculty ratio, with rotation of participants to another faculty after a lunch break. B, Day 2 is a summative experience, with participants repeating procedures from day 1 and being assessed by 2 faculty not seen on day 1.
eTable. Example Procedural Assessment Instrument - Cricothyroidotomy
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Bowyer MW, Andreatta PB, Armstrong JH, Remick KN, Elster EA. A Novel Paradigm for Surgical Skills Training and Assessment of Competency. JAMA Surg. 2021;156(12):1103–1109. doi:10.1001/jamasurg.2021.4412
Can general surgeons demonstrate improved trauma skills accuracy and independence following a standardized surgical skills curriculum?
In this cohort study of 65 general surgeons who were assessed for expert criteria for 24 trauma skills during a standardized cadaveric course with repeated-measures design and real-time formative and summative assessment, accurate performance of skills improved from 1% to 99% with 78% of participants demonstrating independence.
Standardized, focused surgical skills training may improve performance and independence of procedural skills; this approach has broad applicability to all specialties with procedural competencies.
Sustainment of comprehensive procedural skills in trauma surgery is a particular problem for surgeons in rural, global, and combat settings. Trauma care often requires open surgical procedures for low-frequency/high-risk injuries at a time when open surgical experience is declining in general and trauma surgery training.
To determine whether general surgeons participating in a 2-day standardized trauma skills course demonstrate measurable improvement in accuracy and independent performance of specific trauma skills.
Design, Setting, and Participants
General surgeons in active surgical practice were enrolled from a simulation center with anatomic laboratory from October 2019 to October 2020. Differences in pretraining/training and posttraining performance outcomes were examined using (1) pretraining/posttraining surveys, (2) pretraining/posttraining knowledge assessment, and (3) training/posttraining faculty assessment. Analysis took place in November 2020.
A 2-day standardized, immersive, cadaver-based skills course, developed with best practices in instructional design, that teaches and assesses 24 trauma surgical procedures was used.
Main Outcomes and Measures
Trauma surgery capability, as measured by confidence, knowledge, abilities, and independent performance of specific trauma surgical procedures; 3-month posttraining skill transfer.
The study cohort included 65 active-duty general surgeons, of which 16 (25%) were women and 49 (75%) were men. The mean (SD) age was 38.5 (4.2) years. Before and during training, 1 of 65 participants (1%) were able to accurately perform all 24 procedures without guidance. After course training, 64 participants (99%) met the benchmark performance requirements for the 24 trauma procedures, and 51 (78%) were able to perform them without guidance. Procedural confidence and knowledge increased significantly from before to after the course. At 3 months after training, 37 participants (56%) reported skill transfer to trauma or other procedures.
Conclusions and Relevance
In this study, direct measurement of procedural performance following standardized training demonstrated significant improvement in skill performance in a broad array of trauma procedures. This model may be useful for assessment of procedural competence in other specialties.
Surgeons who care for trauma patients in rural, humanitarian, and military settings require competency in a broad-based skill set that transcends traditional training or practice.1-3 Modern surgical trainees have experienced a steady decline in open surgical experience, both in general and trauma surgery.4,5 This reality has led to concerns that graduating surgical resident physicians, trauma fellows, and many practicing general surgeons are inadequately prepared to treat traumatic injuries.6-8 This concern is magnified for the active-duty general surgeon, whose typical peacetime practice rarely, if ever, includes trauma and is limited in both volume and complexity.6 A 2020 review of the case logs of 57 trauma, general, and vascular surgeons at 4 major military hospitals with residency training programs found that only 0.3% of cases were relevant to combat casualty (trauma) care with the number of total mean cases per surgeon of only 92 per year.9
To address the challenges of acquiring and maintaining relevant trauma knowledge, skills, and abilities, the US Department of Defense, in partnership with the American College of Surgeons, has initiated a data-driven Clinical Readiness Program.10-13 This approach sustains trauma currency and competency in military general surgeons through periodic assessment of knowledge and procedural skills; structured training and retraining; and measurement of readiness based on the predicted transferability of day-to-day surgical practice to the requirements for deployed casualty trauma care. Following review of existing trauma skills courses,14,15 we developed and implemented a comprehensive procedural skills course that uses evidence-based education principles in a standardized approach to teach and assess performance of 24 trauma surgical procedures. The course expands and reimagines a 1-day fresh cadaver-based trauma skills course, Advanced Surgical Skills for Exposures in Trauma (ASSET), developed by the American College of Surgeons Committee on Trauma.16 Requirements for this expanded course, titled ASSET+, were derived from identified gaps in the capabilities of deployed surgeons and determined by consensus agreement of military trauma surgeons who had deployed in support of combat missions. Over 2 days, participants engage in immersive hands-on training and assessment activities with 1-to-1 oversight by 4 experienced trauma surgeons. The ASSET+ course differs from the existing ASSET course, as well as the complimentary Advanced Trauma Operative Management course, by virtue of not only the 1:1 ratio, but more significantly by the inclusion of both a formative and summative assessment and the opportunity for each student to perform and be assessed on every procedure to a trauma surgeon standard. We hypothesized that this immersive cadaver-based skills course would measurably improve the accurate and independent performance of specific trauma surgical skills.
This study received expedited institutional review board approval from Uniformed Services University of the Health Sciences, School of Medicine without need for participant consent. The study cohort included active-duty general surgeons (without trauma surgery fellowship training or active practice of trauma) who completed the ASSET+ course. We used a repeated-measures study design to examine the differences between pretraining/training and posttraining performance outcomes. The training state of the participants was the only independent variable. We measured 4 dependent variables, each chosen owing to its known significance as a contributor to trauma surgery capability: confidence, knowledge, abilities, and independence.17-21 Participants were enrolled from October 2019 to October 2020.
Throughout 18 months, 8 ASSET+ courses were conducted with 30 experienced trauma surgeon faculty, all of whom were current ASSET and Advanced Trauma Life Support instructors. All performance data were captured in real time using iPads (Apple) and the Ace Surgery App (Ace Metrics) and sent to a secure server. We controlled instructor variability by precourse 1-to-1 briefing of faculty by a professional psychometrician on how to use the instrumentation, 1-to-1 mentorship by master instructors, psychometrician coaching and oversight during the course, comparative analyses of new faculty scoring compared with the assigned mentor’s scoring (interrater reliability), and internal consistency of new faculty scores for learners who were also scored by other faculty on different procedures. Scores from all faculty are examined for internal consistency for each course. Outliers were identified and remediated until they were able to meet the requirements for assessors.
Posttraining skill transfer was assessed by a survey sent to each participant by email at 3 months following course completion. A pilot examination of skills degradation was conducted in a small cohort (n = 8) of general surgeons who completed ASSET+ at least 6 months earlier.
The ASSET+ courses were conducted throughout 2 days, during which participants rotated between tables to enable instruction and assessment of each participant by 4 separate experienced trauma surgeon instructors (Figure). The course provided immersive experiences in procedural skills deemed essential for deployed surgeons caring for trauma patients. Participants completed a pretraining procedural knowledge examination and procedural confidence survey prior to course start. The course manual was mailed to each participant 1 month before each course. During day 1 (training with formative assessment), participants were coached and assessed 1-to-1 by their assigned instructors while performing sequential trauma procedures in fresh cadavers within time limits consistent with actual situations. The procedural dissections were preceded by interactive, scenario-based PowerPoint (Microsoft) presentations that included indications, contraindications, and step-by-step static image and video demonstrations of standardized procedure performance. Immediately following each procedure, instructors completed a performance assessment on a tablet application to evaluate procedural ability and independence. Benchmarks for each procedure were defined as the completion of sequenced tasks to the quality standard of an experienced trauma surgeon. Participants were paired with a different instructor midday and received detailed feedback verbally (from both instructors) during session breaks and by emailed assessment reports for review before day 2.
Day 2 focused on summative assessment without guidance from instructors, and 2 new instructors were assigned to each participant to assess their independent ability to perform the same sequential trauma procedures from day 1 using new fresh cadaveric specimens. All assessment reports were sent by email to the participants. The course concluded with participant completion of the posttraining procedural knowledge examination and procedural confidence survey.
Participants were asked to self-rate their precourse and postcourse procedural confidence for the 24 trauma procedures taught in the course on a 5-interval rating scale. Scores ranged from 1, very little confidence, to 5, a lot of confidence.
Two 40-item multiple choice examinations (pretest and posttest) with validated content covering the 24 trauma procedures were created. Factor analysis confirmed measurement validity for a single factor, with no crossloadings. Reliability was confirmed through test-retest and Chronbach α measure of internal consistency less than .70. We established the benchmark passing score as 28 (70%). The knowledge tests were administered before and after training through Google forms.
Procedural abilities were assessed for each step required for completion of the procedures, with the standard defined as accurate performance. Critical tasks were derived from previously validated subject matter consensus task deconstruction for each of the procedures.21,22 Specific tasks included identification of anatomical structures, procedural sequencing, accurate approach and incisions, technique and instrument use, completeness of required procedural tasks, protection of neurovascular structures, avoidance of iatrogenic injury, and ability to manage complications. The scale for task assessment included not applicable, does not meet performance standards, partially meets performance standards, meets performance standards, and exceeds performance standards. Scores for task assessments were grounded on the performance accuracy and skill of experienced trauma surgeons performing each task. As an example, the assessment tool for cricothyroidotomy is included as the eTable in the Supplement. To account for cadaver habitus and anatomical variation, case difficulty was assessed for each procedure, and an evidence-based weighting factor was applied to specific procedure scores. Psychometric support for skills performance assessment instruments included content, convergent, criterion, and predictive validity evidence (P < .05) and test-retest reliability more than 0.90.
The maximum score by procedure for all tasks performed accurately and independently was 100. The benchmark scores for all surgical skills were set to 90, which reflected accurate and independent performance of all critical tasks. Participants could score greater than 100 if they also performed efficiently.
Independent performance by procedure was assessed using a graded scale of ability to perform each step relative to guidance required: not applicable, required substantial guidance, required minimal guidance, no guidance required, and efficient/no guidance required. The standard was defined as independent performance.
Participants were surveyed at 3-month intervals after training to evaluate the use of course skills and abilities in the performance of trauma procedures or other operations. Further, 8 general surgeons who were at least 6 months from completion of the ASSET+ course repeated the summative experience from day 2 of the course. Results were compared with the postcourse performance benchmark recorded for each learner.
Paired t tests were used to analyze the association of the training with procedural confidence, knowledge, and ability to perform each procedure accurately and independently. Statistical significance was set at P value less than .05. We calculated Cohen d as a measure of effect size for each procedure (with effect sizes categorized as medium, ≥0.5; large, ≥0.8; and very large, ≥1.4).23 Mean total procedural skills scores were calculated and compared with the performance benchmarks for accuracy and independence. Descriptive statistics are reported for transfer and retention outcomes. All statistics were performed using SPSS Statistics version 27 (IBM). Analysis took place in November 2020.
The study cohort included 65 active-duty general surgeons, of which 16 (25%) were women and 49 (75%) were men. Nine surgeons (14%) were African American, 9 (14%) were Asian American, 6 (9%) were Latin American, and 41 (63%) were White. There were significant improvements between pretraining and posttraining for procedural confidence (medium effect size) and knowledge (very large effect size) (Table 1). Procedural independence scores increased between formative and summative evaluation, with a very large effect size (Table 1). During training (day 1), 1 participant (1%) were able to perform all procedures to the defined standard without guidance. After training (day 2), 51 participants (78%) were able to perform all procedures to the defined standard without guidance, while the remaining 14 (22%) required minor guidance.
There were significant improvements in performance scores for all procedures, with calculated effect sizes ranging from 0.59 to 1.56 (Table 2). During training, all participants reached the benchmark performance score for only 1 procedure (cricothyroidotomy). After training, 64 of 65 surgeons (98.5%) met the performance benchmark (score, ≥90) for all procedures, and benchmark aggregate performance was reached for 23 of 24 procedures (95.8%) (Table 2). Only thigh fasciotomy was slightly below benchmark at 88. The standard deviation of performance scores was significantly smaller after training for 23 of 24 procedures (95.8%), with the exception of trauma laparotomy.
Of 41 surgeons who responded to the posttraining survey, 19 (46.3%) reported direct skill transfer to trauma procedures performed after training and 10 (24.4%) reported skill transfer to other operations (eg, nontrauma vascular exposures, intervention for intra-abdominal bleeding, and colon mobilization) (Table 3). Accounting for overlap, 23 individuals (56.1%) overall reported skill transfer to trauma or other procedures. Ten surgeons (24.4%) performed more than 2 trauma procedures in their subsequent practice.
In the retention pilot study with 8 surgeons at 6 to 18 months (mean [SD], 10 [3.2] months) posttraining, knowledge remained above benchmark, while procedural abilities remained at or above the benchmark for 21 of 24 procedures (87.5%). Vascular exposures of popliteal and subclavian arteries fell slightly below the benchmark score.
Prior to training, the majority of participants were not confident in their abilities to treat trauma patients, and during training, only 1% were able to perform all 24 procedures to standard without guidance. Following the ASSET+ course, 99% of participants met the benchmark performance requirements for 24 trauma procedures, and 78% were able to perform them without guidance. Confidence and knowledge scores also significantly increased. To our knowledge, this is the first report to demonstrate the association of trauma skills training through standardized training and direct measurement of procedural performance following training.
Of the 24 trauma procedures performed in ASSET+, most are low-frequency, high-impact procedures that are used uncommonly by trauma surgeons and rarely by general surgeons. The primary objective of ASSET+ is to ensure the trauma readiness of military general surgeons who deploy to austere and hostile environments without other trauma support. The results of this study indicate that nontrauma-trained general surgeons can achieve benchmark standards of performance in these procedures, giving important input to the debate among trauma and military surgeons about whether military trauma surgery should be the exclusive domain of trauma specialists or should include general surgeons.24-26 Performance outcomes improved significantly for all participants across all procedures, and smaller standard deviations between pretraining and posttraining performance scores indicate less variation between participants after training (with the exception of trauma laparotomy). This exception may be the result of consistent general surgeon experience with elective laparotomy, enabling transfer of these skills to the trauma context at the outset of the course.
Within 3 months of course completion, 56% of survey respondents had performed trauma procedures or applied skills from the ASSET+ course. The retention pilot suggested that knowledge and skills may be retained up to 18 months. Further work with a larger posttraining group is needed to define the retention interval further, yet these initial pilot findings, coupled with our previous work on skills retention,27,28 have prompted consideration of course renewal every 2 years for military surgeons.
Strengths of this educational intervention include the standardized content, instructional design that incorporates best practices in experiential learning, and rigorous performance assessment for each participant by multiple experts. The course appeals to multiple learning styles by using comprehensive multimodal learning activities and 1-to-1 direct table oversight with experienced trauma instructors. The assessment instruments are sufficiently precise to identify variance within individual procedural performance (how individuals varied in their performance across procedures), as well as procedural performance between individuals (how performance on 1 procedure varied between individuals). The detailed assessments provide a common basis for instructor feedback to participants. Over 2 days, each participant was the primary surgeon on all 24 procedures, and 4 different surgeons independently evaluated each participant. This method ensures comprehensive and objective assessment of participant abilities. This education model addresses weaknesses in existing trauma training skills courses, which include variable participant:faculty ratios (typically 2:1 or 4:1), inability of participants to perform all procedures because of limitations with a single cadaver, and single formative participant assessment.
Study limitations include the short interval between formative and summative assessment (<24 hours) and the small size of the retention pilot. Thus, this study cannot demonstrate long-term retention of specific procedural skills in the entire cohort. Results may have been more favorable because the participants are particularly motivated, given the probability of deployment. Although concerns with assessment reliability between instructors were attenuated by rigorous training of faculty, data monitoring for outlier scores during the course, and 4-instructor evaluation of each participant, variations between the perceptions and judgments of individual instructors may be present. Study reproducibility is challenged by the expense (at least double the cost of a 1-day cadaver course) and opportunity costs for participants and faculty.
Previous research suggests that even experienced trauma surgeons may lack readiness to perform many of the 24 trauma procedures to benchmarked standards and may benefit from periodic refresher training to maintain competency.28 Thus, the ASSET+ model may have relevance for the introduction and sustainment of trauma operative skills for surgeons in civilian trauma, rural surgery, and global surgery. Further, this novel education and assessment paradigm may be applicable to procedural training for all medical specialties and may serve as a framework for higher stakes assessments toward board certification. Such a structured framework enables movement beyond formative to summative evaluation of procedural skills.
Corresponding Author: Mark W. Bowyer, MD, Department of Surgery, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Rd, Bethesda, MD 20814 (email@example.com).
Accepted for Publication: July 4, 2021.
Published Online: September 15, 2021. doi:10.1001/jamasurg.2021.4412
Author Contributions: Drs Bowyer and Andreatta had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Bowyer, Andreatta, Elster.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Bowyer, Andreatta, Armstrong, Elster.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Bowyer, Andreatta.
Obtained funding: Elster.
Administrative, technical, or material support: Bowyer, Andreatta, Armstrong, Elster.
Supervision: Bowyer, Andreatta, Elster.
Conflict of Interest Disclosures: None reported.
Disclaimer: The contents of this presentation are the sole responsibility of the authors and do not necessarily reflect the views, opinions or policies of Uniformed Services University of the Health Sciences, the US Department of Defense, or the US Departments of the Army, Navy, or Air Force. Mention of trade names, commercial products, or organizations does not imply endorsement by the US government.