[Skip to Navigation]
Sign In
Figure 1.  Comparison of Mean Entrustable Professional Activity Ratings by Postgraduate Year for FO vs RO, FO vs RA, and RO vs RE Response Data
Comparison of Mean Entrustable Professional Activity Ratings by Postgraduate Year for FO vs RO, FO vs RA, and RO vs RE Response Data

Com indicates communicating with physicians, health care personnel, and patients; EL, exploratory laparotomy for trauma; FO, faculty observation; IH, open inguinal hernia repair; Inf, collecting information; LA, laparoscopic appendectomy; LC, laparoscopic cholecystectomy; Op-B, performing basic operations; Op-C, performing complex operations; RA, resident autonomy; RE, resident expectation; RH, right hemicolectomy; RO, resident observation; and Urg, identifying urgencies and emergencies. Error bars represent 95% CIs.

Figure 2.  Comparison of Mean Entrustable Professional Activity Ratings by Postgraduate Year for FO vs FE, RA vs RE, and RA vs FE Response Data
Comparison of Mean Entrustable Professional Activity Ratings by Postgraduate Year for FO vs FE, RA vs RE, and RA vs FE Response Data

Com indicates communicating with physicians, health care personnel, and patients; EL, exploratory laparotomy for trauma; FE, faculty expectation; FO, faculty observation; IH, open inguinal hernia repair; Inf, collecting information; LA, laparoscopic appendectomy; LC, laparoscopic cholecystectomy; Op-B, performing basic operations; Op-C, performing complex operations; RA, resident autonomy; RE, resident expectation; RH, right hemicolectomy; and Urg, identifying urgencies and emergencies. Error bars represent 95% CIs.

Figure 3.  Mean Differences (Δ) in Entrustable Professional Activity Ratings Among Individual Respondents by Postgraduate Year
Mean Differences (Δ) in Entrustable Professional Activity Ratings Among Individual Respondents by Postgraduate Year

Com indicates communicating with physicians, health care personnel, and patients; EL, exploratory laparotomy for trauma; FE, faculty expectation; FO, faculty observation; IH, open inguinal hernia repair; Inf, collecting information; LA, laparoscopic appendectomy; LC, laparoscopic cholecystectomy; Op-B, performing basic operations; Op-C, performing complex operations; RA, resident autonomy; RE, resident expectation; RH, right hemicolectomy; RO, resident observation; and Urg, identifying urgencies and emergencies. Dashed reference lines indicate Δ = 0.

Figure 4.  Comparison of Stratified FO With Stratified FE Ratings
Comparison of Stratified FO With Stratified FE Ratings

A, Mean ratings stratified by specialty within the Op-B global entrustable professional activities (EPAs). B, Mean ratings stratified by specialty within the Op-C global EPA. C, Mean ratings stratified by faculty years in practice among operative EPAs. D, Proportion of respondents indicating residents independently capable (score of ≥4) in global EPAs. E, Proportion of respondents indicating residents independently capable (score of ≥4) in operative EPAs. Com indicates communicating with physicians, health care personnel, and patients; Bar/MIS indicates bariatric and minimally invasive; CRS, colorectal; CTS, cardiothoracic; EL, exploratory laparotomy for trauma; Endo, endocrine; GS, general surgery; IH, open inguinal hernia repair; Inf, collecting information; LA, laparoscopic appendectomy; LC, laparoscopic cholecystectomy; Op-B, performing basic operations; Op-C, performing complex operations; Onc, surgical oncology; Ped, pediatric; PGY, postgraduate year; Plas, plastic; RH, right hemicolectomy; Transpl, transplant; Trauma, trauma and critical care; Urg, identifying urgencies and emergencies; and Vasc, vascular. Scores on a scale of 1 to 5.

Table.  Comparison of Survey Response Data
Comparison of Survey Response Data
1.
Perone  JA, Fankhauser  GT, Adhikari  D,  et al.  Who did the case? Perceptions on resident operative participation.  Am J Surg. 2017;213(4):821-826.PubMedGoogle ScholarCrossref
2.
Bell  RH  Jr.  Why Johnny cannot operate.  Surgery. 2009;146(4):533-542.PubMedGoogle ScholarCrossref
3.
Bucholz  EM, Sue  GR, Yeo  H, Roman  SA, Bell  RH  Jr, Sosa  JA.  Our trainees’ confidence: results from a national survey of 4136 US general surgery residents.  Arch Surg. 2011;146(8):907-914.PubMedGoogle ScholarCrossref
4.
Mattar  SG, Alseidi  AA, Jones  DB,  et al.  General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors.  Ann Surg. 2013;258(3):440-449.PubMedGoogle ScholarCrossref
5.
American Board of Surgery. Booklet of Information—Surgery 2016-2017. Philadelphia, PA: American Board of Surgery; 2017.https://www.absurgery.org/xfer/BookletofInfo-Surgery.pdf. Accessed June 19, 2017.
6.
Accreditation Council for Graduate Medical Education and American Board of Surgery. The General Surgery Milestones Project. http://www.acgme.org/acgmeweb/portals/0/pdfs/milestones/surgerymilestones.pdf. Published July 2015. Accessed September 9, 2016.
7.
Hirschl  RB.  The making of a surgeon: 10,000 hours?  J Pediatr Surg. 2015;50(5):699-706.PubMedGoogle ScholarCrossref
8.
ten Cate  O.  Entrustability of professional activities and competency-based training.  Med Educ. 2005;39(12):1176-1177.PubMedGoogle ScholarCrossref
9.
Sterkenburg  A, Barach  P, Kalkman  C, Gielen  M, ten Cate  O.  When do supervising physicians decide to entrust residents with unsupervised tasks?  Acad Med. 2010;85(9):1408-1417.PubMedGoogle ScholarCrossref
10.
George  BC, Teitelbaum  EN, Meyerson  SL,  et al.  Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance.  J Surg Educ. 2014;71(6):e90-e96.PubMedGoogle ScholarCrossref
11.
Rekman  J, Gofton  W, Dudek  N, Gofton  T, Hamstra  SJ.  Entrustability scales: outlining their usefulness for competency-based clinical assessment.  Acad Med. 2016;91(2):186-190.PubMedGoogle ScholarCrossref
12.
Peyre  SE, MacDonald  H, Al-Marayati  L, Templeman  C, Muderspach  LI.  Resident self-assessment versus faculty assessment of laparoscopic technical skills using a global rating scale.  Int J Med Educ. 2010;1:37-41. doi:10.5116/ijme.4bf1.c3c1Google ScholarCrossref
13.
Lyle  B, Borgert  AJ, Kallies  KJ, Jarman  BT.  Do attending surgeons and residents see eye to eye? An evaluation of the Accreditation Council for Graduate Medical Education Milestones in General Surgery Residency.  J Surg Educ. 2016;73(6):e54-e58.PubMedGoogle ScholarCrossref
14.
Alameddine  MB, Claflin  J, Scally  CP,  et al.  Resident surgeons underrate their laparoscopic skills and comfort level when compared with the rating by attending surgeons.  J Surg Educ. 2015;72(6):1240-1246.PubMedGoogle ScholarCrossref
15.
Friedell  ML, VanderMeer  TJ, Cheatham  ML,  et al.  Perceptions of graduating general surgery chief residents: are they confident in their training?  J Am Coll Surg. 2014;218(4):695-703.PubMedGoogle ScholarCrossref
16.
Klingensmith  ME, Lewis  FR.  General surgery residency training issues.  Adv Surg. 2013;47:251-270.PubMedGoogle ScholarCrossref
17.
Rosenbaum  L.  Leaping without looking—duty hours, autonomy, and the risks of research and practice.  N Engl J Med. 2016;374(8):701-703.PubMedGoogle ScholarCrossref
18.
Wojcik  BM, Fong  ZV, Patel  MS,  et al.  The resident-run minor surgery clinic: a pilot study to safely increase operative autonomy.  J Surg Educ. 2016;73(6):e142-e149.PubMedGoogle ScholarCrossref
19.
Kempenich  JW, Willis  RE, Rakosi  R, Wiersch  J, Schenarts  PJ.  How do perceptions of autonomy differ in general surgery training between faculty, senior residents, hospital administrators, and the general public? a multi-institutional study.  J Surg Educ. 2015;72(6):e193-e201.PubMedGoogle ScholarCrossref
20.
Teman  NR, Gauger  PG, Mullan  PB, Tarpley  JL, Minter  RM.  Entrustment of general surgery residents in the operating room: factors contributing to provision of resident autonomy.  J Am Coll Surg. 2014;219(4):778-787.PubMedGoogle ScholarCrossref
21.
Meyerson  SL, Teitelbaum  EN, George  BC, Schuller  MC, DaRosa  DA, Fryer  JP.  Defining the autonomy gap: when expectations do not meet reality in the operating room.  J Surg Educ. 2014;71(6):e64-e72.PubMedGoogle ScholarCrossref
Original Investigation
Pacific Coast Surgical Association
April 2018

Use of Entrustable Professional Activities in the Assessment of Surgical Resident Competency

Author Affiliations
  • 1Department of Surgery, David Geffen School of Medicine, UCLA (University of California, Los Angeles)
JAMA Surg. 2018;153(4):335-343. doi:10.1001/jamasurg.2017.4547
Key Points

Question  How are resident capabilities, autonomy, and expectations in a series of entrustable professional activities perceived by surgical faculty and residents?

Findings  In this survey of 31 core surgical faculty members and 39 categorical general surgery residents, faculty members perceived that senior residents underperformed expectations in operative entrustable professional activities but performed at a higher level than the level of autonomy entrusted to them.

Meaning  In establishing competency-based curricula, entrustable professional activities provide a framework for graduated autonomy to improve the resident operative experience.

Abstract

Importance  Competency-based assessments of surgical resident performance require metrics of entrustable autonomy.

Objectives  To designate entrustable professional activities (EPAs) in global performance and in specific operations, and to identify differences in perceived capability, autonomy, and expectations between surgical faculty and residents.

Design, Setting, and Participants  This survey study was conducted from August 9, 2016, through August 24, 2016, in the Department of Surgery at the UCLA David Geffen School of Medicine. The survey instrument consisted of 5-point Likert scales for assessing perceptions of entrustability for 5 global and 5 operative EPAs. Faculty members were surveyed regarding resident capabilities and expected capabilities by postgraduate year. Residents were surveyed regarding their own capabilities, actual autonomy entrusted in the last EPA performed, and expected capabilities.

Main Outcomes and Measures  Differences in mean ratings were assessed across 7 comparison domains.

Results  Among 78 total faculty members, 31 (40%) participated in the survey. Among 49 residents, 39 (80%) participated in the survey. Residents generally rated their global EPA performance higher than the faculty did (mean, 3.7 vs 2.8; P < .01), but operative EPA performance ratings were equivalent (mean, 2.7 vs 2.4; P < .12). Faculty members perceived senior residents as underperforming expectations in operative EPAs. Most faculty members (80%) expected residents not to be independently capable of performing complex operations by graduation. Faculty members perceived residents in postgraduate years 4 and 5 to have greater operative capability than the level of autonomy entrusted to those residents (95% CI, 3.3-3.5 vs 1.9-2.2).

Conclusions and Relevance  Global and operative EPAs are practical for developing competency-based curricula. Graduated autonomy should be granted to improve the operative experience for residents.

Introduction

The competency of surgical residents in operative skills has been the subject of significant research and debate in recent years. Studies have intimated that surgical residents lack confidence and are not capable of operating independently at the time of graduation.1-4 In response, the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Surgery have implemented directives to increase resident operative experience and to improve competency-based assessments in training programs.5,6

Hirschl7 advised that introducing graded responsibility significantly enhances competency-based assessments for surgical residents. Nevertheless, the breadth of responsibility surgeons must assume is formidable, and concrete grades of responsibility remain unclear. Quiz Ref IDAssessments of “entrustable professional activities” (EPAs)8(p1176) provide valuable benchmarks for skills acquisition, characterizing the degree to which a responsibility may be consigned to a trainee to execute autonomously.9-11 These EPAs have not yet been formally incorporated into evaluations of graded responsibility and autonomy in surgical training.

In the present study, we examined select EPAs in global performance and in specific operations to assess resident development in the surgical training program at the UCLA David Geffen School of Medicine. Faculty members and residents were surveyed regarding their perceptions of capability, entrusted responsibility, and resident expectations of these EPAs. We sought to identify the differences in these perspectives on specific EPAs between surgical faculty and residents.

Methods
Setting and Participants

This survey study took place within the Department of Surgery at the UCLA David Geffen School of Medicine from August 9, 2016, through August 24, 2016. Electronic surveys were generated and emailed to 78 core academic surgical faculty members and to 49 categorical general surgery residents. All faculty participants are engaged in clinical education of residents. This study was conducted with the approval of the UCLA institutional review board. Survey respondents provided informed written consent to participate in this study.

Survey Instrument

From the ACGME Milestones6 domains and competencies, these 5 EPAs of global resident performance were selected: collecting information (Inf); performing basic operations (Op-B); performing complex operations (Op-C); identifying urgencies and emergencies (Urg); and communicating with physicians, health care personnel, and patients (Com). See eTable 1 in the Supplement for the ACGME Milestones corresponding with each global EPA. In addition, consensus among us (this study’s authors) resulted in the identification of 5 specific operations of graduated complexity about which to survey resident performance. These operative EPAs included the following: open inguinal hernia repair (IH), laparoscopic appendectomy (LA), laparoscopic cholecystectomy (LC), exploratory laparotomy for trauma (EL), and right hemicolectomy (RH; open or laparoscopic based on the experience and interpretation of the rater).

The survey instrument was formulated to rate resident performance or expected resident performance in the 10 total EPAs using a 5-point Likert scale based on the levels of autonomy entrusted to residents by faculty.9,10Quiz Ref ID This scale included the following general ratings: 1—not safe to perform the task (safe only to observe), 2—safe to perform under direct supervision, 3—safe to perform under indirect supervision, 4—safe to perform independently with oversight, 5—safe to supervise others, and 0—irrelevant to the rater’s practice or unable to rate. Ratings of 0 were excluded from the data analysis.

The survey instrument was generated electronically, and specific sections were delivered according to demographic identifiers. To avoid bias in survey responses, we elicited participants’ opinions without specific instruction or orientation regarding the survey’s subject matter. The survey is shown in eAppendix in the Supplement. Participants principally identified themselves as faculty or as residents. Faculty members were asked to specify their clinical specialty and number of years in practice after training. In the faculty portion of the survey, the first section (faculty observation [FO]) asked faculty members to rate the observed performance of residents by postgraduate year (PGY) in each of the global and operative EPAs. In the next section (faculty expectation [FE]), faculty members were asked to rate the expected performance of residents by PGY level.

In the residents’ portion of the survey, residents were initially asked to indicate their PGY level. In the next section (resident observation [RO]), residents were asked to rate their own capabilities in each of the global and operative EPAs. In the following section (resident autonomy [RA]), residents were asked to rate the level of autonomy entrusted to them the last time they performed each of the EPAs. In the final section (resident expectation [RE]), residents were asked to rate the expected performance for their PGY level.

Statistical Analysis

Survey data were compiled electronically in a secured web-based spreadsheet. Statistical calculations and plots were performed with Excel, version 14.7.4 (Microsoft Corp). Responses in each of the 2 faculty sections and 3 resident sections were compared (Figure 1 and Figure 2). The Table shows domains generated by specific comparisons among the 5 total sections. The mean ratings by PGY level for each EPA were compared by unpaired, 2-tailed t tests. These analyses generated linear plots of mean ratings with 95% CIs for each domain component. Within the domains of RO vs RE, FO vs FE, and RA vs RE, we calculated the mean difference in ratings per individual response and then plotted the mean difference by PGY level.

Within the domain of FO vs FE, the global EPA ratings of Op-B and Op-C were stratified by faculty specialty and by faculty member years of practice. Furthermore, we identified and examined clinical specialties with high or low proportions of core faculty involved in educational program administration. The PGY levels at which faculty members rated residents as independently capable of performing each EPA (4 or 5 rating) were compared with the PGY levels at which faculty members expected residents to be independently capable. Differences in faculty expectation and resident expectation (FE vs RE) were analyzed separately. Differences in mean ratings within this domain were plotted. Comparisons of FO with RE and RO with FE were excluded, as they would generate no logical or relevant conclusion. The RO vs RA comparison was also excluded, as its results may be inferred from the other comparisons presented.

Results
Settings and Participants

Among 78 total faculty members, 31 (40%) participated in the survey. See eTable 2 in the Supplement for the numbers, specialties, and years of practice of faculty participants. Among 49 residents, 39 (80%) participated in the survey. The rates of resident participation by PGY level are shown in eTable 3 in the Supplement.

Survey Responses and Data Analysis

In the FO vs RO comparison domain (Figure 1A), Quiz Ref IDresidents at all PGY levels rated the observed resident performance in all global EPAs significantly higher than did faculty members (mean, 3.7 vs 2.8; P < .01), whereas there was no significant difference among the ratings in most operative EPAs (mean, 2.7 vs 2.4; P > .12). In the FO vs RA domain (Figure 1B), faculty ratings of resident performance at all PGY levels were significantly lower than the levels of autonomy reported by residents in global EPAs (mean, 2.8 vs 3.4; P < .01). Faculty ratings of resident performance were significantly higher than the levels of autonomy reported by residents at PGY levels 4 and 5 in the operative EPAs of LA (mean, 3.8 vs 3.0; P = .03), LC (mean, 3.4 vs 2.3; P < .05), EL (mean, 3.0 vs 2.2; P = .05), and RH (mean, 3.1 vs 2.3; P = .04). In the RO vs RE domain (Figure 1C), resident perceptions of capability and expected capability were not significantly different in any EPA at any PGY level. In the FO vs FE domain (Figure 2A), faculty members rated residents at PGY levels 3 to 5 significantly lower than expected in the operative EPAs of IH (mean, 3.0 vs 3.6; P = .02), EL (mean, 2.6 vs 3.3; P = .01), and RH (mean, 2.8 vs 3.4; P = .01). In the RA vs RE domain (Figure 2B), residents at PGY levels 4 and 5 reported significantly lower ratings of autonomy than they expected for their level in all operative EPAs (mean, 2.5 vs 4.3; P < .05). In the RA vs FE domain (Figure 2C), residents at the PGY levels 1 to 3 reported higher ratings of autonomy in global EPAs than those expected by faculty members (mean, 3.1 vs 2.3; P < .03), whereas residents at PGY levels 4 and 5 reported significantly lower ratings of autonomy in all operative EPAs than those expected by faculty members (mean, 2.5 vs 3.9; P < .02).

Figure 3 shows comparisons of the mean differences in individual responses by PGY level. These trends suggest residents and faculty are more likely to perceive that higher-level residents underperform expectations in operative EPAs. There is no such trend among the global EPAs (Figure 3A and B). The most profound mean differences were among residents at PGY levels 3 to 5, who reported that the actual levels of autonomy entrusted in the operative EPAs were substantially lower than the residents expected (Figure 3C).

Mean faculty ratings by clinical specialty of observed (FO) and expected (FE) resident performance of basic and complex operations are shown in Figure 4. The highest FO performance ratings in the Op-B global EPA were in trauma surgery, surgical oncology, and plastic surgery, whereas the highest FE ratings were reported in colorectal surgery, trauma surgery, and pediatric surgery (Figure 4A). The highest FO performance ratings in the Op-C EPA were in surgical oncology, general surgery, and pediatric surgery, whereas the highest FE ratings were reported in colorectal surgery, trauma surgery, and surgical oncology (Figure 4B). When stratified by faculty years of practice, the greatest differences between FO and FE ratings in operative EPAs were among faculty members who had been in practice for 15 to 19 years (Figure 4C).

Quiz Ref IDIn global EPAs, faculty members observed that residents collected information, identified urgencies, and communicated effectively without supervision at a median of PGY level 4. Faculty members observed that residents performed basic operations without supervision at a median PGY level 5. Faculty members expected residents to communicate effectively at a median PGY level 3, to collect information and identify urgencies at a median PGY level 4, and to perform basic operations at a median PGY level 5 (Figure 4D). A total of 68 faculty members (80%) observed that residents required supervision to perform complex operations by graduation; just 48 faculty members (less than 60%) had this expectation. In specific operative EPAs, median PGY levels of observed independence were 4 for LA; 5 for IH, LC, and RH; and never for EL, whereas for expected independence for all EPAs were PGY level 4 (Figure 4E). Faculty members perceived that residents reached independence at later PGY levels than expected in all operative EPAs. Quiz Ref IDApproximately 1 in 5 faculty members indicated they expected residents to require supervision in all 5 operative EPAs for the duration of their training.

A comparison of faculty and resident expectations (FE vs RE) of resident capability found that the expectations of residents were broadly higher than those of faculty members in all global EPAs (mean, 3.9 vs 3.0; P < .01). There was no significant difference between PGY level 5 resident expectations and faculty member expectations in global EPAs; however, a trend suggested that PGY level 5 resident expectations may be higher than those of faculty in Op-B (mean, 5.0 vs 4.2; P = .06) and Op-C (mean, 3.8 vs 3.0; P = .16) EPAs. Few significant differences in expectations were found between faculty and residents in operative EPAs (eFigure in the Supplement).

Discussion

The ACGME and the American Board of Surgery have identified the need for competency-based skills assessments in surgical training. As these skills assessments have become routine, several themes of concern and discord have emerged in the surgical community regarding the capability of residents and expectations of their performance. Because of the granularity and complexity of our data, we have targeted 3 distinct themes from recently published surgical education literature to discuss our findings. First, we address the issue of resident operative performance. Second, we address the discordance of faculty and resident evaluations of resident performance. Last, we address resident perception of autonomy in the context of performance evaluations.

In the past few years, the operative skill set of new graduates from surgical residency programs has become a contentious issue in public discussions. Recent studies suggest that residents generally lack confidence and are not capable of operating independently by graduation.2-4 Our results demonstrated no significant difference in faculty ratings of observed and expected performance in most global EPAs. Faculty members perceived senior residents to underperform expectations in most operative EPAs and in the global EPA of performing complex operations. In addition, we found that most faculty members did not expect residents to achieve independence in performing complex operations during residency. More than 90% of graduates from our program in the past 10 years pursued postgraduate fellowship training. The preponderance of fellowship training has led some to conclude that faculty perceptions of general surgery residents may reflect an expectation of subsequent operative skills acquisition during specialty training. Indeed, at our institution, we observed that faculty members in general surgical specialties without fellowships (ie, colorectal, trauma, surgical oncology, pediatric surgery, and general surgery) generally had higher expectations of global performance in basic and complex operations than did faculty members in specialties with fellowships (ie, transplant, endocrine, vascular, and plastics). Furthermore, specialties with faculty members who were involved in residency program administration gave higher ratings of both observed and expected global performance in basic and complex operations. These global ratings may provide insights into institution-specific attitudes toward resident participation in specialty services.

The discordance of resident and faculty evaluations of resident performance has been the subject of studies in the past 10 years. Some authors have described a tendency among surgical residents to overestimate their operative skill set,1 whereas others have shown that residents underestimate it.12-14 In specific operative skills, residents have a greater degree of confidence than suggested in earlier studies.15 We observed that, compared with faculty ratings, residents rated their performance in most global EPAs higher than faculty ratings, but not in most operative EPAs. Nevertheless, residents perceived that they met their own performance expectations in all EPAs. Residents in our program receive both real-time, structured operative performance evaluations and summative Milestones assessments from our Clinical Competency Committee; accordingly, this finding may be a reflection of the unstructured nature of global performance assessments. Residents at PGY level 5 overestimated their performance in the Op-B and Op-C global EPAs, but the only significant difference among operative EPAs was in LA. These findings support the notion that a rater’s perception of a global assessment in the performance of operations is distinct from an assessment of discrete operative EPAs. We infer from these findings that operative EPAs provide a more direct, actionable means of assessing resident performance. Although global assessments do not accurately signify residents’ acquisition of skills, they may provide insights into prevailing global faculty attitudes within a training program.

Perhaps the most contentious issue in recent discussions in surgical education is resident autonomy. Several researchers in medical and surgical education have determined that providing autonomy is essential to resident training.7,16-18 There are several challenges for surgical faculty, including regulatory hurdles imposed by institutions or by governments, patient safety concerns, public opinion and perceptions thereof, and a prevailing culture of despondency among faculty regarding resident work-hours limitations.17,19-21 Our findings suggest that another challenge to this autonomy issue is faculty perception of autonomy entrusted to residents. In comparing FO with RA (Figure 1B), we determined that residents at PGY levels 1 to 3 indicated a greater degree of autonomy entrusted than that implied by faculty ratings of residents’ global capabilities, whereas there was high concordance among global EPAs for residents at PGY levels 4 to 5. This association was opposite among operative EPAs. Faculty ratings of the capabilities of residents at PGY levels 1 to 3 in operative EPAs had a high degree of concordance with resident ratings of autonomy entrusted, but there is discord among most operative EPAs for residents at PGY levels 4 to 5. This notion is also evident in comparisons of resident perceptions of autonomy with both faculty and resident expectations (Figure 2B and C; Figure 3C). These results suggest that faculty members perceive residents at PGY levels 4 and 5 to have greater capability and expect these residents to have greater capability than the degree of autonomy actually entrusted. From these findings, we recommend further characterizing specific EPAs to guide entrustment of graded autonomy, with a special focus on operative skills at PGY levels 4 and 5.

Discussing public opinion, public policy, and resident work hours is beyond the scope of this article, but we believe a resident’s experience will improve with more exclusive experience with a faculty member. To improve the resident experience within the current program structure, we encourage surgical faculty to devise formal, graded resident autonomy protocols for the crucial steps of operations that residents perform frequently. In the longer term, we suggest a restructuring of the senior years of residency from hospital service–oriented management to faculty-centered practice emulation. As core general surgery training is expected to condense in the near future, subsequent years of surgical training may accommodate this new structure.

The ACGME, American Board of Surgery, and residents all demand increasingly granular metrics of competency in operative performance. We suggest that operative EPAs provide a framework for these metrics and should be expanded to a core set to suit the changing practice profile of graduates from US general surgery residency programs. Our governing bodies will face the challenge of determining threshold competency metrics in essential EPAs for graduating residents to earn an endorsement of their independence. The EPAs regarding global operative performance provide perceptual insights into the culture of faculty and residents within a program, although they do not reliably indicate resident operative proficiency.

Limitations

This study has several limitations. The survey was conducted at the beginning of the academic year, and as a result, evaluations by PGY level may be a skewed interpretation of overall resident performance. The faculty survey responses characterize a single, static perception of a hypothetical resident’s capability or performance; these responses do not refer to any specific resident, group of residents, or a resident’s operative performance in a particular case. Surgeons of various specialties participated in this survey and may have rated resident capabilities in operations that are not routinely relevant to the surgeons’ scopes of practice. Some surgical specialties are underrepresented in this study. We anticipate expanding this pilot survey study to include faculty members and residents in a variety of program settings.

Conclusions

Faculty members perceived that senior residents underperformed in operative EPAs, and most faculty expected residents to still require supervision in complex operations by graduation. Faculty expectations differed by clinical specialty, years in practice, and prevalence of fellowship training. Global and operative EPAs provide a practical means to develop and enact competency-based curricula in general surgery training. Objectively graduated autonomy should be granted to improve the operative experience for residents, especially those at the PGY levels 4 and 5.

Back to top
Article Information

Corresponding Author: O. Joe Hines, MD, Department of Surgery, David Geffen School of Medicine, UCLA, 10833 Le Conte Ave, 72-180 CHS, Los Angeles, CA 90095 (joehines@mednet.ucla.edu).

Accepted for Publication: August 13, 2017.

Published Online: November 15, 2017. doi:10.1001/jamasurg.2017.4547

Author Contributions: Drs Wagner and Hines had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Wagner, Lewis, Quach, Donahue, Hines.

Acquisition, analysis, or interpretation of data: Wagner, Lewis, Tillou, Agopian, Hines.

Drafting of the manuscript: Wagner, Lewis.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Wagner.

Administrative, technical, or material support: Wagner, Quach, Hines.

Study supervision: Lewis, Tillou, Agopian, Donahue, Hines.

Conflict of Interest Disclosures: None reported.

Meeting Presentation: This paper was presented at the 88th Annual Meeting of the Pacific Coast Surgical Association; February 20, 2017; Indian Wells, California.

References
1.
Perone  JA, Fankhauser  GT, Adhikari  D,  et al.  Who did the case? Perceptions on resident operative participation.  Am J Surg. 2017;213(4):821-826.PubMedGoogle ScholarCrossref
2.
Bell  RH  Jr.  Why Johnny cannot operate.  Surgery. 2009;146(4):533-542.PubMedGoogle ScholarCrossref
3.
Bucholz  EM, Sue  GR, Yeo  H, Roman  SA, Bell  RH  Jr, Sosa  JA.  Our trainees’ confidence: results from a national survey of 4136 US general surgery residents.  Arch Surg. 2011;146(8):907-914.PubMedGoogle ScholarCrossref
4.
Mattar  SG, Alseidi  AA, Jones  DB,  et al.  General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors.  Ann Surg. 2013;258(3):440-449.PubMedGoogle ScholarCrossref
5.
American Board of Surgery. Booklet of Information—Surgery 2016-2017. Philadelphia, PA: American Board of Surgery; 2017.https://www.absurgery.org/xfer/BookletofInfo-Surgery.pdf. Accessed June 19, 2017.
6.
Accreditation Council for Graduate Medical Education and American Board of Surgery. The General Surgery Milestones Project. http://www.acgme.org/acgmeweb/portals/0/pdfs/milestones/surgerymilestones.pdf. Published July 2015. Accessed September 9, 2016.
7.
Hirschl  RB.  The making of a surgeon: 10,000 hours?  J Pediatr Surg. 2015;50(5):699-706.PubMedGoogle ScholarCrossref
8.
ten Cate  O.  Entrustability of professional activities and competency-based training.  Med Educ. 2005;39(12):1176-1177.PubMedGoogle ScholarCrossref
9.
Sterkenburg  A, Barach  P, Kalkman  C, Gielen  M, ten Cate  O.  When do supervising physicians decide to entrust residents with unsupervised tasks?  Acad Med. 2010;85(9):1408-1417.PubMedGoogle ScholarCrossref
10.
George  BC, Teitelbaum  EN, Meyerson  SL,  et al.  Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance.  J Surg Educ. 2014;71(6):e90-e96.PubMedGoogle ScholarCrossref
11.
Rekman  J, Gofton  W, Dudek  N, Gofton  T, Hamstra  SJ.  Entrustability scales: outlining their usefulness for competency-based clinical assessment.  Acad Med. 2016;91(2):186-190.PubMedGoogle ScholarCrossref
12.
Peyre  SE, MacDonald  H, Al-Marayati  L, Templeman  C, Muderspach  LI.  Resident self-assessment versus faculty assessment of laparoscopic technical skills using a global rating scale.  Int J Med Educ. 2010;1:37-41. doi:10.5116/ijme.4bf1.c3c1Google ScholarCrossref
13.
Lyle  B, Borgert  AJ, Kallies  KJ, Jarman  BT.  Do attending surgeons and residents see eye to eye? An evaluation of the Accreditation Council for Graduate Medical Education Milestones in General Surgery Residency.  J Surg Educ. 2016;73(6):e54-e58.PubMedGoogle ScholarCrossref
14.
Alameddine  MB, Claflin  J, Scally  CP,  et al.  Resident surgeons underrate their laparoscopic skills and comfort level when compared with the rating by attending surgeons.  J Surg Educ. 2015;72(6):1240-1246.PubMedGoogle ScholarCrossref
15.
Friedell  ML, VanderMeer  TJ, Cheatham  ML,  et al.  Perceptions of graduating general surgery chief residents: are they confident in their training?  J Am Coll Surg. 2014;218(4):695-703.PubMedGoogle ScholarCrossref
16.
Klingensmith  ME, Lewis  FR.  General surgery residency training issues.  Adv Surg. 2013;47:251-270.PubMedGoogle ScholarCrossref
17.
Rosenbaum  L.  Leaping without looking—duty hours, autonomy, and the risks of research and practice.  N Engl J Med. 2016;374(8):701-703.PubMedGoogle ScholarCrossref
18.
Wojcik  BM, Fong  ZV, Patel  MS,  et al.  The resident-run minor surgery clinic: a pilot study to safely increase operative autonomy.  J Surg Educ. 2016;73(6):e142-e149.PubMedGoogle ScholarCrossref
19.
Kempenich  JW, Willis  RE, Rakosi  R, Wiersch  J, Schenarts  PJ.  How do perceptions of autonomy differ in general surgery training between faculty, senior residents, hospital administrators, and the general public? a multi-institutional study.  J Surg Educ. 2015;72(6):e193-e201.PubMedGoogle ScholarCrossref
20.
Teman  NR, Gauger  PG, Mullan  PB, Tarpley  JL, Minter  RM.  Entrustment of general surgery residents in the operating room: factors contributing to provision of resident autonomy.  J Am Coll Surg. 2014;219(4):778-787.PubMedGoogle ScholarCrossref
21.
Meyerson  SL, Teitelbaum  EN, George  BC, Schuller  MC, DaRosa  DA, Fryer  JP.  Defining the autonomy gap: when expectations do not meet reality in the operating room.  J Surg Educ. 2014;71(6):e64-e72.PubMedGoogle ScholarCrossref
×