Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020

This quality improvement study assesses progress from 2019 to 2020 in a Core Entrustable Professional Activities entrustment decision-making program for graduating medical students entering residency.


Introduction Problem Description
Gaps in readiness for indirect supervision upon transition to residency have been identified for important resident responsibilities. 1,2Entrustable Professional Activities (EPAs) 3 have been proposed as a framework to advance competency-based medical education (CBME), 3 focusing assessment on observable day-to-day professional activities 4 and the level of supervision required for each learner.

Rationale
Aiming to address these gaps and to advance CBME in UME, the Association of American Medical Colleges (AAMC) convened a panel to draft a set of Core EPAs for Entering Residency 21,22 (Table 1).
In a survey of internal medicine program directors, most respondents indicated that graduating students "must" or "should" possess skills to perform most Core EPAs without direct supervision. 23 Association of Program Directors in Surgery statement concludes, "Students should achieve entrustability" in Core EPAs. 24Feasibility of small-scale (approximately 4 students per year at each of 4 schools), time-variable advancement from UME to GME based in part on Core EPAs has been demonstrated in the Education in Pediatrics Across the Continuum initiative. 25,26 2014, the AAMC convened 10 US medical schools to explore feasibility of implementing the Core EPA framework, including summative entrustment decision-making, for entire classes of students.Pilot schools established steering committees; concept groups for curriculum and assessment, 27 faculty development, 28 and entrustment 29 ; and 13 EPA-specific workgroups.Early pilot work focused on developing tools for curriculum and assessment, 30,31 fostering faculty development, 32 and establishing entrustment committee structures modeled on the GME clinical competency committee. 33ained faculty were convened to make determinations about readiness for indirect supervision looking at multimodal performance data, including Workplace Based Assessments (WBAs) with entrustment-supervision scales. 34,35These determinations were generally completed within a day after direct observation in the workplace. 36,37

Specific Aims
Lomis et al 38 described a principal aim of the pilot: to develop theoretical summative determinations of each student's readiness to perform 13 Core EPAs in 2019 "to inform the feasibility of

JAMA Network Open | Medical Education
implementing the Core EPAs construct in UME programs."In an evaluation of decision-making for the 2019 graduating cohort, fewer than half of EPA-specific entrustment determinations had a finding of "ready for indirect supervision," and more than one-quarter had a report of "could not make an entrustment determination." 39This study aimed to evaluate what further progress was made with the 2020 graduating cohort in implementing programmatic assessment 14,40,41 and establishing a summative entrustment process using AAMC Core EPAs.

Methods
This quality improvement study was reported using the Standards for Quality Improvement Reporting Excellence (SQUIRE) 2.0 reporting guidelines.Human Research Protection Program staff at the AAMC determined that this study was exempt from further institutional review board (IRB) review and informed consent because it did not constitute human participants research as defined in 45

Initial Intervention Steps and Their Evolution Over Time
The AAMC and each institution (through letters from the dean of the medical school and the curriculum committee) committed to a 5-year pilot, initiated in July 2014.The AAMC and all 10 institutions subsequently agreed to a 2-year extension.
The first year served as a planning phase, in which participating institutions developed as a community, established working groups, developed guiding principles, and planned curricula, assessment strategies, faculty development, and pathways to entrustment. 38Each school agreed to pilot a minimum of 4 EPAs.
With the entering class of 2015, institutions endeavored to initiate curricula, assessments, and faculty development.Each institution developed plans to render nonbinding entrustment determinations for graduating students.Schools that initiated implementation for the entering class of 2015 developed plans for trained faculty groups to make theoretical readiness determinations starting with the 2019 graduating class.
The entrustment workgroup of the pilot used an iterative process of discussions, data collection, and reflection to describe principles, plans, and activities related to the entrustment process 29 ; choices schools made about the entrustment process, why choices were made, and challenges with the entrustment process 33 ; data considered for each EPA; and results of entrustment decision-making for the first graduating cohort. 39e

Context
Implementing a centralized entrustment process at a medical school involves a major curricular change and poses a variety of cultural, logistical, analytic, psychometric, and ethical challenges. 33aluation of entrustment decision-making for the 2019 graduating cohort highlighted multifactorial challenges in assessment of some of these activities in the workplace, 39 suggesting the need to consider curriculum content revisions and increase availability of WBAs and other assessment data.

Interventions
Schools implemented various improvements based on this first cohort of data collection.These changes included increasing the number of required WBAs for some EPAs, increasing the number of end-of-rotation assessments mapped to EPAs, enhancing curriculum, enhancing data visualization for the entrustment process, providing additional faculty development, and using alternative methods, such as simulation to assess skills.For the study of interventions, 4 schools piloted entrustment decision-making and shared deidentified data for some or all graduating students in both 2019 and 2020.

Measures
The entrustment workgroup, AAMC staff, and Core EPA team leaders collaborated to create a data set for evaluation of entrustment decision-making outcomes across sites.These groups jointly determined that individual-level, deidentified data would be shared for multischool analysis.
Among the uniform set of items at the individual level collected for each EPA-specific instance of entrustment decision-making were readiness determinations (1 of 4 choices: ready for indirect supervision, progressing but not yet ready for indirect supervision, not progressing toward readiness for indirect supervision, or could not make a determination) and number of WBAs available for that determination (choices of 0, 1-3, 4-10, 11-15, and >15).Based on preliminary analysis of volumes of

JAMA Network Open | Medical Education
WBAs available, we created a dichotomous variable for WBAs available (0-3 vs Ն4).Schools also incorporated other available assessments into the entrustment process, as described previously. 33,39

Statistical Analysis
Data were analyzed for EPAs that each participating school considered in both 2019 and 2020.Each school considered all students or a similarly sized, randomly selected subset in both years.
Proportional representation from each school was roughly similar by year.We compiled descriptive statistics and assessed between-year differences in percentages using 2-sample test of proportions, and we assessed associations between entrustment determinations and number of WBAs using χ 2 , with a 2-sided P < .05considered significant.All analyses were performed using Stata statistical software version 17 (StataCorp).3).Proportions with determinations that they were ready for indirect supervision for 2019 ranged from, for example, 0 of 125 students for EPA 13 to 164 of 220 (74.5%) students for EPA 7 (Table 3).

Contextual Elements That Interacted With Interventions
The creation of toolkits including key functions, associated competencies, and behavioral expectations for each EPA helped to develop a shared mental model across schools. 30The pilot engaged in a "goldfish bowl" training exercise for level-setting prior to starting to make formal entrustment determinations.The formative nature of the pilot was associated with the robustness of implementation.Entrustment determinations did not have high-stakes summative implications.
School-specific differences regarding final-year rotation requirements could also have been associated with these outcomes. 42Suspension of face-to-face clinical activities due to the COVID-19 pandemic in 2020 may have been associated with the numbers of WBAs for some EPAs.
Entrustment committees met face to face in 2019 and virtually in 2020, which may also have been associated with outcomes. 16Experiences working as a team in 2019 may have been associated with The distribution of types of entrustment determinations are presented only among students for whom determinations were made for 3 choices of "not progressing towards readiness," "progressing towards readiness," and "ready for indirect supervision," with "could not make a determination" data excluded.See eTable in Supplement 1 for Entrustable Professional Activity (EPA)-specific changes in 2020 vs 2019 in proportions deemed ready.The change in the proportion ready for indirect supervision was significant for EPAs 1, 2, 3, 6, 7, and 12.
reduced challenges in working virtually to some extent.Teams were more experienced in the process in 2020 than in 2019.Faculty also may have recalibrated as they became more used to the process, returning to a "stance of presuming readiness." 7

Observed Associations of Interventions and Relevant Contextual Elements with Outcomes
As shown in Table 4, distribution of entrustment determinations by WBA availability varied across EPAs.The presence of 4 or more WBAs (vs 0-3 WBAs) was generally associated with higher proportions of ready and progressing determinations and lower proportions of not progressing determinations or could not make determination outcomes.For example, for 141 determinations with 4 or more WBAs vs 41 determinations with 0 to 3 WBAs in EPA 2, there were 97 determinations (68.8%) vs 3 determinations (7.3%) that were ready, 43 determinations (30.5%) vs 3 determinations that were progressing, 1 determination (0.7%) vs 0 determinations that were not progressing, and 0 determinations vs 35 determinations (85.4%) that could not be made (P < .001)(Table 4).
However, this was not so for every EPA.For example, for EPAs 1, 5, and 6, ready for indirect supervision determinations were made for approximately two-thirds of students whose EPA-specific <.001 0-3

Unintended Consequences
Entrustment under the Core EPA Pilot guiding principles was a complex intervention.It shed light on systems required for programmatic assessment and data compilation and highlighted a lack of workplace roles for students for several key clinical tasks.

Details About Missing Data
Due to varying time frames for implementation, 2 schools provided entrustment data to the AAMC for graduating students in 2020 only (and so were not eligible to be included in this study).Due to local differences in implementation, data compilation issues, and disruptions related to the COVID-19 pandemic, the remaining 4 schools did not generate data for the multi-institutional data set in either 2019 or 2020.For schools that implemented more than the minimum of 4 EPAs, we excluded EPA-specific data that pertained to a given EPA implemented at a school and considered by its entrustment committee in only 1 of these years.

Summary
In this quality improvement study, the proportions of decision-making instances for which a determination could be made, that were ready for indirect supervision, and that were informed by 4 or more WBAs increased in 2020 compared with 2019 overall and also on an EPA-specific basis for EPAs 1 to 3 and 6.We also observed increases in 2 of these 3 measures for EPAs 5, 7, 9, and 12.
Together, these increases suggest progress in implementation of a program of assessment using the Core EPAs framework.After reviewing first cohort data, schools increased the number of WBAs required and found additional places for curriculum and assessments, which was associated with more available data for entrustment committees to consider for the second cohort.Schools also worked to improve display of data and training for entrustment committees.However, in contrast to progress made for 8 of 13 EPAs, determinations for the remaining 5 EPAs (EPAs 4, 8, 10, 11, and 13) remained challenging given that WBA availability did not increase and fewer than 10% of students considered for each of these EPAs were determined to be ready for indirect supervision in either year.

Interpretation
The highest proportions of readiness for indirect supervision was observed among a subgroup of EPAs broadly taught and assessed throughout the UME curricula, including EPAs 1 to 3, 5 to 7, and 9.
Meyer et al 44 contextualized Core EPAs within the Reporter-Interpreter-Manager-Educator (RIME) observed in 2020 that there were WBAs available for these EPAs for the most part and most students for whom entrustment decision-making was undertaken in these EPAs were determined to be ready to perform these activities with indirect supervision.Relatively high proportions of students in our study for whom entrustment decision-making was undertaken in EPAs 7 and 9 were determined to be ready for indirect supervision for these 2 EPAs.Meyer et al 44 suggested that these EPAs aligned with the high-level educator role; however, pilot schools and others 43 found that these EPAs were "well represented in the UME experience." 45As 4 and 8 may align better with the senior level subinternship curriculum. 42,43,45However, even at this level, opportunities to perform these EPAs may be limited at participating schools or supervision may be "not sufficiently intentional to collect evidence robust enough to substantiate entrustment decisions." 45As 10, 11, and 13 involve high-level skills (ie, manager or educator in the RIME framework).Our findings suggest that for these skills, current clinical environments may not provide students the opportunity for meaningful workplace participation or assessment. 33,43,45A 10 (urgent care) is commonly taught through didactics and simulation, but meaningful opportunities for students to demonstrate and be assessed on skills in the clinical environment prior to graduation may be quite limited.EPA 12 includes several procedural skills (eg, intravenous line, bladder catheterization, cardiopulmonary resuscitation, and bag-mask ventilation) 30 for which the role of physicians and residents, nurses, and other health professionals varies by specialty and location; there were few WBAs available in this study; and simulation is commonly used in teaching and assessment.

Limitations
This study has several limitations.The primary data for the main outcome measure came from 4 schools and 2 cohorts of a formative pilot of Core EPAs implementation.Furthermore, except for EPA 1 (for which we had data from all students in our data set), we had EPA-specific data for various subsets of students in the entire data set because of school differences in particular EPAs implemented.A further limitation of our study is that restrictions on uses of the multischool data set created for our study precluded identification of single-school data sets or cross-school comparisons, which may have been informative.
WBAs were available generally in numbers below what would be ideal for truly high-stakes decisions. 9,34,36,37Availability of WBAs may also be a surrogate for overall volume of assessment data typically collected on third-year clerkships. 43The validity and reliability of entrustment-focused assessments is limited 36,37 and has been challenged on the grounds of subjectivity 46 measurementrelated issues, 8,47 practical challenges, 7,45 and default to presumption of readiness. 7

Conclusions
Our findings highlight substantial challenges in making prospective determinations about readiness of graduating medical students for indirect supervision in the Core EPAs framework at the scale of entire medical school classes. 7,8,33,45Results also suggest important gaps in readiness for a subset of Core EPAs (ie, EPAs 4, 8, 10, 11, and 13). 45Among next steps suggested by these results are improvement in curriculum and assessment specific to these EPAs at participating schools; further improvements in systems for programmatic assessment and data curation; and consideration of revision of what are considered Core EPAs on a specialty-specific basis for incoming interns. 45entifying opportunities for direct observation, feedback, and faculty development related to assessment of these activities may help improve readiness.Nonetheless, there may be limited opportunity for meaningful participation and observation of these activities with advancement to JAMA Network Open | Medical Education performance of activities under indirect supervision prior to graduation for a variety of reasons involving culture and policy.It may be useful to implement a postmatch handoff process so that orientation boot camps, mentorship, and initial-year residency responsibilities may be tailored to specific educational needs of each incoming resident.Awareness of these gaps remains important for program directors to maintain patient safety and support medical school graduates' educational needs upon starting residency.UME and GME educators need a shared mental model regarding required skills for incoming residents and what responsibility lies with medical schools vs specialtyspecific organizations. 48 CFR §46 given that the AAMC access only deidentified data.At 2 of 4 participating institutions (Columbia University Vagelos College of Physicians and Surgeons and Vanderbilt University School of Medicine), the study was submitted for IRB review as a new study; at the other 2 participating institutions (Florida International University Herbert Wertheim College of Medicine and McGovern Medical School at UTHealth Houston), the study was submitted for IRB review as a modification of a preexisting, IRB-reviewed protocol for the entire Core EPAs pilot study.At all participating institutions, this study was deemed exempt from further IRB review and informed consent as defined in 45 CFR §46.
to local circumstances (eg, curriculum overhaul or Liaison Committee for Medical Education site visit), some schools elected to extend planning and begin implementation with the entering class of 2016 or 2017 (corresponding to the 2020 or 2021 graduating class).A decision to extend the pilot was made in 2018.An initial round of entrustment determinations and compilation of data was completed in 2019 (for the graduating class of 2019), and a second round was completed in 2020 (for the graduating class of 2020).

Figure
Figure.Distribution of Entrustment Determinations

Table 1 .
Entrustment Decision-Making Outcomes a bThe percentage is the number of students who were ready plus those progressing plus those not progressing divided by the number of students with data for the EPA.

Table 2 .
Availability of ≥4 WBAs a WBA data missing for 1 student for EPA 5 due to an oversight.

Table 3 .
Percentage of Students Determined as Ready for Indirect Supervision by EPA and RIME Category a Total No. varies because not all students had data for each EPA.

Table 4 .
43trustment Determination by WBA Availability (2019 and 2020 Combined) (continued) EPAs 1, 2, 3, 5, and 6 align with the reporter or interpreter level.Colbert-Getz et al43found that EPAs 1, 2, 5, 6, and 9 were commonly addressed in clerkship narrative assessments.With process improvement put in place after the 2019 cohort of entrustment decision-making, we b χ 2 test of association.c No students with 4 or more WBAs for this EPA.JAMA Network Open | Medical Education framework.