Predictors of Future Success in Otolaryngology Residency Applicants

Objective To evaluate the information available about otolaryngology residency applicants for factors that may predict future success as an otolaryngologist.

Design Retrospective review of residency applications; survey of resident graduates and otolaryngology clinical faculty.

Setting Otolaryngology residency program.

Participants Otolaryngology program graduates from 2001 to 2010 and current clinical faculty from Barnes-Jewish Hospital/Washington University School of Medicine.

Main Outcome Measure Overall ratings of the otolaryngology graduates by clinical faculty (on a 5-point scale) were compared with the resident application attributes that might predict success. The application factors studied are United States Medical Licensing Examination part 1 score, Alpha Omega Alpha Honor Medical Society election, medical school grades, letter of recommendation, rank of the medical school, extracurricular activities, residency interview, experience with acting intern, and extracurricular activities.

Results Forty-six graduates were included in the study. The overall faculty rating of the residents showed good interrater reliability. The objective factors, letters of recommendation, experience as an acting intern, and musical excellence showed no correlation with higher faculty rating. Rank of the medical school and faculty interview weakly correlated with faculty rating. Having excelled in a team sport correlated with higher faculty rating.

Conclusions Many of the application factors typically used during otolaryngology residency candidate selection may not be predictive of future capabilities as a clinician. Prior excellence in a team sport may suggest continued success in the health care team.

The goals of otolaryngology specialty training are to train young physicians to become independent medical practitioners who are equipped to effectively diagnose and treat patients with maladies of the head and neck. Annually, the leaders of approved residency training programs select trainees from qualified senior medical students and medical graduates. Residents enter a 5- to 7-year training program and, with the program director's approval, are eligible to take the American Board of Otolaryngology (ABOto) certifying examination. If the training program is successful, graduates exemplify the desired characteristics and skills. Although the exact qualities of a well-prepared otolaryngologist might be debated, the characteristics listed on a typical hospital credentialing form are generally accepted:

• Knowledgeable

• Good judgment

• Ethical

• Competent and skillful

• Cooperative

• Thorough

• Empathetic

• Available

• Works well with others

The Accreditation Council for Graduate Medical Education (ACGME) Core Competencies for resident education also emphasize similar characteristics:

• Patient care

• Medical knowledge

• Practice-based learning

• Interpersonal and communication skills

• Professionalism

• Systems-based practice

Despite carefully planned eligibility screening and personal interviews, not all residents who complete training exhibit the excellence that is desired by their mentors and the leaders of the training programs. There are many costs associated with an underperforming resident or a resident who departs a training program.¹ In this era of increasing complexity and public scrutiny of health care, selection of future otolaryngologists who exhibit a wide range of excellent characteristics becomes increasingly important.

Although many studies have investigated qualities that may or may not predict “success” during residency,^2-10 few have looked at how factors used to select residents correlate with performance as a practicing physician,¹¹ and the results are often contradictory. In addition, various parameters are used to determine “success,” which may or may not be meaningful determinants of a competent practitioner.

Defining excellence in a physician is a complex judgment. However, faculty who train residents have no difficulty in making such a judgment: “ . . . I know it when I see it.” In this study we compared the overall faculty rating of graduates of the Barnes-Jewish Hospital/Washington University in St Louis (Missouri) otolaryngology residency program with the residency application of these same individuals when they applied as medical students or medical school graduates. Success, in this context, is defined as having characteristics of an excellent physician after graduation from residency training.

Forty-six consecutive graduates from the Department of Otolaryngology–Head and Neck Surgery at Washington University from 2001 to 2010 were identified. One resident left the program during this time and was not included in the analysis. This study was approved by the Washington University institutional review board (identification number 201204130).

Each of the current faculty clinicians (N = 25) in otolaryngology was asked to rate these 46 graduates with regard to their opinion of his or her overall quality as a clinician:

Faculty members were not required to rate every graduate, and they were asked to rate only those with whom they had personal experience. No other instructions were provided; each rater acted independently. These subjective evaluations were tabulated for each graduate. Other characteristics of excellence such as academic achievement and leadership were not specifically addressed.

Clinical faculty within the department were asked their opinion about factors that might predict positive outcome of postgraduate training. Factors to be were most likely to be predictive are as follows:

• United States Medical Licensing Examination (USMLE) score

• Alpha Omega Alpha Honor Medical Society (AOA)

• Percentage of “honors” or equivalent grade in clinical rotations

• Letters of recommendation

• Quality of the medical school

• Residency interview by faculty

• Performance as an acting intern (fourth-year medical student)

• Extracurricular activities and/or talents

• Musical training or experience

• Athletic training or experience

Many of these factors are thought to represent academic excellence and are frequently used for evaluation purposes for residency positions (USMLE scores, AOA election, medical school grades, letters of recommendation, medical school rank). Performance as an acting intern, prolonged specific experience with the applicant, was thought to provide additional insight. In addition, on subjective ranking within our department in the past, this has affected final rank list position both positively and negatively. Extracurricular activities were also included, as they can be thought to provide “extra” insight into the applicant's experience and interests.

The residency applications for the graduate cohort were reviewed by one of us (R.A.C.) for the identified parameters.

The USMLE part 1 scores were available for almost all graduates (44 of 46). Nationally, USMLE Part 1 scores ranged from 185 (pass) to 275 with a mean of approximately 216.

Most, but not all, medical schools offer AOA membership, recognizing superior performance in medical school: generally this represents the top 10% of the graduating class. Not all medical schools have AOA, reported as “yes” or “no.” If AOA information was not available, this was recorded as “no.”

Residency applicants were senior medical students and qualified medical school graduates invited for personal interviews. A committee of faculty members interviewed the applicants in one-on-one interviews of 25 to 30 minutes. Each interviewer rated the applicant on a 7-point modified Likert scale: 1, poor; 4, average; 6, excellent; and 7, outstanding.

The percentage of honors in clinical rotations was expressed as the number of honors divided by the total number of clinical rotations.

Many students did clinical rotations or acting internships at Washington University/Barnes Jewish Hospital. This rotation was tabulated as “yes” or “no.”

An arbitrary classification was used to assign a score to the letter by a single reviewer (R.A.C.):

The US News & World Report ranking of medical schools (n = 150) was used in this study.¹² The schools were placed into deciles with the top 15 schools being in the top decile and the bottom 15 schools being in the lowest decile.

The applications were found to be relatively deficient in information detailing extracurricular activities. The graduates were then contacted to fill out a simple survey regarding their preresidency experience in music and athletics (extracurricular activities were not assessed).

The 18 directors of the ABOto (2010) were asked about their preresidency qualifications. It was assumed that these otolaryngologists were successful graduates of otolaryngology programs. They were recognized by their peers as leaders in otolaryngology. They have had no serious medical practice transgressions and no serious personal transgressions.

The overall faculty rating scale was compared with each variable described herein using the R² coefficient of determination to characterize the proportion of the outcome (“success”) that can be explained by the specific variable. The reported values range from 0 (no correlation) to 1 (perfect correlation). Because this analysis of these data involves multiple comparisons, there is an increase in the possibility of a type I error. A conservative Bonferroni correction would yield an α = 0.006 (α = 0.05).

Forty-six residency graduates were identified (11 women and 35 men) between 2001 and 2010. The subjective faculty rating (Figure 1) had good interrater reliability. The residency applications were complete in 44 of 46. The extracurricular survey responses were obtained for 40 of 46.

Comparison of the faculty rating with the possible predictive factors showed some correlations. Where applicable, the data from the ABOto directors are included for comparison. There is no significant correlation between faculty rating scale and USMLE score (Figure 2), AOA status (Figure 3), percentage of honors in medical school clinical rotations (Figure 4), letters of recommendation (Figure 5), having been an acting intern at our institution (Figure 6), and excellence in music (Figure 7). There is a weak correlation with the interview score (P = .04, R² = 0.1) (Figure 8) and rank of the medical school (P = .03, R² = 0.18) (Figure 9). Excellence in athletics, specifically a team sport, is significantly correlated with the faculty rating scale (P < .001, R² = 0.32) (Figure 10). The ABOto directors had a similar distribution to the resident graduates and did not vary significantly in any particular variable (Figures 2, 3, 7, 9, and 10).

The purpose of a residency training program is to train clinically competent and successful otolaryngologists. Depending on the emphasis of the particular program, clinical strength and/or academic and leadership pursuits may be variably included in a definition of a successful outcome. Much has been studied regarding the variables that could potentially predict such an outcome; however, most studies have focused on comparisons of the application and assessment tools within residency.^2-4,6-8 In this study, we attempted to correlate factors available at the time of the potential resident's application to our program with the overall faculty rating of this same individual after graduation with regard to clinical competence.

The rating scale used here is arbitrary; however, we feel this captures the “essence” of a resident. Those involved with resident education can fairly easily distinguish a “great” graduate from a “good” or (gulp) “poor” trainee. In this capacity, a graduate can be determined to be excellent or outstanding regardless of the career path he or she has chosen. The determination of success as a clinician in our study does not depend on factors such as leadership positions or academic appointment, but rather subjective qualities as would be assessed by a hospital credentialing committee, physician peers, and the public. Such data provide the opportunity for a practical and meaningful assessment of the factors used to choose residency candidates.

In this, as in other studies investigating predictive factors of success, a bias certainly exists in that the participants (ie, residency graduates) are a highly selected group a priori. All have strong academic backgrounds (undergraduate and graduate education) and are generally highly motivated individuals. Determining statistically significant differences within this cohort to predict success is certainly challenging. Our rating scale assessed each individual separately and did identify differences in faculty perception. This method could introduce bias favoring a more personable individual. However, teamwork and interpersonal skills are important qualities in a physician. This is reflected in the ACGME Core Competencies, specifically patient care, professionalism, communication and interpersonal skills, and systems-based learning.

Typically, with residency applications, much emphasis is placed on the objective data available. However, in this study, there is no correlation between successful outcome and USMLE scores, medical school grades, or AOA status. Several studies have determined certain of these characteristics to have some predictive value,^2,4,5,11,13 yet others find no clear significant factors.⁷ USMLE score was predictive of success in several trials; however, an outcome measure of these trials was in-service training examination scores.^2,14 It bears consideration that ability with test-taking may predict further success with testing. While assessing fund of knowledge is certainly a critical component of residency training and board certification, after reaching a certain cutoff, the importance of the particular score may have less relevance. A pass/fail distinction exists for USMLE; however, further distinctions are not well defined and are not supported by this organization.¹⁵ This calls into question the relative weight that these parameters should exhibit during evaluation of a potential residency trainee.

The rank of the medical school attended by the otolaryngology resident applicant showed correlation, albeit weak, with the faculty rating scale. One could speculate that graduates from top medical schools are already a highly selected group and thus could suggest future success. One earlier study, however, had no correlation with medical school reputation but rather with the home institution.⁹ Our associated medical school is ranked in the top decile; this may introduce bias because we typically have had our graduates well represented among the residents.

Subjective variables (letters of recommendation, residency interview, experience with acting intern, and dean's letters) have been deemed important parts of the residency application. The residency interview has been shown to have poor predictive value for future performance,^6,15,16 and this holds true in our study as well. Having functioned as an acting intern did not predict, nor did the letter of recommendation. Both would suggest a potential deeper insight into a residency applicant. Perhaps a ceiling effect where all applicants are “above average” limits the usefulness of this information. Several studies have taken the opposite approach and have investigated factors that may be predictive of future problems.^14,17-19 In one study, negative comments in the medical school dean's letter were predictive of future problems within residency; no other factors studied were predictive.¹⁴ Low score on the Medical Council of Canada Clinical Skills Examination was predictive of retained complaints by patients during the first 2 to 12 years of practice.¹⁷

In our study, the most predictive variable of future success is excellence in athletics, specifically team sports. A recent evaluation of general surgery applicants showed that the presence of a “special skill”—athletics, music, and/or hobby—was negatively correlated with attrition.² An earlier study,⁷ also of general surgery residents, noted more graduates with a history of team sports in the top 10 on a faculty rating scale compared with none in the bottom 10. Much emphasis is placed in health care and in the operating environment, particularly on the importance of teamwork. If a resident applicant has a history of success in a team setting, it stands to reason that he or she is likely to continue to thrive in this environment.

Given the difficulties with relying on the objective and subjective data that current applications are composed of, how should a residency program assess a potential trainee? One argument would be that some would shine, regardless of the process used in the evaluation, and this is borne out by this study, as well as the conglomerate of other studies. An alternative, however, would be to use job-analysis techniques, manual dexterity testing, personality profiles, and/or other metrics.^15,16,20 Currently, these strategies are controversial and are potentially regarded as negative by the applicant. They do, however, offer potentially meaningful objective data by which to assess a candidate. A recent study by Prager et al¹⁶ investigated the feasibility of behavioral-based questions within the framework of the traditional otolaryngology residency interview. Lee et al¹⁵ explored options of using evidence-based objective criteria with behavioral-based questions for noncognitive traits in ophthalmology residency applications. Manual dexterity testing and visuospacial skills are not currently evaluated during the residency application process, yet are arguably considerable factors for surgical (and certain nonsurgical) fields of medicine.

The present study is limited by encompassing a single institution and a single subspecialty. In addition, the data are not complete for all residency graduates. For some earlier graduates, recall bias may exist, and newer faculty may not have had experience with these graduates.

In conclusion, the traditional measures of residency applicants are not necessarily predictive of success. Excellence in a team sport may be predictive of future excellence in medicine. Given the generally high quality of otolaryngology applicants, we both benefit from, and are limited by, a ceiling effect.

Correspondence: Richard A. Chole, MD, PhD, Department of Otolaryngology–Head and Neck Surgery, Washington University School of Medicine, 660 S Euclid Ave, Campus Box 8115, St Louis, MO 63110 (rchole@wustl.edu).

Submitted for Publication: April 26, 2012; accepted May 29, 2012.

Author Contributions: Both authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Chole and Ogden. Acquisition of data: Chole. Analysis and interpretation of data: Chole and Ogden. Drafting of the manuscript: Chole and Ogden. Critical revision of the manuscript for important intellectual content: Chole. Statistical analysis: Chole. Administrative, technical, and material support: Chole. Study supervision: Chole.

Previous Presentation: This study was presented at the Annual Meeting of the Society of University Otolaryngologists; November 12, 2010; Scottsdale, Arizona.

Additional Contributions: We thank the residents and faculty in the Department of Otolaryngology–Head and Neck Surgery, Washington University School of Medicine, for their expertise and assistance.