[Skip to Content]
[Skip to Content Landing]
Original Investigation
December 6, 2016

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Author Affiliations
  • 1University of California at San Francisco
  • 2American Board of Internal Medicine, Philadelphia, Pennsylvania
  • 3Hess Consulting, Lévis, Québec, Canada
  • 4Accreditation Council for Graduate Medical Education, Chicago, Illinois
  • 5Commonwealth Medical College, Scranton, Pennsylvania
JAMA. 2016;316(21):2253-2262. doi:10.1001/jama.2016.17357
Key Points

Question  What information do milestone-based ratings add to the former nondevelopmental rating system for internal medicine residents regarding their competence, medical knowledge, and professionalism?

Findings  In a cross-sectional study involving 21 284 internal medicine residents, milestone-based assessment was correlated with nondevelopmental ratings, but with a greater difference across the training years; it was also correlated with American Board of Internal Medicine certification examination scores.

Meaning  These findings provide some preliminary evidence to support the validity of milestone-based assessment.


Importance  US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed.

Objective  To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings.

Design, Setting, and Participants  Cross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3).

Exposures  Program director ratings on the RAES and milestone ratings.

Main Outcomes and Measures  Correlations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems.

Results  Corresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values <.05). Linear regression showed ratings differed more between PGY-1 and PGY-3 years using milestone ratings than the RAES (all P values <.001). Of the 6260 residents who attempted the certification examination, the 618 who failed had lower ratings using both systems for medical knowledge than did those who passed (RAES difference, −0.9; 95% CI, −1.0 to −0.8; P < .001; milestone medical knowledge 1 difference, −0.3; 95% CI, −0.3 to −0.3; P < .001; and medical knowledge 2 difference, −0.2; 95% CI, −0.3 to −0.2; P < .001). Of the 26 PGY-3 residents with milestone ratings indicating deficiencies on either of the 2 medical knowledge subcompetencies, 12 failed the certification examination. Correlation of RAES ratings for professionalism with residents’ lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45; P < .001).

Conclusions and Relevance  Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.