Association of Demographic and Program Factors With American Board of Surgery Qualifying and Certifying Examinations Pass Rates

Key Points

Question  Is there an association between sociodemographic factors and American Board of Surgery board examination passage rates?

Findings  In this study of 662 individuals, trainees of Hispanic ethnicity were more likely to not attempt either of the board examinations (qualifying and certifying examinations), and examinees who were married with children were more likely to fail the examinations. White examinees were more likely to pass the qualifying examination on the first attempt; white examinees, non-Hispanic ethnicity examinees, and single women compared with women with children during internship were more likely to pass the certifying examination on the first attempt.

Meaning  In a national sample of trainees, we observed adverse impact based on sociodemographic factors on passing the board certification examinations that needs further exploration.

Importance  American Board of Surgery board certification requires passing both a written qualifying examination and an oral certifying examination. No studies have been conducted assessing the effect of sociodemographic variables on board passage rates.

Objective  To evaluate if trainee sociodemographic factors are associated with board passage rates.

Design, Setting, and Participants  This national and multi-institutional prospective observational cohort study of 1048 categorical general surgery trainees starting in 2007-2008 were surveyed. Data collection began in June 2007, follow-up was completed on December 31, 2016, and analysis began September 2018.

Main Outcomes and Measures  Survey responses were linked to American Board of Surgery board passage data.

Results  Of 662 examinees who had complete survey and follow-up data, 443 (65%) were men and 459 (69%) were white, with an overall board passage rate of 87% (n = 578). In a multinomial regression model, trainees of Hispanic ethnicity were more likely to not attempt the examinations (vs passed both) than non-Hispanic trainees (odds ratio [OR], 4.7; 95% CI, 1.5-14). Compared with examinees who were married with children during internship, examinees who were married without children (OR, 0.3; 95% CI, 0.1-0.8) or were single (OR, 0.4; 95% CI, 0.2-0.9) were less likely to fail the examinations. Logistic regression showed white examinees compared with nonwhite examinees (black individuals, Asian individuals, and individuals of other races) (OR, 1.8; 95% CI, 1.03-3.0) and examinees who performed better on their first American Board of Surgery In-Training Examination (OR, 1.03; 95% CI, 1.02-1.05) were more likely to pass the qualifying examination on the first try. White examinees compared with nonwhite examinees (OR, 1.8; 95% CI, 1.1-2.8), non-Hispanic compared with Hispanic examinees (OR, 2.4; 95% CI, 1.2-4.7), and single women compared with women who were married with children during internship (OR, 10.3; 95% CI, 2.1-51) were more likely to pass the certifying examination on the first try.

Conclusions and Relevance  Resident race, ethnicity, sex, and family status at internship were observed to be associated with board passage rates. There are multiple possible explanations for these worrisome observations that need to be explored. Tracking demographics of trainees to help understand passage rates based on demographics will be important. The American Board of Surgery already has begun addressing the potential for unconscious bias among board examiners by increasing diversity and adding implicit bias training.

The certification process for general surgeons in North America requires passing 2 examinations administered by the American Board of Surgery (ABS). The qualifying examination is a multiple choice examination that residents are eligible to take after their fourth year of clinical training. After passing the qualifying examination and graduating from an accredited US or Canadian residency program, examinees are then eligible for the certifying examination. The certifying examination is an in-person examination that is split into 3 consecutive 30-minute sessions using case-based scenarios. An average of 1400 examinees took the qualifying and certifying examinations each year over the last 5 years, with an average pass rate of 85% and 79%, respectively.¹ Candidates are board eligible for 7 years after graduating residency, and candidates who pass both examinations are board certified.

Board certification is not mandatory by law to practice surgery and is distinct from required state licensing; however, many institutions will only consider hiring job applicants if they are board certified or board eligible.² Board certification among graduates also has important consequences for residency programs. The Accreditation Council for Graduate Medical Education requires 65% of a residency program’s graduates over a 5-year period to pass both examinations on their first attempt for the residency to maintain accreditation.³ Several studies have been published attempting to identify predictive factors of passing the board examinations and evaluating interventions to improve pass rates.^3-14 These smaller studies have focused on the effect of mock oral examinations, prior standardized test performance, and training factors (eg, case volume, fellowship training). Previously, there have been few sociodemographic data on board examinees, and currently it is unknown if there are any differences in board examination passage rates based on resident race, ethnicity, family status, or residency program factors.

This study is a follow-up of a national prospective cohort study of all categorical general surgery interns entering general surgery residency training in the 2007-2008 academic year. Interns in this class were surveyed about sociodemographic information, medical school experience, and expectations of life as a surgeon. We then linked these data with ABS data on board examination passage rates. This is the first study using data from a national sample of US general surgery trainees to analyze the likelihood of passing the board examinations based on a variety of sociodemographic and program variables, to our knowledge.

All general surgery interns in the entering class of 2007-2008 who participated in the National Study of Expectations and Attitudes of Residents in Surgery and who completed our survey were included.¹⁵ The National Study of Expectations and Attitudes of Residents in Surgery study has been well described previously in the literature.^15-20 Interns who participated in this survey were followed up for 8 years, using linkable data provided by the ABS. This study was initially approved by the Yale School of Medicine institutional review board and was then transferred and reapproved by the Weill Cornell Medicine institutional review board.¹⁶ Completion of the survey constituted implied consent, and the survey included information that respondents were going to be followed up and may be recontacted. Data collection began in June 2007, and analysis began September 2018.

Development of the survey was based on qualitative interviews of general surgery residents who left their training programs, as well as prior literature on attrition and collaboration with the ABS.¹⁵ Survey questions gathered information about intern demographics, expectations for residency training, reasons why interns chose specific residency programs, and expectations of life as an attending surgeon.

We performed a secondary analysis of the data we collected as part of the National Study of Expectations and Attitudes of Residents in Surgery cohort; our primary study end point was the pass rate for ABS-administered qualifying and certifying examinations. We explored what sociodemographic and program factors, if any, were associated with passing these examinations compared with either failing or making no attempt to take them. Secondary end points included what sociodemographic and program factors, if any, were associated with passing these examinations on the first attempt. For this study, only those interns who completed the sociodemographic questionnaire and finished training were included. Residents who finished training in 2016 also were excluded as these residents were not followed up long enough to obtain reliable board passage data.

Cohort characteristics were examined. Univariate analyses comparing sociodemographic and program factors among those who (1) did not attempt the board examinations, (2) passed both examinations, or (3) failed either examination were performed. Differences across groups were assessed using χ² tests. Univariate analyses also were done comparing sociodemographic and program factors between those who passed the qualifying and certifying examinations on the first try vs those who did not. Differences between groups were assessed using χ² tests.

A multinomial regression analysis was performed to ascertain the likelihood of either making no attempt at the boards compared with board passage as well as failing the boards compared with board passage as predicted by interns’ answers to sociodemographic and program questions. This model included individual intern characteristics, including sex, race, ethnicity, family status (married with children, married without children, unmarried with partner, and single), graduation from a foreign medical school, residency finish year (2012, 2013-2015), and first ABS In-Training Examination (ABSITE) performance as well as training program characteristics, including type (academic, community, military), size (more than 5 graduating chiefs vs less), and geographic location (Northeast, South, Midwest, West). Logistic regression analyses were performed, including the same independent variables described above to ascertain the likelihood of passing the qualifying and certifying examinations on the first attempt (vs not) by sociodemographic and program variables. Interactions between participant characteristics and program size and type were examined and tested for significance. With respect to passing the certifying examination on the first attempt, the interaction between sex and family status was tested to be significant (P = .03). Therefore, the likelihood of passing the certifying examination on the first attempt by family status was further broken down into 2 subgroups by sex. All statistical tests were 2-sided; statistical significance was defined as a P value less than .05. The statistical analysis was performed using SAS version 9.3 (SAS Institute Inc).

Of 1048 categorical general surgery interns who started their training in 2007, 836 (80%) had linkable survey data. Of these, 672 (64%) finished residency and therefore were eligible to take the board examinations. We excluded those who had missing demographic information and those who finished residency in 2016, as they did not have sufficient follow-up, leaving a final cohort of 662 (63%). This cohort was 65% (n = 433) male and 69% (n = 459) white. Overall board passage rate was 87% (n = 578) (Table 1).

Univariate analysis of passing both qualifying and certifying examinations showed no significant differences based on sex, race, ethnicity, family status during internship, medical school location, residency program location, program size, or program type. There was a difference in first ABSITE performance between the no attempt, passed both, and failed either groups (mean 55th vs 58th vs 44th percentile; F₂ = 7.86; P < .001). There was also a difference in passage rates based on the year trainees completed residency, with a 90.1% (374 of 415) passage rate in trainees who finished in 2012 compared with an 82.6% (204 of 247) passage rate in those who finished in 2013-2015 (P = .02) (Table 2).

A multinomial regression analysis assessing the likelihood of making no attempt or failing either examination vs passing both showed that graduated trainees of Hispanic ethnicity were more likely to not attempt either examination compared with those of non-Hispanic ethnicity (odds ratio [OR], 4.66; 95% CI, 1.51-14.4). Compared with examinees who were married with children during internship, examinees who were married without children (OR, 0.3; 95% CI, 0.11-0.78) or were single (OR, 0.36; 95% CI, 0.16-0.85) were less likely to fail the examinations. Examinees who performed better on their first ABSITE examination were less likely to fail the board examinations (OR, 0.98; 95% CI, 0.96-0.99) (Table 3).

With respect to passing the qualifying examination on the first try, univariate analysis showed those who performed better on the first ABSITE they took (mean 60th vs 39th percentile; t = −7.29; P < .001) passed the examination on the first try. Table 4 provides a full list of statistically significant univariate associations. In a logistic regression analysis, white examinees were more likely to pass the qualifying examination on the first try compared with nonwhite examinees (black individuals, Asian individuals, and individuals of other races) (OR, 1.75; 95% CI, 1.03-2.95) and examinees who performed better on their first ABSITE were more likely to pass the qualifying examination on the first try (OR, 1.03; 95% CI, 1.02-1.05) (Table 4).

With respect to passing the certifying examination on the first try, univariate analysis showed a higher percentage of white compared with nonwhite examinees (367 [85%] vs 137 [75%]; χ²₁ = 8.09; P = .004), non-Hispanic compared with Hispanic ethnicity examinees (471 [83%] vs 33 [67%]; χ²₁ = 7.31; P = .007), and single women compared with women who were unmarried with a partner, married with no children, or married with children during internship (84 [90%] vs 52 [75%] vs 34 [76%] vs 5 [56%]; P = .01) passed the certifying examination on the first try. On univariate analysis, there was no difference in first-year ABSITE performance between those who passed the certifying examination on their first attempt compared with those who did not (mean 54th vs 59th percentile; t = −1.69; P = .09). In a logistic regression analysis, white examinees compared with nonwhite examinees (OR, 1.76; 95% CI, 1.11-2.79), non-Hispanic examinees compared with Hispanic examinees (OR, 2.35; 95% CI, 1.18-4.67), and single women compared with women who were married with children at the time of their internship (OR, 10.26; 95% CI, 2.08-50.63) were more likely to pass the certifying examination on the first try. Notably family status during internship had no effect on the likelihood of male examinees passing the certifying examination on their first try (Table 5). A sensitivity analysis was performed in which we transformed ABSITE score from a continuous into a categorical variable, dividing the percentile scores into tertiles. This had no effect on the results in any of our regression models (eTables 1-5 in the Supplement).

To our knowledge, this is the first study to describe the association of a variety of sociodemographic and program variables with the likelihood of passing the qualifying and certifying examinations for ABS certification. We observed concerning associations between examinee race, ethnicity, sex, and family status during internship on rates of taking the boards and board passage.

Based on these data, there appears to be an association between several sociodemographic factors including race, ethnicity, parental status, and duration of training, and attempting the ABS qualifying and certifying examinations, as well as passing the examinations on the first attempt. Some of our findings were not statistically significant on univariate analysis and became so after adjustment in our multivariable regression models, likely owing to some degree of confounding between sociodemographic variables that were only controlled for in a multivariate analysis. A 2012 study showed that medical school graduates entering general surgery training who were underrepresented racial minorities were nearly twice as likely to not be board certified than graduates who were white.²¹ A limitation of the previous study is that the reason for not being board certified was not ascertained. It is possible that the disparity they observed was due to other factors besides passing the examinations, such as failure to complete residency training or a purposeful choice by the graduate to forego board certification. Either way this is concerning, as there is already underrepresentation in surgery. Our data are the first to show sociodemographic disparities in attempting and passing the ABS examinations to become board certified.

The associations observed between race, ethnicity, and board passage were stronger in the first passage rate of the certifying examination compared with the qualifying examination. It is important to note the 2 examinations are designed to test different qualities in the examinee, with the qualifying examination more testing applied knowledge and the certifying examination testing judgment and effective communication, building on the knowledge component tested on the qualifying examination. However, it is equally important to note the qualifying examination is a written multiple choice examination, whereas the certifying examination is conducted in-person, and it is impossible to blind examiners to their perception of the race and ethnicity of the examinee. This makes the administration and grading of the examination susceptible to implicit bias. Furthermore, these examinations are taken in series; an examinee cannot sit for the certifying examination until they have passed the qualifying examination, essentially filtering the pool of examinees to only those who have the fund of knowledge to succeed on the certifying examination.

There were performance differences observed in the qualifying examination, with a higher percentage of white compared with nonwhite examinees and a higher percentage of non-Hispanic compared with Hispanic examinees who passed the examination on the first attempt, by 4.5% and 4.2%, respectively. However, in a pool of ostensibly more qualified candidates now taking an in-person examination, this difference in first-time pass rates more than doubled to 9.7% between white and nonwhite examinees and nearly quadrupled to 15.6% between non-Hispanic and Hispanic examinees. Although we cannot know for certain, with such a stark difference in first-time pass rates between written and in-person examinations, albeit designed to test different things, we cannot ignore the possibility that implicit bias is playing a role in how examinees are graded on their performance during the certifying examination. However, there are other possible explanations for this finding besides implicit bias on the part of the examiners. Prior data have shown that residents who identified as an underrepresented minority were less likely to feel they could count on their resident peers for help and that they were less likely to feel like they fit in at their programs.²² In conjunction with other data that suggest mock oral examinations lead to modest improvement in certifying examination performance and that residency program administration of mock oral examinations heavily depends on the participation of senior residents, it is plausible that underrepresented minority residents are not receiving adequate program-level preparation for the certifying examination.^14,23

Regardless of the potential causes of our observation, we believe it is important to highlight these associations and identify them as problematic. The impact of these findings cannot be understated, especially as we are addressing the likely related problem of our failure to retain underrepresented minorities in academic surgery.^24,25 These findings also have significant consequences for residency programs, as first passage rates of program graduates are being used as a metric to maintain Accreditation Council for Graduate Medical Education accreditation. Adverse impact on pass rates for underrepresented minorities on their first attempt at the board examinations could lead to a disastrous feedback loop in which residency programs either lose accreditation or become less willing to accept minorities into their programs, subsequently leading to even less representation of these groups in surgery.

To address this issue, we believe all board examiners should have implicit bias training and that the pool of examiners should more closely resemble the pool of examinees with respect to sociodemographic factors. The ABS has already taken steps to do this. However, there are limitations to implicit bias training. There are data that show training has some success at changing individual beliefs, but a meta-analysis suggests it may not be effective at improving institutional-level inequities. Furthermore, some research has shown implicit bias training may reinforce biases.²⁶ This is why we believe that tracking examinee sociodemographic factors in a prospective manner is important to assess how our interventions are impacting inequities in examination scores. The surgical community cannot begin to address and fix this problem if we do not know what passage rates are.

We observed that sex and marital status at the time of internship also had a significant association with whether an individual passed the board examinations. Examinees who were married with children were more likely to fail either examination than their married but childless and single counterparts. Women who were single during internship were 10 times more likely than women with children to pass the certifying examination on their first try. The association between having children and first-time certifying examination passage was not observed in men, and implicit bias on the part of examiners is an unlikely cause. This observation could highlight a potential consequence of a broader sociologic phenomenon that although married women with children are working more, they are still expected to fulfill their traditional gender role of primary caregiver to their children.²⁷ According to a survey of US households between 2013 and 2017 conducted by the US Bureau of Labor Statistics, women with children younger than 6 years spent an average of 1.1 hours per day of direct physical care to children compared with 26 minutes for men.²⁸ Although this is a survey of the general population, there is no reason to believe the results would be different in a subgroup analysis of surgeons and surgeons in training. These statistics likely translate to a substantial advantage in examination preparation for men regardless of childbearing status and single or childless women compared with women with children. This is a significant societal problem, with consequences that go beyond board passage. Female surgeons are more likely to leave residency training later in their training, less likely to reach senior academic leadership positions, and earn on average $40 000 per year less than men.^29,30 Although the larger societal issue of traditional gender family roles is difficult to address, the surgical community should strive to change by providing equal opportunities for both men and women to succeed. Improved pay parity and increased flexibility in scheduling would be large strides in empowering professional women both in the workplace and at home, potentially leading to improved gender diversity in academic surgery.

The main strength of our study is that we used a large national cohort of surgery residents from many institutions to obtain our results, which are likely generalizable. A limitation of our study is that study participants were not reassessed during the study period, and answers to their sociodemographic questions (particularly their family status) may have changed from the time they were interns to the time they took their board examinations. Also, we could not ascertain how examinees prepared for examinations or their performance during residency and on prior standardized examinations, all of which could be potentially confounding variables. However, other studies cited throughout this article have assessed the potential effect of these other variables. Our aim specifically was to assess differences in demographic and program variables, as this has not been done previously, and the ABS does not keep track of these data. A statistical limitation of our study is that our multivariable model contains more variables than recommended given the few outcomes we studied, limiting its power. However, we did try to limit the number of variables to those factors we thought would be most important.

We found in a national cohort of general surgery trainees that race, ethnicity, sex, and family status during internship were significantly associated with attempting and passing the ABS certification examinations overall and on the first attempt. Every effort should be made to retain underrepresented minorities and women in surgery, as there is currently a significant lack of representation of these groups within academic surgery. The adverse impact observed for these groups in passing board examinations is a potential contributing factor to this lack of diversity. These findings have significant potential impact on training programs, as the board passage rate on examinees’ first attempts has become a metric to maintain program accreditation. It is important to highlight that we do not know why the observed differences exist, and implicit bias by examiners is not the only potential explanation. Going forward, it is imperative to track these variables and outcomes to ensure equal and fair treatment of the examinees and to address the potential for implicit bias among examiners by administering training and amplifying diversity among the examiner pool. Our finding that women who were single during internship were 10 times more likely to pass the certifying examination on the first try compared with women who were married with children during internship, a finding not observed in men, likely reflects larger societal issues around traditional gender roles in families. Therefore, it will likely be more difficult to address at an institutional level. It is incumbent upon us to view training and examination preparation through this lens, and it should be of paramount importance to ensure all trainees have adequate time and resources to prepare for the ABS examinations.

Corresponding Author: Heather L. Yeo, MD, MHS, MBA, MS, Department of Surgery, NewYork-Presbyterian/Weill Cornell Medical Center, 525 E 68th St, PO Box 172, New York, NY 10065 (hey9002@med.cornell.edu).

Accepted for Publication: July 21, 2019.

Published Online: October 16, 2019. doi:10.1001/jamasurg.2019.4081

Author Contributions: Drs Yeo and Sosa had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Yeo, Mao, Sosa.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Yeo, Dolan.

Critical revision of the manuscript for important intellectual content: Yeo, Mao, Sosa.

Statistical analysis: Mao.

Obtained funding: Yeo.

Administrative, technical, or material support: Yeo, Dolan.

Supervision: Yeo, Dolan, Sosa.

Conflict of Interest Disclosures: Dr Yeo serves on a medical advisory board for SurvivorNet. Dr Dolan reports a grant from the National Institutes of Health Agency for Healthcare Research and Quality outside the submitted work. Dr Sosa is a member of the data monitoring committee of the Medullary Thyroid Carcinoma Consortium Registry supported by Novo Nordisk, GlaxoSmithKline, AstraZeneca, and Eli Lilly and Company. No other disclosures were reported.

Funding/Support: The original cohort study was partially supported by the Robert Wood Johnson Foundation.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.