Informal Peer Interaction and Practice Type as Predictors of Physician Performance on Maintenance of Certification Examinations

Importance  Physicians can demonstrate mastery of the knowledge that supports continued clinical competence by passing a maintenance of certification examination (MOCEX). Performance depends on professional learning and development, which may be enhanced by informal routine interactions with colleagues. Some physicians, such as those in solo practice, may have less opportunity for peer interaction, thus negatively influencing their examination performance.

Objective  To determine the relationship among level of peer interaction, group and solo practice, and MOCEX performance.

Design, Setting, and Participants  Longitudinal cohort study of 568 surgeons taking the 2008 MOCEX. Survey responses reporting the level of physicians’ peer interactions and their practice type were related to MOCEX scores, controlling for initial qualifying examination scores, practice type, and personal characteristics.

Exposures  Solo practice and amount of peer interaction.

Main Outcomes and Measures  Scores on the MOCEX and pass-fail status.

Results  Of the 568 surgeons in the study sample, 557 (98.1%) passed the examination. Higher levels of peer interaction were associated with a higher score (β = 0.91 [95% CI, 0.31-1.52]) and higher likelihood of passing the examination (odds ratio, 2.58 [1.08-6.16]). Physicians in solo (vs group) practice had fewer peer interactions (β = −0.49 [95% CI, −0.64 to −0.33), received lower scores (β = −1.82 [−2.94 to −0.82]), and were less likely to pass the examination (odds ratio, 0.22 [0.06-0.77]). Level of peer interaction moderated the relationship between solo practice and MOCEX score; solo practitioners with high levels of peer interaction achieved an MOCEX performance on a par with that of group practitioners.

Conclusions and Relevance  Physicians in solo practice had poorer MOCEX performance. However, solo practitioners who reported high levels of peer interaction performed as well as those in group practice. Peer interaction is important for professional learning and quality of care.

Medical specialty boards have implemented comprehensive assessment systems for ensuring the continued clinical competence of practicing physicians. The American Board of Surgery (ABS) along with other specialty boards members of the American Board of Medical Specialties use a maintenance of certification (MOC) process to assess physicians’ medical knowledge and patient care–related competencies. Certification and recertification are critical processes for ensuring continued professional knowledge and skill.¹ Member board certification by the American Board of Medical Specialties is associated with better quality of care,^2-4 better patient outcomes,^5-8 and fewer disciplinary actions against the physician.^9-11 The MOC is also associated with better-quality care^12,13; in fact, compliance with standard practices declines as a function of time elapsed since the last recertification examination.¹⁴ The MOC is thus an important system for encouraging lifelong learning by the physician and for promoting the provision of high-quality care.

Performance on the MOC examination (MOCEX) is a measure of physicians’ ongoing professional learning and development. Professional learning includes deliberative study and training (eg, continuing medical education [CME] courses),^15,16 but research has also demonstrated the considerable value of informal or implicit learning, for example learning through experience^17,18 or learning through social interaction.^19-21 In many professions such as teaching,^22,23 law,^24,25 and business,^26-28 interactions with peer professionals play an important part in professional learning. Peer interactions may be particularly important for physicians. Physicians frequently seek clinical information from colleagues rather than journals or databases.^29-31 Interaction with peers improves physicians’ awareness of current evidence, spreads expertise that can be applied to future cases, and increases confidence in the appropriateness of an approach for an individual patient.^32,33 Through discussions and interactions with peers, specialized knowledge is meaningfully integrated and internalized.²³ This process is true even for the ordinary daily interactions that occur within communities of practice.²²

Some physicians may have limited opportunity for interaction with peers. Physicians in solo practice may be at risk for limited peer interaction, and given the importance of peer interaction for professional learning, this consequence of solo practice may partly explain lower MOCEX performance among solo practitioners.³⁴ We therefore examined the relationship among the level of informal routine peer interaction, solo practice, and MOCEX performance.

The ABS uses an MOCEX to assess ongoing medical knowledge and patient-care competencies. The MOCEX consists of approximately 200 single-best-answer questions that comprehensively test general surgical knowledge and is part of a comprehensive framework designed to evaluate physicians across the 6 core competencies proposed by the Accreditation Council for Graduate Medical Education and endorsed by the American Board of Medical Specialties.³⁵ Surgeons pass an MOCEX at 10-year intervals to maintain certification and in many cases to maintain hospital credentials and privileges.³⁶ The success rate during the period since the ABS MOCEX was first administered in 1980 has ranged from 90% to 95%.³⁴

We surveyed the cohort of general surgeons who took the ABS MOCEX in 2008. Each surgeon was given the opportunity to participate in a survey at the time that they registered for the examination (to minimize any response bias from future poor examination performance). We also obtained demographic and other background information at the time of registration. We used the actual forms with which physicians register for the examination and earn recertification, so variables from these registration forms are highly accurate. We linked the survey responses and background data with surgeons’ initial qualifying examination scores and 2008 MOCEX scores.

In 2008, 1632 surgeons took the MOCEX. Of those, 623 (38.2%) consented to participate in the survey, and we had full demographic data, survey data, initial qualifying examination performance data, and MOCEX recertification performance data for 568 (91.2%). Our study sample consisted of the 568 surgeons for whom we had complete data and who gave written informed consent. This study was approved by the institutional review board of the University of Pennsylvania.

We examined 2 dependent variables. The first dependent variable was the 2008 MOCEX score for each surgeon. The examination score was the quotient of the number of items correctly answered divided by the total number of items multiplied by 100, which gave a value from 0 to 100. The second dependent variable was the dichotomous pass-fail examination result, determined by the ABS using an equating method that accounts for the difficulty of the examination and the ability of the cohort. The cutoff point for failing the examination in 2008 was a score of 65.

We used 2 main independent variables and calculated an interaction term combining them. The first independent variable was a dichotomous variable indicating whether the surgeon was in solo practice or not. This variable was created using information from the 2008 MOCEX application forms. The second independent variable was the level of informal routine peer interactions with medical colleagues measured via a self-reported scale completed by the study participants. Scale development followed standard practices for survey research. We developed a conceptual framework using interviews with content experts and existing research on peer interaction and professional learning and determined that no existing survey scale directly assessed the construct of interest.^37-39 Scale items were developed through an iterative process through examining the scholarly literature and learning from experts and consisted of the following questions:

1. “On average, how frequently do you speak with other medical doctors about medical matters?”

2. “How many MD colleagues did you interact with last week about a medical matter?”

3. “How many MD colleagues did you interact with last week about any issue (including socializing)?”

We tested the survey scale for reliability using the Cronbach α, a measure that examines the degree to which items within a scale capture the same latent construct.⁴⁰ The value for the Cronbach α was 0.75, considered an acceptable reliability for surveys.^41,42 We also tested the survey scale for construct validity using factor analysis to determine that the items in the scale varied together. The factor loadings for the items were all greater than 0.4, which is generally accepted as evidence that the items are assessing the same latent construct.

We used several control variables that could account for systematic differences between surgeons who chose solo vs group practice. The first control variable was each surgeon’s score (range, 0-100) on the initial ABS qualifying examination, taken immediately after the completion of residency. This value provides a rigorous control for baseline differences in test-taking abilities, human capital, and the demographic variables highly correlated with these. The second control variable was the percentage of surgeons (0%-100%) within each surgeon’s residency program who passed the initial qualifying examination from 1975 to 2000; this commonly used control variable captures the quality of the residency program of each surgeon. The third control variable was the number of years that had passed since the qualifying examination (this value was correlated at 0.97 with the number of times the respondent had taken a recertification examination). The fourth control variable was the number of hours that a physician had spent in CME in the prior 2 years. The final control variable was the current practice area for each surgeon (clinical or nonclinical).

We used standard descriptive summary statistics to characterize the sample. Differences in examination score and pass-fail status by solo practice and level of peer interaction were evaluated using χ² tests, as was the difference in the level of peer interaction by practice type. To illustrate relationships of interest, we divided respondents into 3 equal groups reporting low, medium, or high levels of peer interaction. The analyses were conducted using the continuous variable; the divided categories were used only to construct the Figures. We estimated the effect of solo practice and level of peer interaction on MOCEX performance using ordinary least squares multivariate regression and logistic regression models. We compared models based on the variance explained. The key variable in explaining variance on the MOCEX score was the initial qualifying examination score, suggesting this is a rigorous control for innate human capital. In addition to including this robust control variable, we also conducted a sensitivity analysis controlling for many known correlates of solo practice, such as age and graduation from a non-US medical school, and for personal characteristics of the respondents, such as personality, job satisfaction, and career engagement. Last, the moderating role of peer interaction on solo practice and examination performance was determined by entering the interaction term into the regression models; this relationship was also graphed. Statistical analysis was performed using commercially available software (STATA, version 12.1; StataCorp). Tests were 2 sided, and P < .05 was considered statistically significant.

Table 1 reports the characteristics of the sample. The 568 surgeons in our study sample were primarily in group practice (71.3%) and worked in a nonrural setting (83.1%). Most of the respondents were born in the United States (82.6%) and attended a US medical school (87.5%). Almost half of the study sample was younger than 55 years; 12.8% of the respondents were women.

The demographic makeup of the study sample was similar to that of the population of general surgeons in the United States during the comparable period, that is, 79% of US general surgeons were in group practice in 2009,⁴³ and 87% were practicing in nonrural settings.⁴⁴ In 2010, 83% of active surgeons had earned their medical degree from a US medical school, approximately 54% were younger than 55 years, and 15% were female.⁴⁵ (Information about the population of general surgeons was collected from publicly available data sources that were not always complete or available for all years, so are not always from 2008, the year of our data collection. As such, we collected data as close to 2008 as possible.) The largest difference between our sample and the population of US general surgeons is in the number of surgeons in solo practice; this group is overrepresented in our sample. This difference possibly resulted from newly graduated surgeons being more likely to join group practices, whereas our sample assessed a group of surgeons who had graduated more than 10 years earlier and were attempting to recertify for the first time since their initial board certification.

In unadjusted bivariate relationships, solo practitioners reported lower levels of peer interaction (Figure 1). Among solo practitioners, 43.6% reported low levels of peer interaction, compared with 24.4% of group practitioners. In contrast, only 18.4% of solo practitioners reported high levels of peer interaction, compared with 37.3% of group practitioners. Medium levels of interaction were similar between solo and group practitioners (31.9% and 29.6%, respectively).

Being in solo practice was associated with fewer peer interactions (β = −0.49 [95% CI, −0.64 to −0.33), lower MOCEX scores (β = −1.82 [95% CI, −2.94 to −0.82]), and decreased odds of passing the examination (odds ratio, 0.22 [0.06-0.77]) (Table 2). In contrast, higher levels of peer interaction predicted higher performance on the MOCEX. A 1-unit increase on the scale assessing frequency of peer interaction was associated with a significant improvement in examination score (β = 0.91 [95% CI, 0.31-1.52]) and significantly increased odds of passing the examination (odds ratio, 2.58 [1.08-6.16]). Figure 2 illustrates this pattern of relationships. For example, surgeons with low levels of peer interaction had a mean MOCEX score of 77.1 and a failure rate of 4.1%, compared with a mean score of 78.9 and failure rate of 1.1% for those with medium and high levels of interaction.

The level of peer interaction moderated the relationship between solo practice and MOCEX performance (Table 2). The coefficient on the interaction term between solo practice and level of peer interaction was positive and significant for the MOCEX score. The coefficient was not significant for odds of passing the examination. These relationships are illustrated in Figure 3, which shows that solo practitioners with low levels of peer interaction had significantly lower examination scores. Solo practitioners with high levels of peer interaction scored on par with surgeons in group practice.

Sensitivity analyses produced the same pattern of results. We found no significant differences between respondents in solo practice and those in group practice on commonly used measures of personality, job satisfaction, or career engagement (analyses not shown). The effect of solo practice persisted even when controlling for demographic and personal factors (analyses not shown).

This study of experienced surgeons’ performance on the MOCEX demonstrated that solo practitioners were likely to have lower levels of informal routine peer interaction and worse examination performance. Our results show that the level of peer interaction moderated the relationship between solo practice and poor examination performance. Solo practitioners who reported high levels of peer interaction scored as well as physicians in group practice. Our results suggest that interactions with peers are a critical part of professional learning and development. Physicians may need to be connected with colleagues through formal learning initiatives like conferences^29,46 and physician professional networks^47,48 in addition to informal daily interaction.

Our cross-sectional study design limits conclusive interpretations or decisive policy recommendations. Nevertheless, our findings suggest that some surgeons are at risk for inadequate opportunities for peer interactions that can enhance professional learning and growth. We identified a performance gap for surgeons in solo practice: mean scores of solo practitioners were 2 percentage points lower than those of group practitioners (77.1 vs 79.0) (Figure 2A), and 4.5% of solo practitioners failed the examination, compared with 0.9% of surgeons in group practice (Figure 2B). A similar performance gap was also shown in a previous study³⁴ that did not measure peer interaction, and so our results contribute the idea that performance differences may be a result of the more limited opportunities for surgeons in solo practice to learn through interactions with peers.

Prior research has identified other factors that contribute to physician learning, focusing primarily on formal learning initiatives like CME⁴⁹ or academic detailing programs.^50,51 During the past 20 years, CME has evolved from a traditional lecture format to a proactive and collaborative process that includes interactions among physician participants.⁴⁹ The CME evolution and related empirical evaluations have shown that medical education activities that involve learner interaction are more likely to result in changes in practice than passive learning activities.^15,16,52 Our study contributes to knowledge about physician professional development by demonstrating that peer interactions are also an important part of physician learning. Although we controlled for the hours spent in CME courses in this study, the type of CME course may matter to examination performance, which would be an interesting area for future research.

Not all potentially relevant variables or controls could be collected or included in this analysis. For example, we did not collect detailed data about the type of peers with whom the respondents regularly interacted. The specific kinds of peer interactions that are beneficial to learning should be explored in future research. Also, we were unable to control for all of the factors likely associated with the choice to practice independently; some of these also may have influenced examination performance. However, our inclusion of the initial qualifying examination score is a rigorous control for many of these factors. Last, many of the study measures were self-reported, and results should be interpreted with this in mind. However, the outcome variable was an objective performance measure that has practical, clinical, and policy implications.

Because this analysis used a cross-sectional sample of convenience, it cannot resolve the causal relationships among solo practice, peer interaction, and examination performance. The moderation analysis provides suggestive evidence that the relationship between solo practice and poor examination performance can be explained in part by a lack of peer interaction. Awareness of this relationship may provide clinicians, medical educators, and policy makers the opportunity to encourage and support regular interactions and discussions with peers for all practicing physicians.

Physicians in solo practice had poorer MOCEX performance. However, solo practitioners who reported high levels of peer interaction performed as well as group practitioners. Peer interaction is important for professional learning and quality care.

Accepted for Publication: December 19, 2013.

Corresponding Author: Melissa A. Valentine, PhD, Department of Management Science and Engineering, Stanford University, Huang Center, 475 Via Ortega, Stanford, CA 94305-4121 (mav@stanford.edu).

Published Online: May 28, 2014. doi:10.1001/jamasurg.2014.183.

Author Contributions: Drs Valentine and Barsade had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Valentine, Barsade, Gal, Rhodes.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Valentine, Barsade, Edmondson, Rhodes.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Valentine, Barsade, Edmondson, Gal, Rhodes.

Administrative, technical, or material support: Barsade, Rhodes.

Study supervision: Barsade, Edmondson.

Conflict of Interest Disclosures: Dr Rhodes is an employee of the American Board of Surgery. No other disclosures were reported.