Association Between Maintenance of Certification Examination Scores and Quality of Care for Medicare Beneficiaries

Background  The relationship between physicians' cognitive skill and the delivery of evidence-based processes of care is not well characterized.Therefore, we set out to determine associations between general internists' performance on the American Board of Internal Medicine maintenance of certification examination and the receipt of important processes of care by Medicare patients.

Methods  Physicians were grouped into quartiles based on their performance on the American Board of Internal Medicine examination. Hierarchical generalized linear models examined associations between examination scores and the receipt of processes of care by Medicare patients. The main outcome measures were the associations between diabetes care, using a composite measure of hemoglobin A_1c, and lipid testing and retinal screening, mammography, and lipid testing in patients with cardiovascular disease and the physician's performance on the American Board of Internal Medicine examination, adjusted for the number of Medicare patients with diabetes and cardiovascular disease in a physician's practice panel; frequency of visits; patient comorbidity, age, and ethnicity; and physician training history and type of practice.

Results  Physicians scoring in the top quartile were more likely to perform processes of care for diabetes (composite measure odds ratio [OR], 1.17; 95% confidence interval [CI], 1.07-1.27) and mammography screening (OR, 1.14; 95% CI, 1.08-1.21) than physicians in the lowest physician quartile, even after adjustment for multiple factors. There was no significant difference among the groups in lipid testing of patients with cardiovascular disease (OR, 1.00; 95% CI, 0.91-1.10).

Conclusion  Our findings suggest that physician cognitive skills, as measured by a maintenance of certification examination, are associated with higher rates of processes of care for Medicare patients.

The overall quality of care for Medicare patients is suboptimal.^1,2 Research has not clearly identified which physician factors affect the quality of care for older patients. A systematic review found a negative relationship between physician experience and performance.³ Several recent studies have found a positive association between performance on some process quality measures and whether a physician was board certified.^4-7 However, the attainment of board certification is a relatively blunt assessment tool, as the majority of US-trained physicians ultimately achieve board certification.⁸ One outstanding question is whether actual cognitive skill, as measured by a reliable high-stakes examination, is associated with quality of care as assessed by validated performance measures. Public support for repeated cognitive testing has been strong, with 87% of patients who were surveyed saying that they believe a physician should take an examination of knowledge periodically.^9,10

However, some in the physician community have questioned the need for a recertification examination, citing the lack of evidence that maintenance of certification (MOC) provides meaningful information about physician competence.¹¹ One Canadian study did find that family physicians' scores on an initial licensing examination had predictive validity for future performance on a number of quality measures up to 6 years after the examination.¹² However, to our knowledge, no study has yet examined the concurrent relationship between cognitive skills and quality of care.

The primary objective of this study was to examine the association between cognitive skills, as measured by performance on a general internal medicine MOC examination, and performance on a set of Center for Medicare and Medicaid Services quality measures for patients with diabetes and mammography screening, as well as lipid testing for cardiovascular disease. Our hypothesis is that internists with higher cognitive skills will provide better quality of care, as assessed by claims-based processes of care measures: hemoglobin A_1c testing, lipid testing, and retinal eye examinations for patients with diabetes, mammography screening for women, and lipid testing for patients with cardiovascular disease.

The sampling method is outlined in the Figure. Potential subjects were internists who initially certified in internal medicine between 1990 and 1995, had a universal physician identification number, and did not hold a subspecialty certificate. We then excluded physicians who did not have between 30 and 800 Medicare beneficiaries in their practice. First, because the response variable at the patient level follows a binomial distribution, we selected a lower limit of 30 cases to ensure that the response variable approximated to a normal distribution. Second, we selected an upper limit of 800 to avoid a situation in which physicians may share 1 universal physician identification number across physicians or groups; 800 beneficiaries represented the 95th percentile for Medicare patient volume. Third, the Center for Medicare and Medicaid Services limits the number of patient records it will release, and we wanted to ensure that we only sampled actively practicing physicians. Finally, we excluded physicians who attempted their first initial certification examination before 1985 or their first MOC examination before 2000, did not take all 3 internal medicine modules on the MOC examination (60 questions per module), or canceled their MOC enrollment. From this cohort, we generated a random sample of 5000 physicians to assemble the patient cohorts for analysis.

Medicare claims data and enrollment files from 2002 and 2003 were used to identify patients seen by the physician cohort, to determine the performance measures for which they were eligible and whether they received them, and to ascertain relevant patient characteristics and comorbidities. All 2002 and 2003 Medicare claims were obtained for the beneficiaries to ensure that services were counted regardless of who provided them and to ensure proper assignment to the appropriate primary care physician. Assignment of beneficiaries to physicians was done using only Part B office visit claims for medical care. Part A outpatient claims were not used for physician assignment in order to keep the study population more homogeneous and because the unique physician-beneficiary linkage is less clear. Since physicians sometimes bill under more than 1 specialty, a provider was considered a primary care physician if the majority of his or her Part B claims were billed as internal medicine, general practice, family practice, or geriatrics. Beneficiaries were assigned to the physician whom they saw most often (or most recently in case of ties) in each calendar year.¹³ Those beneficiaries who were assigned to the same physician, from the cohort described above, in both 2002 and 2003 were included in the study sample. Therefore, a patient had to have a minimum of 2 visits to the physician: 1 in 2002 and 1 in 2003. This requirement eliminated patients who switched physicians late in 2002 or who were new patients in 2003. Visits to specialists were not used for assignment. Eligible Medicare beneficiaries had 24 months of fee-for-service Part B coverage in 2002 and 2003 and were alive as of December 31, 2003. We then restricted the physician cohort only to those physicians who had actually taken the MOC examination between 2000 and 2005. This resulted in a final study sample of 3602 physicians and a total of 220 340 beneficiaries eligible for 1 or more of the performance measures. The study was approved by the New England Institutional Review Board.

Recognizing the need for the periodic assessment of practicing physicians, the American Board of Internal Medicine (ABIM) instituted time-limited internal medicine certificates in 1990.^10-14 All internists must now complete the MOC program every 10 years to receive a new certificate. To renew a certificate, physicians must complete a set of formative self-assessment activities and pass a rigorous recertification examination. The ABIM MOC examination consists of 180 single-best-answer questions. The examination evaluates the extent of the candidate's knowledge and clinical judgment (ie, cognitive skills) in areas in which an internist should demonstrate competence, including treatment of both common and uncommon conditions that have important consequences for patient care (www.abim.org). The examination also includes questions about diagnosis, treatment, and prevention.

The quality-of-care measures that were used in this study were receipt of at least 2 hemoglobin A_1c measurements in the previous year and a lipid test and retinal eye examination within the past year for patients with diabetes; mammography for women aged 65 through 74 years within the previous year; and lipid testing within the past year for patients with cardiac disease. Receipt of each service was determined using claims-based Current Procedural Terminology and International Classification of Diseases, Ninth Revision, Clinical Modification, codes. For diabetes, we used the perfect score method by combining all 3 processes of care for each patient to create a composite measure, assigning a value of 1 if all 3 measures had been performed and 0 if 2 or fewer measures had been performed.¹³

The candidates were divided into 3 percentile groups (<25th, 25-75th, and >75th) based on their first attempt on their ABIM MOC examination. We used the interquartile approach because the relationship between physicians' examination score and their quality of performance is not linear. Second, the cutpoints of less than 25th, 25th through 75th, and greater than 75th percentiles are a standard method to measure scorelike variables^15,16 and are used widely in practice.^17,18 The equated passing score for the MOC examinations was approximately 350.

Bivariate analysis was performed to assess associations between patient characteristics and MOC scores. Patient characteristics were divided into 3 domains: demographics, comorbidities, and access to care during the previous 12 months. Demographics included age, sex (female vs male), race (white vs other), Medicaid status, and comorbidities, including history of percutaneous transluminal coronary angioplasty, coronary artery bypass graft, heart attack, heart failure, renal failure, depression, peripheral vascular disease, hypertension, chronic obstructive pulmonary disease, cancer, liver disease, dementia, diabetes, smoking, and obesity status. We also calculated a Deyo score for each patient.¹⁹ The access-to-care domain includes the total number of primary care physician visits; any cardiologist, endocrinologist, or nephrologist visits (yes/no); hospital emergency department visits (yes/no); or hospitalizations (yes/no). The Mantel-Haenszel χ² test was used to compare dichotomous and categorical variables, and the Kruskal-Wallis rank test was used to compare continuous variables.

We used the hierarchical generalized linear model to evaluate associations between physician MOC scores and physician performance on quality measures. Because patients seeing the same physician are not independent, it is necessary to use a multilevel model to properly account for the clustering of patients by physician. The log-odds of the quality measures were modeled as a function of the MOC score represented by 2 dummy variables, with the lowest-score category as the reference.^20-22 Our intent was not to develop a predictive model for each measure; rather, it was to focus on associations between the MOC score and quality measures with and without adjustment for patient and physician characteristics. Physician characteristics included sex (female vs male), years since graduation, country of medical school (United States vs other), type of practice, and condition-specific patient volume. Patient characteristics were included to differentiate the effect of patient behavior from physician behavior.

Accordingly, we sequentially constructed 3 models for each measure. The first model was fitted without adjustment for either patient or physician characteristics; the second model was fitted with adjustment for patient demographics, comorbidity, and access to care; and the third model was fitted with adjustment for patient and physician characteristics. All hierarchical generalized linear models were fitted with a random physician-specific effect to account for within-physician correlation of the measures. The hierarchical generalized linear models separated the within-physician variation from the between-physician variation. We calculated 95% confidence intervals for each estimate from the models. All statistical analyses were conducted using hierarchical linear modeling version 5.0 (Scientific Software International Inc, Lincolnwood, Illinois) and SAS version 9.02 (SAS Institute Inc, Cary, North Carolina).

Between 2002 and 2003, the study cohort of 3602 general internists who took a recertification examination cared for more than 220 000 Medicare patients with diabetes, patients with cardiovascular disease, and women who were eligible for mammography screening. Table 1 displays the physician characteristics and bivariate results. Of note, approximately 37% of the cohort were women, 26% had trained in a foreign medical school, and nearly 23% failed the initial certification examination on their first attempt. Approximately 60% of these physicians worked in practices of 10 or fewer physicians, with nearly 20% working as solo practitioners.

Table 1 shows that physicians who were not graduates of a US or Canadian medical school, who had graduated more than 20 years ago from medical school, and who were solo practitioners were more likely to score in the lowest quartile on the MOC examination. Table 2 highlights important patient variables and receipt of processes of care. Of note, patients with diabetes who were male, African American, or Hispanic; who received fewer visits; or who did not see a specialist were less likely to receive “perfect care” (eg, all 3 diabetes process-of-care measures). The same relationships held true for mammography (sex not applicable) and lipid testing in patients with cardiovascular disease. The relationship with the Deyo comorbidity score varied by condition: patients with diabetes and a Deyo score of 3 or higher were more likely to receive all 3 processes of care, whereas patients with cardiovascular disease were modestly less likely to receive a lipid test.

Table 3 provides the bivariate analyses of physician characteristics and performance on the quality measures. Regarding the diabetes and mammography measures, physicians who were not graduates of a US or Canadian medical school, who had graduated more than 20 years ago from medical school, who were male, and who were solo practitioners did worse on the performance measures for diabetes and mammography. For example, those physicians in the lowest quartile had a compliance rate 6.2% lower than physicians in the top quartile for mammography screening and 5.0% lower for the diabetes composite measure. Table 3 also provides an additional analysis that breaks down the MOC score into 5 levels. Physicians who scored below an equated score of 300 (with a passing score of approximately 350) had a compliance rate that was 8.6% lower for mammography screening and 9.3% for the diabetes composite measure compared with physicians with an equated score greater than 600.

Table 4 shows the results of the hierarchical models. After adjustment for patient and physician characteristics, there was a significant association for performance on the diabetes and mammography quality-of-care process measures and examination scores. By rough estimation, patients with diabetes were 17% more likely to receive all 3 diabetes processes of care and women were 14% more likely to undergo mammography if they were cared for by physicians who scored in the top quartile compared with the lowest quartile group. The 1 measure that did not show a significant association with MOC scores was lipid testing for patients with peripheral vascular and cardiovascular disease. However, more than 59% of all patients with this condition saw a cardiologist subspecialist, compared with only 6% of the patients with diabetes who saw an endocrinologist. Therefore, performance on this measure could have been confounded by cardiologist performance.

To our knowledge, this is one of the first studies to examine the relationship between physician cognitive skills, as measured by the ABIM MOC examination, and basic quality-of-care performance measures. We found a positive correlation between examination scores and performance on all but 1 quality-of-care performance measure. Regarding lipid testing among patients with cardiac disease, nearly 60% of the patients were also cared for by a cardiologist, thus mitigating the influence of the general internist. Our results are consistent with those of a previous study that examined quality of care in fee-for-service Medicare patients, but our study extends understanding of the role of physician factors in several important ways.⁷ First, we specifically explored the relationship of levels of physician knowledge and quality of care, finding a positive association. We used a direct measure of physician competence in cognitive skill, instead of only demographic characteristics, which have been used in previous studies.³ Second, we looked at performance on key quality-of-care measures in close time proximity to performance on a knowledge examination. Third, we used hierarchical models to specifically examine the physician as the unit of analysis and to adjust for multiple patient and physician demographic factors previously shown to affect quality.^7,13

Our results, like those of Tamblyn et al,¹² who investigated a licensure examination in Canada, raise an important question as to why we found a relationship between physician competency in knowledge and performance on basic quality-of-care processes that many investigators believe should be handled by nonphysicians.^23-26 One potential explanation is that many physician offices currently lack effective systems to ensure that these processes of care occur without direct physician involvement.^27,28 Therefore, the burden to order these studies still falls to the physician, and physicians with better cognitive skills may be more effective in remembering to do the “right thing” or in creating systems. Alternatively, physicians with higher levels of cognitive skills competency may either gravitate toward or create better systems of care delivery. Further more, high performance on the examination may be a marker for more effective physician behaviors in other competencies that promote better care. Previous work supports this hypothesis. Ramsey et al²⁹ found that physician peers rated colleagues who scored higher on the certification examination more favorably. Shea et al³⁰ found that examinees who scored higher on the examination received higher ratings in other competencies from their program directors.

Also, research in cognitive psychology has also deepened our understanding of the link between “knowing” and “action.”³¹ The first step in the clinical reasoning process is to successfully integrate information obtained from the patient with the physician's existing medical knowledge to create a representation of the patient's problem. Physician competence in cognitive skills is an essential element for accurate diagnosis and clinical decision making. Next, physicians should evaluate their initial problem representation to ensure that they have not missed important information from the patients or to identify a gap in their knowledge that necessitates using information resources to acquire new knowledge. However, research suggests that physicians too often skip this important evaluation step and move directly to the last step: action.³² Our understanding of this clinical reasoning process demonstrates the strong link between cognitive skills, the cognitive skills of integration and synthesis, and actual patient care actions. The ABIM examination is a highly reliable and valid evaluation of clinical decision-making skills and currently represents the only summative evaluation method in the MOC program. Our findings and those of other studies suggest that cognitive skill is an important foundational competency, one that is essential to facilitate other care activities.

This study has several limitations. First, we assessed only the technical aspects of quality. Second, the quality measures used are quite modest in stringency; eg, only 2 hemoglobin A_1c determinations were required per year, and the measures for lipid testing and mammography were only required once in a 1-year period. Yet, despite this leniency in the measures, we still found a significant relationship between knowledge and performance for diabetes care and mammography screening. Third, we limited our evaluation to physicians with a minimum number of Medicare beneficiaries, and this restriction did not allow an analysis of the association between very low practice volume and knowledge. In a previous study, we found that low-volume physicians performed more poorly on basic diabetic processes of care, and we excluded low-volume providers from this study.¹³

Fourth, our study examined only the competency of knowledge and did not directly investigate competency in quality improvement and systems-based practice. Fifth, our study looked only at process of care measures, not at outcome measures. A recent study questioned the link between process and outcome measures and mortality rates in the hospital setting.³³ However, other studies found that attribution of performance at the physician level is higher with process measures than with outcome measures.^34,35 Sixth, the physician cohort was limited to those who first certified between 1990 and 1995. Therefore, we cannot examine the relationship of knowledge and performance with older physicians. Given the recent meta-analysis showing a decline in knowledge and performance over time, the relationship between knowledge and quality of care should be an area for future investigation.³ Yet, even among a relatively young cohort of physicians, we still found a relationship between cognitive skills and quality-of-care performance. Finally, we excluded physicians who did not take an MOC examination. We fitted additional models with a dummy variable for physicians who did not take the test and found that there was no difference in performance between physician groups with and without MOC scores. However, the distribution of scores from the physicians' initial certification examination was similar to that of the analyzed cohort and thus likely explains the lack of an association.

Our study findings help to build the evidence link between cognitive skills and quality of care. The MOC examination is part of the medical profession's self-regulatory activities to demonstrate to the public physicians' commitment to maintaining competence in cognitive skills. If the relationship between the examination performance and clinical performance is confirmed in other studies, then using an assessment of cognitive skills through a reliable examination as a quality measure may be reasonable. Clearly, further work is needed to explore the relationship between cognitive skills and competency in other skills, such as accuracy in diagnosis and treatment decisions, and between various competencies and other domains of quality.

Correspondence: Eric S. Holmboe, MD, American Board of Internal Medicine, 510 Walnut St, Ste 1700, Philadelphia, PA 19106 (eholmboe@abim.org).

Accepted for Publication: December 10, 2007.

Author Contributions: Dr Holmboe had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Holmboe, Meehan, Tate, Ho, and Lipner. Acquisition of data: Holmboe, Meehan, Tate, Ho, Starkey, and Lipner. Analysis and interpretation of data: Holmboe, Wang, Meehan, Tate, and Lipner. Drafting of the manuscript: Holmboe and Starkey. Critical revision of the manuscript for important intellectual content: Holmboe, Wang, Meehan, Tate, Ho, and Lipner. Statistical analysis: Wang, Tate, Ho, and Lipner. Administrative, technical, and material support: Meehan, Tate, Ho, and Starkey. Study supervision: Holmboe, Meehan, and Lipner.

Financial Disclosures: Drs Holmboe and Lipner and Ms Starkey are all employed by ABIM. Drs Wang, Meehan, and Ho and Ms Tate are all employed by Qualidigm. Qualidigm received payment from ABIM Foundation to conduct the analyses.

Funding/Support: This study was supported by a grant from ABIM.