Improving Asthma Care Through Recertification: A Cluster Randomized Trial

Background  As part of recertification, the American Board of Internal Medicine requires its diplomats to complete at least 1 practice improvement module (PIM). We assessed whether completing an asthma-specific PIM resulted in improved patient outcomes.

Methods  Practices were the unit of randomization in this cluster randomized trial. Physicians in the intervention group were asked to complete the PIM through its planning phase. The primary outcome was the dispensing of an inhaled corticosteroid (ICS) after a postintervention visit for asthma. Secondary outcomes included patient reported processes of care, asthma-related heath care use, and asthma severity. Analyses were adjusted for baseline rates at the cluster-level as well as for individual sociodemographic characteristics.

Results  Eight practices (19 internists) were randomized to the intervention group and 8 practices (21 internists) to the control group. For the primary outcome, ICS fill rates, patients seen by intervention group physicians were not more likely to fill an ICS prescription in the postintervention period than patients seen by control group physicians (adjusted odd ratio [AOR], 1.00; 95% confidence interval [CI], 0.64-1.56). Patients seen for asthma by intervention group physicians were less likely to receive a written action plan than patients seen by control group physicians (AOR, 0.67; 95% CI, 0.48-0.93); however, they were more likely to discuss potential asthma triggers (AOR, 1.62; 95% CI, 1.08-2.42) and had lower self-reported asthma severity measures (unadjusted P = .03). Per-protocol analysis supported the latter 2 associations.

Conclusion  A PIM designed to improve asthma care did not improve filling of ICS prescriptions but may have lessened asthma severity through an increased discussion of asthma triggers.

Trial Registration  clinicaltrials.gov Identifier: NCT00115284

In 1995, the American Board of Internal Medicine (ABIM) ad hoc Committee on Assessment of Practice Performance recommended that the ABIM assess practice performance in addition to assessing knowledge in its maintenance of certification program.^1,2 These recommendations were developed in the setting of a national movement to measure and monitor quality³ and with the hope that physician participation would be more likely to effect practice change.⁴ Initially, maintenance of certification included the option of completing a practice improvement module (PIM); however, in January 2006, this became a recertification requirement for all internists with time-limited certification.⁵ Modules have been developed for a number of conditions, including asthma, diabetes, and hypertension.

The overall design of each PIM was based on the chronic care model⁶ and idealized design of the clinical office practice.⁷ Each PIM is intended to improve care by having physicians adopt quality improvement in their practice, in part through increased awareness of the office microsystem.^8,9 In completing a PIM, physicians evaluate their (and their practice’s) management of a specific disease condition; they develop and implement a plan to improve care for that condition; and they measure the impact of the implemented plan on subsequent care. However, despite the current requirement that recertifying internists complete a PIM, it is not currently known whether these modules result in objective improvement in disease outcomes. We chose a cluster-randomized study design to determine whether an asthma PIM improved disease-specific outcomes.

The trial was performed at a large vertically integrated health care system in southeastern Michigan. All participating providers were practicing, board-certified internists and members of a large multispecialty medical group. Written consent was required of all participating physicians. The study was approved by the health system's institutional review board and was compliant with its Health Insurance Portability and Accountability Act policy.

This study was designed in accordance with the recommendations of the Consolidated Standards of Reporting Trials for cluster randomized trials.¹⁰ Practicing, board-certified general internists were recruited at the multiple practice sites (ie, clinics) throughout the health system. In the consent process, physicians were notified that if they were randomized to the intervention group they would be asked to complete an asthma PIM. The recruitment goal was 40 internists (20 internists per study arm), although the number of consenting physicians and their practice location determined the number and size of the practice clusters. As PIMs were designed to improve the disease-specific performance at both the physician level and the practice level, participating internists were grouped according to practice, and the practices were randomized such that all participating physicians at a given practice site were randomized to either the intervention arm or the control arm. Likewise, participating physicians within a practice who were randomized to the intervention group could work together to develop a practice improvement plan for their practice. Outcomes were assessed in patients seen for asthma by both intervention and control group physicians and accounted for higher-level clustering by physician and practice. Neither physicians nor researchers were informed of their intervention assignment until after randomization.

To ensure comparable patient populations in each study arm, practice randomization was stratified by rate of use of asthma controller medications (ie, greater than and less than 70% of asthmatic patients who were using a controller medication)¹¹ and practice location (ie, urban vs suburban). Patients with asthma who received their care from enrolled providers were notified by mail of their physician's participation in the study; however, patients were not aware of their physician's group assignment.

There were 2 phases of the asthma PIM: a data collection phase (ie, physicians collected a minimum of 10 patient surveys, performed at least 10 chart reviews, and completed 1 practice review) and an improvement plan phase (ie, physicians reviewed the results of the data collection phase, selected practice processes to improve, submitted the plan to the ABIM, implemented the plan in their practice, and measured the impact of the plan). We requested that participating physicians in the intervention group complete the PIM through plan implementation. All participating physicians in the intervention group received the asthma PIM contemporaneously and were initially allowed 45 days to complete the PIM; this period was extended to 90 days when it was clear that additional time was required. Control group physicians continued to provide their usual care and did not complete an asthma PIM. After this 90-day period, we began assessing the differences between the patients who were seen by intervention group physicians and the patients who were seen by control group physicians.

The primary outcome of this study was the likelihood of filling 1 or more inhaled corticosteroid (ICS) prescriptions after a visit for asthma. Secondary patient-level outcomes included the likelihood of each of the following after a physician visit for asthma: (1) filling the prescription for 1 or more short-acting β-agonist canisters; (2) filling the prescription for 1 or more oral corticosteroid medications; (3) reporting a prior prescription of an asthma controller medication (defined as an “inhaler or pill that is not used for quick relief, but instead is used to control asthma”); (4) receiving a written action plan for asthma exacerbations; (5) receiving instruction on the use of a peak flow meter; (6) reporting instructions regarding proper inhaler technique; (7) having a discussion regarding asthma triggers; (8) experiencing nocturnal asthma symptoms in the 4 weeks before the survey; (9) using a rescue inhaler regularly in the 4 weeks before the survey; and (10) being instructed to quit smoking (among current smokers with asthma). Additional secondary outcomes that were examined but not prespecified included the likelihood of asthma-related hospitalization in the 3 months after the asthma visit; the likelihood of an asthma-related emergency department visit in the 3 months after the asthma visit; measures of asthma severity and control; and both the Physical Functioning Scale and the General Health Scale from the 36-Item Short-Form Health Survey (SF-36).¹² These latter 2 dimensions of the SF-36 were used because they were the most strongly correlated with asthma severity in a previous study.¹³

Patient-level data regarding sociodemographics, processes of care, medication dispensings, asthma control and symptoms, and perception of general health were assessed by both patient surveys and electronic claims data. Because this study was a cluster-randomized trial, with practice as the unit of randomization, we assessed both utilization data and survey data on 600 patients with asthma who were seen at these practices to adjust postintervention study results. This double cross-sectional approach for cluster-randomized studies has been described in detail elsewhere.¹⁴ The survey assessed process indicators consistent with the primary and secondary outcomes, as well as asthma control, asthma severity, and general health, using validated instruments from the Asthma Control Test,¹⁵ the Asthma Symptom Utility Index,¹⁶ and the SF-36, respectively. A total of 358 of 600 patients (60%) responded to the baseline survey; 14 of 358 respondents (4%) reported that they did not have a diagnosis of asthma. Baseline rates of medication dispensing were assessed using electronic claims data. These data were used to identify the proportion of patients with a prescription fill in the 3 months after their last asthma-related visit to a study physician in the preintervention period.

To assess study outcomes, the same survey was administered to all patients seen by intervention and control group physicians for an asthma-related visit (N = 741) (ie, primary or secondary diagnosis coded as International Classification of Diseases, Ninth Revision, 493.xx) in the 6 months after the intervention (ie, the postintervention period). Some patients from the preintervention cohort may have been included in the postintervention cohort owing to the double cross-sectional design of the trial. Of the 741 postintervention surveys that were mailed, 496 (67%) were returned; 49 of the 496 respondents (10%) reportedthat they did not have a diagnosis of asthma. Electronic claims data were used to identify the proportion of patients with an ICS fill in the 3 months after their first asthma-related visit to a study physician in the postintervention period. Information on patient age, sex, race, and marital status was also available from electronic data. Race/ethnicity categories were entered at the time of registration into the health system; usually this information was self-identified but, on occasion, could have been assigned by health care personnel. We used diagnosis and procedural codes to calculate an adaptation of the Charlson Comorbidity Index, an estimate of concomitant comorbidity.¹⁷ To estimate median household income and other socioeconomic status variables, we used a commercially available package (Mapping Solutions LLC, Lansing, Michigan). The components included MapInfo Professional and MapMarker (MapInfo Corp, Troy, New York), bundled with Allocate/Solocast (SRC LLC, Orange, California). MapMarker geocoding software assigned longitude and latitude coordinates to the address record, which was then mapped to the census block using MapInfo Professional. Allocate/Solocast are data retrieval and reporting engines, which then linked census-block level demographic information to the mapped address. An individuals' household income was taken as the median household income for their census block using year 2000 data from the US Census Bureau.

Analyses were specific to the cluster-randomized design and accounted for correlation of outcome measures within practice clusters.¹⁴ This study had 80% power to detect an 18% difference in ICS/controller use (50% vs 68%) between treatment arms, assuming an intraclass correlation coefficient of 0.05 for patients seen in the same practice. Differences in the sociodemographic characteristics of patients seen for asthma in the preintervention and postintervention periods in the 2 study arms were compared using generalized estimating equation (GEE) logistic regression for categorical variables and GEE linear regression for continuous variables. The clusters for the GEE analyses were defined by physicians nested within practice locations. Differences in the primary and secondary outcomes between study arms were fit first with a univariable model and then with a multivariable model using logistical regression to estimate adjusted odds ratios (AORs) and 95% confidence intervals (CIs). Multivariable models were adjusted for baseline asthma severity and stratifying variables in practice clusters, baseline rates for the outcomes of interest in practice clusters, and individual sociodemographic characteristics, including age, sex, race, and median household income. We selected these individual characteristics to include in the models based on their prior association with asthma outcomes.^18-20 Differences in asthma control, asthma severity, physical functioning, and general health were assessed using linear regression. Because many physicians in the intervention arm did not complete the PIM as directed, we repeated our analyses comparing patients who were seen by the 5 intervention physicians who completed the intervention as directed (ie, per-protocol analysis) with patients who were seen by control group physicians. P < .05 was considered statistically significant. Data were analyzed using a commercially available software package.²¹

Eight practices, consisting of 19 participating physicians, were randomized to the intervention arm, and 8 practices, consisting of 21 physicians, were randomized to the control arm (Figure). No differences in age, sex, length of board certification, or location of practice (ie, urban or suburban) were found between intervention arm and control arm physicians (data not shown). Of the 19 physicians in the intervention group, only 5 (26%) completed the PIM through submitting the practice improvement plan (Table 1). Of the remaining 14 physicians, 10 initiated some portion of the PIM; 4 physicians (21%) in the intervention group did not submit any material.

Baseline differences between patients seen by intervention and control group physicians are shown in Table 2. No statistically significant differences were observed in the baseline variables that were used to adjust the postintervention results. Postintervention differences in the sociodemographic composition of patients seen by physicians in both study groups are shown in Table 3. There was no statistically significant difference in the sociodemographic characteristics between patients seen by intervention group and control group physicians (P values not shown).

Table 4 presents the intention-to-treat analysis comparing the odds of study outcomes between treatment arms. For the primary outcome, ie, ICS use, patients seen by intervention group physicians were not more likely than patients seen by control group physicians to have a fill of an ICS in the 3 months after the visit for asthma in the postintervention period (AOR, 1.00; 95% CI, 0.64-1.56). Patients of physicians in the intervention group were less likely to report discussing a written action plan for asthma exacerbations (36% vs 46%; AOR, 0.67; 95% CI, 0.48-0.93) but more likely than patients of physicians in the control group to discuss possible asthma triggers with their physician (77% vs 70%; AOR, 1.62; 95% CI, 1.08-2.42). In the unadjusted analysis, patient-reported asthma severity (ie, Asthma Symptom Utility Index score) was significantly lower in patients seen by physicians in the intervention group (P = .03) but was of borderline significance after adjustment (P = .09) (Table 5).

Because many physicians in the intervention arm did not complete the PIM as directed, we compared postintervention study outcomes between patients who were seen by the 5 intervention physicians who completed the intervention as requested and all patients who were seen by control group physicians (ie, per-protocol analysis). Patients who were seen by intervention physicians were more likely to report discussing potential asthma triggers with their physician (AOR, 1.55; 95% CI = 1.01-2.37) (Table 6). Also, patients who were seen by physicians in the intervention group had a lower index of asthma severity and a better index of disease control after intervention (P< .001 and P = .01, respectively) than patients who were seen by control group physicians (Table 7).

In this study, we did not observe an improvement in our primary outcome measure: filling of ICS prescriptions. However, we did observe an improvement in patient self-reported asthma severity in patients who were seen by intervention group physicians compared with patients who were seen by control group physicians. Although asthma severity was an a priori outcome at the time of data collection (and all measured outcomes are shown), it was not prespecified at the time of study registration, nor was it the primary outcome. These findings, albeit speculative, provide the first evidence that recent recertification requirements implemented by the ABIM may improve relevant disease outcomes.

If traditional, didactic continuing medical education may be of limited utility in improving physician behavior or health outcomes, practice-based interventions may be more effective.^22-24 In 1 study, physicians were asked to record those learning activities that prompted practice changes.²⁵ Investigators found that learning by reviewing the treatment of 1 or more patients was 37% more likely than medical literature review to result in a change in practice. In light of the recent evidence suggesting that physicians do not accurately self-assess their performance,²⁶ structured feedback, as occurs with the PIM, may provide a stronger impetus for behavioral change.²⁷ In a meta-analysis of interventions used in disease management, provider feedback appeared effective in improving adherence to guidelines but resulted in modest improvements in disease control.²⁸

In this study, the exact process through which the PIM may have improved asthma severity was not known. The only significant postintervention differences that we noted for patients seen by intervention group physicians were increased discussions of asthma triggers but lower rates of receiving a written action plan when compared with patients seen by control group physicians. We did not observe a significant difference in the primary outcome measure: ICS use.

Despite the limited differences in process measures, there are a number of reasons why we believe that the observed differences in asthma severity resulted from the intervention. First, differences in asthma severity between treatment arms persisted after baseline severity measures within practice clusters were controlled for. Second, the strength of the association increased after limiting the analysis to those physicians who completed the PIM as requested (this finding also included other measures of disease control, such the Asthma Control Test score). Third, other measures of asthma severity that preceded the postintervention index visit, such as historical asthma-related emergency department visits and hospitalizations, did not differ between patients seen by physicians in both study arms, suggesting that patients seen by both groups of physicians were similar in disease severity before the postintervention index visit. Finally, patterns of rescue medication use (ie, lower short-acting β-agonist and oral steroid use) after the postintervention visit were consistent with subsequent less severe asthma in patients seen by intervention group physicians when compared with those seen by control group physicians.

It is important to note that in completing the PIM, physicians chose the process measure that they wished to improve. Accordingly, physicians may have identified and targeted different deficiencies to improve. We assumed that given the well-described benefits of ICS use,^29-32 most physicians would choose ICS use as the process to improve. In particular, more than 90% of patients reported a previous prescription for a controller medication, yet only 50% of patients with asthma filled an ICS prescription at baseline. This suggested that actual ICS use was inappropriately low even after other asthma controller medication use was accounted for (data not shown). However, physicians completing the asthma PIM adjudicated ICS use by abstracting patient charts and were not privy to pharmacy fill information or rates of adherence. Because patients and physicians tend to overreport^33,34 and overestimate³⁵ compliance, respectively, it is quite possible that physicians did not perceive a deficiency in ICS use and therefore did not target it for improvement.

Although we did not observe a change in the primary outcome measure, if the net effect of multiple process changes (measured and unmeasured) resulted in improved asthma care, we might expect to see an improvement in an end measure, such as asthma severity, rather than a single, intermediate-process measure. However, patients who were seen by intervention physicians were significantly more likely to report discussing potential asthma triggers with their physician than were patients who were seen by control group physicians. Therefore, it is also plausible and possible that a resulting decrease in trigger exposure among patients seen by intervention group physicians resulted in less severe asthma symptoms, as has been demonstrated in intensive home-based interventions.³⁶

In focusing on 1 aspect of asthma-related care to improve, such as discussing asthma triggers, other processes may have been neglected by physicians in the intervention group. Neglecting to discuss these other aspects of asthma care may explain why patients who were seen by intervention physicians were significantly less likely to report receiving written action plans. The long-term effects of these potential trade-offs were not evaluated and will need further study.

This study must be interpreted in light of its other limitations as well. Randomizing practices may result in an inequitable distribution of patients, which can confound outcome differences between treatment arms.¹⁴ To prevent and control potential confounding, we stratified our randomization and adjusted for baseline intercluster differences. Although we found no difference in baseline or patient-level sociodemographic characteristics by study arm, other important differences may have persisted in spite of our efforts. Despite this limitation, we believe that the cluster-randomized design was the most appropriate model for a practice-based intervention.

Next, most of the physicians in the intervention group did not complete the intervention as intended. One of the barriers identified was the time that physicians had to complete the PIM through the planning stage, which was initially shorter than the time allowed by the ABIM in actual practice. As a result, time was extended (ie, from 45 days to 90 days) to more closely replicate the actual time allowed when a module for maintenance of certification is being completed. Extending the time allotted to complete the module did not markedly increase the number of physicians who completed the required PIM components. Because some physicians in our study had time-unlimited ABIM certification, we also examined whether PIM completion rates were different for this group when compared with the intervention group physicians who were required to complete a PIM for recertification. Surprisingly, completion rates were not different between groups (P > .99, data not shown). Therefore, allocated time and current ABIM requirements did not seem to affect completion rates.

Anecdotally, some clinicians expressed difficulty in having sufficient numbers of patients return surveys, a problem well known to researchers. Given low patient response rates on surveys in general, it may be onerous for physicians to meet the requisite number of surveys to complete a PIM. This component of the PIM also had the lowest rate of completion in our study. However, now that PIMs are a requirement for ABIM recertification, a formal analysis of factors related to completion could be performed on a much larger group of physicians than was involved in our study.

Although there were few intervention physicians who completed the PIM as requested, our results suggested that the intervention may have affected clinically important outcomes (ie, asthma severity and control), especially among patients whose physicians developed a plan to improve asthma care in their practice. However, the overall clinical relevance of these changes is uncertain. Moreover, the lack of multiple-process measure improvements proximal to the time of the intervention could also be construed as evidence against the intervention's robustness, since this would have been the time of peak effect. Finally, given sample size limitations, there was no prespecified plan to adjust for the multiple comparisons of secondary outcomes. However, in this study, we show all outcome measures examined, and post hoc adjustment using a conservative criterion, such as the Bonferroni correction, would still suggest that the PIM reduced asthma severity.

In summary, despite many physicians in our study not completing the PIM as directed, our findings suggest that asthma severity may be lowered in patients whose physicians complete an asthma PIM. Although the effect appeared modest, as pointed out by Shojania and Grimshaw,³⁷ perhaps this is the magnitude of improvement that we should rationally expect from quality improvement efforts. Further studies will be needed to confirm our findings and to identify modifiable barriers to physicians completing PIMs.

Correspondence: L. Keoki Williams, MD, MPH, Center for Health Services Research, Henry Ford Hospital, One Ford Place, 3A CHSR, Detroit, MI 48202 (kwillia5@hfhs.org).

Accepted for Publication: June 13, 2007.

Author Contributions: Ms Simpkins had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Simpkins, Divine, and Williams. Acquisition of data: Simpkins, Wang, and Williams. Analysis and interpretation of data: Simpkins, Divine, Wang, Holmboe, Pladevall, and Williams. Drafting of the manuscript: Simpkins, Divine, Holmboe, and Williams. Critical revision of the manuscript for important intellectual content: Simpkins, Divine, Holmboe, Pladevall, and Williams. Statistical analysis: Divine, Wang, and Pladevall. Obtained funding: Williams. Administrative, technical, and material support: Simpkins. Study supervision: Williams.

Financial Disclosure: Dr Holmboe is an employee of the ABIM.

Funding/Support: This research was funded by the ABIM Foundation. Additional funding was provided by the National Institute of Heart, Lung, and Blood (Drs Pladevall and Williams); the National Institute of Allergy and Infectious Diseases (Dr Williams); and the National Institutes of Diabetes, Digestive, and Kidney Disease (Drs Pladevall and Williams), National Institutes of Health, Bethesda, Maryland.

Role of the Sponsor: Dr Holmboe, acting in the role of coinvestigator, was involved in the analysis and interpretation of data and critical revision of the manuscript for important intellectual content.

Previous Presentation: This work was presented in part at the 13th Annual Health Maintenance Organization Research Network Conference; March 20, 2007; Portland, Oregon.