Figure 1. A CONSORT (Consolidated Standards of Reporting Trials) diagram of the selection process. Eligible medical centers required (1) an affiliated institutional review board, (2) 4 or more eligible clinics, and (3) no ongoing potentially overlapping quality improvement project. The 4-clinic requirement was relaxed toward the end of recruitment. Eligible clinics were noncontract (owned and operated by the Department of Veterans Affairs [VA]), delivered primary care, used the VA's electronic health record system, and provided Internet access to all providers.
Figure 2. Adjusted differences in improvement in the clinical indicators, intervention minus control group. (Change measured as postintervention minus preintervention values, except for the physiological differences, for which the signs are reversed because a decrease signifies improvement among intervention clinics minus the change among control clinics.) With the use of generalized mixed-effects regression models, these differences and the associated 95% CIs and P values reflect adjustment for patient clustering within clinics, patient-level covariates (age, sex, race, and comorbidity score), and random clinic effects (to account for correlation of patient measures within individual clinics). The comorbidity score was calculated using a composite score based on the following 12 major health conditions: hypertension, hyperlipidemia, diabetes mellitus, heart failure, emphysema, depression, chronic kidney disease, stroke, cancer (excluding skin, prostate, and thyroid), peripheral vascular disease, dementia, and asthma. Levels of low-density lipoprotein cholesterol (LDL-C) and hemoglobin A1c (HbA1c) are provided for eligible patients who had a measurement obtained within 18 months of the index visit. The laboratory level or prescription was measured at the time of the index visit. ACEI indicates angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker. *Includes adults with diabetes mellitus. To convert LDL-C levels to millimoles per liter, multiply by 0.0259.
Levine DA, Funkhouser EM, Houston TK, Gerald JK, Johnson-Roe N, Allison JJ, Richman J, Kiefe CI. Improving Care After Myocardial Infarction Using a 2-Year Internet-Delivered InterventionThe Department of Veterans Affairs Myocardial Infarction–Plus Cluster-Randomized Trial. Arch Intern Med. 2011;171(21):1910-1917. doi:10.1001/archinternmed.2011.498
Author Affiliations: Departments of Medicine and Neurology, University of Michigan and Ann Arbor Veterans Affairs (VA) Healthcare System, Ann Arbor (Dr Levine); Departments of Medicine (Drs Levine and Funkhouser) and Surgery (Dr Richman), University of Alabama at Birmingham School of Medicine, and VA Research Enhancement Award Program, Birmingham VA Medical Center (Ms Johnson-Roe and Dr Richman), Birmingham; Department of Quantitative Health Sciences (Drs Houston, Allison, and Kiefe) and Transitions, Risks, and Actions in Coronary Events Center for Outcomes Research and Education (Drs Allison and Kiefe), University of Massachusetts Medical School, Worcester; Center for Health Quality, Outcomes, and Economic Research, Bedford VA Medical Center, Bedford, Massachusetts (Dr Houston); and Division of Public Health Policy and Management, Mel and Enid Zuckerman College of Public Health, University of Arizona, Tucson (Dr Gerald).
Background Cardiovascular risk reduction in ambulatory patients who survive myocardial infarction (MI) is effective but underused. We sought to evaluate a provider-directed, Internet-delivered intervention to improve cardiovascular management for post-MI outpatients.
Methods The Department of Veterans Affairs (VA) MI-Plus study was a cluster-randomized trial involving 168 community-based primary care clinics and 847 providers in 26 states, the Virgin Islands, and Puerto Rico, from January 1, 2002, through December 31, 2008, with the clinic as the randomization unit. We collected administrative data for 15 847 post-MI patients and medical record data for 10 452 of these. A multicomponent, Internet-delivered intervention included quarterly educational modules, practice guidelines, monthly literature summaries, and automated e-mail reminders delivered to providers for 27 months. Main outcome measures included percentage of patients who achieved each of 7 clinical indicators, a composite score of the 7 clinical indicators, and mean low-density lipoprotein cholesterol and hemoglobin A1c levels.
Results Clinics had a median of 3 providers (interquartile range, 2-6), with a median of 50.0% of providers (33.3%-66.7%) participating in the study. Patients in intervention clinics had greater improvements (from 70.0% to 85.5%) in the percentages prescribed β-blockers than patients in control clinics (71.9% to 84.0%; adjusted improvement gain for intervention vs control, 2.6%; 95% CI, 0.1%-4.1%). We found nonsignificant differences in improvements favoring patients in intervention clinics for 5 of 6 remaining clinical indicators and levels of low-density lipoprotein cholesterol and hemoglobin A1c.
Conclusion A longitudinal, Internet-delivered intervention improved only 1 of 7 clinical indicators of cardiovascular management in ambulatory post-MI patients.
Each year, 1.3 million Americans experience a myocardial infarction (MI) or death from coronary heart disease (CHD).1 In 2008, almost 8 million US adults were MI survivors.1 Randomized trials have demonstrated that aspirin, β-blockers, angiotensin-converting enzyme inhibitors, and hydroxymethylglutaryl–coenzyme A reductase inhibitors (statins) can reduce morbidity and mortality for MI survivors.2- 6 Population-based evidence suggests that better medical treatment and secondary prevention of CHD has led to substantial declines in CHD mortality during the past 3 decades.7- 10 Despite these advances, providers often fail to adhere to evidence-based guidelines for the management and secondary prevention of CHD.11- 16
Multicomponent interventions can increase adherence to clinical practice guidelines.17,18 In patients with CHD, multicomponent interventions have been shown to improve in-hospital care19- 22 but have yielded mixed results in primary care settings.23- 25 Although guidelines usually focus on CHD in isolation, ambulatory patients who have survived MI (post-MI patients) frequently have multiple comorbidities that may interfere with adherence to CHD guidelines.26 Furthermore, interventions to improve adherence are often limited by their short duration and lack of reinforcement, limiting sustainability over time.27,28 Reliance on academic detailing and site visits may also make many interventions expensive and impractical.23,24 Delivering an intervention for longer periods using the Internet (eg, online spaced education) may address some of these limitations.29,30 Moreover, online spaced education may allow intensive interventions to be widely disseminated with minimal marginal costs.
To test whether a longitudinal, multicomponent, Internet-delivered intervention, including educational cases, guidelines, monthly updates, and e-mail reminders, could improve guideline adherence and reduce CHD risk factors among post-MI patients with multiple comorbidities treated in primary care settings, we undertook a national cluster-randomized trial of community-based outpatient clinics affiliated with the Department of Veterans Affairs (VA).
The VA MI-Plus study was a cluster-randomized trial of post-MI patients with randomization at the clinic level. We were funded through the VA Health Services Research and Development office and by a parallel National Institutes of Health study.26 The institutional review board at each medical center approved the study.
Our intervention targeted VA community-based primary care outpatient clinics.31 We targeted all adults 18 years or older discharged from a participating facility with a primary or secondary diagnosis of acute or previous MI (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM ] codes 410.xx or 412.xx) and subsequently treated at a facility-affiliated clinic from January 1, 2002, through December 31, 2008. The index visit was the most recent clinic visit during the relevant study period, that is, January 1, 2002, through November 30, 2004, for the preintervention evaluation and February 1, 2007, through December 31, 2008, for the postintervention evaluation.
A panel of 12 experts generated 7 clinical indicators of post-MI monitoring, treatment, and target goals using nominal group technique (eTable 1).32,33 Monitoring indicators included measurement of low-density lipoprotein cholesterol (LDL-C) levels (all patients) and hemoglobin A1c (HbA1c) levels (all patients with diabetes mellitus) within 18 months of the index visit. Treatment indicators included prescriptions at the index visit for β-blockers, statins, and angiotensin-converting enzyme inhibitors or angiotensin II receptor blockers. Target goals included last LDL-C level of less than 100 mg/dL and last HbA1c level of less than 8% in patients with diabetes mellitus. (To convert LDL-C to millimoles per liter, multiply by 0.0259; to convert HbA1c to a proportion of total hemoglobin, multiply by 0.01.) Although these clinical indicators are process measures, each is a strong independent predictor of clinical outcomes in post-MI patients.2- 6,34
For the primary analysis, we extracted demographic information (age, race, and sex), diagnoses, medications, and laboratory data from more than 3 million records using the VA Decision Support System files (National Patient Care Data, Austin Information Technology Center, Austin, Texas). Comorbidities were defined using inpatient or outpatient ICD-9-CM codes during the 12 to 18 months before the index visit (eTable 2). Because a growing percentage of post-MI patients have contraindications to CHD therapies but often receive them,12,35 and some contraindications may also be indications (eg, the use of β-blockers in post-MI patients with heart failure), we adjusted for patient comorbidity rather than excluding patients from the denominator for the primary analysis (described in the “Statistical Analysis” subsection).
We performed a secondary analysis to determine receipt of aspirin and to confirm our findings in post-MI patients who were ideal candidates (ie, having no contraindications) for available indicators (for definitions, see eTable 3). We used patient-level data obtained from the VA Office of Quality and Performance (eAppendix).
The intervention included a multicomponent Web site and pushed e-mail cues with educational content.31 Eight case-based, interactive educational modules represented the core of the intervention. Each module reviewed the evidence and guidelines for 1 or more of the 7 clinical indicators developed by our expert panel, presented a clinical scenario followed by a series of questions, and provided tailored feedback to health care providers based on their responses. The Web site also included relevant clinical guidelines, monthly summaries of pertinent peer-reviewed manuscripts, downloadable practice tools, and patient educational materials. The Web site used service-oriented architecture and design principles refined previously.36,37 Iterative usability sessions refined the content. A site map of the VA MI-Plus intervention, which was available from November 1, 2004, through January 25, 2007, is found in eFigure 1. Proactive e-mails, demonstrated to increase participation in Web-delivered provider interventions,38 notified providers of new updates and materials.
When modules were released, providers in control clinics were sent a link to an existing VA Web site that contained links to a wide range of clinical guidelines for various medical conditions (http://www.healthquality.va.gov/). All enrolled intervention and control health care providers could obtain continuing education credits for reviewing materials on the Web site (for screen shots of the VA MI-Plus Intervention Web site, see eFigures 2-6) and receive a subscription to The Medical Letter.
The primary outcome was the percentage of patients who achieved each of the 7 clinical indicators. As a secondary outcome, we created a patient-level composite quality indicator score summing and then normalizing to a total of 100 those clinical indicators for which the patient was eligible. All patients were eligible for 3 contributors to the composite score (β-blocker use, angiotensin-converting enzyme inhibitor/angiotensin II receptor blocker use, and LDL-C level measurement), whereas diabetes mellitus and hyperlipidemia each added 2 additional condition-specific indicators. We also calculated mean levels of LDL-C and HbA1c. All outcomes were based on the most recent test or clinic visit. User tracing of server logs assessed intervention uptake. Provider participation was measured by randomization group.
Recruitment has been described elsewhere.31 Briefly, among 66 eligible medical centers, we recruited 48 (72.7%) affiliated with 219 clinics, including 957 providers in 26 states, the Virgin Islands, and Puerto Rico; of these, 168 clinics (76.7%) were randomized (ie, had ≥1 of its providers enroll); of the 847 providers in these 168 clinics, 401 (47.3%) participated (Figure 1). Providers were recruited by e-mail, postal mail, fax, and telephone. A clinic was enrolled and randomized when the first eligible provider at that clinic logged on to the study Web site. All providers at a clinic were randomized to the same arm, but only those who logged on were enrolled. Using further e-mail reminders, we recruited providers throughout the 27 months of the intervention. Although we recruited continuously, all clinics had at least 12 months of exposure to the intervention or control Web site. After Web site posting, components remained available for the duration of the study. All providers completed an online consent to participate.
The planned sample size of 200 clinics provided at least 80% power with 2-sided α = .05 to detect a difference of improvement of 5% in each clinical indicator attributable to the intervention. Our post hoc analysis suggests that we had 80% power to detect a difference of 5%, assuming 168 clinics and an average of at least 33 patients per clinic per period.
We used an intention-to-treat approach for our main analysis, basing our outcome measures on the entire eligible patient population in each of the 168 clinics regardless of the patient's provider enrolling on our Web site. Analyses were at the patient level but accounted for clustering of patients within clinics (unit of randomization). Baseline clinic characteristics were compared between the intervention and control groups using χ2 tests, unpaired, 2-tailed t tests, or Wilcoxon rank sum tests as appropriate.
Performance improvement was calculated as the change (before vs after the intervention) in the proportion of patients receiving each clinical indicator or as the mean change in the LDL-C or HbA1c level. The differences in performance improvement for each indicator were analyzed at the patient level by randomization group. Generalized linear models examined the associations between indicator performance, LDL-C and HbA1c levels, and the main independent variable, randomized group, accounting for patient clustering within clinic and time of observation (before vs after the intervention), with and without adjustment for patient age, race, sex, and comorbidity using a summary count of the 12 comorbidities considered. The P value for the randomization group × time interaction assessed the statistical significance of the intervention effect. Analyses were performed before and after stratification by clinic size (number of providers) and by urban or rural location.
For the secondary analysis, using manually abstracted VA performance data, we examined receipt of aspirin and performance by ideal candidates using the same approach as for the primary analysis except we adjusted only for patient clustering within clinics because other patient-level covariates were unavailable. We conducted another secondary analysis to assess whether there was a dose-response effect manifested as a correlation between degree of participation in the intervention and improved performance (eAppendix and eTable 5).
The median number of providers per clinic (interquartile range) was 3 (2-6); most of the clinics were urban (Table 1). Clinic characteristics did not differ by intervention group. Patients (n = 15 847) were mostly male with a mean (SD) age of 66 (10) years, 68.5% white, and 8.7% black (Table 2). The most common comorbidities were hypertension and hyperlipidemia.
The median percentage of providers from all 168 clinics who logged on to the intervention or control Web sites was 50.0%, higher for intervention than for control clinics (58.6% vs 50.0%; P = .03). Provider Web site activity was greater at intervention compared with control clinics (median number of Web site visits per provider, 3.7 vs 0.8; median number of Web site pages accessed per provider, 57.7 vs 6.2; P < .001 for both) (Table 1). Among the intervention group, the median number of completed case modules (maximum, 8) was 1.8 (interquartile range, 1-3) modules per provider and 3.3 (interquartile range, 2-5) modules per participating provider. Among providers who enrolled, the median time between the first log-on to the end of the study (active enrollment) was similar in intervention and control groups (17.0 vs 15.8 months; P = .2). However, the median time of active participation (time from the first to last log-on) and the proportion of providers with at least 12 months of active participation were higher in intervention vs control clinics (P < .001 for both).
Clinic size was inversely correlated with all measures of provider activity, including proportion of providers logged on (r = −0.55), mean Web site visits per provider (r = −0.30), and mean number of Web site pages accessed per provider (r = −0.24) (all, P < .001). Correlations were similar across randomization groups.
Over time, all quality measures trended toward improved performance, and improvement was slightly greater in intervention vs control clinics for all 9 measures (Table 3). After adjustment for patient demographics, comorbidity score, and clustering, intervention clinic patients had greater improvements than control clinic patients for the prescription for β-blockers (adjusted improvement gain, 2.6%; 95% CI, 0.1%-4.1%). Small statistically nonsignificant differences in improvements favoring patients in intervention clinics were observed after adjustment in 7 of the 8 remaining measures (Figure 2). Results were similar in small or large clinics and in rural or urban clinics (data not shown). The composite quality indicator score tended to improve more for all patients or for patients with diabetes mellitus and hyperlipidemia (those at highest cardiovascular risk) in intervention vs control clinics, but these gains were not statistically significant (P = .12 and P = .09, respectively, for the group × time interaction terms).
Our analysis using medical record–abstracted data (n = 10 452) showed consistent results, again favoring the intervention for the LDL-C level, with borderline significance (P = .07) (eTable 4). Our analysis assessing a dose-response effect of provider participation revealed trends all as expected, but only statistically significant for some indicators, particularly when defined by the number of Web pages accessed per provider. For example, findings of a mean LDL-C level of less than 100 mg/dL increased significantly more from before the intervention to after the intervention as the level of provider participation increased, whether the latter was defined as the number of Web site visits per provider (P = .02) or as the number of pages accessed per provider (P = .01) (eTable 5).
Our longitudinal Internet-delivered intervention improved performance on only 1 of 7 quality indicators for care of post-MI patients, namely, prescription of β-blockers. These findings were observed in small and large and in rural and urban clinics and persisted after adjustment for patient demographics and comorbidity.
Few studies have assessed multicomponent interventions to improve CHD management in the primary care setting, and results have been mixed.25 Among 20 primary care practices, an intervention with academic detailing, quality improvement facilitation, and audit and feedback demonstrated nonsignificant improvements in blood pressure in patients with CHD but no improvements in lipid levels or β-blocker prescriptions.23 An 18-month intervention with tailored care plans and academic detailing did not improve blood pressure or lipid levels but did reduce the number of hospitalizations in patients with CHD treated in 48 primary care practices.24 Taken together with our results, these studies show the challenges of improving processes of care or clinical outcomes with intensive interventions in primary care.39
Our study has several strengths. Our large-scale intervention was disseminated to 168 community-based outpatient clinics across a wide geographic distribution. Our “light-touch” longitudinal intervention combined low-intensity and high-dissemination ability40,41 and had low staffing and resource requirements. We used online spaced education with automated e-mail reminders to generate knowledge transfer, increase learning efficiency, and change provider behavior. Also, our system included downloadable patient support tools. We used e-mail marketing and continuing medical education credits to enhance engagement. The pushed cues and reminders allowed us to improve on our experience with provider participation and to achieve sustained intervention activity by some intervention providers.37,42 Although our intervention was multicomponent, once designed it was sustained with minimal effort. It is scalable to other settings at a cost that is modest compared with other quality improvement interventions. Although the improvement in the number of β-blocker prescriptions attributable to the intervention was modest, published estimates3,9 indicate that an additional 3% of CHD deaths may be prevented or postponed if the β-blocker prescription improvements are maintained for at least 6 months.
Still, our intervention improved only 1 of the studied process measures. Several factors may account for this. Consistent with previous research,43 the high baseline performance—higher than in other reports9,23,24—may have limited efforts to further improve performance. Also, as is common in implementation studies, some intervention clinics had low-intervention fidelity (eg, the intervention delivered was never received for providers in the intervention group who did not enroll). Our intervention might have had a much larger effect had baseline performance been lower or had it successfully engaged more providers. Indeed, in a post hoc, as-treated analysis, the intensity of provider participation in our intervention was positively associated with clinical indicator performance (eTable 5). During our study period, there were intense efforts by VA and non-VA health systems to improve performance on our indicators. Our findings of large improvements in performance in the control and intervention groups (Table 3) suggest that these secular changes may have minimized our ability to detect the effect of our intervention. Moreover, interventions targeted solely at providers may be less effective than those involving teams or systems reengineering.44,45
Several limitations merit discussion. The VA findings may not be generalizable to other clinical settings or to women. However, the VA is the largest integrated health care delivery system in the United States, with 153 medical centers and more than 900 outpatient clinics providing care to about 5.5 million veterans in 2008.46 Moreover, the VA offers unique uniformity of data and geographic diversity for intervention studies. Although our providers may be a select group owing to participation in a quality improvement study, their performance on the indicators studied was similar to that of all VA clinic providers. For example, performance on the indicator for control of lipid levels was 59.4% and 63.6% in 2005 (baseline) for the intervention and control groups, respectively, vs 55% in 2004 for all VA clinic providers. Similarly, these performances rose to 70.7% and 68.3% in 2007 through 2008 in our study compared with 68% in 2007 for all VA clinics. We did not evaluate differences in patient-centered or clinical outcomes, such as cardiovascular events, because our sample size did not provide sufficient power to detect a clinically meaningful difference. Still, our study indicators are commonly used to assess provider adherence to CHD guidelines and to determine pay for performance in VA and non-VA health systems.
Our study has potential implications for implementation research. Disease management programs may improve performance in patients with CHD and/or heart failure, but they are resource intensive.47 Electronic health records and clinical decision support may help providers manage chronic disease, but their benefits on ambulatory care quality are uncertain, and most ambulatory practices do not currently have electronic health records.48 In addition, most electronic health records do not integrate educational systems and patient support tools. The current ubiquity of Internet access positions the Internet as a promising tool for implementation research and quality improvement. Given major initiatives to increase the meaningful use of health information technology and to improve health information technology systems with performance bonuses for tools that ensure patient-centered appropriate care,49 multicomponent, Internet-delivered interventions may be feasible and cost-effective strategies to improve ambulatory care quality. Some previous Internet-delivered provider educational interventions have improved single performance measures for which baseline performance was low.36,50 However, consistent with previous research,37,42 our results suggest that light-touch educational interventions are less likely to be effective quality improvement strategies when baseline performance is high, when societal efforts targeting the same goals are under way, or when targeting a complex set of process measures.
In conclusion, we demonstrated that a longitudinal provider-directed, multicomponent, Internet-delivered intervention improved only 1 of 7 clinical indicators of cardiovascular management performance for post-MI outpatients. Light-touch educational interventions may improve quality less effectively when baseline performance is high or when targeting a complex set of process measures.
Correspondence: Deborah A. Levine, MD, MPH, Division of General Medicine, Department of Medicine, University of Michigan, 300 N Ingalls St, Room 7C27, Ann Arbor, MI 48109 (firstname.lastname@example.org).
Accepted for Publication: August 12, 2011.
Author Contributions:Study concept and design: Levine, Houston, Johnson-Roe, Allison, and Kiefe. Acquisition of data: Houston, Gerald, Johnson-Roe, and Kiefe. Analysis and interpretation of data: Levine, Funkhouser, Houston, Gerald, Allison, Richman, and Kiefe. Drafting of the manuscript: Levine, Funkhouser, Houston, Gerald, and Richman. Critical revision of the manuscript for important intellectual content: Houston, Gerald, Johnson-Roe, Allison, Richman, and Kiefe. Statistical analysis: Levine, Funkhouser, Allison, Richman, and Kiefe. Obtained funding: Houston, Allison, and Kiefe. Administrative, technical, and material support: Levine, Gerald, and Johnson-Roe.
Financial Disclosure: None reported.
Funding/Support: This study was supported by grant IHD-04-387 from the VA Health Services Research and Development office and grant R01 HL70786-02 from the National Institutes of Health for a parallel study.