Shared Electronic Vascular Risk Decision Support in Primary Care: Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness (COMPETE III) Randomized Trial

Background Computerized decision support systems (CDSSs) linked with electronic medical records (EMRs) are promoted as an effective means of improving patient care. However, very few high-quality studies are set in routine, community-based clinical care, and no consistent evidence of an effect on patient outcomes has been found.

Methods A randomized controlled trial among EMR-using primary care practices in Ontario, Canada. Patients 55 years or older with previous vascular events, diabetes mellitus, hypertension, or hypercholesterolemia were randomized to the Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness (COMPETE III) CDSS intervention or to usual care. The intervention included personally tailored electronic vascular risk monitoring and treatment advice shared between the physician and patient, risk calculation, and a clinical resource. The primary outcome was a composite score of 8 recommended process outcomes at 1 year. Data collectors were blinded to group allocation. Analysis used the intention-to-treat principle with multiple imputation for missing data.

Results We randomized and included in the analysis 1102 patients in 49 community-based physician practices (53.4% female; mean age, 69.1 years; 28.0% with a previous vascular event). The intervention group (545 [49.5%]) had a significantly greater improvement in mean process composite, with a difference of 4.70 (P < .001) on a 27-point scale. Intervention patients had significantly higher odds of rating their continuity of care (4.18; P < .001) and their ability to improve their vascular health (3.07; P < .001) as improved. Despite this improvement, the clinical outcomes—vascular events, clinical variables, and quality of life—were not improved.

Conclusion Despite favorable reviews and important improvements in the complex processes required to reduce vascular risk, clinical outcomes remain unchanged.

Trial Registration clinicaltrials.gov Identifier: NCT00132145

Health care everywhere is troubled with problems of variable quality, efficiency, and access to care, made worse by a lack of high-quality pragmatic trials dealing with complex interventions for chronic diseases.^1-4 Issues particularly relevant in primary care include difficulties in accessing all relevant patient information owing to fragmentation of care, overwhelming amounts of clinical guidance not linked to specific patients, an inability to identify and contact high-risk patients for recommended care, and a lack of infrastructure to support effective management of patients with chronic disease. Electronic medical records (EMRs) as information management tools were designed to overcome many of these problems.^5-9 Although EMRs are intensively promoted as a key component of health care by governments in Europe, Canada, and the United States, a large survey¹⁰ of primary care physicians found that only 23% to 28% of those in North America reported using an EMR in their practices and 11% to 20% produced prescriptions electronically. Implementation of EMRs and its associated computerized decision support systems (CDSSs) in practice often meets with limited success because of protracted and expensive development, poor usability or integration into practitioner workflow, and difficulties in the mastery of frequently evolving electronic tools.^11-13

Evidence supporting the effectiveness of CDSS and other health information technologies is sparse, especially in the area of chronic diseases management.^8,14-16 Publications throughout the past decade note a consistent lack of positive effect of computerized interventions on patient outcomes.^13,17 Trials in this area tend to be difficult to design, monitor, and complete because of technology development and pretesting requirements, multifaceted interventions, complex organization, and cost.^18-21 Many health information technology studies to date have used suboptimal designs, lacked study power, suffered major biases, and failed to examine clinically important outcomes.^22-24

The Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness (COMPETE III) was designed as a pragmatic randomized trial of a multiple risk factor intervention to lower vascular risk and prevent future vascular events. Vascular disorders are responsible for the largest numbers of deaths, disability, and hospitalizations in North America.²⁵ Moreover, the treatment and prevention of vascular diseases is an enormous economic burden estimated at $503 billion in the United States for 2010.²⁶ Despite good evidence of important effects on patients' length and quality of life, effective multiple risk factor secondary prevention of chronic vascular disorders remains underused in hospital and primary care settings.^27-29 The importance of primary prevention of multiple risk factors in high-risk patients may be lessened in terms of decreased mortality and events but is also highly recommended.^30,31

We undertook a pragmatic randomized controlled trial of a CDSS-anchored intervention in primary care to assess its effects on relevant patient and physician outcomes. Our hypothesis was that older adults at increased risk of vascular events, if connected with their family physicians via an electronic network (Web and telephone based) sharing an individualized vascular tracking, advice, and support program, would show improvement in their vascular care and outcomes.

COMPETE III was designed as a pragmatic, 2-arm, randomized, controlled, multicentered clinical trial. We recruited community-based primary care practices using any EMR system certified by the province. Family physicians screened their patient lists for eligible patients (those 55 years or older with a diagnosis of diabetes mellitus, hypercholesterolemia, hypertension, previous myocardial infarction, angina or coronary artery disease, stroke or transient ischemic attack, or peripheral vascular disease who had attended a visit within the past 12 months). Exclusion criteria included cognitive impairment, lack of fluency in English, or being in a nursing home at the time of the study. Invitation packages were sent to eligible patients by mail; only those who replied that they were interested in being contacted about the study were directly contacted by study personnel. Trained interviewers telephoned interested patients to obtain informed consent to participate. Those who were agreeable consented verbally and were immediately randomized using an allocation-concealed online program. Patients completed and returned written consent forms. Randomization was by patient with stratification by physician using a block size of 6.

The study was approved by the independent research ethics committees of St Joseph's Healthcare–Hamilton and Hamilton Health Sciences (05-228), Hamilton, Ontario, Canada, and Elizabeth Bruyere Health Centre, Ottawa, Ontario, Canada.

The COMPETE III intervention centered on a Web-based individualized vascular tracking and advice-decision support system (CIIIVT) outlining 8 of the top vascular risk factors (blood pressure, low-density lipoprotein cholesterol levels, weight, aspirin or equivalent therapy, smoking, exercise, diet, and psychosocial index) plus 2 additional risk factors particularly important for patients with diabetes (hemoglobin A_1c and urine albumin levels).³² The interface, which is shown in Figure 1, displayed the patient's current and previous values for each risk factor, the relevant target, the last time it had been checked, and brief advice summaries. Physicians or clinical resource staff could update the patient's tracker profile data at any time; the decision support algorithms ran nightly to update the recommendations and the color highlighting. Color highlights allowed rapid focus on risk factors requiring attention because red meant that the risk factor was overdue for measurement and that the last value was off target, yellow meant that the timing or the value was off target, and green meant that the time since measurement and the value were within target limits. The CIIIVT was shared by patients and their physicians, and the targets were based on the latest prognostic evidence.³² Patients were encouraged to book quarterly visits to review their CIIIVT with their physician; the study provided them with requisitions for recommended laboratory tests at intervals as suggested by the tracker. In addition to having Web access, patients were mailed a color print version of their tracker page more than a week before their next physician visit with a suggestion to take it with them to the visit. The intent was to have the results available for the physician visit. Physicians could view the CIIIVT while using their EMR. Intervention patients also had telephone access to a clinical resource person (a pharmacist or a nurse) who provided advice and liaison with the physician.

Control group patients received their usual care from their family physician. Patients were to be followed up for 12 months.

Data were collected by the following 2 methods: primary care medical record review using trained reviewers with regular checks for reliability and computer-assisted telephone interview with the patients. The former captured most risk factor information, vascular history, and vascular events. The latter captured perceived usefulness, ease of use, quality of life, and psychosocial, exercise, and diet information. Physicians completed brief written questionnaires on the intervention at the study end. Laboratory requisitions were given to all patients for the 12-month final checks of low-density lipoprotein cholesterol, hemoglobin A_1c, and urine albumin levels. The primary outcome was based on the change in process composite score (PCS). The PCS was calculated as the sum of the frequency-weighted process score for each of the 8 main risk factors, with a total possible score of 27 (Table 1). Process was said to have been met if the medical record review or patient interview identified that a risk factor had been checked. The additional 2 process variables, hemoglobin A_1c and urine albumin levels, were analyzed separately as secondary outcomes for patients with diabetes. The PCS was measured at baseline and study end, with the difference forming the individual primary outcome scores.

The main secondary outcome was a clinical composite score (CCS) based on the same 8 risk factors, measured in 2 ways. The on-target CCS was a comparison of the mean number of clinical variables on target between the groups at study end,and the less restrictive improvement CCS was a comparison between groups of the percentage of patients with improvement. Other secondary outcomes included actual vascular events (hospitalization for acute coronary syndrome, coronary artery bypass grafting, percutaneous coronary intervention, stroke, or peripheral vascular disease), analyses of the individual components of the PCS and CCS, ratings of usability, continuity of care, patient ability to manage vascular risk, and quality of life using the EQ-5D.³³

The nature of the intervention made blinding patients, physicians, and clinical resource staff impossible. However, data collection was blinded, with medical record reviewers and telephone interviewers unaware of group allocation. In addition, data analysis and manuscript preparation were kept blinded until the end of analysis. Data analysis was performed using strict intention-to-treat principles.

All study personnel and partners consented to comply with the COMPETE III code of conduct, which is based on current data security and confidentiality recommendations of the International Organization for Standardization. Given the real-practice setting of this pragmatic randomized controlled trial, missing data were unavoidable, and multiple imputation methods were used for accurate statistical inference.

A total sample size of 1100 participants was sufficient for 90% power to detect a minimal clinically important difference of 1 PCS point between study arms, assuming an SD of 5 points, a 2-tailed t test for difference between means, a significance level of 5%, and a possible withdrawal rate of as much as 10%. Analyses of the primary PCS outcome, the secondary CCS, and the EQ-5D score used a generalized estimating equation with an exchangeable correlation structure to account for clustering within physicians. Other outcomes used descriptive statistics and χ² tests or exact tests, as appropriate.

All statistical analyses were performed with commercially available software (SAS, version 9.2 [SAS Institute, Inc, Cary, North Carolina] and StatsDirect [StatsDirect LTD, Altrincham, England]).

We randomized 1102 patients (mean age, 69.1 years; 53.4% female, 28.0% with a previous vascular event) from 49 family physician practices from February 1 through September 30, 2005. Physicians were 42.9% female, had a mean (SD) age of 47.2 (9.5) years, and had spent a mean of 20.8 years in practice. Participating practices, which were located in 18 sites across Ontario, used 5 different EMR products. Most saw more than 30 patients daily. Baseline characteristics of patients are presented in Table 2. Blood pressure, low-density lipoprotein cholesterol level, aspirin use, and smoking were reasonably well controlled at baseline for many patients. Although the recruitment process clarified that the main study intervention was Web based, 28.7% of all study patients later reported that they never used the Internet. Mean (SD) follow-up was 51.7 (3.5) weeks and was secured in 99.1% of all participants. Five deaths (3 in the intervention group and 2 in the control group) were not included in the statistical analysis. The study flow diagram is shown in Figure 2. Intervention patients interacted with the clinical resource person a mean (SD) of 2.19 (1.12) times.

Patients had high expectations at baseline in terms of the intervention's assistance in improving their vascular health and care, expectations that were tempered by study end (decline of 3.4 in the 30-point summary perceived usefulness score). Despite training on the Web tracker, 86.7% of intervention patients preferred the paper tracker pages compared with the Web version, citing primarily lack of interest in and dislike of computers. Intervention patients were much more likely to rate their continuity of care (odds ratio [OR], 4.18; 95% CI, 3.04-5.76; P < .001) and ability to improve their vascular health (3.07; 2.37-3.99; P < .001) as improved compared with controls.

The intervention had a significantly greater improvement in mean PCS (maximum possible score of 27), with a difference of 4.67 (95% CI, 3.63-5.71; P < .001) (Table 3). Significantly more patients improved by at least 3 points on this score (67.7% vs 25.5%; P < .001 (Table 3). Family physician visits were increased by a mean (SD) of only 1.22 (4.10) visits during the year in the intervention group, suggesting that the CIIIVT improved the efficiency of each visit in that more vascular risk factor modification was addressed in each visit. Despite considerable improvement in PCS, the clinical outcomes of blood pressure, cholesterol levels, body mass index, exercise, diet, and psychosocial scores showed no significant difference between groups. Only prescribing of aspirin therapy improved (OR, 1.44; 95% CI, 1.07-1.94; P = .02) (Table 4). Similarly, CCS, measured by the mean (SD) number of variables on target (4.70 [1.53] vs 4.62 [1.44]) (Table 4) or the proportions of patients who improved by 1 and 3 clinical variables (91.7% vs 90.6% and 36.9% vs 33.5%, respectively) (Table 4) showed no significant difference between groups.

For the diabetes subgroup (n = 244), intervention patients had a significantly greater improvement in the recommended monitoring of hemoglobin A_1c and urine albumin levels. However, neither the hemoglobin A_1c nor urine albumin values were significantly improved in the intervention compared with the control group.

Quality of life, as measured by the EQ-5D score, was not significantly changed. Fifty-seven patients had vascular events during follow-up: 27 (5.0%) in the intervention group and 30 (5.4%) in the control group. A χ² statistic suggested no difference between the groups (P = .75). Exit questionnaire responses were examined for potential explanations of the disconnect between process and clinical outcome results. Only a moderate proportion (49.3%) of intervention patients rated their knowledge of their vascular risk status as improved. Physicians noted improved knowledge of vascular disease benchmarks but highlighted technical difficulties with integration of the tracker system with their own EMR.

Because patient-important clinical outcomes were not affected, a preplanned economic analysis was not completed.

Although a CDSS integrated with EMRs has been touted as a leading solution for improving quality of care and chronic disease management, it has rarely been subjected to rigorous analysis using patient-important outcomes. Even more unusual are randomized trials in the setting of distributed small primary care practices in the community; however, these supply most of the ongoing clinical care in North America. Our results suggest that a shared, individualized CDSS for managing vascular disease is highly regarded by patients and physicians and improves the processes of multiple risk factor modification but does not change important clinical markers or events in a 1-year follow-up. This conclusion is not inconsistent with those of other high-quality studies addressing such a complex area. The most recent Cochrane review on multiple risk factor intervention for cardiovascular disease addressed only primary prevention³⁴ and found that mortality was not decreased in the 10 trials addressing clinical outcomes. The most recent systematic review¹³ of higher-quality evidence of CDSSs found 13 trials examining cardiovascular disease management or prevention. Only 4 examined clinical events or quality of life as outcomes, and none noted improvement in patient outcomes. Similarly, a recent report³⁵ indicates that higher-quality evidence from randomized trials is not associated with improved patient outcomes from health information technology interventions to improve medication management.

Several factors may be important in explaining the divide between our process success and the failed clinical outcome. Full technical integration of CDSSs with the various EMRs in use in the community, which would allow for a single interface of data entry and receipt of patient-specific advice, was not fully achieved with each participating EMR. Future CDSS endeavors may need to awaitEMRs with better integration potential. Second, the enrollment of patients of variable vascular risk status may have diluted the impact of the intervention. Future studies might focus on high-risk patients. Third, lifestyle modifications (diet, weight, exercise, and psychological health) are a prominent,³⁶ perhaps dominant, part of vascular risk management,^32,37,38 but community-based primary care is poorly equipped to provide and monitor lifestyle prescriptions. Organized referral programs dealing with lifestyle issues, similar to cardiac rehabilitation programs, might have improved the success of this study. Fourth, the accepted quality improvement paradigm of allowing process outcomes to act as surrogates for clinical outcomes in chronic disease management may be flawed. This possibility is supported by the most recent systematic review of CDSSs in clinical care, in which process improvements were frequent but rarely led to improvements in clinical outcomes.¹³ Vascular disease, with its requirement of lifelong attendance to multiple risk factors, may be more resistant to control than other chronic diseases. Fifth, although both patients and physicians noted improved knowledge of vascular disease and targets for risk factors and improved continuity of care and patient self-efficacy, the lack of patient use of the Web-based tracker may have limited its impact.

Our study has several potential limitations. Practice-based trials are often performed as cluster randomized trials. However, these designs are not feasible when practices refuse to enroll if randomization to the control group is a possibility, and the subsequent potential for patient selection bias and outcome assessment bias may counteract the potential benefit of cluster randomization.³⁹ We dealt with clustering by stratifying by physician practice and checking the intraclass correlation, which was very low. Second, our composite scores, although sensible, have not been validated.

The COMPETE III trial has implications for clinical practice and health care policy. Implementation of electronic medical and health records is expensive, and an integrated CDSS is also a complex, costly intervention. Clinicians are correct to remain skeptical about the cost-effectiveness of these systems and should continue to demand evidence that they improve patient outcomes. To date, the benefit of both innovations has been greatly overstated and is not supported by the type of health technology appraisal that is routinely applied in most other domains of health care. Second, although some patients may be interested in electronic health care, our surveys suggest that only a small percentage of older adults use the Internet for their own health care management. Canada, the United States, and other developed countries continue to spend billions of dollars trying to computerize health care. No one doubts the value of a legible medical record, organized storage of cumulative medical data, or the ability to query a patient population. However, health information technology policymakers should demand rigorous research in this area with full assessments of benefits, harms, and costs. Problems with the usability of health information technology, training and support, quality standards, and knowledge content might resolve with further scrutiny.

We found that, despite favorable ratings from patients in terms of continuity of care and self-efficacy in handling their personal vascular risks and important improvements in the complex processes required to reduce vascular risk, clinical variables were not consistently improved. Large investments in CDSSs may not be warranted until the CDSSs and their associated EMRs have matured to a higher quality.

Corresponding Author: Anne Holbrook, MD, PharmD, MSc, FRCPC, Division of Clinical Pharmacology and Therapeutics, 105 Main St E, P1 Level, Hamilton, ON L8N 1G6, Canada (holbrook@mcmaster.ca).

Accepted for Publication: July 8, 2011.

Author Contributions: All the authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis; all authors read and approved the final manuscript. Study concept and design: Holbrook. Acquisition of data: Holbrook, Troyan, Keshavjee, and Chan. Analysis and interpretation of data: Holbrook, Pullenayegum, Thabane, Foster, Keshavjee, Dolovich, Gerstein, Demers, and Curnew. Drafting of the manuscript: Holbrook and Troyan. Critical revision of the manuscript for important intellectual content: Holbrook, Pullenayegum, Thabane, Keshavjee, Chan, Dolovich, Gerstein, Demers, and Curnew. Statistical analysis: Pullenayegum, Thabane, and Foster. Obtained funding: Holbrook. Administrative, technical, and material support: Keshavjee and Chan. Study supervision: Holbrook and Troyan.

Funding/Support: This study was supported by the Ontario Ministry of Health and Long-term Care's Primary Healthcare Transition Fund competition.

Role of the Sponsors: The sponsors had no role in the design and conduct of the study; in the collection, management, analysis, or interpretation of the data; or in the preparation, review, or approval of the manuscript.