eFigure 1. CV Wizard Provider View (Prioritizes Patient Risks and Provides Decision Support Suggestions)
eFigure 2. CV Wizard Patient View (Prioritizes Patient Risks and Presents the Information Visually)
eMethods. Reversible Risk Calculations
eTable. Clinic Characteristics by Study Group at Baseline
Data Sharing Statement
Customize your JAMA Network experience by selecting one or more topics from the list below.
Gold R, Larson AE, Sperl-Hillen JM, et al. Effect of Clinical Decision Support at Community Health Centers on the Risk of Cardiovascular Disease: A Cluster Randomized Clinical Trial. JAMA Netw Open. 2022;5(2):e2146519. doi:10.1001/jamanetworkopen.2021.46519
Does a clinical decision support system (CDSS) proven to reduce cardiovascular risk in integrated care settings also reduce cardiovascular risk in community health centers?
In this cluster randomized clinical trial of 18 578 eligible patients, although CDSS adoption rates were low, CDSS use was associated with significantly improved reversible risk of cardiovascular disease among patients with the highest levels of baseline risk.
The use of a CDSS in community health centers has the potential to improve reversible risk of cardiovascular disease among socioeconomically vulnerable high-risk patients; strategies to increase CDSS adoption in this setting are needed.
Management of cardiovascular disease (CVD) risk in socioeconomically vulnerable patients is suboptimal; better risk factor control could improve CVD outcomes.
To evaluate the impact of a clinical decision support system (CDSS) targeting CVD risk in community health centers (CHCs).
Design, Setting, and Participants
This cluster randomized clinical trial included 70 CHC clinics randomized to an intervention group (42 clinics; 8 organizations) or a control group that received no intervention (28 clinics; 7 organizations) from September 20, 2018, to March 15, 2020. Randomization was by CHC organization accounting for organization size. Patients aged 40 to 75 years with (1) diabetes or atherosclerotic CVD and at least 1 uncontrolled major risk factor for CVD or (2) total reversible CVD risk of at least 10% were the population targeted by the CDSS intervention.
A point-of-care CDSS displaying real-time CVD risk factor control data and personalized, prioritized evidence-based care recommendations.
Main Outcomes and Measures
One-year change in total CVD risk and reversible CVD risk (ie, the reduction in 10-year CVD risk that was considered achievable if 6 key risk factors reached evidence-based levels of control).
Among the 18 578 eligible patients (9490 [51.1%] women; mean [SD] age, 58.7 [8.8] years), patients seen in control clinics (n = 7419) had higher mean (SD) baseline CVD risk (16.6% [12.8%]) than patients seen in intervention clinics (n = 11 159) (15.6% [12.3%]; P < .001); baseline reversible CVD risk was similarly higher among patients seen in control clinics. The CDSS was used at 19.8% of 91 988 eligible intervention clinic encounters. No population-level reduction in CVD risk was seen in patients in control or intervention clinics; mean reversible risk improved significantly more among patients in control (−0.1% [95% CI, −0.3% to −0.02%]) than intervention clinics (0.4% [95% CI, 0.3% to 0.5%]; P < .001). However, when the CDSS was used, both risk measures decreased more among patients with high baseline risk in intervention than control clinics; notably, mean reversible risk decreased by an absolute 4.4% (95% CI, −5.2% to −3.7%) among patients in intervention clinics compared with 2.7% (95% CI, −3.4% to −1.9%) among patients in control clinics (P = .001).
Conclusions and Relevance
The CDSS had low use rates and failed to improve CVD risk in the overall population but appeared to have a benefit on CVD risk when it was consistently used for patients with high baseline risk treated in CHCs. Despite some limitations, these results provide preliminary evidence that this technology has the potential to improve clinical care in socioeconomically vulnerable patients with high CVD risk.
ClinicalTrials.gov Identifier: NCT03001713
Evidence-based management of modifiable risk factors for cardiovascular disease (CVD) can substantially reduce CVD-related morbidity and mortality risks. However, a deficit persists between recommended and observed CVD risk management, especially among socioeconomically vulnerable patients.1-6 One reason for this is that primary care clinicians must consider multiple factors affecting CVD risk for a given patient7 and determine which to address to optimally affect that patient’s risk within a brief encounter.8-15
Electronic health record (EHR)–based clinical decision support systems (CDSS) address such barriers by alerting clinicians when patients have uncontrolled CVD risks and suggesting treatment options.16-31 CV Wizard, for example, is a nonproprietary, web-based CDSS developed at HealthPartners Institute, a large, nonprofit, integrated health care system.32-36 The CV Wizard algorithms reflect current CVD care guidelines37-43 and account for a patient’s blood pressure (BP), laboratory results, distance from goals, medications, and comorbidities. At HealthPartners Institute and in similar settings, rates of use and reported user satisfaction were high, and use was associated with significant decreases in CVD risk measures.32-36
Little evidence exists on the effects of CDSS in underresourced settings23-31,33,44-47 such as community health centers (CHCs), which serve more than 38 million socioeconomically vulnerable US residents annually. Implementing a CDSS that has proven effective in other settings could enhance CVD risk management in CHCs. Evidence is needed about the effect of CDSS such as CV Wizard in CHCs, whose patients have high rates of uncontrolled CVD risk and medical and social complexity.2-4,48,49 This cluster randomized clinical trial is one of the first to assess whether a CDSS developed in an integrated care setting improves outcomes in CHCs.
OCHIN Inc is a national nonprofit operating the largest network of community-based care organizations in the country. Its members (96 CHC organizations running 493 clinic sites in 14 states as of September 2018) share an OCHIN Epic EHR. In 2017 to 2018, CV Wizard was set up to work in this EHR50 and pilot-tested in 2 OCHIN Inc member organizations (9 clinics).
The Kaiser Permanente Northwest institutional review board approved all research activities and monitored study progress. A data and safety monitoring board monitored safety outcomes. The institutional review board granted a waiver for obtaining patient consent in this cluster randomized clinical trial, and all OCHIN Inc members sign an agreement that their EHR data may be used for research. The original and current institutional review board–approved study protocols are available in Supplement 1. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.
Our power calculations conservatively estimated needing 30 clinics per group. They then varied effect size associated with group × time interaction and intraclass correlation to assess power to identify a 1.5%, 2.0%, or 3.0% absolute reduction in risk score over time in intervention vs control sites.
In Spring 2018, 70 OCHIN Inc member clinics run by 15 CHC organizations were recruited (10 more than needed, in case of dropout). Eligible clinics served at least 35 adults with hypertension annually and were not the pilot sites. They were cluster randomized 1:1 to the intervention or the control group by organization (to avoid contamination between clinics in the same organization) as follows. Random numbers were generated for each organization, which were then assigned to groups above or below the random numbers’ median. This process was repeated 5 times. The grouping with the most even distribution of organization size (number of encounters in the prior year) and patient characteristics (percentage of tobacco users and percentage with hypertension) was used to assign intervention and control status.
CV Wizard was activated in the intervention and control organizations in September 2018. It ran invisibly in the control clinics to collect data without giving access to the tool. Patients seen at study clinics in the first 6 months after tool activation were followed up for at least 1 year during an 18-month comparison period (September 20, 2018, to March 15, 2020). Control sites received the CDSS after follow-up.
At the point of care, CV Wizard identified patients aged 40 to 75 years with diabetes or atherosclerotic CVD and at least 1 uncontrolled CVD risk factor, or high reversible risk (>10% ten-year risk of a cardiovascular event), calculated based on other CVD risk factors. Reversible risk was calculated as the difference between the patient’s current state and the expected risk should the patient achieve control goals recommended by national guidelines (details are provided hereinafter).43 When a patient met CDSS eligibility criteria, rooming staff saw an EHR alert containing a link to the CV Wizard interfaces. The recommended workflow was to then view (1 click) and print (1 click) the interfaces, which show the patient’s 10-year CVD risk and how they could lower it by following personalized care recommendations. Users could access a clinician version (eFigure 1 in Supplement 2) and a patient version (eFigure 2 in Supplement 2), the latter in either English or Spanish. Tool use was defined as the proportion of primary care encounters among eligible patients where the CDSS tool’s output was viewed and/or printed. Tool use counts included use at the index visit or subsequent study period visits, excluding the patient’s last visit during the study period.
Data on 10-year CVD risk and reversible risk were collected through CV Wizard’s web service for all eligible patients. Additional EHR-extracted data came from the Accelerating Data Value Across a National Community Health Center Network (ADVANCE) Clinical Research Network, a Patient-Centered Clinical Research Network member, including demographic characteristics, BP, medications, laboratory values, diagnostic codes, and clinic characteristics. Outcomes were assessed using encounter data from all postindex visits.
The target population consisted of patients meeting the aforementioned CV Wizard eligibility criteria excluding those with current or recent pregnancy, active cancer, or hospice or palliative care. Analyses included target patients with an index visit at a study clinic in the 6 months after CV Wizard activation and 1 or more postindex encounters during 12 months of follow-up. This ensured adequate follow-up data for the study analyses.
Patients newly diagnosed with diabetes during the follow-up period (347 [1.9%]) were excluded from CVD risk analyses because diabetes may be diagnosed more often with CDSS use. Because the presence of diabetes substantially raises total CVD risk estimation, this exclusion removed the possibility of confounding based on the likelihood of a new diabetes diagnosis.
The primary outcomes were 1-year change in total CVD risk and 1-year change in reversible CVD risk. Total CVD risk was estimated using the American College of Cardiology/American Heart Association pooled risk equations, which include age, race and ethnicity, sex, systolic BP, total cholesterol level, high-density lipoprotein cholesterol level, and diabetes, smoking, and antihypertensive medication status.51 Race and ethnicity data were extracted from the EHR; these data are collected by the study clinics as part of patient care. These data were considered relevant to the study because rates of uncontrolled CVD risk differ across racial/ethnic groups. Reversible CVD risk was calculated as follows (with additional details in the eMethods in Supplement 2 and methods described previously32-36). Standardized equations estimated the potential reduction in CVD risk if a patient’s uncontrolled risk factors reached evidence-based thresholds. Change was calculated by subtracting reversible risk at follow-up from that at index visit; negative values represent favorable changes. This approach focuses clinical attention on patients with high reversible CVD risk, rather than all patients with high total CVD risk.34 It also focuses on risk factors not adequately addressed by American College of Cardiology/American Heart Association equations: change in hemoglobin A1c (HbA1c) level or body mass index and aspirin use. This estimation of patient-level reversible CVD risk, although imprecise, is likely superior to the demonstrably erroneous estimates of CVD benefits and risks broadly used in primary care and to intuitive estimates of benefit or risk.32,33,52 In exploratory analyses, these outcomes were stratified by baseline risk of less than 10%, 10% to less than 20%, and at least 20%, because analyses in other settings indicated CV Wizard’s potentially greater impact among patients with higher baseline risk,34 patients with lower baseline risk were less likely to improve, and clinical implications for these groups differ. Secondary outcomes assessed change in BP, low-density lipoprotein (LDL) level, and HbA1c levels for those above goal at baseline; analyses included patients with at least 1 follow-up value.
Descriptive statistics compared baseline characteristics of intervention vs control groups and CHC organizations. We used a 2-tailed χ2 test, unparied t test, and nonparametric Wilcoxon rank sum test as appropriate. The threshold for statistical significance was P < .05 and was 2-tailed.
The extent of CDSS adoption drives its population-level impact,53,54 and therefore analyses differentiated between the tool’s population impact and its impact when used. Intention-to-treat (ITT) analyses (all targeted patients at intervention vs control organizations) assessed population impact. The effect of treatment on the treated (ETOT; also termed per protocol) analyses assessed impact when used, comparing patients in the intervention CHC group with matched controls (each group matched separately to control patients) based on exposure to the CDSS.
In 3-level random-intercept models (encounter nested within patient and clinic), the intraclass correlation coefficients at the clinic level were 0.05 for CVD risk, 0.04 for both reversible risk and BP, and less than 0.03 for HbA1c and LDL levels. In 4-level random-intercept models (encounter nested within patient, clinic, and organization), intraclass correlation coefficients at the organization level were at least 0.02 for reversible risk, BP, and HbA1c and LDL levels, and 0.05 for CVD risk. Because of the complexity of fitting the 4-level models, and the low organization-level intraclass correlation coefficient, all models were fit using a 3-level random intercept.
Differences in outcome changes were assessed using multilevel mixed models adjusted for these individual-level fixed effects: baseline CVD risk; distribution of eligible patients by age, race and ethnicity, sex, rural-urban commuting area status, and federal poverty level at index visit; and number of ambulatory visits during the follow-up period. These variables, selected a priori, were confirmed by descriptive analyses showing significant differences between baseline intervention and control organization and patient characteristics. Residual distributions indicated that linear-mixed models were appropriate for all outcomes except the reversible risk model that included all study patients, which had a negative binomial distribution.
To assess the CDSS tool’s impact when it was used, analyses were conducted among patients for whom the tool was ever used during follow-up. To assess association with increasing use, we also considered 3 categories of tool use (never, once, and more than once).
Per protocol analyses require adjusting for loss to follow-up and off-protocol therapies or treatments,55 but if group differences vary too greatly, model misspecification can yield biased estimates. Propensity score methods are an alternative.56 In ETOT analyses of change in CVD risk, patients in the intervention organizations were matched to those in control organizations based on age, federal poverty level, outcome of interest at baseline, race and ethnicity, sex, count of ambulatory care visits after the index visit, time from the index visit to the last study period visit, and clinic rural or urban status. The BP, HbA1c, and LDL analyses also matched on presence of diabetes. Propensity scores were estimated using nearest-neighbor matching with replacement. Analyses of change in total and reversible CVD risk included all patients meeting study inclusion criteria; for other outcomes, analyses were restricted to patients with uncontrolled baseline risk. Residual distributions indicated that linear mixed models were appropriate for all outcomes. All analyses were performed using Stata, version 15.1 (StataCorp LLC).
A total of 18 578 eligible patients were seen at the study clinics during the study period. The mean (SD) age was 58.7 (8.8) years, and there were 9490 (51.1%) women and 9088 (48.9%) men. In terms of race and ethnicity, 4934 (26.6%) were Hispanic, 3351 (18.0%) were non-Hispanic Black, 8434 (45.4%) were non-Hispanic White, 1038 (5.6%) were non-Hispanic in another racial group, and 821 (4.4%) did not have documented race and ethnicity data (Table 1). Randomization (Figure) yielded 42 intervention clinics from 8 organizations (11 159 patients) and 28 control clinics from 7 organizations (7419 patients). Distribution of patient and clinic characteristics differed significantly between intervention and control organizations (3891 [34.9%] vs 4543 [61.2%] non-Hispanic White, respectively; urban clinic location, 35 of 42 [83.3%] vs 15 of 28 [53.6%]) (Table 1 and eTable in Supplement 2).
CV Wizard was used at 34.7% of index encounters (clinic range, 0%-59.0%). Among patients for whom it was used, it was used a mean of 2.4 (1.9) times during follow-up. It was used at 19.8% of all 91 988 eligible encounters during the study period, including index and follow-up encounters.
In the ITT analysis, patients in control clinics had significantly higher mean (SD) baseline 10-year CVD risk (16.6% [12.8%]) than patients in intervention clinics (15.6% [12.3%]; P < .001) (Table 2). Change in this risk did not differ significantly between intervention vs control clinics overall. In a subgroup analysis, mean 10-year CVD risk did not improve among patients with baseline CVD risk of less than 20% (risk <10%, 1.8% [95% CI, 1.0-2.5] vs 1.3% [95% CI, 0.7-2.0], P < .001; risk of 10% to <20%, 0.3% [95% CI, −0.1 to 0.7] vs 0.6% [95% CI, 0.2-1.0], P < .001]). Among those with baseline risk of at least 20%, the mean 10-year risk improved significantly more among patients in control clinics (−1.4% [95% CI, −1.6% to −1.2%]) than those in intervention clinics (−1.0% [95% CI, −1.2% to −0.8%]; P = .004).
In ETOT analyses, among patients with baseline CVD risk of less than 10%, risk did not improve in any CDSS use categories (Table 3). However, the mean change in CVD risk was significantly greater among patients in intervention clinics with a baseline risk of at least 20% (−0.9% [95% CI, −1.2% to −0.7%]) compared with matched control patients (−0.3% [95% CI, −0.5% to −0.1%]; P < .001) (Table 4).
Patients in control clinics had significantly higher mean (SD) baseline reversible risk (9.7% [10.0%]) than those in intervention clinics (7.9% [9.0%]; P < .001). Overall, in ITT analyses, reversible risk improved significantly more among patients in control clinics (mean, −0.1% [95% CI, −0.3% to −0.02%]) compared with patients in intervention clinics (mean, 0.4% [95% CI, 0.3%-0.5%]; P < .001).
In ITT analyses, change in reversible risk did not differ across study groups in patients with baseline risk of less than 10% or at least 20%. Reversible risk improved significantly more among patients in control clinics (mean, −1.2% [95% CI, −1.5% to –1.0%]) compared with patients in intervention clinics (mean, −0.7% [95% CI, −0.9% to −0.5%]; P = .001) with a baseline reversible risk of at least 10% to less than 20% (Table 2).
In ETOT analyses, no improvement was seen among patients with a baseline reversible CVD risk of less than 10%, regardless of tool use. Among patients with a baseline risk of at least 10%, no significant differences were seen when the tool was used once or never (Table 3), but when it was used more than once, reversible CVD risk for patients in intervention clinics improved significantly more than that of patients in control clinics (mean, −1.7% [95% CI, −2.0% to −1.3%] vs 0.1% [95% CI, −0.5% to −0.2%]; P < .001). When stratified, no improvement was seen among those with baseline risk of less than 10% (Table 4), but patients in intervention clinics improved significantly more than patients in control clinics among those with baseline risk of at least 10% to less than 20% (mean, −0.7% [95% CI, −0.9% to −0.4%] vs −0.2% [95% CI, −0.5% to 0.04%], respectively; P = .04) and at least 20% (mean, −4.4% [95% CI, −5.2% to −3.7%] vs −2.7% [95% CI, −3.4% to −1.9%], respectively; P = .001).
In ITT analyses, no significant difference was seen in change in systolic BP or diastolic BP (DBP) or in HbA1c or LDL levels among those with high baseline measures of each biomarker (Table 2). In ETOT analyses, LDL levels did not improve significantly more in patients in intervention or control clinics in any tool use categories. Although mean DBP improved significantly more in patients in intervention clinics (−3.2 [95% CI, −3.5 to −2.8] mm Hg) than in control clinics (−2.5 [95% CI, −2.8 to −2.2] mm Hg; P = .01) when the tool was used more than once, this difference was not clinically significant. Mean levels of HbA1c decreased significantly less among patients for whom the tool was used once (−0.5% [95% CI, −0.7% to −0.4%]) compared with controls (−1.0% [95% CI, −1.2% to −0.9%]; P < .001) and among those for whom it was used more than once (−0.7% [95% CI, −0.8% to −0.6%]) compared with controls (−1.0% [95% CI, −1.0% to −0.8%]; P < .001). We note that for most patients with more than 1 uncontrolled risk factor, the CDSS would emphasize more effective ways to reduce CVD risk than tightening control of HbA1c levels.
CV Wizard was effective in integrated care settings primarily serving insured patients; this trial assessed its impact in CHCs. Because CDSS use rates affect population-level outcomes, ITT and ETOT analyses were conducted. In ITT results, total CVD risk did not improve significantly more in patients in intervention clinics overall but improved more among patients in intervention than control clinics among those with a baseline risk of greater than 20%. No consistent, significant impact on reversible CVD risk was seen. In ETOT analyses, however, among patients for whom the tool was used at least once, CVD risk decreased significantly more among those in the highest baseline risk group compared with controls. Although this risk reduction was modest (absolute improvement of 4.4% vs 2.7%), if maintained over time it could represent a population-level reduction in cardiovascular events.57,58 Among those with a baseline reversible risk of at least 10%, intervention patients improved significantly more than controls when the tool was used more than once, suggesting a possible dose-response effect.
The ITT results are unsurprising given the overall tool adoption rate and may explain why these findings contrast with the largely positive earlier findings for this CDSS in better-resourced health care systems.33,34,59 In those studies, use of CV Wizard improved glucose levels and BP control in adults with diabetes, overall CVD risk in adults without diabetes or heart disease,33,34 and BP management in patients aged 6 to 18 years.59 In those studies, however, CDSS results were printed at 70% to 80% of targeted encounters. Many other CDSS studies were unable to demonstrate impact owing to low adoption23,59,60 (eg, a recent implementation in Belgium that had single-digit use rates and no improvement in targeted outcomes).53
Factors that affect point-of-care CDSS use include workflow integration, competing clinical demands, number of clicks to access the CDSS, and clinician confidence in the validity of the advice provided.23,24 This CDSS achieved much higher use rates in centralized care systems with established tool use workflows. The present study included numerous care organizations in which heterogeneity in rooming protocols impeded training and sustained high CDSS use. Future studies should identify strategies for increasing CDSS use in CHCs. Analyses designed to understand CDSS adoption in this setting will be presented in future reports.
In this cluster randomized clinical trial, randomization accounted for organization size, but could not balance on other characteristics, so analyses controlled for baseline factors likely to affect outcomes. Other variables may have affected outcomes. In addition, intraclass correlation coefficients for key study outcomes indicated high heterogeneity across intervention and control clinics, which dilutes power to detect intervention effects. This study was conducted in a heterogenous network of CHCs sharing a single EHR. Extrapolation to different settings requires caution. CV Wizard supports both CDS and shared decision-making, but these analyses did not assess which elements were used. Similarly, even if the tool’s output was viewed or printed, we do not know how it was used to engage individual patients; however, further analyses are underway.
This CDSS intervention appeared to have a benefit for CVD risk when it was consistently used for CHC patients with high baseline risk. Future research is needed on how CDSS tools are used in clinical encounters and to develop strategies to increase CDSS use in CHCs and similar settings. Despite limitations, these results provide preliminary evidence that this technology has the potential to improve clinical care among CHC patients with high CVD risk.
Accepted for Publication: November 11, 2021.
Published: February 4, 2022. doi:10.1001/jamanetworkopen.2021.46519
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2022 Gold R et al. JAMA Network Open.
Corresponding Author: Rachel Gold, PhD, MPH, Center for Health Research, Kaiser Permanente Northwest, 3800 N Interstate Ave, Portland, OR 97227 (email@example.com).
Author Contributions: Drs Gold and Larson had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Gold, Larson, Sperl-Hillen, Heintzman, Middendorf, Appana, Thirumalai, Bava, Yosuf, O’Connor.
Acquisition, analysis, or interpretation of data: Gold, Larson, Sperl-Hillen, Boston, Sheppler, Heintzman, McMullen, Appana, Thirumalai, Romer, Davis, Hauschildt, Scott, Moore, O’Connor.
Drafting of the manuscript: Gold, Larson, Sperl-Hillen, Boston, Davis, O’Connor.
Critical revision of the manuscript for important intellectual content: Gold, Larson, Sperl-Hillen, Boston, Sheppler, Heintzman, McMullen, Middendorf, Appana, Thirumalai, Romer, Bava, Yosuf, Hauschildt, Scott, Moore, O’Connor.
Statistical analysis: Larson, Boston, Scott, O’Connor.
Obtained funding: Gold, Heintzman, Appana.
Administrative, technical, or material support: Sperl-Hillen, Boston, Sheppler, McMullen, Middendorf, Appana, Thirumalai, Bava, Yosuf, Hauschildt, Moore, O’Connor.
Supervision: Gold, Boston, Heintzman, McMullen.
Conflict of Interest Disclosures: Dr Gold reported receiving grants from the National Institutes of Health (NIH) during the conduct of the study. Dr Boston reported receiving grants from the National Heart, Lung, and Blood Institute (NHLBI) during the conduct of the study. Dr Sheppler reported receiving grants from the NHLBI during the conduct of the study. Dr Heintzman reported grants from the National Institute on Aging during the conduct of the study and serving as lead clinician scientist at OCHIN Inc, a nonprofit network of community health centers. Dr McMullen reported receiving grants from the NIH during the conduct of the study. Ms Middendorf reported receiving grants from the NHLBI during the conduct of the study. Mr Thirumalai reported receiving grants from HealthPartners Institute during the conduct of the study. Ms Romer reported receiving grants from the NHLBI during the conduct of the study. Ms Bava reported receiving grants from the NHLBI during the conduct of the study. Ms Yosuf reported receiving grants from the NHLBI during the conduct of the study. Ms Hauschildt reported receiving grants from the NHLBI during the conduct of the study. Ms Moore reported receiving grants from the NHLBI during the conduct of the study. No other disclosures were reported.
Funding/Support: This study was supported by the award R01HL133793 from the NHLBI and in part by award P30DK092924 from the NIH (Drs Sperl-Hillen and O’Connor).
Role of the Funder/Sponsor: The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Meeting Presentations: This paper was presented virtually at the American Medical Informatics Association 2021 Annual Symposium (November 1, 2021); the 49th Annual North American Primary Care Research Group Meeting (November 22, 2021); and the 14th Annual Conference on the Science of Dissemination and Implementation in Health (December 14-16, 2021).
Data Sharing Statement: See Supplement 3.
Additional Contributions: Lauren Crain, PhD, HealthPartners Institute, consulted on the analysis of this project. Laurel Nightingale, MPH, MPP, University of Minnesota, and Heidi Ekstrom, MA, HealthPartners Institute, and M. J. Dunne, MA, Oregon Health and Science University, David Killaby, MPA:HA, William Pinnock, MS, and Jessica Black, CMS, OCHIN Inc, provided project management support. Joan Nelson, MPH, PA, Kaiser Permanente Northwest, was a practice coach on the project. Debra McCauley, BS, AAS, and Christina Wood, BS, HealthPartners Institute, were programmers on the project.
Additional Information: This work was conducted with the Accelerating Data Value Across a National Community Health Center Network (ADVANCE) Clinical Research Network (CRN). OCHIN Inc, leads the ADVANCE network in partnership with Health Choice Network, Fenway Health, and Oregon Health & Science University. ADVANCE is funded through contract RI-CRN-2020-001 from the Patient-Centered Outcomes Research Institute.