There were 7867 COVID-19 hospitalizations, 675 inpatient deaths, and 8712 within-household COVID-19 transmissions from May to December 2020. Error bars indicate 95% CIs. CDC indicates US Centers for Disease Control and Prevention; CRS, COVID-19 risk score; PROVID, probability of COVID-19 infection.
CRS indicates COVID-19 risk score; PROVID, probability of COVID-19 infection.
High NDI is defined as an NDI at or greater than the 75th percentile of all adults in the study. CRS indicates COVID-19 risk score; PROVID, probability of COVID-19 infection.
eTable 1. Implementation of the Advisory Committee on Immunization Practices (ACIP) guidelines
eTable 2. Probability of COVID-19 Infection (PROVID) Model Coefficients
eTable 3. COVID-19 Risk Score (CRS) Model Coefficients
eTable 4. Probability of COVID-19 Infection (PROVID) Deciles
eTable 5. COVID-19 Risk Score (CRS) Deciles
eAppendix. Calculation of the Abbreviated Laboratory-Based Acute Physiology Score (abLAPS)
eTable 6. Characteristics of Patients by Probability of COVID-19 Infection (PROVID) and COVID-19 Risk Score (CRS) Levels
eFigure 1. Estimated Avoidable Hospitalizations among Patients Vaccinated in First 8 Months by Vaccination Prioritization Order in Populations with Varying Age and Race Distributions
eFigure 2. Estimated Avoidable Household COVID-19 Deaths among Patients Vaccinated in First 8 Months by Vaccination Prioritization Order in Populations with Varying Age and Race Distributions
eFigure 3. Estimated Avoidable Household COVID-19 Transmissions among Patients Vaccinated in First 8 Months by Vaccination Prioritization Order in Populations with Varying Age and Race Distributions
eFigure 4. Cumulative Percent of Patients Vaccinated by Race, Vaccination Month and Strategy in Populations with Varying Age and Race Distributions
eTable 7. Consolidated health economic evaluation reporting standards (CHEERS) statement checklist
eTable 8. Age and Race/Ethnicity Distribution across U.S. States
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Kipnis P, Soltesz L, Escobar GJ, Myers L, Liu VX. Evaluation of Vaccination Strategies to Compare Efficient and Equitable Vaccine Allocation by Race and Ethnicity Across Time. JAMA Health Forum. 2021;2(8):e212095. doi:10.1001/jamahealthforum.2021.2095
What is the association of different vaccine allocation strategies with COVID-19–related morbidity and mortality and their distribution by racial and ethnic groups across time?
In this decision analytical model, the use of risk-based, age-based, and US Centers for Disease Control and Prevention (CDC)–phased vaccine allocation strategies was simulated. Risk-based strategies were associated with the largest estimated reductions in nonelective hospitalizations, death, and household transmissions compared with the CDC- and age-based strategies, with a similar proportion of Hispanic and Black patients being vaccinated early in the process compared with the CDC strategy.
The study findings suggest that risk-based vaccine prioritization strategies could have the greatest effectiveness on reducing COVID-19–related deaths and household transmissions while ensuring equitable vaccine distribution.
Identifying the most efficient COVID-19 vaccine allocation strategy may substantially reduce hospitalizations and save lives while ensuring an equitable vaccine distribution.
To simulate the association of different vaccine allocation strategies with COVID-19–associated morbidity and mortality and their distribution across racial and ethnic groups.
Design, Setting, and Participants
We developed and internally validated the risk of COVID-19 infection and risk of hospitalization models on randomly split training and validation data sets. These were used in a computer simulation study of vaccine prioritization among adult health plan members who were drawn from an integrated health care delivery system. The study was conducted from January 3, 2021, to June 1, 2021, in Oakland, California, and the data were analyzed during the same period.
Main Outcomes and Measures
We simulated the association of different vaccine allocation strategies, including (1) random, (2) a US Centers for Disease Control and Prevention (CDC) proxy, (3) age based, and (4) combinations of models for the risk of adverse outcomes (CRS) and COVID-19 infection (PROVID), with COVID-19-related hospitalizations between May 1, 2020, and December 31, 2020, that were randomly permuted by month across 250 simulations and assessed vaccine allocation by race and ethnicity and the neighborhood deprivation index across time.
The study included 3 202 679 adult patients (mean [SD] age, 48.2 [18.0] years; 1 677 637 women [52.4%]; 1 525 042 men [47.6%]; 611 154 Asian [19.1%], 206 363 Black [6.4%], 642 344 Hispanic [20.1%], and 1 390 638 White individuals [43.4%]), of whom 36 137 (1.1%) were positive for SARS-CoV-2. A risk-based strategy (CRS/PROVID) showed the largest avoidable hospitalization estimates (4954; 95% CI, 3452-5878) followed by age-based (4362; 95% CI, 2866-5175) and CDC proxy (4085; 95% CI, 2805-5109) strategies. Random vaccination showed substantially lower reductions in adverse outcomes. Risk-based strategies also showed the largest number of avoidable COVID-19 deaths (joint CRS/PROVID) and household transmissions. Risk-based (PROVID) and CDC proxy strategies were estimated to vaccinate the highest percentage of Hispanic and Black patients in 8 months (joint CRS/PROVID: 642 570 [100%] Hispanic, 185 530 [90%] Black; PROVID: 642 570 [100%] Hispanic, 198 480 [96%] Black; CDC proxy: 605 770 [95%] Hispanic and 151 772 [74%] Black) compared with an age-based approach (438 423 [68%] Hispanic, 154 714 [75%] Black). Overall, the PROVID and joint CRS/PROVID risk-based strategies were estimated to be followed by the most patients from areas with high neighborhood deprivation index being vaccinated early.
Conclusions and Relevance
In this simulation modeling study of adults from a large integrated health care delivery system, risk-based strategies were associated with the largest estimated reductions in COVID-19 hospitalizations, deaths, and household transmissions compared with the CDC proxy and age-based strategies, with a higher proportion of Hispanic and Black patients were estimated to be vaccinated early in the process compared with the CDC strategy.
SARS-CoV-2 infection and COVID-19 disease have resulted in more than 22 million US COVID-19 cases and 400 000 deaths,1 severely straining hospitals, disrupting communities, amplifying social disparities, and contributing to social unrest worldwide.2-6 Vaccination remains the key to controlling the COVID-19 disease,7 with the recent development of effective COVID-19 vaccines representing a critical step forward.8,9 Given the limitations in vaccine supply in early 2021, the Advisory Committee on Immunization Practices (ACIP) recommended phased COVID-19 vaccine allocation in the US10,11 (eTables 1-5 in the Supplement).
While the ACIP allocation hierarchy sought to balance exposure risk factors, morbidity, and mortality along with social and equity priorities, implementing this strategy proved challenging, with many states pivoting to alternate approaches like age-based stratification. A key limitation has been the availability of data needed to implement the ACIP guidelines.12,13 While routine electronic health record (EHR) data could be used to inform vaccination, to our knowledge few studies have evaluated their potential role in improving allocation, particularly under inadequate vaccine supply constraints. In this study, we used EHR data from an integrated health care delivery system to simulate the potential association of different vaccine allocation strategies with estimated COVID-19–related morbidity, mortality, household transmission, and racial and ethnic group vaccine distribution.
This computer simulation study adheres to the health economic modeling report guidelines (Consolidated Health Economic Evaluation Reporting Standards; eTable 7 in the Supplement).14 Kaiser Permanente Northern California (KPNC) is an integrated health care delivery system that serves 4.5 million members. This project was approved by the KPNC institutional review board, which waived the requirement for individual informed consent because of the large number of study participants.
The overall cohort included adult (≥18 years) KPNC patients with continuous membership or utilization (unless they died) between February 1, 2020 and December 3, 2020. We captured members’ self-reported race and ethnicity (documented at the time of insurance enrollment by medical assistants in the outpatient setting and/or admission clerks for inpatients); comorbidities15-17; and neighborhood deprivation index (NDI), a composite index ranging from −5 to +5, with more positive values indicating worsening socioeconomic status (eg, poverty and unemployment).18 We also calculated 2 previously validated general illness risk-adjustment indices: the Comorbidity Point Score, version 2 (COPS2), to assess 1-year comorbid disease burden and the abbreviated Laboratory-Based Acute Physiology Score (abLAPS), which is an outpatient laboratory-based severity of illness score (eAppendix in the Supplement).19,20
Between the first 2 COVID-19 waves in Northern California, we developed 2 scores to classify patients within COVID-19 risk groups using data from February 1, 2020, to September 30, 2020: (1) the PROVID (probability of COVID-19 infection) to measure the likelihood of a patient having a positive SARS-CoV-2 polymerase chain reaction test result and (2) the COVID-19 Risk Score (CRS) to measure the risk of adverse outcomes (ie, hospitalization, death) among patients with COVID-19.21 The PROVID score used a logistic regression model with a positive SARS-CoV-2 test result as the outcome and predictors of age, sex, race and ethnicity,22 COPS2, abLAPS, and NDI. The model was fitted on a random training set (70% of the data) and validated on the remaining 30%, with a validation area under the receiver operating characteristic curve of 0.71 (95% CI, 0.71-0.72). The CRS score quantified the risk of nonelective hospitalization within 30 days of a SARS-CoV-2 positive test result using a logistic regression model for patients with COVID-19 that included sex and truncated power function splines at 3 selected knots23 for age and COPS2.21 We fitted the CRS model on 75% of the data and evaluated its performance on the remaining 25%. The model had a validation area under the curve of 0.82 (95% CI, 0.81-0.83) for 30-day nonelective hospitalization.
We calculated CRS and PROVID scores for all adult KPNC members in the cohort and divided patients into deciles establishing low- (<40th percentile), medium- (40th to 60th percentile), and high-risk (>60th percentile) groups for each score. We characterized the demographic, clinical, and hospital utilization characteristics of the 9 joint CRS-PROVID risk groups based on the interactions between the 3 CRS and 3 PROVID risk strata (eTables 1-5 in the Supplement).
Because the intent was to explore the association of vaccination strategies with adverse outcomes from a public health perspective, our primary outcomes were COVID-19 hospitalizations, deaths, and household transmissions. A hospitalization was classified as COVID-19 related if the physician designation during hospital treatment was either confirmed active COVID or resolved COVID. COVID-19 deaths were inpatient deaths that occurred during COVID-19–related hospitalizations. To estimate COVID-19 household transmission, we grouped KPNC members into households according to their address in EHR data. In each household, we identified the first member with a positive SARS-CoV-2 test result and quantified COVID-19 transmission within a household based on the number of household members who tested positive within 4 weeks of that first household positive test result. As secondary outcomes, we evaluated the rate of vaccination by NDI and among 4 broad racial and ethnic groups (White, Black and African American, Hispanic, and Asian individuals).
The detailed data sources at KPNC allowed us to conduct a computer simulation study which is easier to understand and requires fewer modeling assumptions compared with a mathematical simulation model. We simulated 6 vaccination strategies assuming a target goal of vaccinating 75% of the 3 million adult KPNC members (2 400 000) over an 8-month period (240 days) with a uniform vaccination rate of 10 000 vaccines administered daily. When priority groups were simulated, we assumed that patients within the same priority grouping were vaccinated in random order and that all group members would be vaccinated before moving to the next group. Finally, we assumed that all phase 1a–eligible patients (ie, health care personnel and long-term care facility residents) completed vaccination before each prioritization strategy began. Four strategies were used. The first was a random vaccine allocation, which was a baseline process with randomly distributed vaccinations. The second was a CDC proxy, which was an ACIP-based process starting with essential workers, defined as patients ages 18 to 65 years with a PROVID score greater than the 80th percentile (1.6%) based on the most recent 5-year estimates from the American Community Survey (2014-2018).24,25 We defined patients with high-risk medical conditions (phase 1c) as those with a CRS score of 20% or greater, as used at KPNC to define higher-risk patients for COVID-19–related interventions. The remaining essential workers were defined as those with a PROVID in the 50th to 80th percentile (0.71%-1.6%). If patients qualified for several vaccination phases, we assigned them to their highest qualified tier (eTable 1 in the Supplement). The third strategy was age based and was developed around age groups in reverse order (age ≥75, 65-74, 55-64, 45-54, 35-44, and 18-34 years). The fourth strategy was risk based and involved a process in which vaccination prioritization was determined by the rate of hospitalizations across the 9 joint CRS/PROVID risk groups, starting with the high PROVID/high CRS group and ending with the low CRS/low PROVID group. We also assessed vaccine prioritization strategies by either risk score alone (PROVID or CRS alone).
We calculated the number of outcomes during the 8-month period of May 1, 2020, through December 31, 2020, among patients that corresponded to each strategy until 2.4 million patients were vaccinated and compared the age-based, CDC proxy, and risk-based strategies with random vaccination order. To estimate the association between allocation strategies and outcomes, we conducted 250 simulations in which we permuted the month of COVID-19–related hospitalization month to simulate random monthly outcome distributions over the vaccination period. We assumed that any vaccination would be 100% effective at preventing hospitalization; thus, we removed hospitalizations that occurred at any time after 1 month from the vaccination date. In each simulation, we calculated the difference between the actual and simulated number of hospitalizations to estimate the number of avoided hospitalizations. We reported the average difference in hospitalizations and deaths across the 250 simulations and the 2.5th and 97.5th percentile as the 95% confidence interval. We followed a similar strategy to simulate SARS-CoV-2 transmissions within households. We permuted the first COVID-19 household test month in 250 simulations and assumed that vaccination would prevent transmission for transmissions that occurred within 2 weeks of receiving the vaccine. Finally, we compared the percentage of vaccinated patients by race and ethnicity and NDI (>75th percentile) to assess equitable distribution of the vaccine within each allocation strategy. Statistical analyses were conducted using SAS, version 7.15 (SAS Institute).
We also varied the underlying age and race distribution of the population based on US census data24 to identify the individual states with the youngest (470 905 [14.7%] younger than age 65 years); oldest (804 063 [25.1%] older than age 65 years); highest percentage Asian (1 220 509 [38.1%]), African American (1 412 716 [44.1%]), Hispanic (1 579 294 [49.3%]) or White (2 972 790 [92.8%] populations); and lowest percentage African American (19 221 [0.6%]) and Hispanic (48 052 [1.5%]) populations (eTable 8 in the Supplement). We then created bootstrap samples of the KPNC data with the corresponding age and race distributions of these states and simulated the association between the vaccine prioritization strategies and adverse outcomes among populations of KPNC patients with these characteristics.
We identified 3 202 679 adults, including 36 137 (1.1%) with positive SARS-CoV-2 test results (Table 118,19,26). Patients who tested positive for SARS-CoV-2 during the study period tended to be younger (mean [SD] age, 43.4 [16.3] vs 48.2 [18.0] years; P < .001), had fewer comorbidities (COPS2 of 14.2 vs 14.7; P < .001), and were less likely to be White (23% vs 44%; P < .001) compared with those who tested negative or without a test. Among those testing positive, 3403 (9.4%) experienced nonelective hospitalization within 30 days of their test. Hospitalized patients with COVID-19 were older (mean [SD] age, 59.5 [17.1] vs 41.8 [15.3] years; P < .001), had more comorbidities (mean COPS2, 30.4 vs 12.6; P < .001) and had more acute illness (mean abLAPS, 1.8 vs 0.3; P < .001) than those not hospitalized.
Patients with a high risk of SARS-CoV-2 infection (higher PROVID scores) were more likely to be Hispanic (60%-70% vs <5%; P < .001 for all comparisons) and Black (12%-16% vs 0%-9%), to test positive for COVID-19 (20-24 vs 4-9 per 1000), and to have higher mean NDI scores (0.4 to 0.5 vs −1.1 to −0.2). Patients with high risk of adverse outcomes (higher CRS scores) were older (age 65-70 years vs 30-50 years; P < .001) and more likely to be male (60% vs 40%-50%; P < .001) than those with low and medium risk (eTable 6 and eFigures 1-4 in the Supplement).
The oldest patients (75 years or older; n = 262 005) represented only 8.2% of the population but accounted for 25.4% of COVID-19 hospitalizations and 50.6% of inpatient deaths. In contrast, patients aged 18 to 54 years (n = 1 993 356) represented 62.2% of members, 34.4% of COVID-19 hospitalizations, and 11.3% of inpatient deaths (Table 2). Patients in the joint high CRS/high PROVID risk group had the highest hospitalization rate (12.1 per 1000 members), followed by those in the high CRS/medium PROVID risk group (6.8 per 1000 members). These 2 groups represented 11% of the population (n = 347 706) and accounted for 38.6% of all COVID-19 hospitalizations and 53.2% of inpatient deaths. Patients in the high PROVID/medium CRS and low PROVID/high CRS represented 28.2% of the population (n = 904 931) and accounted for another 38.6% of the COVID-19 hospitalizations and 42.1% of inpatient deaths.
Over the 8-month study period, there were 7867 COVID-19 hospitalizations and 675 inpatient deaths. The simulated random allocation strategy was associated with estimates of 2645 (95% CI, 1766-3574; Figure 1) avoidable hospitalizations and 232 (95% CI, 143-317) avoidable inpatient deaths. Comparatively, the joint CRS/PROVID and CRS risk-based vaccination strategies were associated with the largest estimates of avoidable COVID-19 hospitalizations (CRS/PROVID: 4954; 95% CI, 3452-5878; CRS: 4470; 95% CI, 3215-5631) and deaths (CRS/PROVID: 505; 95% CI, 343-596; CRS: 535; 95% CI, 338-619). The age-based and CDC proxy approaches also showed higher estimated avoidable hospitalizations and deaths than random allocation (age-based: 4362; 95% CI, 2866-5175; CDC proxy: 4085; 95% CI, 2805-5109).
The random allocation strategy was associated with an estimated 2301 (95% CI, 1696-2967) household transmissions prevented. A PROVID-based risk approach showed the highest estimated avoidable household transmissions (3176; 95% CI, 2496-3632), followed by the CDC proxy (2793; 95% CI, 2173-3352), the joint CRS/PROVID (2538; 95% CI, 1663-3460), the CRS (1796; 95% CI, 882-2832), and age-based (1722; 95% CI, 892-2549) strategies. In sensitivity analyses, results were similar when varying the age and race and ethnicity distributions, with the joint CRS/PROVID risk-based strategy showing the largest estimated reduction in hospitalizations and deaths (eTable 6 and eFigures 1-4 in the Supplement).
The risk-based joint CRS/PROVID, PROVID risk allocation, and CDC proxy strategies vaccinated the highest percentage of Hispanic and Black patients in 8 months (CRS/PROVID: 642 570 [100%] Hispanic, 185 530 [90%] Black; PROVID: 642 570 [100%] Hispanic, 198 480 [96%] Black; CDC proxy: 605 770 [95%] Hispanic and 151 772 [74%] Black) compared with an age-based approach (438 423 [68%] Hispanic, 154 714 [75%] Black; Figure 2). Overall, the PROVID and CRS/PROVID risk-based strategies resulted in the most patients from areas with high neighborhood deprivation being vaccinated early (Figure 3).
Using data from a large, diverse, and integrated health care delivery system, we simulated the implementation of different COVID-19 vaccine allocation strategies and compared their association with estimates of vaccine allocation effectiveness and equity. We found that risk-based strategies that identified patients at high risk for adverse outcomes were associated with the highest estimates of avoidable hospital admissions and deaths, followed by the CDC proxy and age-based strategies. We also found that a risk-based approach that identified patients at high risk for COVID-19 infection, followed by the CDC proxy strategy, achieved the goal of earlier vaccinations being administered to Hispanic and Black patients. Simulating the association of these differing prioritization strategies with adverse outcomes could help identify and mitigate some of the potential racial and ethnic and socioeconomic inequities that have been potentiated during the COVID-19 pandemic. Finally, we found that a risk-based allocation strategy that prioritized the risk of infection was associated with the highest estimates of avoidable household transmission, followed by the CDC proxy and joint risk-based approach. Overall, our analyses suggest that risk-based strategies using available EHR data could inform optimal approaches for improving vaccination effectiveness, efficiency, and equity.
Recent studies on COVID-19 vaccine distribution have focused on policies and practices to support the equity, transparency, accountability, availability, and access to COVID-19 vaccines.27 While Schmidt et al28 compared different US approaches for implementing the CDC vaccine distribution recommendations to reduce inequity, other studies have highlighted the challenges of implementing the CDC strategy. These include its complexity, political considerations, and equity.12,29,30 Mathematical modeling studies also showed similar findings that prioritizing older adults could lead to reduced adverse outcomes,31,32 while alternate strategies that target younger adults, who are more likely to be essential workers or have higher infection potential, could reduce overall transmission.33,34 Another recent study compared the efficacy of risk-based, age-based, and CDC-phased vaccine allocation approaches to reduce COVID-19-related deaths35; however, the risk-based approach was based on only a single model, the CDC approach was defined by age alone, and the study did not evaluate other salient outcomes.
Concerns about the equity of vaccination approaches have been confirmed, as emerging data revealed racial and ethnic inequities in vaccine allocation.12 For example, as of March 1, 2021, and including data from 41 states, there was consistent evidence of Hispanic and Black patients receiving smaller shares of vaccines compared with their rates of infection and hospitalization.36 In some locales, Asian patients, who also have increased rates of infection compared with White patients, also had lower relative rates of vaccination.22 While concerns have been raised about the legal implications of vaccine allocation by race and ethnicity, legally permissible strategies could use NDI, which integrates income levels, education, employment, and housing quality, to address vaccination.37 However, the urgent need to understand and simulate the vaccine allocation process and its resulting effect on marginalized patient groups remains.6,38,39
While the US is entering a period of decreasing vaccination rates, vaccine supply remains highly constrained across the world today. Thus, the estimates produced in this simulation can help inform vaccine allocation approaches in locales with limited vaccine supply or potentially in future periods of vaccine distribution for COVID-19 or other similar pandemics. In addition, risk-based approaches could be used to inform public health and community engagement campaigns that are designed to increase vaccination uptake among groups that remain undervaccinated.
The major strength of this study was the ability to find an association between detailed population-level data with SARS-CoV-2 test results and COVID-19 hospital admissions, deaths, and household transmission through facilitating the use of 2 risk models and simulating key outcomes. The study also quantifies the effectiveness and equity of these approaches in a large and diverse patient population. This was possible because of the high degree of integration at KPHC and because it has made a concerted effort to capture race and ethnicity data in the outpatient setting.
Our study also has important limitations. First, the CDC vaccine allocation process we simulated is only a proxy because we lacked access to high-quality information about essential workers’ status; however, the risk scores we developed could be valuable as a proxy to other health systems that similarly lack employment data Second, we also used prior COVID-19 hospitalization data, those before the availability of any vaccine outside of clinical trials, to understand the association of vaccination allocation and future hospitalizations. This assumed that the association of future surges on hospitalization and death would be similar to what we have already observed. While future COVID-19 sequelae might differ from that observed in 2020, we compared all strategies using the same assumptions and hospitalization data. Third, we did not simulate a scenario in which vaccinated patients were less likely to transmit the virus.40 Fourth, we did not consider deaths associated with the vaccine, which have proven to be extremely rare.41 Fifth, we did not consider different vaccine uptake levels among different racial and ethnic groups.42 Sixth, our estimates of household transmission should be viewed as provisory, as they likely reflect differential testing patterns and characteristics across households. Finally, the simulation does not consider any differential efficacy or reduction in the efficacy of differing vaccines over time.
In this simulated modeling study of adults from a large integrated health care delivery system, risk-based strategies were associated with the largest estimated reductions in COVID-19 hospitalizations, deaths, and household transmissions compared with the CDC proxy and age-based strategies, with similar proportions of Hispanic and Black patients being vaccinated early in the process compared with the CDC strategy.
Accepted for Publication: June 19, 2021.
Published: August 20, 2021. doi:10.1001/jamahealthforum.2021.2095
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Kipnis P et al. JAMA Health Forum.
Corresponding Author: Patricia Kipnis, PhD, Division of Research, Kaiser Permanente, 2000 Broadway Ave, Oakland, CA 94612 (email@example.com).
Author Contributions: Dr Kipnis had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Kipnis, Escobar, Myers, Liu.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: All authors.
Critical revision of the manuscript for important intellectual content: Kipnis, Escobar, Liu.
Statistical analysis: Kipnis, Soltesz, Myers.
Obtained funding: Escobar, Liu.
Administrative, technical, or material support: Escobar, Liu.
Conflict of Interest Disclosures: Dr Escobar reported grants from AstraZeneca outside the submitted work. Dr Liu reported grants from the National Institute of General Medical Sciences during the conduct of the study. No other disclosures were reported.
Funding/Support: Dr Liu was funded by grant R35GM128672 from the National Institutes of Health, The Permanente Medical Group, and Kaiser Foundation Hospitals.
Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: We thank Tracy Lieu, MD, MPH, Yi-Fen Irene Chen, MD, and Stephen Parodi, MD, Kaiser Permanente Northern California. These individuals were not compensated for their contributions.