JAMA Health Forum – Health Policy, Health Care Reform, Health Affairs | JAMA Health Forum | JAMA Network
[Skip to Navigation]
Figure 1.  Estimated Avoidable Hospitalizations, Deaths, and Household COVID-19 Transmissions Among Patients Vaccinated in First 8 Months by Vaccination Prioritization Order
Estimated Avoidable Hospitalizations, Deaths, and Household COVID-19 Transmissions Among Patients Vaccinated in First 8 Months by Vaccination Prioritization Order

There were 7867 COVID-19 hospitalizations, 675 inpatient deaths, and 8712 within-household COVID-19 transmissions from May to December 2020. Error bars indicate 95% CIs. CDC indicates US Centers for Disease Control and Prevention; CRS, COVID-19 risk score; PROVID, probability of COVID-19 infection.

Figure 2.  Cumulative Percentage of Patients Vaccinated by Race, Vaccination Month, and Strategy
Cumulative Percentage of Patients Vaccinated by Race, Vaccination Month, and Strategy

CRS indicates COVID-19 risk score; PROVID, probability of COVID-19 infection.

Figure 3.  Percentage of Total Vaccinated With High Neighborhood Depravation Index (NDI) by Vaccination Month and Strategy
Percentage of Total Vaccinated With High Neighborhood Depravation Index (NDI) by Vaccination Month and Strategy

High NDI is defined as an NDI at or greater than the 75th percentile of all adults in the study. CRS indicates COVID-19 risk score; PROVID, probability of COVID-19 infection.

Table 1.  Characteristics of Kaiser Foundation Health Plan Adult Members as of February 1, 2020
Characteristics of Kaiser Foundation Health Plan Adult Members as of February 1, 2020
Table 2.  COVID-19 Vaccine Allocation Strategies
COVID-19 Vaccine Allocation Strategies
1.
US Centers for Disease Control and Prevention. United States COVID-19 cases and deaths by state. Accessed February 20, 2021. https://covid.cdc.gov/covid-data-tracker/#cases_casesper100klast7days
2.
Ogedegbe  G, Ravenell  J, Adhikari  S,  et al.  Assessment of racial/ethnic disparities in hospitalization and mortality in patients with COVID-19 in New York City.   JAMA Netw Open. 2020;3(12):e2026881. doi:10.1001/jamanetworkopen.2020.26881 PubMedGoogle Scholar
3.
Bravata  DM, Perkins  AJ, Myers  LJ,  et al.  Association of intensive care unit patient load and demand with mortality rates in US Department of Veterans Affairs hospitals during the COVID-19 pandemic.   JAMA Netw Open. 2021;4(1):e2034266. doi:10.1001/jamanetworkopen.2020.34266 PubMedGoogle Scholar
4.
Whaley  CM, Pera  MF, Cantor  J,  et al.  Changes in health services use among commercially insured US populations during the COVID-19 pandemic.   JAMA Netw Open. 2020;3(11):e2024984. doi:10.1001/jamanetworkopen.2020.24984 PubMedGoogle Scholar
5.
Holland  KM, Jones  C, Vivolo-Kantor  AM,  et al.  Trends in US emergency department visits for mental health, overdose, and violence outcomes before and during the COVID-19 pandemic.   JAMA Psychiatry. 2021;78(4):372-379. doi:10.1001/jamapsychiatry.2020.4402 PubMedGoogle ScholarCrossref
6.
Emanuel  EJ, Persad  G, Upshur  R,  et al.  Fair allocation of scarce medical resources in the time of Covid-19.   N Engl J Med. 2020;382(21):2049-2055. doi:10.1056/NEJMsb2005114 PubMedGoogle ScholarCrossref
7.
Greenwood  B.  The contribution of vaccination to global health: past, present and future.   Philos Trans R Soc Lond B Biol Sci. 2014;369(1645):20130433. doi:10.1098/rstb.2013.0433 PubMedGoogle Scholar
8.
Oliver  SE, Gargano  JW, Marin  M,  et al.  The Advisory Committee on Immunization Practices’ interim recommendation for use of Pfizer-BioNTech COVID-19 vaccine—United States, December 2020.   MMWR Morb Mortal Wkly Rep. 2020;69(50):1922-1924. doi:10.15585/mmwr.mm6950e2 PubMedGoogle ScholarCrossref
9.
Oliver  SE, Gargano  JW, Marin  M,  et al.  The Advisory Committee on Immunization Practices’ interim recommendation for use of Moderna COVID-19 vaccine—United States, December 2020.   MMWR Morb Mortal Wkly Rep. 2021;69(5152):1653-1656. doi:10.15585/mmwr.mm695152e1 PubMedGoogle ScholarCrossref
10.
Smith  JC.  The structure, role, and procedures of the U.S. Advisory Committee on Immunization Practices (ACIP).   Vaccine. 2010;28(suppl 1):A68-A75. doi:10.1016/j.vaccine.2010.02.037 PubMedGoogle ScholarCrossref
11.
Dooling  K, Marin  M, Wallace  M,  et al.  The Advisory Committee on Immunization Practices’ updated interim recommendation for allocation of COVID-19 vaccine—United States, December 2020.   MMWR Morb Mortal Wkly Rep. 2021;69(5152):1657-1660. doi:10.15585/mmwr.mm695152e2 PubMedGoogle ScholarCrossref
12.
Jean-Jacques  M, Bauchner  H.  Vaccine distribution—equity left behind?   JAMA. 2021;325(9):829-830. doi:10.1001/jama.2021.1205 PubMedGoogle ScholarCrossref
13.
US Centers for Disease Control and Prevention. People with certain medical conditions. Accessed May 31, 2021, 2021. https://www.cdc.gov/coronavirus/2019-ncov/need-extra-precautions/people-with-medical-conditions.html
14.
Husereau  D, Drummond  M, Petrou  S,  et al.  Consolidated health economic evaluation reporting standards (CHEERS) statement.   Eur J Health Econ. 2013;14(3):367-372. doi:10.1007/s10198-013-0471-6 PubMedGoogle ScholarCrossref
15.
van Walraven  C, Austin  PC, Jennings  A, Quan  H, Forster  AJ.  A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data.   Med Care. 2009;47(6):626-633. doi:10.1097/MLR.0b013e31819432e5 PubMedGoogle ScholarCrossref
16.
Elixhauser  A, Steiner  C, Harris  DR, Coffey  RM.  Comorbidity measures for use with administrative data.   Med Care. 1998;36(1):8-27. doi:10.1097/00005650-199801000-00004 PubMedGoogle ScholarCrossref
17.
Deyo  RA, Cherkin  DC, Ciol  MA.  Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.   J Clin Epidemiol. 1992;45(6):613-619. doi:10.1016/0895-4356(92)90133-8 PubMedGoogle ScholarCrossref
18.
Messer  LC, Laraia  BA, Kaufman  JS,  et al.  The development of a standardized neighborhood deprivation index.   J Urban Health. 2006;83(6):1041-1062. doi:10.1007/s11524-006-9094-x PubMedGoogle ScholarCrossref
19.
Escobar  GJ, Greene  JD, Scheirer  P, Gardner  MN, Draper  D, Kipnis  P.  Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases.   Med Care. 2008;46(3):232-239. doi:10.1097/MLR.0b013e3181589bb6 PubMedGoogle Scholar
20.
van Walraven  C, Escobar  GJ, Greene  JD, Forster  AJ.  The Kaiser Permanente inpatient risk adjustment methodology was valid in an external patient population.   J Clin Epidemiol. 2010;63(7):798-803. doi:10.1016/j.jclinepi.2009.08.020 PubMedGoogle ScholarCrossref
21.
Rouillard  S, Liu  VX, Corley  DA.  COVID-19 and long-term planning for procedure-based specialties during extended mitigation and suppression strategies.   Gastroenterology. 2021;160(1):4-9. doi:10.1053/j.gastro.2020.05.047 PubMedGoogle ScholarCrossref
22.
Escobar  GJ, Adams  AS, Liu  VX,  et al.  Racial disparities in COVID-19 testing and outcomes : retrospective cohort study in an integrated health system.   Ann Intern Med. 2021;174(6):786-793. doi:10.7326/M20-6979 PubMedGoogle ScholarCrossref
23.
The Elements of Statistical Learning.  Data Mining, Inference, and Prediction. Springer-Verlag; 2017.
24.
US Census Bureau. American Community Survey, 2018; American Community Survey 5-year estimates, table B03002. Accessed May 31, 2021. https://data.census.gov/cedsi/
25.
Rho  HJ, Brown  H, Fremstad  S. A basic demographic profile of workers in frontline industries. Accessed May 31, 2021. https://cepr.net/wp-content/uploads/2020/04/2020-04-Frontline-Workers.pdf
26.
Escobar  GJ, Gardner  MN, Greene  JD, Draper  D, Kipnis  P.  Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system.   Med Care. 2013;51(5):446-453. doi:10.1097/MLR.0b013e3182881c8ePubMedGoogle ScholarCrossref
27.
Shen  AK, Hughes  R  IV, DeWald  E, Rosenbaum  S, Pisani  A, Orenstein  W.  Ensuring equitable access to COVID-19 vaccines in the US: current system challenges and opportunities.   Health Aff (Millwood). 2021;40(1):62-69. doi:10.1377/hlthaff.2020.01554 PubMedGoogle ScholarCrossref
28.
Schmidt  H, Weintraub  R, Williams  MA,  et al. Equitable allocation of COVID-19 vaccines: an analysis of allocation plans of CDC's jurisdictions with implications for disparate impact monitoring. Accessed May 31, 2021. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3803582
29.
Brewster  J. Here are the states breaking from CDC guidelines on vaccine priority. Accessed June 10, 2021. https://www.forbes.com/sites/jackbrewster/2020/12/24/here-are-the-states-breaking-from-cdc-guidelines-on-vaccine-priority/?sh=1fa8532a3804
30.
Martin  EG, Birkhead  GS, Holtgrave  DR.  Maintaining a focus on health equity during the COVID-19 vaccine rollout.   J Public Health Manag Pract. 2021;27(3):226-228. doi:10.1097/PHH.0000000000001359 PubMedGoogle ScholarCrossref
31.
Foy  BH, Wahl  B, Mehta  K, Shet  A, Menon  GI, Britto  C.  Comparing COVID-19 vaccine allocation strategies in India: a mathematical modelling study.   Int J Infect Dis. 2021;103:431-438. doi:10.1016/j.ijid.2020.12.075 PubMedGoogle ScholarCrossref
32.
Minoza  JMA, Bongolan  VP, Rayo  JF.  COVID-19 agent-based model with multi-objective optimization for vaccine distribution.   arXiv. Posted January 27, 2021. https://arxiv.org/abs/2101.11400Google Scholar
33.
Mulberry  N, Tupper  P, Kirwin  E, McCabe  C, Colijn  C.  Vaccine rollout strategies: the case for vaccinating essential workers early.   medRxiv. Posted February 25, 2021. doi:10.1101/2021.02.23.21252309Google Scholar
34.
Grauer  J, Löwen  H, Liebchen  B.  Strategic spatiotemporal vaccine distribution increases the survival rate in an infectious disease like Covid-19.   Sci Rep. 2020;10(1):21594. doi:10.1038/s41598-020-78447-3 PubMedGoogle ScholarCrossref
35.
Ioannou  GN, Green  P, Fan  VS,  et al.  Development of COVIDVax model to estimate the risk of SARS-CoV-2-related death among 7.6 million US veterans for use in vaccination prioritization.   JAMA Netw Open. 2021;4(4):e214347. doi:10.1001/jamanetworkopen.2021.4347 PubMedGoogle Scholar
36.
Ndugga  N AS, Pham  O. Latest data on COVID-19 vaccinations: race/ethnicity. Accessed March 11, 2021. https://www.kff.org/racial-equity-and-health-policy/issue-brief/how-are-states-addressing-racial-equity-in-covid-19-vaccine-efforts/
37.
Schmidt  H, Gostin  LO, Williams  MA.  Is it lawful and ethical to prioritize racial minorities for COVID-19 vaccines?   JAMA. 2020;324(20):2023-2024. doi:10.1001/jama.2020.20571 PubMedGoogle ScholarCrossref
38.
Persad  G, Peek  ME, Emanuel  EJ.  Fairly prioritizing groups for access to COVID-19 vaccines.   JAMA. 2020;324(16):1601-1602. doi:10.1001/jama.2020.18513 PubMedGoogle ScholarCrossref
39.
Wood  S, Schulman  K.  Beyond politics—promoting Covid-19 vaccination in the United States.   N Engl J Med. 2021;384(7):e23. doi:10.1056/NEJMms2033790 PubMedGoogle Scholar
40.
Peiris  M, Leung  GM.  What can we expect from first-generation COVID-19 vaccines?   Lancet. 2020;396(10261):1467-1469. doi:10.1016/S0140-6736(20)31976-0 PubMedGoogle ScholarCrossref
41.
Dutta  S, Kaur  RJ, Charan  J,  et al.  Serious adverse events reported from the COVID-19 vaccines: a descriptive study based on WHO database.   medRxiv. Posted March 24, 2021. doi:10.1101/2021.03.23.21253433Google Scholar
42.
Webb Hooper  M, Nápoles  AM, Pérez-Stable  EJ.  No populations left behind: vaccine hesitancy and equitable diffusion of effective COVID-19 vaccines.   J Gen Intern Med. 2021. doi:10.1007/s11606-021-06698-5 PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 2,054
    Original Investigation
    August 20, 2021

    Evaluation of Vaccination Strategies to Compare Efficient and Equitable Vaccine Allocation by Race and Ethnicity Across Time

    Author Affiliations
    • 1Division of Research, Kaiser Permanente, Oakland, California
    • 2The Permanente Medical Group, Oakland, California
    JAMA Health Forum. 2021;2(8):e212095. doi:10.1001/jamahealthforum.2021.2095
    Key Points

    Question  What is the association of different vaccine allocation strategies with COVID-19–related morbidity and mortality and their distribution by racial and ethnic groups across time?

    Findings  In this decision analytical model, the use of risk-based, age-based, and US Centers for Disease Control and Prevention (CDC)–phased vaccine allocation strategies was simulated. Risk-based strategies were associated with the largest estimated reductions in nonelective hospitalizations, death, and household transmissions compared with the CDC- and age-based strategies, with a similar proportion of Hispanic and Black patients being vaccinated early in the process compared with the CDC strategy.

    Meaning  The study findings suggest that risk-based vaccine prioritization strategies could have the greatest effectiveness on reducing COVID-19–related deaths and household transmissions while ensuring equitable vaccine distribution.

    Abstract

    Importance  Identifying the most efficient COVID-19 vaccine allocation strategy may substantially reduce hospitalizations and save lives while ensuring an equitable vaccine distribution.

    Objective  To simulate the association of different vaccine allocation strategies with COVID-19–associated morbidity and mortality and their distribution across racial and ethnic groups.

    Design, Setting, and Participants  We developed and internally validated the risk of COVID-19 infection and risk of hospitalization models on randomly split training and validation data sets. These were used in a computer simulation study of vaccine prioritization among adult health plan members who were drawn from an integrated health care delivery system. The study was conducted from January 3, 2021, to June 1, 2021, in Oakland, California, and the data were analyzed during the same period.

    Main Outcomes and Measures  We simulated the association of different vaccine allocation strategies, including (1) random, (2) a US Centers for Disease Control and Prevention (CDC) proxy, (3) age based, and (4) combinations of models for the risk of adverse outcomes (CRS) and COVID-19 infection (PROVID), with COVID-19-related hospitalizations between May 1, 2020, and December 31, 2020, that were randomly permuted by month across 250 simulations and assessed vaccine allocation by race and ethnicity and the neighborhood deprivation index across time.

    Results  The study included 3 202 679 adult patients (mean [SD] age, 48.2 [18.0] years; 1 677 637 women [52.4%]; 1 525 042 men [47.6%]; 611 154 Asian [19.1%], 206 363 Black [6.4%], 642 344 Hispanic [20.1%], and 1 390 638 White individuals [43.4%]), of whom 36 137 (1.1%) were positive for SARS-CoV-2. A risk-based strategy (CRS/PROVID) showed the largest avoidable hospitalization estimates (4954; 95% CI, 3452-5878) followed by age-based (4362; 95% CI, 2866-5175) and CDC proxy (4085; 95% CI, 2805-5109) strategies. Random vaccination showed substantially lower reductions in adverse outcomes. Risk-based strategies also showed the largest number of avoidable COVID-19 deaths (joint CRS/PROVID) and household transmissions. Risk-based (PROVID) and CDC proxy strategies were estimated to vaccinate the highest percentage of Hispanic and Black patients in 8 months (joint CRS/PROVID: 642 570 [100%] Hispanic, 185 530 [90%] Black; PROVID: 642 570 [100%] Hispanic, 198 480 [96%] Black; CDC proxy: 605 770 [95%] Hispanic and 151 772 [74%] Black) compared with an age-based approach (438 423 [68%] Hispanic, 154 714 [75%] Black). Overall, the PROVID and joint CRS/PROVID risk-based strategies were estimated to be followed by the most patients from areas with high neighborhood deprivation index being vaccinated early.

    Conclusions and Relevance  In this simulation modeling study of adults from a large integrated health care delivery system, risk-based strategies were associated with the largest estimated reductions in COVID-19 hospitalizations, deaths, and household transmissions compared with the CDC proxy and age-based strategies, with a higher proportion of Hispanic and Black patients were estimated to be vaccinated early in the process compared with the CDC strategy.

    Introduction

    SARS-CoV-2 infection and COVID-19 disease have resulted in more than 22 million US COVID-19 cases and 400 000 deaths,1 severely straining hospitals, disrupting communities, amplifying social disparities, and contributing to social unrest worldwide.2-6 Vaccination remains the key to controlling the COVID-19 disease,7 with the recent development of effective COVID-19 vaccines representing a critical step forward.8,9 Given the limitations in vaccine supply in early 2021, the Advisory Committee on Immunization Practices (ACIP) recommended phased COVID-19 vaccine allocation in the US10,11 (eTables 1-5 in the Supplement).

    While the ACIP allocation hierarchy sought to balance exposure risk factors, morbidity, and mortality along with social and equity priorities, implementing this strategy proved challenging, with many states pivoting to alternate approaches like age-based stratification. A key limitation has been the availability of data needed to implement the ACIP guidelines.12,13 While routine electronic health record (EHR) data could be used to inform vaccination, to our knowledge few studies have evaluated their potential role in improving allocation, particularly under inadequate vaccine supply constraints. In this study, we used EHR data from an integrated health care delivery system to simulate the potential association of different vaccine allocation strategies with estimated COVID-19–related morbidity, mortality, household transmission, and racial and ethnic group vaccine distribution.

    Methods

    This computer simulation study adheres to the health economic modeling report guidelines (Consolidated Health Economic Evaluation Reporting Standards; eTable 7 in the Supplement).14 Kaiser Permanente Northern California (KPNC) is an integrated health care delivery system that serves 4.5 million members. This project was approved by the KPNC institutional review board, which waived the requirement for individual informed consent because of the large number of study participants.

    Study Population

    The overall cohort included adult (≥18 years) KPNC patients with continuous membership or utilization (unless they died) between February 1, 2020 and December 3, 2020. We captured members’ self-reported race and ethnicity (documented at the time of insurance enrollment by medical assistants in the outpatient setting and/or admission clerks for inpatients); comorbidities15-17; and neighborhood deprivation index (NDI), a composite index ranging from −5 to +5, with more positive values indicating worsening socioeconomic status (eg, poverty and unemployment).18 We also calculated 2 previously validated general illness risk-adjustment indices: the Comorbidity Point Score, version 2 (COPS2), to assess 1-year comorbid disease burden and the abbreviated Laboratory-Based Acute Physiology Score (abLAPS), which is an outpatient laboratory-based severity of illness score (eAppendix in the Supplement).19,20

    COVID-19 Risk Scores

    Between the first 2 COVID-19 waves in Northern California, we developed 2 scores to classify patients within COVID-19 risk groups using data from February 1, 2020, to September 30, 2020: (1) the PROVID (probability of COVID-19 infection) to measure the likelihood of a patient having a positive SARS-CoV-2 polymerase chain reaction test result and (2) the COVID-19 Risk Score (CRS) to measure the risk of adverse outcomes (ie, hospitalization, death) among patients with COVID-19.21 The PROVID score used a logistic regression model with a positive SARS-CoV-2 test result as the outcome and predictors of age, sex, race and ethnicity,22 COPS2, abLAPS, and NDI. The model was fitted on a random training set (70% of the data) and validated on the remaining 30%, with a validation area under the receiver operating characteristic curve of 0.71 (95% CI, 0.71-0.72). The CRS score quantified the risk of nonelective hospitalization within 30 days of a SARS-CoV-2 positive test result using a logistic regression model for patients with COVID-19 that included sex and truncated power function splines at 3 selected knots23 for age and COPS2.21 We fitted the CRS model on 75% of the data and evaluated its performance on the remaining 25%. The model had a validation area under the curve of 0.82 (95% CI, 0.81-0.83) for 30-day nonelective hospitalization.

    We calculated CRS and PROVID scores for all adult KPNC members in the cohort and divided patients into deciles establishing low- (<40th percentile), medium- (40th to 60th percentile), and high-risk (>60th percentile) groups for each score. We characterized the demographic, clinical, and hospital utilization characteristics of the 9 joint CRS-PROVID risk groups based on the interactions between the 3 CRS and 3 PROVID risk strata (eTables 1-5 in the Supplement).

    Outcomes

    Because the intent was to explore the association of vaccination strategies with adverse outcomes from a public health perspective, our primary outcomes were COVID-19 hospitalizations, deaths, and household transmissions. A hospitalization was classified as COVID-19 related if the physician designation during hospital treatment was either confirmed active COVID or resolved COVID. COVID-19 deaths were inpatient deaths that occurred during COVID-19–related hospitalizations. To estimate COVID-19 household transmission, we grouped KPNC members into households according to their address in EHR data. In each household, we identified the first member with a positive SARS-CoV-2 test result and quantified COVID-19 transmission within a household based on the number of household members who tested positive within 4 weeks of that first household positive test result. As secondary outcomes, we evaluated the rate of vaccination by NDI and among 4 broad racial and ethnic groups (White, Black and African American, Hispanic, and Asian individuals).

    Vaccination Strategy Simulation

    The detailed data sources at KPNC allowed us to conduct a computer simulation study which is easier to understand and requires fewer modeling assumptions compared with a mathematical simulation model. We simulated 6 vaccination strategies assuming a target goal of vaccinating 75% of the 3 million adult KPNC members (2 400 000) over an 8-month period (240 days) with a uniform vaccination rate of 10 000 vaccines administered daily. When priority groups were simulated, we assumed that patients within the same priority grouping were vaccinated in random order and that all group members would be vaccinated before moving to the next group. Finally, we assumed that all phase 1a–eligible patients (ie, health care personnel and long-term care facility residents) completed vaccination before each prioritization strategy began. Four strategies were used. The first was a random vaccine allocation, which was a baseline process with randomly distributed vaccinations. The second was a CDC proxy, which was an ACIP-based process starting with essential workers, defined as patients ages 18 to 65 years with a PROVID score greater than the 80th percentile (1.6%) based on the most recent 5-year estimates from the American Community Survey (2014-2018).24,25 We defined patients with high-risk medical conditions (phase 1c) as those with a CRS score of 20% or greater, as used at KPNC to define higher-risk patients for COVID-19–related interventions. The remaining essential workers were defined as those with a PROVID in the 50th to 80th percentile (0.71%-1.6%). If patients qualified for several vaccination phases, we assigned them to their highest qualified tier (eTable 1 in the Supplement). The third strategy was age based and was developed around age groups in reverse order (age ≥75, 65-74, 55-64, 45-54, 35-44, and 18-34 years). The fourth strategy was risk based and involved a process in which vaccination prioritization was determined by the rate of hospitalizations across the 9 joint CRS/PROVID risk groups, starting with the high PROVID/high CRS group and ending with the low CRS/low PROVID group. We also assessed vaccine prioritization strategies by either risk score alone (PROVID or CRS alone).

    Statistical Analyses

    We calculated the number of outcomes during the 8-month period of May 1, 2020, through December 31, 2020, among patients that corresponded to each strategy until 2.4 million patients were vaccinated and compared the age-based, CDC proxy, and risk-based strategies with random vaccination order. To estimate the association between allocation strategies and outcomes, we conducted 250 simulations in which we permuted the month of COVID-19–related hospitalization month to simulate random monthly outcome distributions over the vaccination period. We assumed that any vaccination would be 100% effective at preventing hospitalization; thus, we removed hospitalizations that occurred at any time after 1 month from the vaccination date. In each simulation, we calculated the difference between the actual and simulated number of hospitalizations to estimate the number of avoided hospitalizations. We reported the average difference in hospitalizations and deaths across the 250 simulations and the 2.5th and 97.5th percentile as the 95% confidence interval. We followed a similar strategy to simulate SARS-CoV-2 transmissions within households. We permuted the first COVID-19 household test month in 250 simulations and assumed that vaccination would prevent transmission for transmissions that occurred within 2 weeks of receiving the vaccine. Finally, we compared the percentage of vaccinated patients by race and ethnicity and NDI (>75th percentile) to assess equitable distribution of the vaccine within each allocation strategy. Statistical analyses were conducted using SAS, version 7.15 (SAS Institute).

    Sensitivity Analysis

    We also varied the underlying age and race distribution of the population based on US census data24 to identify the individual states with the youngest (470 905 [14.7%] younger than age 65 years); oldest (804 063 [25.1%] older than age 65 years); highest percentage Asian (1 220 509 [38.1%]), African American (1 412 716 [44.1%]), Hispanic (1 579 294 [49.3%]) or White (2 972 790 [92.8%] populations); and lowest percentage African American (19 221 [0.6%]) and Hispanic (48 052 [1.5%]) populations (eTable 8 in the Supplement). We then created bootstrap samples of the KPNC data with the corresponding age and race distributions of these states and simulated the association between the vaccine prioritization strategies and adverse outcomes among populations of KPNC patients with these characteristics.

    Results

    We identified 3 202 679 adults, including 36 137 (1.1%) with positive SARS-CoV-2 test results (Table 118,19,26). Patients who tested positive for SARS-CoV-2 during the study period tended to be younger (mean [SD] age, 43.4 [16.3] vs 48.2 [18.0] years; P < .001), had fewer comorbidities (COPS2 of 14.2 vs 14.7; P < .001), and were less likely to be White (23% vs 44%; P < .001) compared with those who tested negative or without a test. Among those testing positive, 3403 (9.4%) experienced nonelective hospitalization within 30 days of their test. Hospitalized patients with COVID-19 were older (mean [SD] age, 59.5 [17.1] vs 41.8 [15.3] years; P < .001), had more comorbidities (mean COPS2, 30.4 vs 12.6; P < .001) and had more acute illness (mean abLAPS, 1.8 vs 0.3; P < .001) than those not hospitalized.

    Patients with a high risk of SARS-CoV-2 infection (higher PROVID scores) were more likely to be Hispanic (60%-70% vs <5%; P < .001 for all comparisons) and Black (12%-16% vs 0%-9%), to test positive for COVID-19 (20-24 vs 4-9 per 1000), and to have higher mean NDI scores (0.4 to 0.5 vs −1.1 to −0.2). Patients with high risk of adverse outcomes (higher CRS scores) were older (age 65-70 years vs 30-50 years; P < .001) and more likely to be male (60% vs 40%-50%; P < .001) than those with low and medium risk (eTable 6 and eFigures 1-4 in the Supplement).

    The oldest patients (75 years or older; n = 262 005) represented only 8.2% of the population but accounted for 25.4% of COVID-19 hospitalizations and 50.6% of inpatient deaths. In contrast, patients aged 18 to 54 years (n = 1 993 356) represented 62.2% of members, 34.4% of COVID-19 hospitalizations, and 11.3% of inpatient deaths (Table 2). Patients in the joint high CRS/high PROVID risk group had the highest hospitalization rate (12.1 per 1000 members), followed by those in the high CRS/medium PROVID risk group (6.8 per 1000 members). These 2 groups represented 11% of the population (n = 347 706) and accounted for 38.6% of all COVID-19 hospitalizations and 53.2% of inpatient deaths. Patients in the high PROVID/medium CRS and low PROVID/high CRS represented 28.2% of the population (n = 904 931) and accounted for another 38.6% of the COVID-19 hospitalizations and 42.1% of inpatient deaths.

    Over the 8-month study period, there were 7867 COVID-19 hospitalizations and 675 inpatient deaths. The simulated random allocation strategy was associated with estimates of 2645 (95% CI, 1766-3574; Figure 1) avoidable hospitalizations and 232 (95% CI, 143-317) avoidable inpatient deaths. Comparatively, the joint CRS/PROVID and CRS risk-based vaccination strategies were associated with the largest estimates of avoidable COVID-19 hospitalizations (CRS/PROVID: 4954; 95% CI, 3452-5878; CRS: 4470; 95% CI, 3215-5631) and deaths (CRS/PROVID: 505; 95% CI, 343-596; CRS: 535; 95% CI, 338-619). The age-based and CDC proxy approaches also showed higher estimated avoidable hospitalizations and deaths than random allocation (age-based: 4362; 95% CI, 2866-5175; CDC proxy: 4085; 95% CI, 2805-5109).

    The random allocation strategy was associated with an estimated 2301 (95% CI, 1696-2967) household transmissions prevented. A PROVID-based risk approach showed the highest estimated avoidable household transmissions (3176; 95% CI, 2496-3632), followed by the CDC proxy (2793; 95% CI, 2173-3352), the joint CRS/PROVID (2538; 95% CI, 1663-3460), the CRS (1796; 95% CI, 882-2832), and age-based (1722; 95% CI, 892-2549) strategies. In sensitivity analyses, results were similar when varying the age and race and ethnicity distributions, with the joint CRS/PROVID risk-based strategy showing the largest estimated reduction in hospitalizations and deaths (eTable 6 and eFigures 1-4 in the Supplement).

    The risk-based joint CRS/PROVID, PROVID risk allocation, and CDC proxy strategies vaccinated the highest percentage of Hispanic and Black patients in 8 months (CRS/PROVID: 642 570 [100%] Hispanic, 185 530 [90%] Black; PROVID: 642 570 [100%] Hispanic, 198 480 [96%] Black; CDC proxy: 605 770 [95%] Hispanic and 151 772 [74%] Black) compared with an age-based approach (438 423 [68%] Hispanic, 154 714 [75%] Black; Figure 2). Overall, the PROVID and CRS/PROVID risk-based strategies resulted in the most patients from areas with high neighborhood deprivation being vaccinated early (Figure 3).

    Discussion

    Using data from a large, diverse, and integrated health care delivery system, we simulated the implementation of different COVID-19 vaccine allocation strategies and compared their association with estimates of vaccine allocation effectiveness and equity. We found that risk-based strategies that identified patients at high risk for adverse outcomes were associated with the highest estimates of avoidable hospital admissions and deaths, followed by the CDC proxy and age-based strategies. We also found that a risk-based approach that identified patients at high risk for COVID-19 infection, followed by the CDC proxy strategy, achieved the goal of earlier vaccinations being administered to Hispanic and Black patients. Simulating the association of these differing prioritization strategies with adverse outcomes could help identify and mitigate some of the potential racial and ethnic and socioeconomic inequities that have been potentiated during the COVID-19 pandemic. Finally, we found that a risk-based allocation strategy that prioritized the risk of infection was associated with the highest estimates of avoidable household transmission, followed by the CDC proxy and joint risk-based approach. Overall, our analyses suggest that risk-based strategies using available EHR data could inform optimal approaches for improving vaccination effectiveness, efficiency, and equity.

    Recent studies on COVID-19 vaccine distribution have focused on policies and practices to support the equity, transparency, accountability, availability, and access to COVID-19 vaccines.27 While Schmidt et al28 compared different US approaches for implementing the CDC vaccine distribution recommendations to reduce inequity, other studies have highlighted the challenges of implementing the CDC strategy. These include its complexity, political considerations, and equity.12,29,30 Mathematical modeling studies also showed similar findings that prioritizing older adults could lead to reduced adverse outcomes,31,32 while alternate strategies that target younger adults, who are more likely to be essential workers or have higher infection potential, could reduce overall transmission.33,34 Another recent study compared the efficacy of risk-based, age-based, and CDC-phased vaccine allocation approaches to reduce COVID-19-related deaths35; however, the risk-based approach was based on only a single model, the CDC approach was defined by age alone, and the study did not evaluate other salient outcomes.

    Concerns about the equity of vaccination approaches have been confirmed, as emerging data revealed racial and ethnic inequities in vaccine allocation.12 For example, as of March 1, 2021, and including data from 41 states, there was consistent evidence of Hispanic and Black patients receiving smaller shares of vaccines compared with their rates of infection and hospitalization.36 In some locales, Asian patients, who also have increased rates of infection compared with White patients, also had lower relative rates of vaccination.22 While concerns have been raised about the legal implications of vaccine allocation by race and ethnicity, legally permissible strategies could use NDI, which integrates income levels, education, employment, and housing quality, to address vaccination.37 However, the urgent need to understand and simulate the vaccine allocation process and its resulting effect on marginalized patient groups remains.6,38,39

    While the US is entering a period of decreasing vaccination rates, vaccine supply remains highly constrained across the world today. Thus, the estimates produced in this simulation can help inform vaccine allocation approaches in locales with limited vaccine supply or potentially in future periods of vaccine distribution for COVID-19 or other similar pandemics. In addition, risk-based approaches could be used to inform public health and community engagement campaigns that are designed to increase vaccination uptake among groups that remain undervaccinated.

    Strengths and Limitations

    The major strength of this study was the ability to find an association between detailed population-level data with SARS-CoV-2 test results and COVID-19 hospital admissions, deaths, and household transmission through facilitating the use of 2 risk models and simulating key outcomes. The study also quantifies the effectiveness and equity of these approaches in a large and diverse patient population. This was possible because of the high degree of integration at KPHC and because it has made a concerted effort to capture race and ethnicity data in the outpatient setting.

    Our study also has important limitations. First, the CDC vaccine allocation process we simulated is only a proxy because we lacked access to high-quality information about essential workers’ status; however, the risk scores we developed could be valuable as a proxy to other health systems that similarly lack employment data Second, we also used prior COVID-19 hospitalization data, those before the availability of any vaccine outside of clinical trials, to understand the association of vaccination allocation and future hospitalizations. This assumed that the association of future surges on hospitalization and death would be similar to what we have already observed. While future COVID-19 sequelae might differ from that observed in 2020, we compared all strategies using the same assumptions and hospitalization data. Third, we did not simulate a scenario in which vaccinated patients were less likely to transmit the virus.40 Fourth, we did not consider deaths associated with the vaccine, which have proven to be extremely rare.41 Fifth, we did not consider different vaccine uptake levels among different racial and ethnic groups.42 Sixth, our estimates of household transmission should be viewed as provisory, as they likely reflect differential testing patterns and characteristics across households. Finally, the simulation does not consider any differential efficacy or reduction in the efficacy of differing vaccines over time.

    Conclusions

    In this simulated modeling study of adults from a large integrated health care delivery system, risk-based strategies were associated with the largest estimated reductions in COVID-19 hospitalizations, deaths, and household transmissions compared with the CDC proxy and age-based strategies, with similar proportions of Hispanic and Black patients being vaccinated early in the process compared with the CDC strategy.

    Back to top
    Article Information

    Accepted for Publication: June 19, 2021.

    Published: August 20, 2021. doi:10.1001/jamahealthforum.2021.2095

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Kipnis P et al. JAMA Health Forum.

    Corresponding Author: Patricia Kipnis, PhD, Division of Research, Kaiser Permanente, 2000 Broadway Ave, Oakland, CA 94612 (patricia.kipnis@kp.org).

    Author Contributions: Dr Kipnis had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Kipnis, Escobar, Myers, Liu.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: All authors.

    Critical revision of the manuscript for important intellectual content: Kipnis, Escobar, Liu.

    Statistical analysis: Kipnis, Soltesz, Myers.

    Obtained funding: Escobar, Liu.

    Administrative, technical, or material support: Escobar, Liu.

    Supervision: Liu.

    Conflict of Interest Disclosures: Dr Escobar reported grants from AstraZeneca outside the submitted work. Dr Liu reported grants from the National Institute of General Medical Sciences during the conduct of the study. No other disclosures were reported.

    Funding/Support: Dr Liu was funded by grant R35GM128672 from the National Institutes of Health, The Permanente Medical Group, and Kaiser Foundation Hospitals.

    Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: We thank Tracy Lieu, MD, MPH, Yi-Fen Irene Chen, MD, and Stephen Parodi, MD, Kaiser Permanente Northern California. These individuals were not compensated for their contributions.

    References
    1.
    US Centers for Disease Control and Prevention. United States COVID-19 cases and deaths by state. Accessed February 20, 2021. https://covid.cdc.gov/covid-data-tracker/#cases_casesper100klast7days
    2.
    Ogedegbe  G, Ravenell  J, Adhikari  S,  et al.  Assessment of racial/ethnic disparities in hospitalization and mortality in patients with COVID-19 in New York City.   JAMA Netw Open. 2020;3(12):e2026881. doi:10.1001/jamanetworkopen.2020.26881 PubMedGoogle Scholar
    3.
    Bravata  DM, Perkins  AJ, Myers  LJ,  et al.  Association of intensive care unit patient load and demand with mortality rates in US Department of Veterans Affairs hospitals during the COVID-19 pandemic.   JAMA Netw Open. 2021;4(1):e2034266. doi:10.1001/jamanetworkopen.2020.34266 PubMedGoogle Scholar
    4.
    Whaley  CM, Pera  MF, Cantor  J,  et al.  Changes in health services use among commercially insured US populations during the COVID-19 pandemic.   JAMA Netw Open. 2020;3(11):e2024984. doi:10.1001/jamanetworkopen.2020.24984 PubMedGoogle Scholar
    5.
    Holland  KM, Jones  C, Vivolo-Kantor  AM,  et al.  Trends in US emergency department visits for mental health, overdose, and violence outcomes before and during the COVID-19 pandemic.   JAMA Psychiatry. 2021;78(4):372-379. doi:10.1001/jamapsychiatry.2020.4402 PubMedGoogle ScholarCrossref
    6.
    Emanuel  EJ, Persad  G, Upshur  R,  et al.  Fair allocation of scarce medical resources in the time of Covid-19.   N Engl J Med. 2020;382(21):2049-2055. doi:10.1056/NEJMsb2005114 PubMedGoogle ScholarCrossref
    7.
    Greenwood  B.  The contribution of vaccination to global health: past, present and future.   Philos Trans R Soc Lond B Biol Sci. 2014;369(1645):20130433. doi:10.1098/rstb.2013.0433 PubMedGoogle Scholar
    8.
    Oliver  SE, Gargano  JW, Marin  M,  et al.  The Advisory Committee on Immunization Practices’ interim recommendation for use of Pfizer-BioNTech COVID-19 vaccine—United States, December 2020.   MMWR Morb Mortal Wkly Rep. 2020;69(50):1922-1924. doi:10.15585/mmwr.mm6950e2 PubMedGoogle ScholarCrossref
    9.
    Oliver  SE, Gargano  JW, Marin  M,  et al.  The Advisory Committee on Immunization Practices’ interim recommendation for use of Moderna COVID-19 vaccine—United States, December 2020.   MMWR Morb Mortal Wkly Rep. 2021;69(5152):1653-1656. doi:10.15585/mmwr.mm695152e1 PubMedGoogle ScholarCrossref
    10.
    Smith  JC.  The structure, role, and procedures of the U.S. Advisory Committee on Immunization Practices (ACIP).   Vaccine. 2010;28(suppl 1):A68-A75. doi:10.1016/j.vaccine.2010.02.037 PubMedGoogle ScholarCrossref
    11.
    Dooling  K, Marin  M, Wallace  M,  et al.  The Advisory Committee on Immunization Practices’ updated interim recommendation for allocation of COVID-19 vaccine—United States, December 2020.   MMWR Morb Mortal Wkly Rep. 2021;69(5152):1657-1660. doi:10.15585/mmwr.mm695152e2 PubMedGoogle ScholarCrossref
    12.
    Jean-Jacques  M, Bauchner  H.  Vaccine distribution—equity left behind?   JAMA. 2021;325(9):829-830. doi:10.1001/jama.2021.1205 PubMedGoogle ScholarCrossref
    13.
    US Centers for Disease Control and Prevention. People with certain medical conditions. Accessed May 31, 2021, 2021. https://www.cdc.gov/coronavirus/2019-ncov/need-extra-precautions/people-with-medical-conditions.html
    14.
    Husereau  D, Drummond  M, Petrou  S,  et al.  Consolidated health economic evaluation reporting standards (CHEERS) statement.   Eur J Health Econ. 2013;14(3):367-372. doi:10.1007/s10198-013-0471-6 PubMedGoogle ScholarCrossref
    15.
    van Walraven  C, Austin  PC, Jennings  A, Quan  H, Forster  AJ.  A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data.   Med Care. 2009;47(6):626-633. doi:10.1097/MLR.0b013e31819432e5 PubMedGoogle ScholarCrossref
    16.
    Elixhauser  A, Steiner  C, Harris  DR, Coffey  RM.  Comorbidity measures for use with administrative data.   Med Care. 1998;36(1):8-27. doi:10.1097/00005650-199801000-00004 PubMedGoogle ScholarCrossref
    17.
    Deyo  RA, Cherkin  DC, Ciol  MA.  Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.   J Clin Epidemiol. 1992;45(6):613-619. doi:10.1016/0895-4356(92)90133-8 PubMedGoogle ScholarCrossref
    18.
    Messer  LC, Laraia  BA, Kaufman  JS,  et al.  The development of a standardized neighborhood deprivation index.   J Urban Health. 2006;83(6):1041-1062. doi:10.1007/s11524-006-9094-x PubMedGoogle ScholarCrossref
    19.
    Escobar  GJ, Greene  JD, Scheirer  P, Gardner  MN, Draper  D, Kipnis  P.  Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases.   Med Care. 2008;46(3):232-239. doi:10.1097/MLR.0b013e3181589bb6 PubMedGoogle Scholar
    20.
    van Walraven  C, Escobar  GJ, Greene  JD, Forster  AJ.  The Kaiser Permanente inpatient risk adjustment methodology was valid in an external patient population.   J Clin Epidemiol. 2010;63(7):798-803. doi:10.1016/j.jclinepi.2009.08.020 PubMedGoogle ScholarCrossref
    21.
    Rouillard  S, Liu  VX, Corley  DA.  COVID-19 and long-term planning for procedure-based specialties during extended mitigation and suppression strategies.   Gastroenterology. 2021;160(1):4-9. doi:10.1053/j.gastro.2020.05.047 PubMedGoogle ScholarCrossref
    22.
    Escobar  GJ, Adams  AS, Liu  VX,  et al.  Racial disparities in COVID-19 testing and outcomes : retrospective cohort study in an integrated health system.   Ann Intern Med. 2021;174(6):786-793. doi:10.7326/M20-6979 PubMedGoogle ScholarCrossref
    23.
    The Elements of Statistical Learning.  Data Mining, Inference, and Prediction. Springer-Verlag; 2017.
    24.
    US Census Bureau. American Community Survey, 2018; American Community Survey 5-year estimates, table B03002. Accessed May 31, 2021. https://data.census.gov/cedsi/
    25.
    Rho  HJ, Brown  H, Fremstad  S. A basic demographic profile of workers in frontline industries. Accessed May 31, 2021. https://cepr.net/wp-content/uploads/2020/04/2020-04-Frontline-Workers.pdf
    26.
    Escobar  GJ, Gardner  MN, Greene  JD, Draper  D, Kipnis  P.  Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system.   Med Care. 2013;51(5):446-453. doi:10.1097/MLR.0b013e3182881c8ePubMedGoogle ScholarCrossref
    27.
    Shen  AK, Hughes  R  IV, DeWald  E, Rosenbaum  S, Pisani  A, Orenstein  W.  Ensuring equitable access to COVID-19 vaccines in the US: current system challenges and opportunities.   Health Aff (Millwood). 2021;40(1):62-69. doi:10.1377/hlthaff.2020.01554 PubMedGoogle ScholarCrossref
    28.
    Schmidt  H, Weintraub  R, Williams  MA,  et al. Equitable allocation of COVID-19 vaccines: an analysis of allocation plans of CDC's jurisdictions with implications for disparate impact monitoring. Accessed May 31, 2021. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3803582
    29.
    Brewster  J. Here are the states breaking from CDC guidelines on vaccine priority. Accessed June 10, 2021. https://www.forbes.com/sites/jackbrewster/2020/12/24/here-are-the-states-breaking-from-cdc-guidelines-on-vaccine-priority/?sh=1fa8532a3804
    30.
    Martin  EG, Birkhead  GS, Holtgrave  DR.  Maintaining a focus on health equity during the COVID-19 vaccine rollout.   J Public Health Manag Pract. 2021;27(3):226-228. doi:10.1097/PHH.0000000000001359 PubMedGoogle ScholarCrossref
    31.
    Foy  BH, Wahl  B, Mehta  K, Shet  A, Menon  GI, Britto  C.  Comparing COVID-19 vaccine allocation strategies in India: a mathematical modelling study.   Int J Infect Dis. 2021;103:431-438. doi:10.1016/j.ijid.2020.12.075 PubMedGoogle ScholarCrossref
    32.
    Minoza  JMA, Bongolan  VP, Rayo  JF.  COVID-19 agent-based model with multi-objective optimization for vaccine distribution.   arXiv. Posted January 27, 2021. https://arxiv.org/abs/2101.11400Google Scholar
    33.
    Mulberry  N, Tupper  P, Kirwin  E, McCabe  C, Colijn  C.  Vaccine rollout strategies: the case for vaccinating essential workers early.   medRxiv. Posted February 25, 2021. doi:10.1101/2021.02.23.21252309Google Scholar
    34.
    Grauer  J, Löwen  H, Liebchen  B.  Strategic spatiotemporal vaccine distribution increases the survival rate in an infectious disease like Covid-19.   Sci Rep. 2020;10(1):21594. doi:10.1038/s41598-020-78447-3 PubMedGoogle ScholarCrossref
    35.
    Ioannou  GN, Green  P, Fan  VS,  et al.  Development of COVIDVax model to estimate the risk of SARS-CoV-2-related death among 7.6 million US veterans for use in vaccination prioritization.   JAMA Netw Open. 2021;4(4):e214347. doi:10.1001/jamanetworkopen.2021.4347 PubMedGoogle Scholar
    36.
    Ndugga  N AS, Pham  O. Latest data on COVID-19 vaccinations: race/ethnicity. Accessed March 11, 2021. https://www.kff.org/racial-equity-and-health-policy/issue-brief/how-are-states-addressing-racial-equity-in-covid-19-vaccine-efforts/
    37.
    Schmidt  H, Gostin  LO, Williams  MA.  Is it lawful and ethical to prioritize racial minorities for COVID-19 vaccines?   JAMA. 2020;324(20):2023-2024. doi:10.1001/jama.2020.20571 PubMedGoogle ScholarCrossref
    38.
    Persad  G, Peek  ME, Emanuel  EJ.  Fairly prioritizing groups for access to COVID-19 vaccines.   JAMA. 2020;324(16):1601-1602. doi:10.1001/jama.2020.18513 PubMedGoogle ScholarCrossref
    39.
    Wood  S, Schulman  K.  Beyond politics—promoting Covid-19 vaccination in the United States.   N Engl J Med. 2021;384(7):e23. doi:10.1056/NEJMms2033790 PubMedGoogle Scholar
    40.
    Peiris  M, Leung  GM.  What can we expect from first-generation COVID-19 vaccines?   Lancet. 2020;396(10261):1467-1469. doi:10.1016/S0140-6736(20)31976-0 PubMedGoogle ScholarCrossref
    41.
    Dutta  S, Kaur  RJ, Charan  J,  et al.  Serious adverse events reported from the COVID-19 vaccines: a descriptive study based on WHO database.   medRxiv. Posted March 24, 2021. doi:10.1101/2021.03.23.21253433Google Scholar
    42.
    Webb Hooper  M, Nápoles  AM, Pérez-Stable  EJ.  No populations left behind: vaccine hesitancy and equitable diffusion of effective COVID-19 vaccines.   J Gen Intern Med. 2021. doi:10.1007/s11606-021-06698-5 PubMedGoogle Scholar
    ×