Association Between mRNA Vaccination and COVID-19 Hospitalization and Disease Severity | Infectious Diseases | JAMA | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Participant Flow Through a Study of Association Between COVID-19 mRNA Vaccination and Hospitalizations and Disease Severity
Participant Flow Through a Study of Association Between COVID-19 mRNA Vaccination and Hospitalizations and Disease Severity

“Syndrome-negative” participants were persons hospitalized without signs or symptoms consistent with acute COVID-19 and who tested negative for SARS-CoV-2 by molecular testing. They were included as a secondary control group because of the theoretical risk of case misclassification in test-negative controls.

Figure 2.  Association Between Hospitalization for COVID-19 and Prior Vaccination With a 2-Dose mRNA Vaccine
Association Between Hospitalization for COVID-19 and Prior Vaccination With a 2-Dose mRNA Vaccine

An adjusted odds ratio (aOR) less than 1.0 indicated that COVID-19 hospitalization was associated with being unvaccinated compared with being fully vaccinated. Vaccine effectiveness for prevention of COVID-19 hospitalization can be estimated from the aORs presented here with the following equation: vaccine effectiveness = (1 − aOR) × 100%. BNT162b2 is the vaccine produced by Pfizer-BioNTech and mRNA-1273 is the vaccine produced by Moderna. mRNA indicates messenger RNA.

aModels were mixed-effects logistic regression models with vaccination status (fully vaccinated vs unvaccinated) as the primary independent variable, case-control status (hospitalized with COVID-19 vs hospitalized without it) as the dependent variable, enrolling site as a random effect, and the following covariables: admission date (biweekly intervals), age group (18-49, 50-64, and ≥65 years), sex, and self-reported race and ethnicity. Models stratified by age group were adjusted for continuous age in years.

bImmunocompromising conditions are defined in the Table.

cAlpha estimates restricted to March to June illness-onset dates (Alpha period); Delta estimates restricted to July to August illness-onset dates (Delta period).

Figure 3.  Association Between Progression to Severe Disease and Prior Vaccination Among Adults Hospitalized With COVID-19
Association Between Progression to Severe Disease and Prior Vaccination Among Adults Hospitalized With COVID-19

An adjusted odds ratio (aOR) less than 1.0 indicated that progression to death or invasive mechanical ventilation after hospital admission for COVID-19 was associated with being unvaccinated compared with being vaccinated.

aModels were adjusted for age group (18-49, 50-64, and ≥65 years), sex, self-reported race and ethnicity, and number of chronic medical comorbidities (0, 1, 2, 3, and ≥4). Models stratified by age group were adjusted for continuous age in years.

bImmunocompromising conditions are defined in the Table.

cAnalysis restricted to COVID-19 case patients with hypoxemia within 24 hours of admission, defined as receiving supplemental oxygen or having an oxygen saturation less than 92% as measured by pulse oximetry.

Figure 4.  Competing Risks Regression of Hospital Discharge for Adults Hospitalized With COVID-19
Competing Risks Regression of Hospital Discharge for Adults Hospitalized With COVID-19

Cumulative incidence of hospital discharge by vaccination status (fully vaccinated with a 2-dose series of mRNA vaccine vs unvaccinated) is shown for patients not immunocompromised (A), those immunocompromised (B), those aged 18 to 64 years (C), and those aged 65 years or older (D). The event of interest was discharge from the hospital before day 28 in the presence of the competing event of death. Patients who remained hospitalized more than 28 days were censored at 28 days. Competing risk models were adjusted for age group (18-49, 50-64, and ≥65 years), sex, self-reported race and ethnicity, and number of medical comorbidities (0, 1, 2, 3, and ≥4). Models by age group were adjusted for continuous age in years. mRNA indicates messenger RNA; and SHR, subdistribution hazard ratio.

Table.  Characteristics of Case Patients With COVID-19 by Vaccination Status and Control Patients Without COVID-19
Characteristics of Case Patients With COVID-19 by Vaccination Status and Control Patients Without COVID-19
Supplement.

eAppendix 1. Investigators and Collaborators

eAppendix 2. Supplementary Tables and Figures

eTable 1. Modified WHO COVID-19 Clinical Progression Scale Used in This Analysis to Assess Disease Severity Among Adults Hospitalized With COVID-19

eTable 2. Underlying Medical Conditions Obtained Through Medical Record Review

eTable 3. Association Between Hospitalization for COVID-19 and Prior Receipt of Full Vaccination With a Two-Dose Series of a mRNA Vaccine, Restricted to Patients Without Immunocompromising Conditions

eTable 4. Association Between Hospitalization for COVID-19 and Prior Receipt of Full Vaccination With a Two-Dose Series of a mRNA Vaccine, Restricted to Patients Without Immunocompromising Conditions and Aged 18-64 Years

eTable 5. Association Between Hospitalization for COVID-19 and Prior Receipt of Full Vaccination With a Two-Dose Series of a mRNA Vaccine, Restricted to Patients Without Immunocompromising Conditions and Aged ≥65 Years

eTable 6. Treatments, Outcomes, and Severity of Illness Among Hospitalized COVID-19 Cases by Vaccination Status

eTable 7. Odds Prior mRNA COVID-19 Vaccination Among COVID-19 Cases Who Received In-Hospital COVID-19 Therapeutics and Those Who Did Not

eFigure 1. Whole Genome Sequencing SARS-CoV-2 Lineage Determination Among COVID-19 Cases by Admission Week

eFigure 2. Highest Severity Level Experienced on the WHO COVID-19 Clinical Progression Scale During the First 28 Days of Hospitalization Among Vaccine Breakthrough COVID-19 Cases and Unvaccinated COVID-19 Cases

1.
Morin  L, Savale  L, Pham  T,  et al; Writing Committee for the COMEBAC Study Group.  Four-month clinical status of a cohort of patients after hospitalization for COVID-19.   JAMA. 2021;325(15):1525-1534. doi:10.1001/jama.2021.3331 PubMedGoogle Scholar
2.
Dagan  N, Barda  N, Kepten  E,  et al.  BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting.   N Engl J Med. 2021;384(15):1412-1423. doi:10.1056/NEJMoa2101765 PubMedGoogle ScholarCrossref
3.
Haas  EJ, Angulo  FJ, McLaughlin  JM,  et al.  Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data.   Lancet. 2021;397(10287):1819-1829. doi:10.1016/S0140-6736(21)00947-8 PubMedGoogle ScholarCrossref
4.
Tenforde  MW, Patel  MM, Ginde  AA,  et al; Influenza and Other Viruses in the Acutely Ill (IVY) Network.  Effectiveness of SARS-CoV-2 mRNA vaccines for preventing Covid-19 hospitalizations in the United States.   Clin Infect Dis. 2021;ciab687. doi:10.1093/cid/ciab687 PubMedGoogle Scholar
5.
Goldshtein  I, Nevo  D, Steinberg  DM,  et al.  Association between BNT162b2 vaccination and incidence of SARS-CoV-2 infection in pregnant women.   JAMA. 2021;326(8):728-735. doi:10.1001/jama.2021.11035 PubMedGoogle ScholarCrossref
6.
CDC COVID-19 Vaccine Breakthrough Case Investigations Team.  COVID-19 vaccine breakthrough infections reported to CDC—United States, January 1–April 30, 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(21):792-793. doi:10.15585/mmwr.mm7021e3 PubMedGoogle ScholarCrossref
7.
Halloran  ME, Longini  IM  Jr, Struchiner  CJ.  Design and interpretation of vaccine field studies.   Epidemiol Rev. 1999;21(1):73-88. doi:10.1093/oxfordjournals.epirev.a017990 PubMedGoogle ScholarCrossref
8.
WHO Working Group on the Clinical Characterisation and Management of COVID-19 infection.  A minimal common outcome measure set for COVID-19 clinical research.   Lancet Infect Dis. 2020;20(8):e192-e197. doi:10.1016/S1473-3099(20)30483-7 PubMedGoogle ScholarCrossref
9.
Hall  VJ, Foulkes  S, Saei  A,  et al; SIREN Study Group.  COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study.   Lancet. 2021;397(10286):1725-1735. doi:10.1016/S0140-6736(21)00790-X PubMedGoogle ScholarCrossref
10.
Sheikh  A, McMenamin  J, Taylor  B, Robertson  C; Public Health Scotland and the EAVE II Collaborators.  SARS-CoV-2 Delta VOC in Scotland: demographics, risk of hospital admission, and vaccine effectiveness.   Lancet. 2021;397(10293):2461-2462. doi:10.1016/S0140-6736(21)01358-1 PubMedGoogle ScholarCrossref
11.
Thompson  MG, Burgess  JL, Naleway  AL,  et al.  Prevention and attenuation of Covid-19 with the BNT162b2 and mRNA-1273 vaccines.   N Engl J Med. 2021;385(4):320-329. doi:10.1056/NEJMoa2107058 PubMedGoogle ScholarCrossref
12.
Cromer  D, Juno  JA, Khoury  D,  et al.  Prospects for durable immune control of SARS-CoV-2 and prevention of reinfection.   Nat Rev Immunol. 2021;21(6):395-404. doi:10.1038/s41577-021-00550-x PubMedGoogle ScholarCrossref
13.
Sadarangani  M, Marchant  A, Kollmann  TR.  Immunological mechanisms of vaccine-induced protection against COVID-19 in humans.   Nat Rev Immunol. 2021;21(8):475-484. doi:10.1038/s41577-021-00578-z PubMedGoogle ScholarCrossref
14.
Patel  MM, Thornburg  NJ, Stubblefield  WB,  et al.  Change in antibodies to SARS-CoV-2 over 60 days among health care personnel in Nashville, Tennessee.   JAMA. 2020;324(17):1781-1782. doi:10.1001/jama.2020.18796 PubMedGoogle ScholarCrossref
15.
Tenforde  MW, Self  WH, Naioti  EA,  et al; IVY Network Investigators; IVY Network.  Sustained effectiveness of Pfizer-BioNTech and Moderna vaccines against COVID-19 associated hospitalizations among adults—United States, March-July 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(34):1156-1162. doi:10.15585/mmwr.mm7034e2 PubMedGoogle ScholarCrossref
16.
Self  WH, Tenforde  MW, Rhoads  JP,  et al; IVY Network.  Comparative effectiveness of Moderna, Pfizer-BioNTech, and Janssen (Johnson & Johnson) vaccines in preventing COVID-19 hospitalizations among adults without immunocompromising conditions—United States, March-August 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(38):1337-1343. doi:10.15585/mmwr.mm7038e1 PubMedGoogle ScholarCrossref
17.
Grijalva  CG, Feldstein  LR, Talbot  HK,  et al; Influenza and Other Viruses in the Acutely Ill (IVY) Network.  Influenza vaccine effectiveness for prevention of severe influenza-associated illness among adults in the United States, 2019-2020: a test-negative study.   Clin Infect Dis. 2021;(May):ciab462. PubMedGoogle Scholar
18.
Tenforde  MW, Talbot  HK, Trabue  CH,  et al.  Influenza vaccine effectiveness against hospitalization in the United States, 2019-2020.   J Infect Dis. 2021;224(5):813-820. doi:10.1093/infdis/jiaa800PubMedGoogle ScholarCrossref
19.
Grijalva  CG, Zhu  Y, Williams  DJ,  et al.  Association between hospitalization with community-acquired laboratory-confirmed influenza pneumonia and prior receipt of influenza vaccination.   JAMA. 2015;314(14):1488-1497. doi:10.1001/jama.2015.12160 PubMedGoogle ScholarCrossref
20.
Lauer  SA, Grantz  KH, Bi  Q,  et al.  The incubation period of coronavirus disease 2019 (COVID-19) from publicly reported confirmed cases: estimation and application.   Ann Intern Med. 2020;172(9):577-582. doi:10.7326/M20-0504 PubMedGoogle ScholarCrossref
21.
Lewnard  JA, Patel  MM, Jewell  NP,  et al.  Theoretical framework for retrospective studies of the effectiveness of SARS-CoV-2 vaccines.   Epidemiology. 2021;32(4):508-517. doi:10.1097/EDE.0000000000001366 PubMedGoogle ScholarCrossref
22.
Food and Drug Administration. CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel. Accessed May 26, 2021. https://www.fda.gov/media/134922/download
23.
Quick  J. nCoV2-2019 sequencing protocol v3 (LoCost) V.3. protocols.io. Accessed May 26, 2021. https://www.protocols.io/view/ncov-2019-sequencing-protocol-%20v3-locost-bh42j8ye
24.
Rambaut  A, Holmes  EC, O’Toole  Á,  et al.  A dynamic nomenclature proposal for SARS-CoV-2 lineages to assist genomic epidemiology.   Nat Microbiol. 2020;5(11):1403-1407. doi:10.1038/s41564-020-0770-5PubMedGoogle ScholarCrossref
25.
Centers for Disease Control and Prevention. Interim clinical considerations for use of COVID-19 vaccines currently authorized in the United States. Accessed June 1, 2021. https://www.cdc.gov/vaccines/covid-19/clinical-considerations/covid-19-vaccines-us.html#footnote-01
26.
Foppa  IM, Haber  M, Ferdinands  JM, Shay  DK.  The case test-negative design for studies of the effectiveness of influenza vaccine.   Vaccine. 2013;31(30):3104-3109. doi:10.1016/j.vaccine.2013.04.026 PubMedGoogle ScholarCrossref
27.
Centers for Disease Control and Prevention. COVID data tracker. Accessed August 6, 2021. https://covid.cdc.gov/covid-data-tracker/#datatracker-home
28.
Fine  JP, Gray  RJ.  A proportional hazards model for the subdistribution of a competing risk.   J Am Stat Assoc. 1999;94(446):496-509. doi:10.1080/01621459.1999.10474144 Google ScholarCrossref
29.
Baden  LR, El Sahly  HM, Essink  B,  et al; COVE Study Group.  Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine.   N Engl J Med. 2021;384(5):403-416. doi:10.1056/NEJMoa2035389 PubMedGoogle ScholarCrossref
30.
Polack  FP, Thomas  SJ, Kitchin  N,  et al; C4591001 Clinical Trial Group.  Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine.   N Engl J Med. 2020;383(27):2603-2615. doi:10.1056/NEJMoa2034577 PubMedGoogle ScholarCrossref
31.
Food and Drug Administration. Vaccines and Related Biological Products Advisory Committee September 17, 2021 meeting announcement. Accessed September 21, 2021. https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-september-17-2021-meeting-announcement
32.
Nanduri  S, Pilishvili  T, Derado  G,  et al.  Effectiveness of Pfizer-BioNTech and Moderna vaccines in preventing SARS-CoV-2 infection among nursing home residents before and during widespread circulation of the SARS-CoV-2 B.1.617.2 (Delta) variant—National Healthcare Safety Network, March 1–August 1, 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(34):1163-1166. doi:10.15585/mmwr.mm7034e3 PubMedGoogle ScholarCrossref
33.
Centers for Disease Control and Prevention. COVID-19 vaccines for moderately to severely immunocompromised people. Accessed September 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/vaccines/recommendations/immuno.html
34.
Grannis  SJ, Rowley  EA, Ong  TC,  et al; VISION Network.  Interim estimates of COVID-19 vaccine effectiveness against COVID-19–associated emergency department or urgent care clinic encounters and hospitalizations among adults during SARS-CoV-2 B.1.617.2 (Delta) variant predominance—nine states, June-August 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(37):1291-1293. doi:10.15585/mmwr.mm7037e2 PubMedGoogle ScholarCrossref
35.
Steensels  D, Pierlet  N, Penders  J, Mesotten  D, Heylen  L.  Comparison of SARS-CoV-2 antibody response following vaccination with BNT162b2 and mRNA-1273.   JAMA. 2021;e2115125. PubMedGoogle Scholar
36.
Dan  JM, Mateus  J, Kato  Y,  et al.  Immunological memory to SARS-CoV-2 assessed for up to 8 months after infection.   Science. 2021;371(6529):eabf4063. doi:10.1126/science.abf4063 PubMedGoogle Scholar
37.
Flannery  B, Kondor  RJG, Chung  JR,  et al.  Spread of antigenically drifted influenza A(H3N2) viruses and vaccine effectiveness in the United States during the 2018-2019 season.   J Infect Dis. 2020;221(1):8-15. doi:10.1093/infdis/jiz543 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    EXPAND ALL
    Biases and Confounding
    Saeed Taheri, Dr. | New Lahijan Scientific Foundation; Lahijan; Iran
    This study should be interpreted with the following considerations of bias and confounding in mind:

    1. It is reasonable to consider that the higher rate of vaccination in the control group is skewed due to selection/prioritization of vaccinated patients with other medical conditions for hospitalization. In the COVID era, patients with more benign conditions (like elective surgeries or conditions with the possibility of outpatient management) were more likely to postpone their hospital attendance until completion of vaccination.

    2. Older patients and those with more comorbidities and as a consequence, a higher possibility of hospitalization for other medical conditions, are
    more likely to access health care or procedures after vaccination. They are also prioritized by the governments for early vaccination.

    3. People not believing in vaccination are likely younger with fewer medical conditions/ comorbidities (Table), and consequently they are less likely to be hospitalized for other reasons. They are also more likely to have more risky behaviors for COVID-19 infection. So the lower rate of vaccination in the COVID-19+ (case) group and higher rate of vaccination in the control group hospitalized for other conditions and therefore, the effects of the lack of vaccination on hospitalization due to COVID-19 infection, might have been overestimated.

    4. Ideally, two groups in observational studies are different in only one factor and comparable in others. In the current study, the case and control groups are different in both COVID-19 infection and reasons for hospitalization. It would have been nice to stratify control group analyses into syndrome-positive and syndrome-negative groups.

    5. In the outcome analysis (figure 3), failing to reach a significantly worse outcome for the unvaccinated patients versus the vaccinated ones in the younger individuals could be due in part to the same selection-bias mentioned above, with young unvaccinated patients being less likely to have serious medical conditions (versus those who go after vaccination). In other words, even in younger patients, vaccination might have resulted in better outcome, here potentially attenuated by difference in the baseline medical conditions.

    CONFLICT OF INTEREST: None Reported
    READ MORE
    Original Investigation
    November 4, 2021

    Association Between mRNA Vaccination and COVID-19 Hospitalization and Disease Severity

    Mark W. Tenforde, MD, PhD1; Wesley H. Self, MD, MPH2; Katherine Adams, MPH1; et al Manjusha Gaglani, MBBS3; Adit A. Ginde, MD, MPH4; Tresa McNeal, MD3; Shekhar Ghamande, MD3; David J. Douin, MD5; H. Keipp Talbot, MD, MPH6; Jonathan D. Casey, MD, MSci7; Nicholas M. Mohr, MD, MS8; Anne Zepeski, PharmD8; Nathan I. Shapiro, MD, MPH9; Kevin W. Gibbs, MD10; D. Clark Files, MD10; David N. Hager, MD, PhD11; Arber Shehu, MD11; Matthew E. Prekker, MD, MPH12; Heidi L. Erickson, MD13; Matthew C. Exline, MD, MPH14; Michelle N. Gong, MD15; Amira Mohamed, MD16; Daniel J. Henning, MD, MPH17; Jay S. Steingrub, MD18; Ithan D. Peltan, MD, MSc19; Samuel M. Brown, MD, MS19; Emily T. Martin, PhD20; Arnold S. Monto, MD20; Akram Khan, MD21; Catherine L. Hough, MD21; Laurence W. Busse, MD22; Caitlin C. ten Lohuis, ACNP-BC23; Abhijit Duggal, MD24; Jennifer G. Wilson, MD25; Alexandra June Gordon, MD25; Nida Qadir, MD26; Steven Y. Chang, MD, PhD26; Christopher Mallow, MD, MHS27; Carolina Rivas, BS27; Hilary M. Babcock, MD, MPH28; Jennie H. Kwon, DO, MSci28; Natasha Halasa, MD, MPH29; James D. Chappell, MD, PhD29; Adam S. Lauring, MD, PhD30; Carlos G. Grijalva, MD, MPH31; Todd W. Rice, MD, MSci7; Ian D. Jones, MD32; William B. Stubblefield, MD, MPH32; Adrienne Baughman, BS32; Kelsey N. Womack, PhD33; Jillian P. Rhoads, PhD33; Christopher J. Lindsell, PhD34; Kimberly W. Hart, MA34; Yuwei Zhu, MD, MS34; Samantha M. Olson, MPH1; Miwako Kobayashi, MD1; Jennifer R. Verani, MD, MPH1; Manish M. Patel, MD1; for the Influenza and Other Viruses in the Acutely Ill (IVY) Network
    Author Affiliations
    • 1CDC COVID-19 Response Team, Atlanta, Georgia
    • 2Vanderbilt Institute for Clinical and Translational Research, Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, Tennessee
    • 3Baylor Scott & White Health, Texas A&M University College of Medicine, Temple
    • 4Department of Emergency Medicine, University of Colorado School of Medicine, Aurora
    • 5Department of Anesthesiology, University of Colorado School of Medicine, Aurora
    • 6Departments of Medicine and Health Policy, Vanderbilt University Medical Center, Nashville, Tennessee
    • 7Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee
    • 8Department of Emergency Medicine, University of Iowa, Iowa City
    • 9Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts
    • 10Department of Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina
    • 11Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland
    • 12Departments of Emergency Medicine and Medicine, Hennepin County Medical Center, Minneapolis, Minnesota
    • 13Department of Medicine, Hennepin County Medical Center, Minneapolis, Minnesota
    • 14Department of Medicine, The Ohio State University, Columbus
    • 15Department of Medicine, Montefiore Health System, Albert Einstein College of Medicine, Bronx, New York
    • 16Department of Medicine, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, New York
    • 17Department of Emergency Medicine, University of Washington, Seattle
    • 18Department of Medicine, Baystate Medical Center, Springfield, Massachusetts
    • 19Department of Medicine, Intermountain Medical Center, Murray, Utah; and University of Utah, Salt Lake City
    • 20School of Public Health, University of Michigan, Ann Arbor
    • 21Department of Medicine, Oregon Health & Science University, Portland
    • 22Department of Medicine, Emory University, Atlanta, Georgia
    • 23Emory Critical Care Center, Emory Healthcare, Atlanta, Georgia
    • 24Department of Medicine, Cleveland Clinic, Cleveland, Ohio
    • 25Department of Emergency Medicine, Stanford University School of Medicine, Stanford, California
    • 26Department of Medicine, University of California–Los Angeles, Los Angeles
    • 27Department of Medicine, University of Miami, Miami, Florida
    • 28Department of Medicine, Washington University, St Louis, Missouri
    • 29Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee
    • 30Departments of Internal Medicine and Microbiology and Immunology, University of Michigan, Ann Arbor
    • 31Department of Health Policy, Vanderbilt University Medical Center, Nashville, Tennessee
    • 32Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, Tennessee
    • 33Vanderbilt Institute for Clinical and Translational Research, Vanderbilt University Medical Center, Nashville, Tennessee
    • 34Department of Biostatistics, Vanderbilt University Medical Center, Nashville, Tennessee
    JAMA. 2021;326(20):2043-2054. doi:10.1001/jama.2021.19499
    Key Points

    Question  Does prior COVID-19 vaccination reduce hospitalizations for COVID-19, and among patients hospitalized for COVID-19, does prior vaccination reduce disease severity?

    Findings  In a case-control study that included 4513 hospitalized adults in 18 US states, hospitalization for a COVID-19 diagnosis compared with an alternative diagnosis was associated with an adjusted odds ratio (aOR) of 0.15 for full vaccination with an authorized or approved mRNA COVID-19 vaccine. Among adults hospitalized for COVID-19, progression to death or invasive mechanical ventilation was associated with an aOR of 0.33 for full vaccination; both ORs were statistically significant.

    Meaning  Vaccination with an mRNA COVID-19 vaccine was significantly less likely among patients with COVID-19 hospitalization and with disease progression, consistent with risk reduction among vaccine breakthrough infections.

    Abstract

    Importance  A comprehensive understanding of the benefits of COVID-19 vaccination requires consideration of disease attenuation, determined as whether people who develop COVID-19 despite vaccination have lower disease severity than unvaccinated people.

    Objective  To evaluate the association between vaccination with mRNA COVID-19 vaccines—mRNA-1273 (Moderna) and BNT162b2 (Pfizer-BioNTech)—and COVID-19 hospitalization, and, among patients hospitalized with COVID-19, the association with progression to critical disease.

    Design, Setting, and Participants  A US 21-site case-control analysis of 4513 adults hospitalized between March 11 and August 15, 2021, with 28-day outcome data on death and mechanical ventilation available for patients enrolled through July 14, 2021. Date of final follow-up was August 8, 2021.

    Exposures  COVID-19 vaccination.

    Main Outcomes and Measures  Associations were evaluated between prior vaccination and (1) hospitalization for COVID-19, in which case patients were those hospitalized for COVID-19 and control patients were those hospitalized for an alternative diagnosis; and (2) disease progression among patients hospitalized for COVID-19, in which cases and controls were COVID-19 patients with and without progression to death or mechanical ventilation, respectively. Associations were measured with multivariable logistic regression.

    Results  Among 4513 patients (median age, 59 years [IQR, 45-69]; 2202 [48.8%] women; 23.0% non-Hispanic Black individuals, 15.9% Hispanic individuals, and 20.1% with an immunocompromising condition), 1983 were case patients with COVID-19 and 2530 were controls without COVID-19. Unvaccinated patients accounted for 84.2% (1669/1983) of COVID-19 hospitalizations. Hospitalization for COVID-19 was significantly associated with decreased likelihood of vaccination (cases, 15.8%; controls, 54.8%; adjusted OR, 0.15; 95% CI, 0.13-0.18), including for sequenced SARS-CoV-2 Alpha (8.7% vs 51.7%; aOR, 0.10; 95% CI, 0.06-0.16) and Delta variants (21.9% vs 61.8%; aOR, 0.14; 95% CI, 0.10-0.21). This association was stronger for immunocompetent patients (11.2% vs 53.5%; aOR, 0.10; 95% CI, 0.09-0.13) than immunocompromised patients (40.1% vs 58.8%; aOR, 0.49; 95% CI, 0.35-0.69) (P < .001) and weaker at more than 120 days since vaccination with BNT162b2 (5.8% vs 11.5%; aOR, 0.36; 95% CI, 0.27-0.49) than with mRNA-1273 (1.9% vs 8.3%; aOR, 0.15; 95% CI, 0.09-0.23) (P < .001). Among 1197 patients hospitalized with COVID-19, death or invasive mechanical ventilation by day 28 was associated with decreased likelihood of vaccination (12.0% vs 24.7%; aOR, 0.33; 95% CI, 0.19-0.58).

    Conclusions and Relevance  Vaccination with an mRNA COVID-19 vaccine was significantly less likely among patients with COVID-19 hospitalization and disease progression to death or mechanical ventilation. These findings are consistent with risk reduction among vaccine breakthrough infections compared with absence of vaccination.

    Introduction

    The COVID-19 pandemic, caused by SARS-CoV-2, continues to be a global public health crisis.1 The messenger RNA (mRNA) COVID-19 vaccines, including mRNA-1273 (Moderna) and BNT162b2 (Pfizer-BioNTech), are highly effective for preventing SARS-CoV-2 infections and COVID-19 hospitalizations.2-5 However, vaccine breakthrough COVID-19 (defined as development of COVID-19 despite prior full vaccination) is now being reported throughout the world.6 Because vaccine effectiveness is less than 100%, breakthrough cases are expected, and as vaccine coverage increases in the population, the ratio of vaccinated to unvaccinated cases will increase.

    A full interpretation of the protective benefits of COVID-19 vaccines must account for protection against SARS-CoV-2 infection, as well as against progression of disease severity after breakthrough infection.7,8 To date, evaluations of COVID-19 vaccines have primarily focused on prevention of symptomatic infection and hospitalizations.2,3,9-11 Once hospitalized, patients with COVID-19 can progress to more severe disease, including respiratory failure and death. SARS-CoV-2 infection in vaccinated persons is expected to trigger memory antibody and cellular responses owing to prior vaccination; these immune responses could mitigate disease progression, possibly preventing life-threatening organ failure and death.12,13 However, the association between prior vaccination and disease progression to the most severe forms of COVID-19 is not well understood.

    To estimate the benefits of mRNA vaccination against severe COVID-19, this study examined the association between prior vaccination and hospitalization for COVID-19, as well as the association between prior vaccination and progression to death or invasive mechanical ventilation among patients hospitalized for COVID-19.

    Methods
    Design and Setting

    This program was conducted by the Influenza and Other Viruses in the Acutely Ill (IVY) Network, a collaboration among 21 US hospitals in 18 states and the Centers for Disease Control and Prevention (investigators and collaborators are listed in eAppendix 1 in the Supplement).4,14 A total of 4513 patients hospitalized at the network hospitals between March 11, 2021, and August 15, 2021, were included. Data on hospitalizations were included for patients enrolled through August 15, 2021, and data on 28-day outcomes after hospitalization were included for patients enrolled through July 14, 2021. This analysis was an update to information previously published using earlier versions of the program’s data.4,15,16 STROBE guidelines for reporting were followed. This case-control study was determined to be a public health surveillance program, with waiver of participant informed consent by all participating institutions and the Centers for Disease Control and Prevention (CDC).

    We used a test-negative case-control design to assess the association between hospitalization for COVID-19 and prior vaccination with an mRNA COVID-19 vaccine. In this analysis, case patients were those hospitalized with COVID-19 and control patients were those hospitalized for other reasons.17-19 In a second analysis among only patients hospitalized with COVID-19, we assessed the association between COVID-19 disease progression and prior vaccination with an mRNA vaccine. In the second analysis, cases and controls were patients hospitalized with COVID-19 with and without progression to death or invasive mechanical ventilation, respectively.

    Participants

    Sites screened hospitalized adults aged 18 years and older for potential eligibility through daily review of hospital admission logs and electronic medical records (eAppendix 2 in the Supplement). COVID-19 cases included patients hospitalized with a clinical syndrome consistent with acute COVID-19 and a positive molecular or antigen test result for SARS-CoV-2 within 10 days after symptom onset.20 We included 2 control groups: “test-negative” controls were persons hospitalized with signs or symptoms consistent with acute COVID-19 but who tested negative for SARS-CoV-2 by molecular testing; and “syndrome-negative” controls were persons hospitalized without signs or symptoms consistent with acute COVID-19 and who tested negative for SARS-CoV-2 by molecular testing and were included as a secondary control group because of the theoretic risk of case misclassification in test-negative controls.21 Sites attempted to capture all cases admitted to the hospital during the surveillance period and targeted a case-control ratio of approximately 1:1 for each group of controls. Controls were selected from lists of eligible participants hospitalized within 2 weeks of enrollment of cases. Information on vaccination status was not collected until after patients were enrolled.

    Data Collection

    Demographic, clinical, and laboratory data were collected by trained personnel through standardized participant (or proxy) interviews and medical record reviews. Data on race and ethnicity were collected because the association between vaccination and COVID-19 may vary by race or ethnicity. Information on race and ethnicity was reported by participants during interviews conducted by research personnel using fixed categories.

    Details of COVID-19 vaccination, including dates and location, vaccine product, and lot number, were ascertained through a systematic process including patient or proxy interview and source verification. Sources of documentation included vaccination cards, hospital records, state vaccine registries, and vaccine records requested from clinics and pharmacies. Vaccine doses were classified as administered if source documentation was identified or if the patient or proxy reported a vaccine dose with a plausible date and location of vaccination.

    Laboratory Analysis

    Upper respiratory specimens were collected from enrolled patients, frozen, and shipped to a central laboratory at Vanderbilt University Medical Center (Nashville, Tennessee). Specimens underwent reverse transcriptase–polymerase chain reaction testing for SARS-CoV-2 nucleocapsid gene targets with standardized methods and interpretive criteria.22 Specimens positive for SARS-CoV-2 with a cycle threshold less than 32 were shipped to the University of Michigan (Ann Arbor, Michigan) for viral whole-genome sequencing using the ARTIC Network version 3 protocol on an Oxford Nanopore Technologies instrument (GridION).23 SARS-CoV-2 lineages were assigned with greater than 80% coverage with Phylogenetic Assignment of Named Global Outbreak Lineages (PANGOLIN).24

    Classification of Vaccination Status

    During this study period, the mRNA COVID-19 vaccines were administered as a series of at least 2 doses; participants were considered fully vaccinated 14 days after receipt of the second dose.25 Vaccination status was classified according to vaccine receipt before a reference date, which was the date of symptom onset for cases and test-negative controls and 4 days before hospital admission for syndrome-negative controls. Participants were classified as unvaccinated if they had received no vaccine doses before the reference date and fully vaccinated if they had received 2 or more mRNA vaccine doses at least 14 days before the reference date. Patients were excluded if they had previously received at least 1 dose of an mRNA vaccine but were not fully vaccinated, if they received a different type of COVID-19 vaccine, such as AD26.COV2.S (Janssen/Johnson & Johnson), or if they were vaccinated with a mixed vaccine schedule (ie, BNT1262b2 vaccine for 1 dose and mRNA-1273 vaccine for 1 dose).

    COVID-19 Severity Classification

    We collected data on severity for patients hospitalized with COVID-19. These outcome data were collected until the earlier of hospital discharge or 28 days after hospital admission. The primary classification of disease severity was a binary measure that divided patients into those who experienced death or invasive mechanical ventilation (progression to high disease severity) and those who did not (no progression to high disease severity).

    As a secondary assessment, we classified COVID-19 severity using a modified version of the World Health Organization COVID-19 Clinical Progression Scale, a commonly used ordinal scale for assessing COVID-19 severity that ranges from uninfected (level 0) and infected but asymptomatic (level 1) to death (level 9) (eAppendix 2 [eTable 1] in the Supplement). We classified severity according to the highest ordinal level that the patient experienced during the first 28 days of hospitalization. In this analysis of hospitalized patients, the highest severity level experienced could range from level 4 to 9, including hospitalized without supplemental oxygen (level 4), with standard supplemental oxygen (level 5), with high-flow nasal cannula or noninvasive ventilation (level 6), with invasive mechanical ventilation (level 7), or with mechanical ventilation and additional organ support (extracorporeal membrane oxygenation, vasopressors, or new kidney replacement therapy; level 8); and in-hospital death (level 9).

    We also evaluated in-hospital receipt of treatments used for severe COVID-19 (corticosteroids, remdesivir, COVID-19 convalescent plasma, tocilizumab, or baricitinib) according to a binary category of no COVID-19 treatments vs 1 or more.

    We also characterized COVID-19 severity by hospital length of stay while accounting for the competing risk of death.

    Statistical Analysis

    For the association between COVID-19 hospitalization and prior vaccination, we compared the odds of being fully vaccinated with an mRNA vaccine (exposed) vs being unvaccinated (unexposed) in cases hospitalized with COVID-19 vs controls hospitalized with other conditions. Patients from the test-negative and syndrome-negative control groups were pooled for this analysis in accordance with a prior analysis showing similar results with each control group individually.4,15 A mixed-effects logistic regression model was generated, treating enrolling site as a random effect and with the following covariables: admission date (biweekly intervals), age, sex, and race and ethnicity. In this model, an adjusted odds ratio (aOR) less than 1.0 indicated that COVID-19 hospitalization was associated with reduced likelihood of vaccination. The aOR was used to estimate vaccine effectiveness for the prevention of COVID-19 hospitalizations via the following equation: vaccine effectiveness = (1 − aOR) × 100%.4,26

    The association between COVID-19 hospitalization and prior vaccination was also evaluated in stratified secondary analyses, including by immunocompetent vs immunocompromised, age group (18-49, 50-64, or ≥65 years), vaccine type (mRNA-1273 or BNT162b2 vaccine), and time between second vaccine dose and illness onset (14-120 days; >120 days). Additional models were constructed to evaluate interactions between vaccination and exposure variable group. For immunocompetent vs immunocompromised status and age groups, we added interaction terms between vaccination and the stratifying variable to the regression model. For vaccine type and time since vaccination, the vaccination exposure variable was replaced with a product variable (unvaccinated, vaccinated with mRNA-1273, and vaccinated with BNT162b2) or time variable (unvaccinated, vaccinated 14-120 days before illness onset, and vaccinated >120 days before illness onset). P values for comparisons across subgroups were calculated for the interaction term between the exposure variable and vaccination status (immunocompetent vs immunocompromised; age group) or with the pwcompare Stata function for pairwise comparisons (vaccine product; time between second vaccine dose and illness onset). In addition, separate assessments were conducted evaluating the association between hospitalization with SARS-CoV-2 variants (B.1.1.7 [Alpha]; B.1617.2 or AY [Delta]) and period of illness onset (March to June 2021, which had predominant Alpha variant circulation; July to August 2021, which had predominant Delta variant circulation).27

    Among patients hospitalized with COVID-19 through July 14, 2021, the association between progression to death or invasive mechanical ventilation and prior mRNA vaccination was calculated with multivariable logistic regression adjusted for the following covariables: age, sex, race and ethnicity, and number of medical comorbidities by category (eTable 2 in the Supplement). The association between death alone and prior vaccination was similarly calculated with multivariable logistic regression evaluating the odds of vaccination among patients with COVID-19 who died vs survived.

    In a subgroup analysis to assess disease progression among COVID-19 patients admitted with hypoxemia, the association between death or invasive mechanical ventilation and prior vaccination was calculated among the subgroup of patients who received oxygen therapy or had oxygen saturation less than 92% as measured by pulse oximetry within 24 hours of hospital admission.

    A multivariable proportional odds model was used to compare the highest severity level experienced on the World Health Organization COVID-19 Clinical Progression Scale between patients with vaccine breakthrough COVID-19 and unvaccinated patients with COVID-19, using the same covariables described earlier. An aOR less than 1.0 for this model indicated lower odds of vaccinated patients’ experiencing higher severity levels on the ordinal scale compared with unvaccinated patients.

    For receipt of in-hospital COVID-19 treatments, a multivariable logistic regression model was constructed to calculate the aOR of vaccination among patients who did vs did not receive at least 1 COVID-19 treatment.

    To assess hospital length of stay, we calculated the probability of hospital discharge within 28 days after admission in vaccinated vs unvaccinated patients with COVID-19, using a Fine-Gray time-to-event analysis. We developed cumulative incidence function curves, with discharge from the hospital as the event of interest, death as the competing event, and patients who remained hospitalized censored at 28 days.28 Point estimates of subdistribution adjusted hazard ratios were reported.

    Missing data for illness onset was imputed according to the median number of days between illness onset and hospital admission for study patients within the same participant group. Imputation was not used for other variables; the number of patients with missing data was reported. Because of the potential for type I error owing to multiple comparisons, findings for analyses of secondary end points should be interpreted as exploratory. Statistical significance was indicated by 95% CIs not containing the null or a 2-sided P < .05. Stata version 16 was used for statistical analysis.

    Results
    Participants

    During March 11, 2021, to August 15, 2021, 5479 patients were enrolled from 21 hospitals; 966 patients were excluded from this analysis, with the most common reasons for exclusion being receipt of at least 1 mRNA vaccine but not being fully vaccinated (n = 547) and receipt of a COVID-19 vaccine other than an mRNA vaccine (n = 194) (Figure 1). The analytic population included 4513 patients (median age, 59 years [IQR, 45-69]; 2202 [48.8%] women; 23.0% non-Hispanic Black individuals, 15.9% Hispanic individuals, and 20.1% with an immunocompromising condition), including 1983 cases with COVID-19 and 2530 controls without it (1359 test-negative controls and 1171 syndrome-negative controls).

    Among 1983 COVID-19 case patients, vaccine breakthrough patients compared with unvaccinated patients tended to be older (median age 67 vs 53 years), were more likely to be White non-Hispanic (64.0% vs 43.0%), and were more likely to be immunocompromised (40.8% vs 11.5%) (Table). Among 1700 fully vaccinated patients (including both COVID-19 cases and controls), 1036 (60.9%) received the BNT162b2 vaccine and 664 (39.1%) received the mRNA-1273 vaccine; 1666 (98.0%) vaccinated patients had source documentation of vaccine doses and 34 (2.0%) had plausible self-report only. Full vaccination was less common in COVID-19 case patients (15.8%) than controls without COVID-19 (54.8%) (absolute difference, −39.0%; 95% CI, −41.5% to −36.4%).

    Among 730 COVID-19 case patient specimens that had SARS-CoV-2 lineage determined, 245 (33.6%) were identified as B.1.1.7 (Alpha) variant, 335 (45.9%) as B.1.617.2 or AY group (Delta) variant, and 150 (20.5%) as other variants. The predominant variant shifted from Alpha to Delta in mid-June 2021 (eFigure 1 in the Supplement), and in this analysis the period between July 1, 2021, and August 15, 2021, was considered dominated by Delta variant circulation.

    Association Between COVID-19 Hospitalizations and mRNA Vaccination

    Overall, COVID-19 hospitalization was strongly associated with a lower likelihood of vaccination, with an aOR of 0.15 (95% CI, 0.13-0.18) (Figure 2). Effect modification was observed by immunocompromised status, with a greater magnitude of association for patients without immunocompromising conditions (aOR, 0.10; 95% CI, 0.09-0.13) than with immunocompromising conditions (aOR, 0.49; 95% CI, 0.35-0.69) (P < .001) (Figure 2; eTable 3 in the Supplement). The magnitude of association was higher for the mRNA-1273 vaccine (aOR, 0.11; 95% CI, 0.08-0.14) than the BNT162b2 vaccine (aOR, 0.19; 95% CI, 0.16-0.23) (P < .001), with this difference largely because of a lower aOR for patients at more than 120 days since vaccination with the mRNA-1273 vaccine (aOR, 0.15; 95% CI, 0.09-0.23; median 141 days from vaccine dose 2 to illness onset) than with the BNT162b2 vaccine (aOR, 0.36; 95% CI, 0.27-0.49; median 143 days from vaccine dose 2 to illness onset) (P < .001). A lower aOR for the mRNA-1273 vaccine compared with the BNT162b2 vaccine for patients with illness onset greater than 120 days after vaccination was observed after restricting to patients without immunocompromising conditions and further stratifying these patients into younger (18-64 years) and older (≥65 years) groups (eTables 3-5 in the Supplement). By SARS-CoV-2 variants sequenced, COVID-19 hospitalization was strongly associated with lower likelihood of vaccination for both the B.1.1.7 (Alpha) variant (aOR, 0.10; 95% CI, 0.06-0.16) and B.1.617.2 or AY (Delta) variant (aOR, 0.14; 95% CI, 0.10-0.21).

    Breakthrough COVID-19 Hospitalizations and Association Between Disease Progression and mRNA Vaccination

    Among 1197 patients hospitalized with COVID-19 between March 11, 2021, and July 14, 2021, 142 (11.9%) were vaccinated breakthrough cases and 1055 (88.1%) were unvaccinated. Compared with unvaccinated cases, vaccine breakthrough cases were older and had more chronic medical conditions (Table). Compared with unvaccinated cases, vaccine breakthrough cases less commonly received ICU-level care (24.6% vs 40.1%; absolute difference, −15.5%; 95% CI, −23.1% to −7.8%; P < .001) and invasive mechanical ventilation (7.7% vs 23.0%; absolute difference, −15.3%; 95% CI, −20.4% to −10.2%; P < .001) (eTable 6 in the Supplement).

    Unvaccinated patients accounted for 93.9% (261/278) of cases with disease progression to death or invasive mechanical ventilation. The composite of death or mechanical ventilation was experienced by 17 of 142 (12.0%) vaccine breakthrough cases and 261 of 1055 (24.7%) unvaccinated cases. Among patients hospitalized with COVID-19, death or invasive mechanical ventilation was associated with a lower likelihood of vaccination (aOR, 0.33; 95% CI, 0.19-0.58) (Figure 3). Restricting to cases admitted with hypoxemia (n = 902, 75.4% of cases), death or mechanical ventilation was also associated with a lower likelihood of vaccination (aOR, 0.30; 95% CI, 0.16-0.58). Receipt of 1 or more COVID-19–related therapeutics during hospitalization was also associated with a lower likelihood of vaccination (aOR, 0.32; 95% CI, 0.20-0.52) (eTable 7 in the Supplement).

    Unvaccinated patients accounted for 91.0% (91/100) of deaths among patients with COVID-19 in this study. Death occurred in 9 of 142 (6.3%) vaccine breakthrough cases and 91 of 1055 (8.6%) unvaccinated patients with COVID-19. Progression to death after COVID-19 hospitalization was associated with a lower likelihood of vaccination (aOR, 0.41; 95% CI, 0.19-0.88).

    According to the World Health Organization COVID-19 Clinical Progression Scale, the highest level of disease severity experienced was significantly lower among vaccine breakthrough cases than unvaccinated cases (aOR, 0.36; 95% CI, 0.25-0.51) (eFigure 2 in the Supplement). Hospital discharge alive within 28 days of hospital admission was experienced by 125 of 142 (88.0%) vaccine breakthrough cases and 814 of 1055 (77.2%) unvaccinated cases (P = .003). In the competing risk model evaluating time to hospital discharge with a competing risk of death, vaccine breakthrough cases had a higher rate of hospital discharge (ie, shorter length of stay) than unvaccinated cases (subdistribution adjusted hazard ratio, 1.73; 95% CI, 1.42-2.10). These findings remained consistent when stratified by age and immunocompromised status (Figure 4).

    Discussion

    In this analysis of adults hospitalized in 21 US hospitals between mid-March and mid-August 2021, vaccination with an mRNA COVID-19 vaccine was significantly less likely among patients with COVID-19 than other conditions and among those with COVID-19 who progressed to death or mechanical ventilation than those with COVID-19 who did not have disease progression. These findings are consistent with risk reduction of developing severe COVID-19 among patients with vaccine breakthrough infections compared with absence of vaccination.

    The aOR in this analysis corresponds to an estimated overall vaccine effectiveness of 85% for mRNA vaccines to prevent COVID-19 hospitalizations. The findings also correspond to an estimated vaccine effectiveness of 90% for the immunocompetent population and 86% for COVID-19 hospitalizations caused by the Delta variant. When the mRNA-1273 and BNT162b2 vaccines were compared, estimated vaccine effectiveness was similar within 120 days of vaccination. In contrast, beyond 120 days, the results corresponded to an estimated effectiveness of 85% for the mRNA-1273 and 64% for the BNT162b2 vaccine to prevent COVID-19 hospitalizations.

    Among patients hospitalized with COVID-19, the outcome of death or invasive mechanical ventilation was associated with a lower likelihood of vaccination. These data suggest that the COVID-19 mRNA vaccines may attenuate disease severity among patients who develop COVID-19 despite vaccination, and the total benefits of vaccination exceed those estimated from the prevention of hospitalization alone.

    These data complement vaccine trials and emerging postmarketing data that suggest receipt of mRNA vaccination is associated with risk reduction of severe COVID-19. The mRNA vaccine clinical trials were not powered to address severe disease, including complications after hospitalization.29,30 Observational postmarketing studies, including this analysis, have consistently demonstrated a strong association between vaccination and risk reductions in COVID-19 hospitalization in immunocompetent individuals, suggesting that the high efficacy observed in mRNA clinical trials translates into beneficial effects in the community setting.2,3,10 As vaccine coverage increases, breakthrough cases are also expected to increase. Concerns about vaccine failure against severe disease are especially likely among patients with complicated comorbidities who are overrepresented in inpatient settings compared with the general population. This analysis demonstrated a strong association between hospitalization for COVID-19 and lower likelihood of vaccination. Moreover, disease progression to critical illness after hospital admission was associated with a lower likelihood of vaccination among a population representing typical hospitalized patients in the US, which included high prevalence of medical comorbidities and multimorbidity.

    Recent surges of COVID-19 cases from the SARS-CoV-2 Delta variant and signs of potential waning protection over time from a 2-dose mRNA vaccine series have prompted policy discussions about additional vaccine doses.16,31,32 Several findings from this study could help inform ongoing policy discussions on implementing booster vaccination and guiding future research. First, the association between mRNA vaccination and reduced risk of COVID-19 hospitalization was substantially weaker in the immunocompromised population than the immunocompetent one, supporting recent recommendations for additional vaccine doses among immunocompromised persons.33 Second, vaccine breakthrough COVID-19 hospitalization appeared to be more common with the BNT162b2 than the mRNA-1273 mRNA vaccine in this analysis. Third, the association between vaccination with the BNT162b2 vaccine and reduced risk of COVID-19 hospitalization declined after 4 months from vaccination, potentially indicating clinically important waning of protection over time, including for severe COVID-19.

    Similar product-specific differences between the mRNA-1273 and BNT162b2 vaccines have also been reported in other recent observational studies in inpatient and outpatient settings.34 Furthermore, recent immunologic studies have shown higher antibody responses after vaccination with mRNA-1273 compared with BNT162b2.16,35 These differences may be related to higher antigen content in the mRNA-1273 vaccine, a longer recommended interval between vaccine doses (4 weeks for mRNA-1273 and 3 weeks for BNT162b2), or both. However, differences in the population vaccinated with the mRNA-1273 and BNT162b2 vaccines could also contribute to observed differences in vaccine breakthrough. Differences in memory B- and T-cell responses between COVID-19 vaccines have not been assessed and may be robust and similar for both mRNA vaccine products.36

    An unresolved question is whether observed differences by product and time since vaccination are due to declining immunity, evasion of immunity by the Delta variant, or a combination of the 2. Disentangling the mechanism of decline in protection could inform decisions on whether improving protection would be better achieved through booster vaccine dosing of the same products or administration of new vaccine formulations with a strain change, as is done with seasonal influenza vaccines.37 These issues are epidemiologically challenging to disentangle with certainty because the Alpha variant preceded Delta circulation, and thus patients infected with the Delta variant also were more likely to have been vaccinated longer ago. The association between prior vaccination and COVID-19 hospitalization was strong for sequenced Alpha and Delta variants. Because the numbers sequenced were smaller than the full cohort numbers, the report also evaluated the magnitude of association by time since vaccination between March and June when the Alpha variant circulated vs July and August when the Delta variant predominated. The association between mRNA vaccination and reduced risk for COVID-19 hospitalization observed during the Delta variant circulation was high for participants with illness onset within 120 days of vaccination and lower for participants with illness onset after that period. This suggests that waning immunity rather than primary evasion by the Delta variant may be a driving mechanism of reduced vaccine protection observed. Whether this decline is restricted to specific high-risk subpopulations or vaccine types or is due to unmeasured confounding warrants further investigation.

    Limitations

    This study has several limitations. First, although several relevant confounders were controlled for, unmeasured confounding in this observational case-control study could have occurred. Second, progression of COVID-19 to high severity was measured with multiple outcomes that considered death, organ failures, oxygen use, and duration of hospitalization. Although these measures do not comprehensively characterize disease severity, they capture life-threatening complications of COVID-19. Third, this analysis included only hospitalized patients and cannot inform whether vaccination attenuates COVID-19 severity among outpatients. Fourth, if vaccine breakthrough cases were systematically more likely to be hospitalized for COVID-19 of lesser severity than unvaccinated patients, our analyses of the association between vaccination and severe disease could be confounded by an admission bias. However, lower risk of progression to severe disease among vaccine breakthrough cases was sustained after the population was limited to patients who were admitted with hypoxemia. Fifth, sample size limitations prevented assessments of disease attenuation stratified by vaccine type, SARS-CoV-2 variant, and time since vaccination.

    Conclusions

    Vaccination with an mRNA COVID-19 vaccine was significantly less likely among patients with COVID-19 hospitalization and with disease progression to death or invasive mechanical ventilation. These findings are consistent with risk reduction of developing severe COVID-19 among vaccine breakthrough infections compared with absence of vaccination.

    Back to top
    Article Information

    Corresponding Author: Mark W. Tenforde, MD, PhD, Centers for Disease Control and Prevention, 1600 Clifton Rd, Mailstop 24/7, Atlanta, GA 30329 (mtenforde@cdc.gov).

    Accepted for Publication: October 13, 2021.

    Published Online: November 4, 2021. doi:10.1001/jama.2021.19499

    Author Contributions: Dr Tenforde had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs Tenforde and Self and Ms Adams contributed equally to this work as lead authors.

    Concept and design: Tenforde, Self, Adams, Ginde, Talbot, Hager, Exline, Gong, Peltan, Brown, Busse, Qadir, Grijalva, Rice, Kobayashi, Verani, Patel.

    Acquisition, analysis, or interpretation of data: Tenforde, Self, Adams, Gaglani, Ginde, McNeal, Ghamande, Douin, Talbot, Casey, Mohr, Zepeski, Shapiro, Gibbs, Files, Hager, Shehu, Prekker, Erickson, Exline, Gong, Mohamed, Henning, Steingrub, Peltan, Brown, Martin, Monto, Khan, Hough, Busse, ten Lohuis, Duggal, Wilson, Gordon, Qadir, Chang, Mallow, Rivas, Babcock, Kwon, Halasa, Chappell, Lauring, Grijalva, Rice, Jones, Stubblefield, Baughman, Womack, Rhoads, Lindsell, Hart, Zhu, Olson, Patel.

    Drafting of the manuscript: Tenforde, Self, Adams, Mohamed, Khan, Mallow, Patel.

    Critical revision of the manuscript for important intellectual content: Tenforde, Self, Adams, Gaglani, Ginde, McNeal, Ghamande, Douin, Talbot, Casey, Mohr, Zepeski, Shapiro, Gibbs, Files, Hager, Shehu, Prekker, Erickson, Exline, Gong, Henning, Steingrub, Peltan, Brown, Martin, Monto, Khan, Hough, Busse, ten Lohuis, Duggal, Wilson, Gordon, Qadir, Chang, Mallow, Rivas, Babcock, Kwon, Halasa, Chappell, Lauring, Grijalva, Rice, Jones, Stubblefield, Baughman, Womack, Rhoads, Lindsell, Hart, Zhu, Olson, Kobayashi, Verani, Patel.

    Statistical analysis: Tenforde, Adams, Talbot, Casey, Monto, Grijalva, Lindsell, Zhu, Olson.

    Obtained funding: Self, Steingrub, Verani, Patel.

    Administrative, technical, or material support: Self, Mohr, Files, Hager, Exline, Gong, Khan, ten Lohuis, Duggal, Wilson, Gordon, Qadir, Mallow, Babcock, Jones, Baughman, Womack, Rhoads, Lindsell, Hart, Kobayashi, Verani, Patel.

    Supervision: Self, Mohr, Shapiro, Exline, Henning, Martin, Monto, Khan, ten Lohuis, Wilson, Gordon, Halasa, Chappell, Grijalva, Hart, Patel.

    Operational support for enrollment: McNeal.

    Data cleaning, data management: Olson.

    Supervision of staff at study site: Kwon.

    Conflict of Interest Disclosures: Dr Self reported receiving grants from the Centers for Disease Control and Prevention (CDC) (principal investigator of the primary funding contract from CDC for this work) during the conduct of the study. Dr Gaglani reported receiving grants from CDC, Vanderbilt University Medical Center, Baylor Scott & White Health (BSWH), and the IVY study during the conduct of the study; grants from CDC–BSWH HAIVEN influenza/COVID-19 vaccine effectiveness study, CDC–BSWH ambulatory US influenza/COVID-19 vaccine effectiveness study, CDC–Abt Associates BSWH RECOVER COVID-19/influenza study, and CDC–Westat BSWH VISION COVID-19/influenza study outside the submitted work; and Pfizer BSWH Independent Grants for Learning & Change for meningococcal vaccination of adolescents and an institutional contract with Janssen (BSWH Observational RSV Study in infants). Dr Ginde reported receiving grants from CDC during the conduct of the study; and grants from the National Institutes of Health (NIH), Department of Defense, and AbbVie outside the submitted work. Dr McNeal reported receiving grants from the CDC HAIVEN study group that become the IVY-3 study group during the conduct of the study. Dr Talbot reported receiving grants from CDC during the conduct of the study. Dr Casey reported receiving grants from NIH K23HL153584 outside the submitted work. Dr Mohr reported receiving grants from CDC during the conduct of the study. Dr Shapiro reported receiving grants from CDC during the conduct of the study. Dr Files reported receiving grants from CDC during the conduct of the study and personal fees from Cytovale and Medpace outside the submitted work. Dr Prekker reported receiving grants from CDC during the conduct of the study. Dr Exline reported receiving a speaking honorarium from Abbott Laboratories outside the submitted work. Dr Gong reported receiving grants from CDC during the conduct of the study; grants from NIH to conduct clinical trials on COVID-19 and non–COVID-19 outside the submitted work; and data and safety monitoring board fees for participating in Regeneron trials outside the submitted work. Dr Henning reported receiving grants from CDC during the conduct of the study. Dr Peltan reported receiving grants from CDC during the conduct of the study; grants from NIH, Intermountain Research and Medical Foundation, and Janssen Pharmaceuticals outside the submitted work; and payment to Intermountain Medical Center for subject enrollment from Regeneron and Asahi Kasei Pharma outside the submitted work. Dr Brown reported receiving grants from CDC during the conduct of the study. Dr Martin reported receiving grants from CDC during the conduct of the study and personal fees from Pfizer outside the submitted work. Dr Khan reported receiving grants from United Therapeutics, Actelion Pharmaceuticals, Eli Lilly, Johnson & Johnson, Regeneron Pharmaceuticals, and Gilead Sciences outside the submitted work. Dr Hough reported receiving grants from CDC during the conduct of the study and grants from NIH outside the submitted work. Dr Wilson reported receiving grants from CDC/Vanderbilt during the conduct of the study. Dr Chang reported receiving personal fees from La Jolla Pharmaceuticals and PureTech Health outside the submitted work. Dr Babcock reported receiving grants from CDC during the conduct of the study. Dr Kwon reported receiving grants from NIH National Institute of Allergy and Infectious Diseases (award 1K23 AI137321-01A1) outside the submitted work. Dr Halasa reported receiving grants from CDC during the conduct of the study; grants from Sanofi outside the submitted work; and hemagglutination inhibition and microneutralization testing, vaccine donation, and grants from Quidel outside the submitted work. Dr Chappell reported receiving grants from CDC during the conduct of the study. Dr Lauring reported receiving consulting fees from Sanofi for an influenza antiviral and fees from Roche as a member of an influenza antiviral trial steering committee outside the submitted work. Dr Grijalva reported receiving a contract from CDC during the conduct of the study; consulting fees from Pfizer, Merck, and Sanofi; a contract from CDC, Campbell Alliance, and the Food and Drug Administration outside the submitted work; and grants from NIH and the Agency for Healthcare Research and Quality outside the submitted work. Dr Rice reported receiving grants from CDC during the conduct of the study and personal fees from Cumberland Pharmaceuticals, Sanofi, and Cytovale outside the submitted work. Dr Lindsell reported receiving grants from CDC to Vanderbilt University during the conduct of the study; grants from NIH to institution, grants from Department of Defense to institution, contracts to Vanderbilt University for research services from bioMérieux, Endpoint Health, and Entegrion. In addition, he had a patent for risk stratification in sepsis and septic shock, issued to Cincinnati Children's Hospital Medical Center. Dr Zhu reported receiving grants from CDC during the conduct of the study. No other disclosures were reported.

    Funding/Support: Primary funding for this study was provided by the CDC (75D30121F00002).

    Role of the Funder/Sponsor: Investigators from CDC were involved in all aspects of the study, including the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The CDC had the right to control decisions about publication via the CDC publication clearance process.

    Group Information: A full list of investigators and collaborators in the Influenza and Other Viruses in the Acutely Ill (IVY) Network is available in eAppendix 1 in the Supplement.

    Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.

    References
    1.
    Morin  L, Savale  L, Pham  T,  et al; Writing Committee for the COMEBAC Study Group.  Four-month clinical status of a cohort of patients after hospitalization for COVID-19.   JAMA. 2021;325(15):1525-1534. doi:10.1001/jama.2021.3331 PubMedGoogle Scholar
    2.
    Dagan  N, Barda  N, Kepten  E,  et al.  BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting.   N Engl J Med. 2021;384(15):1412-1423. doi:10.1056/NEJMoa2101765 PubMedGoogle ScholarCrossref
    3.
    Haas  EJ, Angulo  FJ, McLaughlin  JM,  et al.  Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data.   Lancet. 2021;397(10287):1819-1829. doi:10.1016/S0140-6736(21)00947-8 PubMedGoogle ScholarCrossref
    4.
    Tenforde  MW, Patel  MM, Ginde  AA,  et al; Influenza and Other Viruses in the Acutely Ill (IVY) Network.  Effectiveness of SARS-CoV-2 mRNA vaccines for preventing Covid-19 hospitalizations in the United States.   Clin Infect Dis. 2021;ciab687. doi:10.1093/cid/ciab687 PubMedGoogle Scholar
    5.
    Goldshtein  I, Nevo  D, Steinberg  DM,  et al.  Association between BNT162b2 vaccination and incidence of SARS-CoV-2 infection in pregnant women.   JAMA. 2021;326(8):728-735. doi:10.1001/jama.2021.11035 PubMedGoogle ScholarCrossref
    6.
    CDC COVID-19 Vaccine Breakthrough Case Investigations Team.  COVID-19 vaccine breakthrough infections reported to CDC—United States, January 1–April 30, 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(21):792-793. doi:10.15585/mmwr.mm7021e3 PubMedGoogle ScholarCrossref
    7.
    Halloran  ME, Longini  IM  Jr, Struchiner  CJ.  Design and interpretation of vaccine field studies.   Epidemiol Rev. 1999;21(1):73-88. doi:10.1093/oxfordjournals.epirev.a017990 PubMedGoogle ScholarCrossref
    8.
    WHO Working Group on the Clinical Characterisation and Management of COVID-19 infection.  A minimal common outcome measure set for COVID-19 clinical research.   Lancet Infect Dis. 2020;20(8):e192-e197. doi:10.1016/S1473-3099(20)30483-7 PubMedGoogle ScholarCrossref
    9.
    Hall  VJ, Foulkes  S, Saei  A,  et al; SIREN Study Group.  COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study.   Lancet. 2021;397(10286):1725-1735. doi:10.1016/S0140-6736(21)00790-X PubMedGoogle ScholarCrossref
    10.
    Sheikh  A, McMenamin  J, Taylor  B, Robertson  C; Public Health Scotland and the EAVE II Collaborators.  SARS-CoV-2 Delta VOC in Scotland: demographics, risk of hospital admission, and vaccine effectiveness.   Lancet. 2021;397(10293):2461-2462. doi:10.1016/S0140-6736(21)01358-1 PubMedGoogle ScholarCrossref
    11.
    Thompson  MG, Burgess  JL, Naleway  AL,  et al.  Prevention and attenuation of Covid-19 with the BNT162b2 and mRNA-1273 vaccines.   N Engl J Med. 2021;385(4):320-329. doi:10.1056/NEJMoa2107058 PubMedGoogle ScholarCrossref
    12.
    Cromer  D, Juno  JA, Khoury  D,  et al.  Prospects for durable immune control of SARS-CoV-2 and prevention of reinfection.   Nat Rev Immunol. 2021;21(6):395-404. doi:10.1038/s41577-021-00550-x PubMedGoogle ScholarCrossref
    13.
    Sadarangani  M, Marchant  A, Kollmann  TR.  Immunological mechanisms of vaccine-induced protection against COVID-19 in humans.   Nat Rev Immunol. 2021;21(8):475-484. doi:10.1038/s41577-021-00578-z PubMedGoogle ScholarCrossref
    14.
    Patel  MM, Thornburg  NJ, Stubblefield  WB,  et al.  Change in antibodies to SARS-CoV-2 over 60 days among health care personnel in Nashville, Tennessee.   JAMA. 2020;324(17):1781-1782. doi:10.1001/jama.2020.18796 PubMedGoogle ScholarCrossref
    15.
    Tenforde  MW, Self  WH, Naioti  EA,  et al; IVY Network Investigators; IVY Network.  Sustained effectiveness of Pfizer-BioNTech and Moderna vaccines against COVID-19 associated hospitalizations among adults—United States, March-July 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(34):1156-1162. doi:10.15585/mmwr.mm7034e2 PubMedGoogle ScholarCrossref
    16.
    Self  WH, Tenforde  MW, Rhoads  JP,  et al; IVY Network.  Comparative effectiveness of Moderna, Pfizer-BioNTech, and Janssen (Johnson & Johnson) vaccines in preventing COVID-19 hospitalizations among adults without immunocompromising conditions—United States, March-August 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(38):1337-1343. doi:10.15585/mmwr.mm7038e1 PubMedGoogle ScholarCrossref
    17.
    Grijalva  CG, Feldstein  LR, Talbot  HK,  et al; Influenza and Other Viruses in the Acutely Ill (IVY) Network.  Influenza vaccine effectiveness for prevention of severe influenza-associated illness among adults in the United States, 2019-2020: a test-negative study.   Clin Infect Dis. 2021;(May):ciab462. PubMedGoogle Scholar
    18.
    Tenforde  MW, Talbot  HK, Trabue  CH,  et al.  Influenza vaccine effectiveness against hospitalization in the United States, 2019-2020.   J Infect Dis. 2021;224(5):813-820. doi:10.1093/infdis/jiaa800PubMedGoogle ScholarCrossref
    19.
    Grijalva  CG, Zhu  Y, Williams  DJ,  et al.  Association between hospitalization with community-acquired laboratory-confirmed influenza pneumonia and prior receipt of influenza vaccination.   JAMA. 2015;314(14):1488-1497. doi:10.1001/jama.2015.12160 PubMedGoogle ScholarCrossref
    20.
    Lauer  SA, Grantz  KH, Bi  Q,  et al.  The incubation period of coronavirus disease 2019 (COVID-19) from publicly reported confirmed cases: estimation and application.   Ann Intern Med. 2020;172(9):577-582. doi:10.7326/M20-0504 PubMedGoogle ScholarCrossref
    21.
    Lewnard  JA, Patel  MM, Jewell  NP,  et al.  Theoretical framework for retrospective studies of the effectiveness of SARS-CoV-2 vaccines.   Epidemiology. 2021;32(4):508-517. doi:10.1097/EDE.0000000000001366 PubMedGoogle ScholarCrossref
    22.
    Food and Drug Administration. CDC 2019-Novel Coronavirus (2019-nCoV) Real-Time RT-PCR Diagnostic Panel. Accessed May 26, 2021. https://www.fda.gov/media/134922/download
    23.
    Quick  J. nCoV2-2019 sequencing protocol v3 (LoCost) V.3. protocols.io. Accessed May 26, 2021. https://www.protocols.io/view/ncov-2019-sequencing-protocol-%20v3-locost-bh42j8ye
    24.
    Rambaut  A, Holmes  EC, O’Toole  Á,  et al.  A dynamic nomenclature proposal for SARS-CoV-2 lineages to assist genomic epidemiology.   Nat Microbiol. 2020;5(11):1403-1407. doi:10.1038/s41564-020-0770-5PubMedGoogle ScholarCrossref
    25.
    Centers for Disease Control and Prevention. Interim clinical considerations for use of COVID-19 vaccines currently authorized in the United States. Accessed June 1, 2021. https://www.cdc.gov/vaccines/covid-19/clinical-considerations/covid-19-vaccines-us.html#footnote-01
    26.
    Foppa  IM, Haber  M, Ferdinands  JM, Shay  DK.  The case test-negative design for studies of the effectiveness of influenza vaccine.   Vaccine. 2013;31(30):3104-3109. doi:10.1016/j.vaccine.2013.04.026 PubMedGoogle ScholarCrossref
    27.
    Centers for Disease Control and Prevention. COVID data tracker. Accessed August 6, 2021. https://covid.cdc.gov/covid-data-tracker/#datatracker-home
    28.
    Fine  JP, Gray  RJ.  A proportional hazards model for the subdistribution of a competing risk.   J Am Stat Assoc. 1999;94(446):496-509. doi:10.1080/01621459.1999.10474144 Google ScholarCrossref
    29.
    Baden  LR, El Sahly  HM, Essink  B,  et al; COVE Study Group.  Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine.   N Engl J Med. 2021;384(5):403-416. doi:10.1056/NEJMoa2035389 PubMedGoogle ScholarCrossref
    30.
    Polack  FP, Thomas  SJ, Kitchin  N,  et al; C4591001 Clinical Trial Group.  Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine.   N Engl J Med. 2020;383(27):2603-2615. doi:10.1056/NEJMoa2034577 PubMedGoogle ScholarCrossref
    31.
    Food and Drug Administration. Vaccines and Related Biological Products Advisory Committee September 17, 2021 meeting announcement. Accessed September 21, 2021. https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-september-17-2021-meeting-announcement
    32.
    Nanduri  S, Pilishvili  T, Derado  G,  et al.  Effectiveness of Pfizer-BioNTech and Moderna vaccines in preventing SARS-CoV-2 infection among nursing home residents before and during widespread circulation of the SARS-CoV-2 B.1.617.2 (Delta) variant—National Healthcare Safety Network, March 1–August 1, 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(34):1163-1166. doi:10.15585/mmwr.mm7034e3 PubMedGoogle ScholarCrossref
    33.
    Centers for Disease Control and Prevention. COVID-19 vaccines for moderately to severely immunocompromised people. Accessed September 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/vaccines/recommendations/immuno.html
    34.
    Grannis  SJ, Rowley  EA, Ong  TC,  et al; VISION Network.  Interim estimates of COVID-19 vaccine effectiveness against COVID-19–associated emergency department or urgent care clinic encounters and hospitalizations among adults during SARS-CoV-2 B.1.617.2 (Delta) variant predominance—nine states, June-August 2021.   MMWR Morb Mortal Wkly Rep. 2021;70(37):1291-1293. doi:10.15585/mmwr.mm7037e2 PubMedGoogle ScholarCrossref
    35.
    Steensels  D, Pierlet  N, Penders  J, Mesotten  D, Heylen  L.  Comparison of SARS-CoV-2 antibody response following vaccination with BNT162b2 and mRNA-1273.   JAMA. 2021;e2115125. PubMedGoogle Scholar
    36.
    Dan  JM, Mateus  J, Kato  Y,  et al.  Immunological memory to SARS-CoV-2 assessed for up to 8 months after infection.   Science. 2021;371(6529):eabf4063. doi:10.1126/science.abf4063 PubMedGoogle Scholar
    37.
    Flannery  B, Kondor  RJG, Chung  JR,  et al.  Spread of antigenically drifted influenza A(H3N2) viruses and vaccine effectiveness in the United States during the 2018-2019 season.   J Infect Dis. 2020;221(1):8-15. doi:10.1093/infdis/jiz543 PubMedGoogle ScholarCrossref
    ×