Analysis of the Effectiveness of the Ad26.COV2.S Adenoviral Vector Vaccine for Preventing COVID-19 | Infectious Diseases | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Effectiveness of the Ad26.COV2.S Vaccine and Comparison of COVID-19 Severity Between Vaccinated and Unvaccinated Patients
Effectiveness of the Ad26.COV2.S Vaccine and Comparison of COVID-19 Severity Between Vaccinated and Unvaccinated Patients

A, Real-world evidence supporting the effectiveness of the Ad26.COV2.S (J&J) COVID-19 vaccine. B, Comparison of COVID-19 severity between vaccinated and unvaccinated patients. J&J indicates Johnson & Johnson; PCR, polymerase chain reaction.

Figure 2.  Kaplan-Meier Analyses of Cumulative Incidence Proportion of SARS-CoV-2 Infections Between Vaccinated and Unvaccinated Individuals
Kaplan-Meier Analyses of Cumulative Incidence Proportion of SARS-CoV-2 Infections Between Vaccinated and Unvaccinated Individuals

A, SARS-CoV-2 infection at any time after vaccination. Cumulative incidence proportion at time t is the estimated proportion of patients who experience the outcome on or before time t (ie, 1 minus the standard Kaplan-Meier survival estimate with onset on any day after the date of vaccination): 60 of 8889 vaccinated and 2236 of 88 898 unvaccinated (log-rank P = .001). B, SARS-CoV-2 infection within 14 days after infection. Cumulative incidence proportion: 18 vaccinated and 473 unvaccinated (log-rank P = 6.93 × 10-4). PCR indicates polymerase chain reaction.

Figure 3.  Temporal Distribution of Positive Polymerase Chain Reaction (PCR) Test Results in Vaccinated and Unvaccinated Individuals
Temporal Distribution of Positive Polymerase Chain Reaction (PCR) Test Results in Vaccinated and Unvaccinated Individuals

A, Distribution of the time to the first positive PCR test result. B, Distribution of the date of positive PCR test results.

Table 1.  Clinical Characteristics of Vaccinated and 1:10 Propensity-Matched Unvaccinated Cohorts
Clinical Characteristics of Vaccinated and 1:10 Propensity-Matched Unvaccinated Cohorts
Table 2.  SARS-CoV-2 Incidence Rates in Vaccinated Cohort and 1:10 Propensity-Matched Unvaccinated Cohorts and Corresponding Vaccine Effectiveness
SARS-CoV-2 Incidence Rates in Vaccinated Cohort and 1:10 Propensity-Matched Unvaccinated Cohorts and Corresponding Vaccine Effectiveness
1.
World Health Organization. Coronavirus disease (COVID-19) pandemic. Accessed April 25, 2021. https://www.who.int/emergencies/diseases/novel-coronavirus-2019
2.
Centers for Disease Control and Prevention. COVID data tracker. Accessed April 25, 2021. https://covid.cdc.gov/covid-data-tracker/
3.
Sadoff  J, Gray  G, Vandebosch  A, et al; ENSEMBLE Study Group. Safety and efficacy of single-dose Ad26.COV2.S vaccine against Covid-19. N Engl J Med. 2021;384(23):2187-2201. doi:10.1056/NEJMoa2101544PubMed
4.
Pawlowski  C, Wagner  T, Puranik  A,  et al.  Inference from longitudinal laboratory tests characterizes temporal evolution of COVID-19-associated coagulopathy (CAC).   eLife. 2020;9:e59209. doi:10.7554/eLife.59209 PubMedGoogle Scholar
5.
Pawlowski  C, Venkatakrishnan  AJ, Ramudu  E,  et al.  Pre-existing conditions are associated with COVID-19 patients’ hospitalization, despite confirmed clearance of SARS-CoV-2 virus.   EClinicalMedicine. 2021;34:100793. doi:10.1016/j.eclinm.2021.100793 PubMedGoogle Scholar
6.
Pawlowski  C, Lenehan  P, Puranik  A,  et al.  FDA-authorized mRNA COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system.   Med (N Y). 2021;2(8):979-992. doi:10.1016/j.medj.2021.06.007PubMedGoogle Scholar
7.
McMurry  R, Lenehan  P, Awasthi  S,  et al.  Real-time analysis of a mass vaccination effort confirms the safety of FDA-authorized mRNA COVID-19 vaccines.   Med (N Y). 2021;2(8):965-978. doi:10.1016/j.medj.2021.06.006PubMedGoogle Scholar
8.
Pawlowski  C, Rincón-Hekking  J, Awasthi  S, et al. Cerebral venous sinus thrombosis is not significantly linked to COVID-19 vaccines or non-COVID vaccines in a large multi-state health system.  J Stroke Cerebrovasc Dis. 2021;16(1):105923. doi:10.1016/j.jstrokecerebrovasdis.2021.105923
9.
Austin  PC.  An introduction to propensity score methods for reducing the effects of confounding in observational studies.   Multivariate Behav Res. 2011;46(3):399-424. doi:10.1080/00273171.2011.568786 PubMedGoogle ScholarCrossref
10.
Austin  PC. A comparison of 12 algorithms for matching on the propensity score.  Stat Med. 2014;33(6):1057-1069. doi:10.1002/sim.6004PubMed
11.
Sahai  H, Khurshid  A.  Statistics in Epidemiology: Methods, Techniques and Applications. CRC Press; 1995.
12.
Centers for Disease Control and Prevention. United States COVID-19 cases and deaths by state over time. Accessed April 28, 2021. https://data.cdc.gov/Case-Surveillance/United-States-COVID-19-Cases-and-Deaths-by-State-o/9mfq-cb36
13.
GISAID. Acknowledging GISAID data contributors. Accessed July 28, 2021. https://www.gisaid.org/help/publish-with-data-from-gisaid/
14.
Centers for Disease Control and Prevention. Variant proportions. Accessed April 24, 2021. https://covid.cdc.gov/covid-data-tracker/#variant-proportions
15.
Centers for Disease Control and Prevention. Underlying medical conditions associated with high risk for severe COVID-19: information for healthcare providers. Accessed April 25, 2021. https://www.cdc.gov/coronavirus/2019-ncov/hcp/clinical-care/underlyingconditions.html
16.
Wagner  T, Shweta  F, Murugadoss  K,  et al.  Augmented curation of clinical notes from a massive EHR system reveals symptoms of impending COVID-19 diagnosis.   eLife. 2020;9:e58227. doi:10.7554/eLife.58227 PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 12,272
    Citations 0
    Original Investigation
    Public Health
    November 2, 2021

    Analysis of the Effectiveness of the Ad26.COV2.S Adenoviral Vector Vaccine for Preventing COVID-19

    Author Affiliations
    • 1nference, Cambridge, Massachusetts
    • 2nference Labs, Murgesh Pallya, Bengaluru, Karnataka, India
    • 3Department of Internal Medicine, Mayo Clinic, Rochester, Minnesota
    JAMA Netw Open. 2021;4(11):e2132540. doi:10.1001/jamanetworkopen.2021.32540
    Key Points

    Question  How effective is the Ad26.COV2.S adenoviral vector vaccine from Johnson & Johnson at preventing SARS-CoV-2 infection?

    Findings  This comparative effectiveness research study found that, through large-scale longitudinal retrospective curation of electronic health records from the multistate Mayo Clinic Health System, the Ad26.COV2.S vaccine had an effectiveness of 74%.

    Meaning  This study suggests that a single dose of the Ad26.COV2.S vaccine appears highly effective at preventing SARS-CoV-2 infection.

    Abstract

    Importance  Continuous assessment of the effectiveness and safety of the US Food and Drug Administration–authorized SARS-CoV-2 vaccines is critical to amplify transparency, build public trust, and ultimately improve overall health outcomes.

    Objective  To evaluate the effectiveness of the Johnson & Johnson Ad26.COV2.S vaccine for preventing SARS-CoV-2 infection.

    Design, Setting, and Participants  This comparative effectiveness research study used large-scale longitudinal curation of electronic health records from the multistate Mayo Clinic Health System (Minnesota, Arizona, Florida, Wisconsin, and Iowa) to identify vaccinated and unvaccinated adults between February 27 and July 22, 2021. The unvaccinated cohort was matched on a propensity score derived from age, sex, zip code, race, ethnicity, and previous number of SARS-CoV-2 polymerase chain reaction tests. The final study cohort consisted of 8889 patients in the vaccinated group and 88 898 unvaccinated matched patients.

    Exposure  Single dose of the Ad26.COV2.S vaccine.

    Main Outcomes and Measures  The incidence rate ratio of SARS-CoV-2 infection in the vaccinated vs unvaccinated control cohorts, measured by SARS-CoV-2 polymerase chain reaction testing.

    Results  The study was composed of 8889 vaccinated patients (4491 men [50.5%]; mean [SD] age, 52.4 [16.9] years) and 88 898 unvaccinated patients (44 748 men [50.3%]; mean [SD] age, 51.7 [16.7] years). The incidence rate ratio of SARS-CoV-2 infection in the vaccinated vs unvaccinated control cohorts was 0.26 (95% CI, 0.20-0.34) (60 of 8889 vaccinated patients vs 2236 of 88 898 unvaccinated individuals), which corresponds to an effectiveness of 73.6% (95% CI, 65.9%-79.9%) and a 3.73-fold reduction in SARS-CoV-2 infections.

    Conclusions and Relevance  This study’s findings are consistent with the clinical trial–reported efficacy of Ad26.COV2.S and the first retrospective analysis, suggesting that the vaccine is effective at reducing SARS-CoV-2 infection, even with the spread of variants such as Alpha or Delta that were not present in the original studies, and reaffirm the urgent need to continue mass vaccination efforts globally.

    Introduction

    As of April 2021, there have been more than 195 million cases of COVID-19 worldwide, with more than 4 million associated deaths.1 After the emergency use authorizations issued by the US Food and Drug Administration (FDA) on February 27, 2021, more than 21 million doses of the Ad26.COV2.S COVID-19 vaccine (from Johnson & Johnson and Janssen) have been administered in the United States.2 This vaccine consists of a single-dose injection of a recombinant, replication-incompetent, human adenovirus type 26 vector encoding the SARS-CoV-2 spike protein. A recent phase 3 trial has demonstrated the effectiveness (66.9% [95% CI, 59.0%-73.4%]) and safety profile of this vaccine.3 Self-resolving, mild to moderate adverse effects were common in participants who received the vaccine, and serious adverse effects occurred rarely, with a frequency comparable to that of placebo.

    As the SARS-CoV-2 vaccines continue to be administered more broadly, it is critical to continuously assess the safety and effectiveness data for various reasons. For example, the interpretation of vaccine trial outcomes is inherently limited by how representative the studied population is of the broader population that will ultimately receive the vaccine. Furthermore, effectiveness is a dynamic process that may be affected by the evolution of the virus. Variations in the spike protein regularly occur and may have the potential to escape the immune response triggered by the vaccine. Finally, the fraction of the population that has been vaccinated may affect the observed effectiveness through herd immunity.

    Previous studies have used recent advances in deep neural networks to perform high-throughput, machine-augmented curation of electronic health record (EHR) systems.4,5 These advances have enabled the rapid assessment of the real-world effectiveness and safety of the messenger RNA (mRNA) vaccines mRNA-1273 (Moderna) and BNT162b2 (Pfizer/BioNTech) as well as a targeted investigation of the incidence of cerebral venous sinus thrombosis among patients receiving COVID-19 vaccines (including Ad26.COV2.S) within the Mayo Clinic Health System.6-8 We believe such approaches are essential for the development and deployment of vaccines and, most importantly, to build trust and transparency for the public. Here, we expand on this effort to conduct a preliminary assessment of the effectiveness of the Ad26.COV2.S vaccine using EHR data from the Mayo Clinic and Mayo Clinic Health System between February 27 and July 22, 2021.

    Methods
    Study Design and Participants

    This was a comparative effectiveness research study of individuals who underwent polymerase chain reaction (PCR) testing for suspected SARS-CoV-2 infection at the Mayo Clinic and hospitals affiliated with the Mayo Clinic Health System between February 27 and July 22, 2021. We limited ourselves to states where the Ad26.COV2.S vaccine was administered (ie, Arizona, Iowa, Minnesota, Florida, and Wisconsin). This study was reviewed by the Mayo Clinic institutional review board and determined to be exempt from the requirement for institutional review board approval (45 CFR 46.104d, category 4). Participants were excluded if they did not have a research authorization on file. This study followed the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) reporting guideline.

    The participant-selection algorithm for this study mirrored that outlined in a previous real-world effectiveness analysis of mRNA COVID-19 vaccines.6 Specifically, in this retrospective study, patients with the following criteria were included: (1) underwent at least 1 SARS-CoV-2 PCR test at the Mayo Clinic between February 27 and July 22, 2021; (2) aged at least 18 years; and (3) and resides in a local area (based on zip code) in which at least 10 patients have received the Ad26.COV2.S vaccine. Matching based on geography and the number of individuals undergoing PCR testing is used to control for any bias in infection rates due to geographical constraints and changes in infection rates.

    Exclusion criteria were as follows: (1) individuals with a positive SARS-CoV-2 PCR test result before the date of vaccine administration or the beginning of the study period (February 27, 2021); (2) individuals with no follow-up days after vaccination (ie, those who received the vaccine dose on the last date of data collection); (3) individuals who received the mRNA-1273 (Moderna) or BNT162b2 (Pfizer/BioNTech) vaccines; and (4) individuals with no research authorization on file. After the application of these inclusion and exclusion criteria, the study population included 8889 vaccinated patients and 230 790 unvaccinated patients (Figure 1A). Propensity score matching of the unvaccinated cohort was performed to match the vaccinated population.

    Effectiveness Analysis
    Propensity Score–Matching Procedure Defining the Unvaccinated Cohort

    We used 1:10 propensity score matching9 to construct an unvaccinated cohort similar to the vaccinated cohort with respect to key risk factors for SARS-CoV-2 infection, as was described previously6: (1) geography (zip code of the patient’s residence), (2) demographic characteristics (age, sex, race, and ethnicity), and (3) records of PCR testing (number of negative PCR test results before February 27, 2021). As described previously, by obtaining the number of negative PCR test results, we intended to capture a combination of factors, including the potential levels of exposure to COVID-19 as well as the willingness and ability of individuals to undergo SARS-CoV-2 testing. Propensity scores were obtained by training regularized logistic regression models for each zip code using the software package sklearn, version 0.20.3 in Python (Python Software Foundation).

    Using these propensity scores, we matched each of the 8889 individuals in the previously defined vaccinated cohort with 10 of the 230 790 eligible unvaccinated individuals, using greedy nearest-neighbor matching without replacement.10 If an unvaccinated individual tested positive for SARS-CoV-2 prior to the vaccination date for a potential matched vaccinated individual, this match was considered invalid; in such cases, the unvaccinated individual was recycled (ie, made available to potentially match to other vaccinated individuals), and a new unvaccinated individual was selected from the pool.

    Evaluation of Vaccine Effectiveness

    Vaccine effectiveness was evaluated as described previously by comparing the rates of positive SARS-CoV-2 PCR test results in the vaccinated and unvaccinated cohorts.6 For each vaccinated individual, the date of study enrollment (day 0) was defined as the date of vaccine administration. For each unvaccinated individual, the date of study enrollment was defined as the date of the vaccine administration for their matched vaccinated individual.

    The cumulative proportional incidence of SARS-CoV-2 infection was compared between vaccinated and unvaccinated patients by Kaplan-Meier analysis. The cumulative proportional incidence at time t is the estimated proportion of patients who experience the outcome on or before time t (ie, 1 minus the standard Kaplan-Meier survival estimate). We considered the cumulative incidence starting at day 1 and day 14 relative to the date of study enrollment (day 0). Significance was assumed to be indicated by P < .05. Statistical significance was assessed with the 2-sided log-rank test. Censored data include individuals who did not test positive for SARS-CoV-2 at the end of the study as reported in the EHR. They were considered in the analysis as negative for SARS-CoV-2 infection.

    We calculated the incidence rate ratio (IRR) of positive SARS-CoV-2 tests between the vaccinated and unvaccinated cohorts. Effectiveness was defined as 100% × (1 − IRR). We considered the incidence rate starting on day 1 after vaccination, day 8 after vaccination, and day 15 after vaccination. Incidence rates were defined as the number of patients testing positive for SARS-CoV-2 in the given period divided by the total number of at-risk person-days contributed in that period. For each individual, at-risk person-days are defined as the number of days in the period in which the individual has not yet tested positive for SARS-CoV-2 or died. The IRR was calculated as the incidence rate of the vaccinated cohort divided by the incidence rate of the unvaccinated cohort, and its 95% CI was computed using an exact approach described previously.11 For comparison, Minnesota, Florida, and Wisconsin had a mean of 840, 4442, and 451 new COVID-19 cases daily, respectively, during the study period.12 These trends are shown in eFigure 1 in the Supplement.2 We also computed the daily prevalence of SARS-CoV-2 variants of concern as a percentage of new daily observations for each state using the GISAID (Global Initiative on Sharing Avian Influenza Data) data set13 by plotting a simple moving mean with a sliding window of 7 days as a function of time for the duration of the study, illustrating the dynamics of SARS-CoV-2 variants (eFigure 2 in the Supplement).14

    COVID-19 Severity Analysis
    Propensity Score Matching Procedure Defining the Unvaccinated Infected Cohort

    We applied 1:10 propensity score matching to construct a SARS-CoV-2–positive unvaccinated cohort similar in baseline clinical covariates to the cohort of patients who were vaccinated and subsequently tested positive for SARS-CoV-2. Patients were matched based on demographic characteristics (age, sex, race, and ethnicity) and comorbid conditions (asthma, cancer, cardiomyopathy, chronic kidney disease, chronic obstructive pulmonary disease, coronary artery disease, heart failure, hypertension, obesity, pregnancy, severe obesity, sickle cell disease, solid organ transplant, stroke or cerebrovascular disease, and type 2 diabetes). This list of comorbid conditions was derived from the list of risk factors for severe COVID-19 illness provided by the Centers for Disease Control and Prevention.15 We used deep neural networks to automatically identify comorbid conditions from the clinical notes. To obtain the propensity scores, we trained an L2-regularized logistic regression model with these features and stratified patients by zip code using the software package sklearn, version 0.20.3 in Python.

    Based on these propensity scores, we matched each of the 60 individuals who tested positive for SARS-CoV-2 after vaccination with 600 of the 2236 individuals who tested positive for SARS-CoV-2 and who were not vaccinated, using greedy nearest-neighbor matching without replacement.

    Disease Severity Analysis

    For each SARS-CoV-2–positive patient in both the vaccinated and unvaccinated cohorts, the index date for the analysis (day 0) was taken to be the date of the first positive PCR test result. Analysis was performed with patients in each cohort with at least 14 days of follow-up after their first positive PCR test result (60 vaccinated and 600 unvaccinated). Severe COVID-19 cases included those with hospital admission, those with intensive care unit (ICU) admission, and those who died. The following parameters were evaluated: (1) 14-day hospital admission rate (ie, the number of patients admitted to the hospital in the 2 weeks after their positive PCR test result); (2) 14-day ICU admission rate (ie, the number of patients admitted to the ICU in the 2 weeks after their positive PCR test result); and (3) 14-day mortality rate (ie, the number of patients who died in the 2 weeks after their positive PCR test result). For each outcome, we report the relative rate (ie, the rate in the vaccinated cohort divided by the rate in the matched unvaccinated cohort), 95% CI for the relative risk, and the Fisher exact test P value. Hospital-free and ICU-free survival were also compared via Kaplan-Meier analysis, with statistical significance assessed using the log-rank test.

    Automated Clinical Data Extraction From Clinical Notes

    We used the previously described BERT (bidirectional encoder representations from transformers)–based neural network model to identify comorbid conditions from the EHR for each patient and classify the sentiment for the phenotypes that appeared in the clinical notes.16 In brief, we applied a phenotype sentiment classification model that had been trained on 18 500 sentences that achieves an out-of-sample accuracy of 93.6% with precision and recall scores above 95%. This classification model identifed 4 classes: (1) “yes” (confirmed diagnosis); (2) “no” (diagnosis ruled out); (3) “maybe” (possibility of disease); and (4) “other” (alternate context [eg, family history of disease]). For each patient, we applied the sentiment model to the clinical notes in the Mayo Clinic EHR. For each comorbidity phenotype, if a patient had at least 1 mention of the phenotype during the period with a confidence score of 90% or greater, then the patient was identified as having the phenotype.

    SARS-CoV-2 Variant Analysis

    The number of cases per state over time was obtained from the Centers for Disease Control and Prevention website.2 The information on variants was obtained from GISAID.13

    Results
    Prevention of SARS-CoV-2 Infection in a Population Vaccinated With Ad26.COV2.S

    Between February 27 and July 22, 2021, in the US, the pandemic entered a new phase characterized by a flattening in the number of cases due to the massive campaign for vaccination and the expansion of new SARS-CoV-2 variants (eFigures 1 and 2 in the Supplement). The Alpha variant was the most common (66.1% in Wisconsin, 71.7% in Florida, and 80.4% in Minnesota until May 31, 2021) but was replaced by the recent Delta variant in June and July (45.1% in Wisconsin, 37.9% in Florida, and 38.2% in Minnesota). A total of 8889 individuals who met the study inclusion criteria received the Ad26.COV2.S COVID-19 vaccine across the Mayo Clinic network (see Methods; Figure 1A). To evaluate the effectiveness of this vaccine in preventing SARS-CoV-2 infection, we identified a cohort of 88 898 unvaccinated individuals using 1:10 propensity score matching (see Methods; Table 1). The vaccinated and unvaccinated cohorts were balanced for sex, race, and age with 4397 (49.5%) vs 44 140 (49.7%) women, 8202 (92.3%) vs 81 458 (91.6%) non-Hispanic or non-Latino individuals, and a mean (SD) age of 52.4 (16.9) vs 51.7 (16.7) years, respectively. The cohorts were matched by zip code, and the distribution of states of residence was correspondingly equivalent. Patients resided in 15 different states. The median follow-up period was 111 days (IQR, 102-131 days) in both groups (Table 1).

    During the study period, 60 of 8889 vaccinated individuals (0.7%) had a positive SARS-CoV-2 PCR test result compared with 2236 of 88 898 unvaccinated individuals (2.5%) (Table 2; Figure 2A). Polymerase chain reaction testing was performed for symptomatic and asymptomatic patients (eg, screening for travel or prior to an intervention). The incidence rates of positive SARS-CoV-2 test results in the vaccinated and unvaccinated cohorts were 0.065 and 0.25 cases per 1000 person-days, respectively, for an IRR of SARS-CoV-2 infection in the vaccinated vs unvaccinated control cohorts of 0.26 (95% CI, 0.20-0.34), indicating an overall vaccine effectiveness of 73.6% (95% CI, 65.9%-79.9%) and a 3.73-fold reduction in SARS-CoV-2 infections (Table 2; Figure 2A).

    The full effectiveness of the Ad26.COV2.S vaccine is expected to be achieved after several weeks.3 Thus, we next analyzed the number of new SARS-CoV-2 infections starting 15 days after the study enrollment date (Table 2; Figure 2B) until the end of the study period (July 22, 2021). Of the 8698 vaccinated individuals with at least 2 weeks of follow-up, 41 (0.5%) tested positive for SARS-CoV-2 15 days or more after vaccination compared with 1561 of 86 495 unvaccinated individuals (1.8%) (a 3.91-fold reduction). This corresponds to a vaccine effectiveness of 74.2% (95% CI, 64.9%-81.6%) in preventing SARS-CoV-2 infection, with onset at least 2 weeks after vaccination (Table 2; Figure 2B). This result is similar to the effectiveness from day 1 and suggests that an immune response occurs more quickly than anticipated (<14 days).

    Infection can occur after vaccination if an individual is exposed to SARS-CoV-2 shortly after vaccination (ie, before building immunity) or if the vaccine fails to elicit protective immunity. To distinguish between these 2 scenarios, we analyzed the distribution of time between vaccination and the first positive PCR test result in these 2 cohorts (Figure 3A). The median times to a positive PCR test result after vaccination were similar in the vaccinated and unvaccinated cohorts: 27 days (IQR, 12-49 days) vs 28 days (IQR, 12-79 days), respectively, with 19 of 60 infections vs 641 of 2236 infections. These results suggest that most infections in the vaccinated cohort are likely not due to viral exposure before the recipient was able to develop immunity. In addition, in Figure 3B, the incidence of infections in vaccinated individuals appears to parallel the increased frequency of the Delta variant (eFigure 2 in the Supplement), raising the concern of decreasing vaccine protection against variants.

    Rate of Severe COVID-19 Disease After Ad26.COV2.S Vaccination

    To understand the effectiveness of the Ad26.COV2.S vaccine in preventing severe COVID-19 (hospitalization, ICU admission, and death), we analyzed the number of hospitalizations, ICU admissions, and deaths among the 60 vaccinated individuals who tested positive for SARS-CoV-2 and the 2236 unvaccinated individuals who tested positive for SARS-CoV-2. A composite outcome showed decreased odds of severe events in vaccinated individuals (odds ratio [OR], 0.33 [95% CI, 0.19-0.65]; P < .001). In subgroup analyses, we observed decreased odds of hospitalization (OR, 0.32 [95% CI, 0.18-0.66]; P = 2.8 × 10−4) and ICU admission (OR, 0.00 [95% CI, 0.00-1.43]; P = .01) but not mortality (OR, 0.83 [95% CI, 0.26-5.20]; P > .99) among vaccinated individuals (eTable 1 in the Supplement). The number of fatal events was low in both groups (1 of 8888 in vaccinated participants vs 12 of 88 886 in unvaccinated participants).

    Currently, vaccination is prioritized for older individuals and at-risk individuals, which biases the comparison with the unvaccinated cohort, who are younger and healthier. To control for this bias, we designed a cohort of unvaccinated patients matching the 60 vaccinated patients who were infected with SARS-CoV-2 after vaccination (Figure 1B; see Methods). Using 1:10 ratio matching, we matched the cohort based on a propensity score derived from demographic characteristics (age, sex, race, and ethnicity) and risk factors for severe COVID-19 illness provided by the Centers for Disease Control and Prevention15 (see Methods). The matching results are summarized in eTable 2 in the Supplement. There were no significant differences between the 2 cohorts in any of the clinical covariates that were included in propensity score matching. We then compared the 14-day hospitalizations, ICU admissions, and deaths between these cohorts (eTable 3 in the Supplement). We did not observe any significant differences in the composite outcome or in any individual analyses. However, only 41 vaccinated patients with COVID-19 had sufficient follow-up (≥14 days) for inclusion in this analysis. In conclusion, our results suggest a reduction in the severity of COVID-19 with the Ad26.COV2.S vaccine, although further analysis with more patients and events will be required for more precise estimates of the protection against hospitalization, ICU admission, and mortality.

    Discussion

    In this study, we assessed the effectiveness of the Ad26.COV2.S COVID-19 vaccine (Johnson & Johnson and Janssen) by analyzing longitudinal health records of 8889 vaccinated individuals and a 1:10 propensity-matched unvaccinated cohort in the multistate Mayo Clinic Health System (Minnesota, Arizona, Florida, Wisconsin, and Iowa). We observed reductions in rates of newly diagnosed COVID-19 infection, with an overall effectiveness of 73.6% (95% CI, 65.9%-79.9%) and effectiveness of 74.2% (95% CI, 64.9%-81.6%) after 14 days. These findings are consistent with the results of the phase 3 trial of Ad26.COV2.S, which demonstrated 66.9% efficacy against moderate to severe COVID-19 with onset at least 14 days after vaccination.3 Our results highlight the effectiveness of the vaccine in a cohort in which the SARS-Cov-2 Alpha and Delta variants were the predominant circulating viruses in comparison with the original clinical trials.

    Infections in vaccinated individuals were observed but were rare (0.7% of vaccinated individuals). They do not appear to be concentrated close to the injection date, which argues against an incubation period needed to generate a protective immune response. A spike of COVID-19 cases seems to be associated with the emergence of the Delta variant in June and July 2021, suggesting that the vaccine fails to trigger a protective immune response in a subset of patients, in particular with the emergence of variants. However, the number of cases is low, and the emergence of the Delta variant is too recent to be definitive.

    Clinical trials have demonstrated that multiple COVID-19 vaccines, including Ad26.COV2.S, are highly efficacious in reducing the risk of severe illness. Our analysis shows a reduction in severe cases, in particular for risks of hospitalization (and trends for reducing the risk of ICU admission and mortality). However, because only 60 individuals tested positive for SARS-CoV-2 after receiving the Ad26.COV2.S vaccine, our study was underpowered for definitive assessment of mortality. As the vaccine is administered to more patients, we will continue to assess the rates of hospitalization, ICU admission, and mortality among individuals who become infected after Ad26.COV2.S vaccination.

    Limitations

    Biases, such as the care-seeking bias, and differences in COVID-19 incidence rates and comorbid conditions were controlled for using propensity-score matching for the number of PCR tests, geographical location, and comorbid conditions. However, this study has several limitations. First, the study was conducted using data from a single health system, with most individuals residing in Minnesota or Wisconsin. The cohorts, which are more than 90% White and approximately 49.5% female, are not demographically representative of the broader US or world population that is now eligible for vaccination. Second, the cohort analyzed is relatively small compared with the population analyzed in the phase 3 trial of the Ad26.COV2.S vaccination and compared with prior real-world analyses of FDA-authorized mRNA vaccines.3,6 Third, our study does not take into account receipt of care outside the Mayo Clinic system, and participants might have received a PCR test that we would not detect in the Mayo Clinic EHR. Finally, sequencing information at the patient level was not available, so the prevalence of the Alpha and Delta variants was approximated by the GISAID population-level prevalence. We have assumed that the rate of such tests is equal in both cohorts, limiting the bias of censored data.

    Conclusions

    Despite these caveats, to our knowledge, this study is the first using EHR data in a propensity score–matched retrospective analysis of the effectiveness of the Ad26.COV2.S vaccine performed after emergency use authorization. Implementation of this framework will allow us to track in real time how the effectiveness of this 1-shot vaccine continues to evolve over the coming weeks and months. This information is particularly important in the context of the emergence of variants that could potentially escape vaccine-induced immunity. The extraction of such data from holistic health records is critical in shaping the global race against the ongoing pandemic.

    Back to top
    Article Information

    Accepted for Publication: September 2, 2021.

    Published: November 2, 2021. doi:10.1001/jamanetworkopen.2021.32540

    Open Access: This is an open access article distributed under the terms of the CC-BY-NC-ND License. © 2021 Corchado-Garcia J et al. JAMA Network Open.

    Corresponding Authors: Tyler Wagner, PhD (tyler@nference.net), and Venky Soundararajan, PhD (venky@nference.net), nference, One Main St, E Arcade, Cambridge, MA 02142.

    Author Contributions: Dr Soundararajan had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs Corchado-Garcia and Zemmour are co–first authors.

    Concept and design: Corchado-Garcia, Zemmour, Lenehan, O’Horo, Badley, Halamka, Virk, Swift, Wagner, Soundararajan.

    Acquisition, analysis, or interpretation of data: Corchado-Garcia, Zemmour, Hughes, Bandi, Cristea-Platon, Lenehan, Pawlowski, Bade, O’Horo, Gores, Williams, Badley, Wagner, Soundararajan.

    Drafting of the manuscript: Corchado-Garcia, Zemmour, Hughes, Bade, Soundararajan.

    Critical revision of the manuscript for important intellectual content: Corchado-Garcia, Zemmour, Hughes, Bandi, Cristea-Platon, Lenehan, Pawlowski, O'Horo, Gores, Williams, Badley, Halamka, Virk, Swift, Wagner, Soundararajan.

    Statistical analysis: Corchado-Garcia, Zemmour, Hughes, Bandi, Cristea-Platon, Pawlowski, Bade.

    Obtained funding: Badley.

    Administrative, technical, or material support: Bade, O’Horo, Badley, Soundararajan.

    Supervision: O’Horo, Gores, Badley, Halamka, Wagner, Soundararajan.

    Conflict of Interest Disclosures: Dr Corchado-Garcia reported receiving personal fees from and holding stock in nference Inc outside the submitted work. Dr Zemmour reported receiving personal fees from nference Inc outside the submitted work. Dr Hughes reported receiving personal fees from nference Inc during the conduct of the study; and personal fees from nference Inc outside the submitted work. Mr Lenehan reported receiving other fees from Janssen (nference collaborates with Janssen on data science projects unrelated to this manuscript, and this relationship did not impact the study design or interpretation of its results) outside the submitted work. Dr Pawlowski reported receiving personal fees from nference Inc outside the submitted work. Dr O’Horo reported receiving personal fees from Elsevier and Bates College; and grants from nference Inc, outside the submitted work. Dr Badley reported being a consultant for AbbVie and Gilead; serving on scientific advisory boards for Freedom Tunnel, Pinetree Therapeutics, Primmune, Immunome, Flambeau Diagnostics, nference, and Zentalis; serving on data safety and monitoring boards for Corvus, Equillium, and Excision Biotherapeutics; and being founder and president of Splissen Therapeutics. Dr Virk reported being an inventor for Mayo Clinic Travel App interaction with Smart Medical Kit and Medical Kit for Pilgrims. Dr Swift reported receiving grants from Pfizer during the conduct of the study. Dr Wagner reported receiving personal fees from and holding stock in nference Inc outside the submitted work. Dr Soundararajan reported other from Janssen (nference collaborates with Janssen and other biopharmaceutical companies on data science initiatives unrelated to this study, and these collaborations had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript outside the submitted work. The Mayo Clinic may stand to gain financially from the successful outcome of this research. This research has been reviewed by the Mayo Clinic Conflict of Interest Review Board and was in compliance with Mayo Clinic Conflict of Interest policies. No other disclosures were reported.

    Additional Information: The data will be made available on reasonable request to the corresponding author. A proposal with detailed description of study objectives and the statistical analysis plan will be needed for evaluation of the reasonability of requests. Deidentified data will be provided after approval from the corresponding author and the Mayo Clinic.

    References
    1.
    World Health Organization. Coronavirus disease (COVID-19) pandemic. Accessed April 25, 2021. https://www.who.int/emergencies/diseases/novel-coronavirus-2019
    2.
    Centers for Disease Control and Prevention. COVID data tracker. Accessed April 25, 2021. https://covid.cdc.gov/covid-data-tracker/
    3.
    Sadoff  J, Gray  G, Vandebosch  A, et al; ENSEMBLE Study Group. Safety and efficacy of single-dose Ad26.COV2.S vaccine against Covid-19. N Engl J Med. 2021;384(23):2187-2201. doi:10.1056/NEJMoa2101544PubMed
    4.
    Pawlowski  C, Wagner  T, Puranik  A,  et al.  Inference from longitudinal laboratory tests characterizes temporal evolution of COVID-19-associated coagulopathy (CAC).   eLife. 2020;9:e59209. doi:10.7554/eLife.59209 PubMedGoogle Scholar
    5.
    Pawlowski  C, Venkatakrishnan  AJ, Ramudu  E,  et al.  Pre-existing conditions are associated with COVID-19 patients’ hospitalization, despite confirmed clearance of SARS-CoV-2 virus.   EClinicalMedicine. 2021;34:100793. doi:10.1016/j.eclinm.2021.100793 PubMedGoogle Scholar
    6.
    Pawlowski  C, Lenehan  P, Puranik  A,  et al.  FDA-authorized mRNA COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system.   Med (N Y). 2021;2(8):979-992. doi:10.1016/j.medj.2021.06.007PubMedGoogle Scholar
    7.
    McMurry  R, Lenehan  P, Awasthi  S,  et al.  Real-time analysis of a mass vaccination effort confirms the safety of FDA-authorized mRNA COVID-19 vaccines.   Med (N Y). 2021;2(8):965-978. doi:10.1016/j.medj.2021.06.006PubMedGoogle Scholar
    8.
    Pawlowski  C, Rincón-Hekking  J, Awasthi  S, et al. Cerebral venous sinus thrombosis is not significantly linked to COVID-19 vaccines or non-COVID vaccines in a large multi-state health system.  J Stroke Cerebrovasc Dis. 2021;16(1):105923. doi:10.1016/j.jstrokecerebrovasdis.2021.105923
    9.
    Austin  PC.  An introduction to propensity score methods for reducing the effects of confounding in observational studies.   Multivariate Behav Res. 2011;46(3):399-424. doi:10.1080/00273171.2011.568786 PubMedGoogle ScholarCrossref
    10.
    Austin  PC. A comparison of 12 algorithms for matching on the propensity score.  Stat Med. 2014;33(6):1057-1069. doi:10.1002/sim.6004PubMed
    11.
    Sahai  H, Khurshid  A.  Statistics in Epidemiology: Methods, Techniques and Applications. CRC Press; 1995.
    12.
    Centers for Disease Control and Prevention. United States COVID-19 cases and deaths by state over time. Accessed April 28, 2021. https://data.cdc.gov/Case-Surveillance/United-States-COVID-19-Cases-and-Deaths-by-State-o/9mfq-cb36
    13.
    GISAID. Acknowledging GISAID data contributors. Accessed July 28, 2021. https://www.gisaid.org/help/publish-with-data-from-gisaid/
    14.
    Centers for Disease Control and Prevention. Variant proportions. Accessed April 24, 2021. https://covid.cdc.gov/covid-data-tracker/#variant-proportions
    15.
    Centers for Disease Control and Prevention. Underlying medical conditions associated with high risk for severe COVID-19: information for healthcare providers. Accessed April 25, 2021. https://www.cdc.gov/coronavirus/2019-ncov/hcp/clinical-care/underlyingconditions.html
    16.
    Wagner  T, Shweta  F, Murugadoss  K,  et al.  Augmented curation of clinical notes from a massive EHR system reveals symptoms of impending COVID-19 diagnosis.   eLife. 2020;9:e58227. doi:10.7554/eLife.58227 PubMedGoogle Scholar
    ×