Durability of the Single-Dose Ad26.COV2.S Vaccine in the Prevention of COVID-19 Infections and Hospitalizations in the US Before and During the Delta Variant Surge

Key Points Question What is the durability of the Ad26.COV2.S COVID vaccine effectiveness before and during the Delta variant surge in the US? Findings Among 422 034 vaccinated and 1 645 397 matched unvaccinated individuals across the US, vaccine effectiveness was estimated to be 76% for COVID-19 infection and 81% for hospitalizations for at least 180 days after vaccination before and during the Delta variant surge. Meaning In this study, the Ad26.COV2.S COVID-19 vaccine was associated with high and durable effectiveness in clinical practice, including against the Delta variant.


Correction for vaccine under-recording
Such reduced sensitivity of exposure assessment leads to an underestimation of the true effectiveness, as our unvaccinated comparator population likely includes patients who were vaccinated at mass vaccination sites, pharmacies, and other organizations without submitting claims to insurance companies and thus our study data source. 11 Uncorrected VE estimates shown in main manuscript tables are based on an implicit assumption of complete capture (no under-recording) of vaccine events, and thus represent a lowest bound of VE for each analysis. For primary analysis bias-corrected effect estimates, we conservatively assumed an under-reporting of 40% (sensitivity of 60%).
Assuming a 40% estimated proportion of vaccination under-reporting in claims data, we used standard algebraic methods to correct relative risk estimates assuming 100% specificity of vaccination recording. 12 Observed and corrected relative risks and 95% confidence intervals were calculated by specifying a 61-day fixed follow-up period that began 14 days post-index among vaccinated and 1:1 PS matched unvaccinated patients. From this fixed follow-up analysis, specific bias correction factors were derived for each outcome and all subgroups of interest, and then applied to hazard ratio and rate ratio-based effectiveness estimates from the primary analysis. 11 Rather than deriving separate under-reporting rates for each subgroup and analysis within the study, we opted to apply one consistent adjustment factor (assuming 40% under-recording) to all corrected estimates, with sensitivity analyses examining the effect of varying this assumption. Supplemental Tables S4 and S5 show corrected VE estimates across a range of assumed under-recording scenarios, from complete capture (no under-recording) to 70% under-recording. We hope to refine this correction approach for future monthly refreshes through further linkage to state registry information and consideration of changes in under-reporting rates by time and geography.

eMethods 5. Month-by-Month Effectiveness Analyses
To assess vaccine effectiveness over calendar time, we examined incident rate ratios separately for each calendar month within the overall National Cohort and High-Delta-incidence States ( Fig  3). Table 2 also shows effectiveness estimates for the combined period June to August 2021 for high-Delta-incidence states, given the emergence of the Delta variant during this period.
Only outcome events and eligible person-time occurring within each person's follow-up period for main effectiveness analyses (March 2021 to August 2021) were included in month-level incident rate ratio calculations. Persons could contribute eligible person-time for multiple months, and/or for partial months, but were excluded from monthly person-time totals if censored from analysis. Figure S3 shows how month-by-month person-time denominators were derived for 2 hypothetical study participants.
Using this approach, monthly incident rates were calculated as the (number of outcome events beginning in each calendar month) / (the total number of person-years within each month for which persons were eligible for outcomes). Incident rate ratios were calculated for each month as the ratio of outcome rates in exposed vs unexposed patients, and effectiveness estimates were then corrected for vaccine under-recording in claims data using the methods described in Suppl. S4.

eMethods 6. HealthVerity Data Linkage Overview
Note: HealthVerity data linkage information is from HealthVerity proprietary white paper HealthVerity De-Identification Overview: HealthVerity uses a mixture of techniques to achieve accurate record linkage, including both straight hashing and Bloom filters (probabilistic). This allows us to be robust to errors and variations in text fields while maintaining a high level of patient privacy.
For fields that are purely numeric, such as date-of-birth or Social Security number (SSN), the data is subjected to a straight hash. We use SHA-256 hashing with a salt provided by a 3rd party server (not owned or managed by HealthVerity).
Free text fields, such as the patient's name, are prone to small errors and variations that make straight hashing a very brittle approach. We use a specific form of Bloom filters based on Durham et al. 13 to encode these fields into a special hash that supports probabilistic matching in the presence of small variations. We extend the existing work as described below to provide additional safeguards on the patient's privacy.
Bloom Filters: Achieving accurate record linkage using such hashes is non-trivial because a patient's record often contains typographical and semantic errors. 14 To overcome this problem, for fields that are likely to contain typographical errors and are not directly attackable using a constraint satisfaction program. HealthVerity will split a patient's identifier into a series of n-grams (an example of which is shown in Figure 1), whereby the patient's identifiers are split into a series of overlapping substrings that are subject to a set of hash functions and mapped into a Bloom Filter (BF), which is a binary array of a fixed length. When each field (e.g., forename, surname, etc.) is mapped into such a structure, it is referred to as a field-level Bloom filter (FBF). It has been shown that FBFs over common patient identifiers can be highly accurate. 15 Yet, it has also been shown that FBF encodings can be insecure because they reveal frequency distributions for field values (e.g., specific names) and bit positions (i.e., n-grams). Specifically, it has been shown that parameter settings recommended in Schnell et al., 15 which consists of 2-grams, a filter length of 500 bits, and 15 hash functions, can be cracked to leak names and geocodes (e.g., 5-digit ZIP) when the adversary has access to some global frequency distribution of such information [Kuzu 2011], such as a voter registration list or address directory. 16 It should be note that evidence suggests the identifiers available to an attacker may not be of the same frequency distribution as those observed in the FBFs. 16 However, it was shown that, even though the distributions may not be equivalent, an attacker can allow for weaker constraints in their cryptanalysis and commit a successful cracking of the system. Given that it is not always possible to determine what data sources an adversary may have access to, this risk analysis aims to account for the worst-case scenario (i.e., when the hashed records and information available to the adversary are derived from the same resource), and thus ensure a system is robust against a constraint satisfaction-based cryptanalysis.
At the same time, it was further shown that as n grows, the likelihood of a successful attack diminishes. This is because as n increases, so too does the number of distinct n-grams, which leads to a smaller quantity of frequency-related information to commit a successful cryptanalysis. While this observation suggests a simple strategy to improve security, it was also found that n and linkage accuracy were inversely correlated. Thus, a more principled mechanism, independent of n has been developed to increase the security of BF-based encodings.
Specifically, HealthVerity uses a record-level Bloom filter (RBF), which merges multiple FBFs to prevent cryptanalysis. 13 HealthVerity has implemented this by mapping all the FBFs into the same bit space, such that it is difficult to discern from which fields such bits were derived. However, each FBF is still computed separately, using a distinct set of salts.

Implementation:
The above section explains the general RBF data structure for the pseudonyms generated for use by HealthVerity. However, it does not provide details on how the RBF is generated or what specifically it is composed of. To do so, let us consider the process of data generation. Figure 2 provides an illustration for how hashing and data transmission process works. It should be noted that the salt for the system is generated by a 3rd party server (not owned or managed by HealthVerity). This is critical because it ensures that HealthVerity cannot run a dictionary attack against the system. Additionally, note that the tokens from the de-identification (DeID) are encrypted and sent to a different 3rd party to perform the record matching process. These are encrypted at rest, and there is no PII data held on this 3rd party server. At no time does HealthVerity have access to any of the DeID tokens, and the only output that we receive for each record is a single HealthVerity ID number, based on the result of the matching.
The RBF is a composite over multiple fields, corresponding to the first and last (i.e., forename and surname) of the patient. The parameters for the RBF is set according to the strategy laid out in Durham et al., 13 specifically: 1. All names are truncated to ensure that the total length of the first and last name does not exceed 20 characters. Preference is given truncating the first name initially unless the last name exceeds 19 characters by itself. In that case, the first initial of the first name is used, along with the first 19 characters of the last name. 2. All names are translated into a set of lowercase trigrams. 3. Names are padded with two null characters at the beginning and end, providing a total of N+2 trigrams for a name with N characters.
4. Each trigram is subject to 10 rounds of hashing, using the salts provided by a 3rd party. 1. The first and last names use a different set of salts. The salts remain consistent for all trigrams in the first name, and for all trigrams in the last name. 2. The first name and last name are mapped into the same bit array.
c. The first round of hashing for a trigram is on the trigram plus the salt. The nine subsequent rounds of hashing are on the output of the previous round plus the next salt.
5. The RBF consists of 208 bits. This length is chosen to ensure that (on average) 50% of the bits are set to 1, derived from a representative sample of data. 6. To further protect the name length, the RBF will have all the bits inverted if the total number of bits set to 1 falls below 104 (50%).
Probabilistic Matching: The RBF method described above allows HealthVerity to implement a highly reliable method of anonymous record linkage. This is because two names that differ by a single typographic error will have RBF hashes that still share most of the same bits. The number of differing bits forms a narrow probability distribution function, allowing us to accurately assess the likelihood that two RBF fields correspond to the same patient.
Using approximate field matching with the RBFs, in concert with additional Bayesian methods to address missing data and changing field values (e.g., a patient moves to a new zip code), These patterns are learned from the observed data and constantly updated within the matching engine. All the probabilities are normalized by the observed frequency of the underlying tokens, so that matching the tokens for a common name like John Smith carries less evidentiary weight than matching the tokens for a truly unique name. The probabilities for the individual tokens are combined into a full joint conditional probability for each patient, and the results normalized across the entire population. HealthVerity currently has a database of tokens for 330M individuals in the United States that new queries are matched against, including numerous variations of the individual tokens. HealthVerity significantly improves the matching accuracy over state-of-the-art heuristic matching methods.
Performance and Validation: The use of Bloom filters produces a significant improvement in matching accuracy, as traditional means of hashing are fully deterministic and as a result are susceptible to significant false negative errors in matching due to source data variance. Based on both simulation and realworld use cases with current client data, traditional methods of deterministic matching can produce a false negative rate of between 7-35%.
The hashing components have undergone extensive analysis for frequency patterns and known attacks as a part of the expert determination in our HIPAA expert determination certifications performed by multiple separate parties (Malin, 2016/2018, El Emam, 2017). In addition, Bloom filter hashing is well established in the literature.
HealthVerity uses probabilistic matching, with probabilities normalized across the population. Currently, there is support for reporting the top ten match candidates for an individual, along with their relative probabilities. The default configuration is set to a threshold to prefer false negatives over false positives by a 10:1 ratio. This threshold may be set differently at a system level, or dynamically changed by the individual users through filtering of the top candidates.
HealthVerity has performed a series of extensive tests of the sensitivity and specificity of our system in simulation, as well as side-channel analysis of the real-world matching rates. Simulation tests were performed on the public data sets of 11M+ real-world individuals, under a variety of conditions and assumptions. Naturally, the results of these tests depend heavily on conditions such as data fidelity, error rates, demographic skews and shifts, and available fields. Under data conditions typical of medical records, the system consistently performed at 98.0% sensitivity and 99.8% specificity, regardless of demographics and scale.
Validating the results on real-world data in process is much more difficult, particularly in a privacy preserving setting. Fortunately, there are side-channels for analysis that allow us to indirectly identify sensitivity and specificity. Comparing identities to census data, in aggregate and in crosstabs, can identify overcounts due to identity fragmentation. Move rates between geographic regions can identify false negatives from a particularly challenging situation for matching. Even in regions where HealthVerity may not have full coverage of a population, the amount of variance in predicted demographic ratios versus ratios observed in the HVIDs can be used to bound the sensitivity rates. A variety of these methods generally agree on an actual sensitivity of 98.2%. Conversely, specificity can be estimated using conflict rates, such as when a single identity has mutually exclusive procedures or diagnoses, the most intuitive of which is post-mortem activity. HealthVerity has access to month of death indicators for a significant portion of the HVIDs. While some medical claims may still be filled in the months immediately following death, any activity more than six months later is an indicator that multiple individuals were given the same HVID. Multiple independent estimates using these types of mutual exclusivity consistently provide an estimated specificity of 99.6-99.7%. One empirical study with limited data was performed with a data partner, using hashed 4-digit SSNs. Real data on approximately 8,000 individuals with entries from multiple data sources were run through HealthVerity's system, and the results were compared using the hashed SSNs as a surrogate for identity. After accounting for possible coincidental matches (since there were less than 10,000 possible hashed SSNs), the sensitivity was estimated at 98.5% and the specificity was estimated at 99.7%. While the sample size was small, these numbers continue to agree with the independent estimates we get from simulated and from side-channel analyses.    Malignancies; n (%) 8        Characteristics reported for population matched with risk-set sampling and propensity scores. Unless otherwise noted, demographic variables are assessed at cohort entry (index) and comorbidities and clinical utilization variables are assessed during the 1 year before cohort entry. ASD; Absolute Standardized Difference *Variables not included in propensity score models; ** Note baseline covariate definition for Type 2 Diabetes differs slightly from subgroup definition; ***Recent medical and pharmacy claims were defined as claims beginning during the 60 days before cohort entry.       Characteristics reported for high-Delta-incidence States (early Delta States) population matched with risk-set sampling and propensity scores. Unless otherwise noted, demographic variables are assessed at cohort entry (index) and comorbidities and clinical utilization variables are assessed during the 1 year before cohort entry. ASD; Absolute Standardized Difference *Variables not included in propensity score models; ** Note baseline covariate definition for Type 2 Diabetes differs slightly from subgroup definition; ***Recent medical and pharmacy claims were defined as claims beginning during the 60 days before cohort entry.