Have vaccine lottery programs in the United States been associated with increases in the COVID-19 vaccination rate?
This cross-sectional study of 403 714 adult participants in the Household Pulse Survey and state daily vaccination rates found that vaccine lottery programs were associated with an increase in COVID-19 vaccination rates overall and in some states but not in others.
These findings suggest that policy makers should take the heterogeneous response to vaccine incentive lottery programs into account when implementing similar policies.
There is limited evidence on whether COVID-19 vaccine incentive lotteries help to increase vaccine uptake.
To estimate whether and to what extent statewide implementation of lottery programs is associated with COVID-19 vaccine uptake.
Design, Setting, and Participants
This cross-sectional study used data from 403 714 adults who answered the question, “have you received a COVID-19 vaccine?” on the Household Pulse Survey (HPS) and state-level daily vaccination rates for 11 states implementing a vaccine lottery program (ie, treated states) and 28 states with no such program (ie, control states). A difference-in-difference (DiD) analysis used responses to the HPS question and compared changes in vaccine uptake between treated and control states. The augmented synthetic control (ASC) analysis compared state-level daily new vaccination rates in treated states with their synthetic versions constructed from a donor pool of control states. Data were analyzed from March 17 to July 5, 2021.
Announcement of the vaccine lottery programs and states implementing such programs.
Main Outcomes and Measures
The outcome variable in the DiD analysis was the binary indicator of self-reported vaccination status. In the ASC analysis, it was the state-level daily new first vaccine dose administered per 100 000 people (logarithm scale).
A total of 403 714 individuals (mean [SD] age, 52.7 [15.7] years; 239 563 [weighted percentage, 51.6%] women; 31 746 [weighted percentage, 11.9%] Black; 39 709 [weighted percentage, 18.2%] Hispanic; 334 034 [weighted percentage, 76.4%] White) responded to the question in HPS regarding vaccination status. Overall, 80 949 respondents (weighted percentage, 28.1%) in the HPS were unvaccinated. A pooled analysis from both methods indicated that the lottery programs were associated with increased vaccinations. The ASC analysis revealed that lottery programs were associated with an increase of 0.208 log points (95% CI, 0.004-0.412 log points), implying an average 23.12% increment in the new daily vaccination rate. In state-specific analyses, both methods suggested that the vaccine lottery programs were helpful in Ohio (0.09 log points; P < .001), Maryland (0.26 log points; P < .001), Oregon (0.15 log points; P = .002), and Washington (0.37 log points; P < .001) but not in Arkansas, Kentucky, and West Virginia. The ASC analysis found that lottery programs were positively associated with vaccinations in New Mexico (0.32 log points; P < .001) and New York (0.33 log points; P = .001).
Conclusions and Relevance
The findings of this study suggest that lottery programs may be associated with decreased COVID-19 vaccine hesitancy, but that success might differ across states. The findings could also be useful in the ongoing debate on persuading US residents who are not yet vaccinated.
COVID-19 vaccines administered in the United States are safe and effective at lowering COVID-19 incidence, death, and hospitalization rates.1 The federal and local governments have put forward various strategies to increase the vaccination rate, including making the vaccine available to the public free of cost. Nevertheless, the vaccination rate in the US is less than the suggested threshold for herd immunity2,3 and is falling. Many states have launched incentive programs to boost vaccination, including vaccine lotteries in which a vaccinated individual is eligible to win a big cash prize.
Previous studies on the association between vaccine incentive programs and vaccine uptake have found mixed results. Cash incentives might help people change their behavioral health decision-making,4,5 while other researchers argue that external incentives could weaken the intrinsic motivation to take the desired action.6 Some maintain that monetary incentives, particularly the big cash prizes, may raise doubts about the safety of the vaccine and perpetuate vaccine skepticism7 and are ethically questionable.8
Ohio announced its lottery program, “Vax-a-Million,” on May 13, 2021, and after the initial media reports of its success, many other states adopted similar programs.9 However, there remains considerable uncertainty about whether such programs are successful. An interrupted time-series analysis of daily vaccinations in Ohio did not find a meaningful association of lottery incentives with vaccine uptake.10 Two recent reports on Ohio’s vaccine lottery program11,12 differ in their conclusion. The available studies are limited to a single state, and the results may not be generalizable to other states, which likely differ in key contextual factors related to public health.
In this article, we use 2 methodological approaches and 2 sources of data to investigate whether lottery-based incentives are associated with an increase in vaccination rates across multiple states in the United States. To our knowledge, this is the first study of its kind evaluating the association of this policy with vaccination rates in multiple states using both the individual responses of vaccination status and state-level daily vaccination data.
The data set for this cross-sectional study was deidentified and publicly available. The institutional review board at the University of Georgia determined that it was not human participant research and therefore waived the requirement for informed consent. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.
Study Population and Data
We conducted a difference-in-difference (DiD) analysis using data from the Household Pulse Survey (HPS), an online, nationally representative survey (average response rate, 7%) conducted by the US Census Bureau and aimed at understanding the impacts of the COVID-19 pandemic on American households.13 The HPS assigns a person-weight to each survey respondent to make the sample representative of the entire US population. We used the microdata with respondents aged 18 and older from survey weeks 27 to 33, corresponding to March 17 to July 5, 2021. The outcome of interest was a binary indicator of individual vaccination status based on the question, “Have you received a COVID-19 vaccine?” We controlled for individual and household characteristics, such as age, gender, race and ethnicity, marital status, household size, presence of children, homeownership status, employment status, supplemental nutrition assistance program (SNAP) recipient status, social security benefits recipient status, unemployment insurance recipient status, educational attainment, and household income. The HPS categorized ethnicity as Hispanic and non-Hispanic and race as Asian alone, Black alone, White alone, and other. We collapsed Asian alone and other in our model because of smaller samples in these 2 groups. Given that Ohio14 ended its vaccine lottery program on June 22, 2021, we limited the timeframe to the program’s end date for Ohio.
In the augmented synthetic control (ASC) analysis, we used state-level data of daily first dose of vaccine administration combined with state-level time-invariant covariates. We obtained the daily first dose of COVID-19 vaccination rate per 100 000 people (7-day moving average, natural logarithm scale) for March 17 to July 5 from Opportunity Insight.15 The study window corresponds to the timeframe of the HPS. We used state-level covariates, such as demographic characteristics (proportion aged ≤14 years, proportion aged ≥65 years, proportion Hispanic residents, and proportion Black residents) and socioeconomic characteristics (proportion of the population with a Bachelor’s degree or greater, proportion of people in the labor force, median household income, unemployment rate, and proportion with private health insurance) from American Community Survey 2019 annual estimates; partisan leaning index from FiveThirtyEight16; and geographic variables (latitude, longitude, and census division) and public-health related characteristics (prepandemic life expectancy at birth) from the US Centers for Disease Control and Prevention. Many of these covariates are associated with vaccine hesitancy17 and are typically used in public health literature.
The following 11 states that announced the vaccine lottery programs with prizes of at least $1 million before June 10, 2021, were considered as the treated states: Arkansas, California, Colorado, Kentucky, Maryland, Ohio, Oregon, New Mexico, New York, Washington, and West Virginia (eTable 1 in the Supplement). The analysis was performed for each state separately and by pooling them together. Illinois and Delaware were excluded because they had other major incentives targeted to increase vaccination rates.
First, we estimated the association between the lottery incentive programs and vaccinations with the DiD approach using linear regression models with fixed effects. The DiD analysis is extensively practiced in policy evaluation; in DiD analyses, a change in outcome in the treated unit(s) in the posttreatment period is compared with the control unit(s).18-20 This approach assumes a similar trend between treated and control units in the absence of treatment. We fitted separate models for each treated state and a model for pooled analysis. In our models, the outcome variable was the binary indicator of vaccination status. The interaction term between treatment status (a dummy labeled Treat) and postintervention period (a dummy labeled Post) is the primary variable of interest. The coefficient of the interaction term (Treat × Post) measures the proportional change in vaccine uptake associated with the vaccine lottery program implementation. We also included state-level fixed effects to control the fixed differences across states and survey week–level fixed effects to absorb the temporal variation in vaccine uptake.
We applied dynamic event study parameterization to partially test parallel trend assumptions and calculate the lead and lag estimates.21 Standard errors were clustered at the state level. To calculate the pooled estimator, combining all treated states, we applied the method by Sun and Abraham.22 The method builds on a growing body of literature on DiD with staggered adoption23-25 and deals with scenarios in which treatment is dynamic and treatment effects are heterogeneous across treated units. The DiD analysis was carried out using Stata version 17 (StataCorp).
Next, we evaluated the association between vaccine lottery programs and vaccination rates with an ASC analysis. The synthetic control method (SCM) is popular in policy evaluations that estimate the counterfactual outcome.26 It addresses the arbitrariness in selecting controls in traditional matching methods by explicitly choosing a data-driven approach.27-29 The SCM method works by creating a synthetic version of each treated state by a weighted combination of control states in the donor pool. The pretreatment outcome of treated states resembles their synthetic counterparts. Building on the SCM, an ASC method has been proposed30 that provides a way to debias the original SCM estimates when a pretreatment fit between a treated and synthetic unit is imperfect. For details on the ASC method, please refer to Ben-Michael et al.30
Because of the high variability in the outcome variable, we modeled the natural log of the daily vaccination rate. We specified the ridge-regularized linear regression as the outcome model and allowed nonnegative weights to improve pretreatment fit while minimizing extrapolation outside the convex hull.30 The ASC estimator would still be the linear combination of the control units, and the regularization parameter directly controls the level of extrapolation by penalizing the distance from SCM weights. The regularization parameter is selected via cross-validation.30 We applied synthetic control with a staggered adoption framework31 to calculate the combined, pooled results in all treated states. This approach accounts for the variation in treatment timing in different treated states. The ASC analysis was carried out using R version 4.0.3 (R Project for Statistical Computing) using the augsynth package.32 Statistical significance was P < .05, and all tests were 2-tailed.
Because of the gradual vaccine rollout programs, a substantial proportion of the population was already eligible to receive vaccines before they became available to the general public. We also anticipated that the enthusiasm brought about by the announcement of lottery programs might fade away over time. We performed sensitivity analyses to assess whether our ASC estimates were robust to change in the sampling window by limiting the time frame from when the general public became eligible for vaccination to 2 weeks after the program announcement.
Table 1 describes the characteristics of the population from the HPS in treated and control states. DiD analysis used 403 714 individuals (mean [SD] age, 52.7 [15.7] years; 239 563 [weighted percentage, 51.6%] women; 31 746 [weighted percentage, 11.9%] Black; 39 709 [weighted percentage, 18.2%] Hispanic; 334 034 [weighted percentage, 76.4%] White). Overall, 80 949 (weighted percentage, 28.1%) were unvaccinated, and 322 765 (weighted percentage, 71.9%) were vaccinated. The 2667 individuals who did not answer the question of interest were dropped. All percentages incorporate the sampling weight provided by the US Census Bureau. The mean (SD) age in the vaccinated group was higher compared with the unvaccinated group (54.5 [15.5] years vs 45.6 [14.3] years; P < .001). The unvaccinated group had a lower percentage of women (49 667 [50.6%] vs 189 896 [52.0%]; P < .001) and a higher percentage of Black individuals (8257 [14.7%] vs 23 489 [10.8%]; P < .001), Hispanic individuals (9758 [20.3%] vs 29 951 [17.4%]; P < .001), SNAP participants (9670 [15.2%] vs 15 547 [7.4%]; P < .001), and unemployment insurance recipients (10 213 [15.1%] vs 22 382 [9.0%]; P < .001) relative to vaccinated group. The unvaccinated group, on average, had a lower household income and education status compared with the vaccinated group. The trend of the daily state-specific first dose of COVID-19 vaccination rate per 100 000 (7-day moving average) from March 17 through July 5, 2021, appeared to decrease after plateauing in in mid-April (eFigure 1 in the Supplement).
In the pooled DiD analysis, we first evaluated the validity of the parallel trend assumption using the dynamic event study approach.22 Figure 1 displays the event study analysis results, in which 1 survey wave before the program announcement was considered a reference week. We found no evidence of a differential trend of vaccinations between treated and control states.
Table 2 presents estimates from the DiD approach. The coefficient of Treat × Post in the pooled analysis column shows the association between lottery program and vaccination from the pooled analysis, considering variation in treatment timing. The result showed that the lottery programs were significantly associated with a 2.1% (95% CI: 0.7%-3.5%; P < .001) increase in vaccine uptake, controlling for observables. Table 2 also provides DiD estimates for each of the treated states. California (2.5%; 95% CI: 1.5%-3.5%; P < .001), Colorado (2.2%; 95% CI: 1.2%-3.2%; P < .001), Maryland (6.4%; 95% CI: 5.4%-7.4%; P < .001), Ohio (5.4%; 95% CI: 4.4%-6.4%; P < .001), Oregon (6.6%; 95% CI: 5.6%-7.6%; P < .001), and Washington (3.8%; 95% CI: 2.8%-4.8%; P < .001) showed a positive and significant association between the lottery program and vaccinations. The magnitude of the association was largest in Oregon, followed by Maryland and Ohio. In contrast, the lottery programs were not associated with vaccinations in New Mexico and New York. DiD estimates for Arkansas, Kentucky, and West Virginia implied the opposite association between the lottery program and vaccine uptake. However, state-specific results could be imprecise and need caution in interpretation given that the event study graphs provide little evidence of parallel pre-trend assumption (eFigure 2 in the Supplement). Among the covariates used for adjustment in the DiD analysis, having a higher education, being a woman, owning a house, and receiving social security benefits were positively associated with the vaccine uptake, while being unemployed, participating in SNAP, and having children was negatively associated.
Summary statistics (mean and SD) of outcome and covariates used in the ASC in treated states and the donor pool of control states are presented in eTable 2 in the Supplement. In general, covariates were comparable, but the treated states had a higher vaccination rate than control states. Each donor states’ relative contributions (ie, weights) to the synthetic version of each treated state are available in eTable 3 in the Supplement. Using the ASC method, we extrapolated outside the convex hull, and thus, the weights could be negative. The weight matrix for the pooled analysis is not shown but is available on request.
Figure 2A shows the estimated relative change in the daily new vaccination rate per 100 000 (log scale) between treated and synthetic states from the pooled analysis, which accounts for the staggered adoption (variation in the treatment timing across states) in multiple states.31,33 We found that the combined outcome of all treated states was associated with an improvement in the daily vaccination rate by 0.208 log points (95% CI, 0.004-0.412 log points). This implied an average 23.12% increment in the daily vaccination rate after the program implementation. Given that the average vaccination rate of treated states on the day of the announcement was 230 per 100 000, a 23.12% increase in the daily new vaccination rate corresponds to approximately 53 additional new vaccine doses per 100 000 people.
In Figure 2B to E and eFigure 3 in the Supplement, we plot the trajectories of daily vaccination in each treated state and its synthetic control. The pretreatment trends of vaccination rates in the treated states closely tracked those in the synthetic control, barring a few exceptions, suggesting that synthetics are generally credible counterfactuals. In the postintervention period, the vaccination in most treated states was higher than their corresponding synthetic controls.
We present gap plots in eFigure 4 in the Supplement for each of the treated states. The corresponding average treatment effects on treated (ATT) and P values are presented in Table 3. A substantial improvement in vaccination rate is evident in New Mexico (0.32 log points; P < .001), New York (0.33 log points; P = .001), Washington (0.37 log points; P < .001), Ohio (0.09 log points; P < .001), Oregon (0.15 log points; P = .002), and Maryland (0.26 log points; P < .001). These translate to 108 000 additional new vaccine doses in New Mexico, 744 000 in New York, 178 000 in Washington, 61 000 in Ohio, 77 000 in Oregon, and 221 000 in Maryland. However, ASC analysis did not find a positive association between lottery programs and vaccination rates in Arkansas, Kentucky, California, Colorado, and West Virginia. In Ohio, where the prize was $5 million, the marginal cost per induced vaccination was $82.
The estimates in the sensitivity analysis were largely consistent with our results. A gap plot from the pooled analysis is available in eFigure 5 in the Supplement. From state-specific analyses, the observed vs synthetic plots and gap plots of daily vaccination rates are available in eFigure 6 and 7 in the Supplement, respectively.
Past work has documented that vaccine hesitancy is a global matter of concern.34-37 The US federal and state governments have launched programs to address misinformation about the safety and effectiveness of the COVID-19 vaccine and adopted incentive programs from free beer to lottery prizes.38 Little empirical evidence exists on the role of lottery incentives in improving vaccination. Our study used sophisticated policy evaluation methods to address that information bottleneck.
The DiD estimate from the pooled analysis suggests that the lottery program was associated with a 2.1% increase in vaccine uptake. When analyzed separately by state, the results were mixed: lottery programs appeared successful in most states, but not in all states. The DiD results, however, need to be interpreted with caution when the treated and control states do not provide evidence of no differential trend during the preintervention period. We also acknowledge that the selection on unobservables could still be an issue in the DiD setting. After the announcement of lottery incentives, the decision to get vaccinated may be informed differentially by unobservable across control and treated units.39
Consistent with the pooled DiD estimate, our pooled ASC analysis revealed a positive association between lottery programs and daily vaccination rates. Given that the adult population in 11 treated states is approximately 85 million,40 a 2.1% estimate from the DiD result implies that the lottery programs may have expanded the vaccine coverage to approximately 1.78 million additional adults. Similarly, our ASC estimate of 53 doses per day per 100 000 people translates into approximately 1.67 million additional adults getting vaccinated for an average of 37 posttreatment days in 11 treated states (calculated as 85 000 000 × 53 × 37/100 000). The results were mixed in the state-specific ASC analyses, with the highest ATT being observed in Washington. Among the states where vaccine uptake was positively associated with lottery programs, we estimate the number of new additional vaccinations to be approximately 108 000, 744 000, 178 000, 61 000, 77 000, and 221 000 for New Mexico, New York, Washington, Ohio, Oregon, and Maryland, respectively. The $5 million lottery program in Ohio ended within the study window, which produced the marginal cost per induced vaccination to be $82.
We recognize that vaccine acceptance may be a much more complex process with political, psychological, cultural, geographical, or socioeconomic elements involved41-45 that could explain the differential results of lottery programs in different states. In particular, we note that the states where vaccine lottery incentives did not succeed—West Virginia, Arkansas, and Kentucky—are heavily Republican,16 the political group less responsive to financial incentives for COVID-19 vaccinations.17 Future research examining vaccinations with politico-economic characteristics, the access and time to go to the nearest vaccine center, and the nature of public-health mobilization implemented by local governments may shed light on this differential response.
Our Ohio lottery program results differ from those in the study by Walkey et al10 and Lang et al12 but corroborate more recent studies.11,46,47 The segmented regression approach in the study by Walkey et al10 does not involve the comparison of treated and control units and, as such, cannot account for factors external to the time series.48,49 Lang et al12 did not match the covariates in deriving the synthetic control. The emerging literature suggests that a good balance in covariates requires lenient assumptions and provides synthetic control estimators with tighter bounds.27,50 Unlike past work, which has focused on a single state, we focused on multiple states and used novel and sophisticated statistical methods. DiD analysis controlled for demographic characteristics that could affect individuals’ vaccination decisions.51
This study has limitations. Although creating an ideal control group is the key idea of synthetic control methods, there are some practical difficulties in creating one. The pretreatment trends of New Mexico, West Virginia, and Washington were somehow different from their synthetic counterparts. We advise caution in making inferences about the results for these states. Moreover, our analysis is subject to possible data collection and reporting issues of statewide daily vaccinations and the potential self-reporting bias in the HPS responses.
In this study, vaccine lottery incentives were associated with increased vaccinations overall and in Ohio, Maryland, Oregon, and Washington but not in Arkansas, Kentucky, and West Virginia. This heterogeneity in results should inform governments when adopting similar policies in the future. The findings may also be relevant to the ongoing debate on how to persuade the millions of US residents who are not yet vaccinated against COVID-19.
Accepted for Publication: October 15, 2021.
Published: December 9, 2021. doi:10.1001/jamanetworkopen.2021.38238
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Acharya B et al. JAMA Network Open.
Corresponding Author: Chandra Dhakal, MS, MSc, Department of Agricultural and Applied Economics, University of Georgia, 207 B Conner Hall, Athens, GA 30602 (firstname.lastname@example.org).
Author Contributions: Messrs Dhakal and Acharya had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Messrs Acharya and Dhakal contributed equally to this work.
Concept and design: Both authors.
Acquisition, analysis, or interpretation of data: Both authors.
Drafting of the manuscript: Both authors.
Critical revision of the manuscript for important intellectual content: Both authors.
Statistical analysis: Both authors.
Administrative, technical, or material support: Dhakal.
Conflict of Interest Disclosures: None reported.
van der Swaluw
et al. Commitment lotteries promote physical activity among overweight adults—a cluster randomized trial. Ann Behav Med
. 2018;52(4):342-351. doi:10.1093/abm/kax017PubMedGoogle ScholarCrossref
SRM. Looking to the empirical literature on the potential for financial incentives to enhance adherence with COVID-19 vaccination. Prev Med
. 2021;145:106421. doi:10.1016/j.ypmed.2021.106421PubMedGoogle Scholar
AM. Behaviorally informed strategies for a national COVID-19 vaccine promotion program. JAMA
. 2021;325(2):125-126.PubMedGoogle Scholar
M. The economic impacts of COVID-19: evidence from a new public database built from private sector data. NBER Working Paper 27431. November 2020. doi:10.3386/w27431
D. Early impacts of the Affordable Care Act on health insurance coverage in Medicaid expansion and non-expansion states. J Policy Anal Manage
. 2017;36(1):178-210. doi:10.1002/pam.21961PubMedGoogle ScholarCrossref
M. Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Econ
. 2016;25(12):1514-1528. doi:10.1002/hec.3258PubMedGoogle ScholarCrossref
J. Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. J Am Stat Assoc
. 2010;105(490):493-505. doi:10.1198/jasa.2009.ap08746Google ScholarCrossref
J. Synthetic controls with staggered adoption. National Bureau of Economic Research Working Paper 28886. doi:10.3386/w28886
HJ. Mapping global trends in vaccine confidence and investigating barriers to vaccine uptake: a large-scale retrospective temporal modelling study. Lancet
. 2020;396(10255):898-908. doi:10.1016/S0140-6736(20)31558-0PubMedGoogle ScholarCrossref
P. Understanding vaccine hesitancy around vaccines and vaccination from a global perspective: a systematic review of published literature, 2007-2012. Vaccine
. 2014;32(19):2150-2159. doi:10.1016/j.vaccine.2014.01.081PubMedGoogle ScholarCrossref
JM. Econometric Analysis of Cross Section and Panel Data. MIT press; 2010.
W. Experimental and Quasi-experimental Designs for Generalized Causal Inference. Houghton Mifflin; 2002.
C. An analysis of factors associated with influenza, pneumoccocal, Tdap, and herpes zoster vaccine uptake in the US adult population and corresponding inter-state variability. Hum Vaccin Immunother
. 2018;14(2):430-441. doi:10.1080/21645515.2017.1403697PubMedGoogle ScholarCrossref