Customize your JAMA Network experience by selecting one or more topics from the list below.
Objectives To examine patterns and trends of undervaccination in children aged 2 to 24 months and to compare health care utilization rates between undervaccinated and age-appropriately vaccinated children.
Design Retrospective matched cohort study.
Setting Eight managed care organizations of the Vaccine Safety Datalink.
Participants Children born between 2004 and 2008.
Main Exposure Immunization records were used to calculate the average number of days undervaccinated. Two matched cohorts were created: 1 with children who were undervaccinated for any reason and 1 with children who were undervaccinated because of parental choice. For both cohorts, undervaccinated children were matched to age-appropriately vaccinated children by birth date, managed care organization, and sex.
Main Outcome Measures Rates of undervaccination, specific patterns of undervaccination, and health care utilization rates.
Results Of 323 247 children born between 2004 and 2008, 48.7% were undervaccinated for at least 1 day before age 24 months. The prevalence of undervaccination and specific patterns of undervaccination increased over time (P < .001). In a matched cohort analysis, undervaccinated children had lower outpatient visit rates compared with children who were age-appropriately vaccinated (incidence rate ratio [IRR], 0.89; 95% CI, 0.89- 0.90). In contrast, undervaccinated children had increased inpatient admission rates compared with age-appropriately vaccinated children (IRR, 1.21; 95% CI, 1.18-1.23). In a second matched cohort analysis, children who were undervaccinated because of parental choice had lower rates of outpatient visits (IRR, 0.94; 95% CI, 0.93-0.95) and emergency department encounters (IRR, 0.91; 95% CI, 0.88-0.94) than age-appropriately vaccinated children.
Conclusions Undervaccination appears to be an increasing trend. Undervaccinated children appear to have different health care utilization patterns compared with age-appropriately vaccinated children.
Immunization is one of the most significant public health achievements of the past 100 years. However, an increasing number of parents have expressed concerns about immunizations, and survey data1-5 have shown that more than 10% of parents report delaying or refusing certain vaccinations for their children. These concerned parents often request alternative vaccination schedules that either increase the time between vaccinations or reduce the number of vaccinations in a single well-child visit. Despite their concerns, however, the safety of alternative vaccination schedules is not known.
In an observational setting, examining the safety of alternative vaccination schedules poses significant methodologic challenges. First, identifying children on intentional alternative vaccination schedules may be difficult because there are many potential causes of undervaccination, including missed opportunities, barriers to health care, missing immunization records, and medical contraindication to vaccination. Because vaccine-hesitant parents have a wide range of opinions on the benefits and risks of vaccination, it is also likely that there are numerous alternative schedules to study. Finally, parents who delay or refuse vaccinations may exhibit different health care–seeking behaviors than parents who allow vaccination according to the recommended schedule of the Advisory Committee on Immunization Practices (ACIP).6 The latter issue is particularly problematic because systems for active safety surveillance rely on automated health care utilization data to identify vaccine adverse events. These potential challenges suggest that, before conducting studies on the safety of alternative vaccination schedules, additional data on the epidemiology of undervaccination are needed.
The first objective of this study was to describe patterns and trends of undervaccination in a large cohort of children from 8 managed care organizations (MCOs) during a 7-year period. The second objective was to compare health care utilization rates between undervaccinated children and children who were age-appropriately vaccinated according to the ACIP schedule. As part of the second objective, we also compared health care utilization rates among a subset of children who were undervaccinated because of parental choice. We believe these results will inform future observational studies that examine the safety of alternative vaccination schedules.
We conducted a retrospective cohort analysis within the Vaccine Safety Datalink, a collaborative project between the Centers for Disease Control and Prevention and 10 MCOs across the United States.7 The participating MCOs comprise a population of more than 9 million members annually (3% of the US population). Using a standardized data dictionary, each MCO prepares electronic data files containing information on demographics, vaccination history, and medical encounters in the outpatient, emergency department (ED), and inpatient settings. Eight MCO sites contributed data to the analyses, and each site's institutional review board approved the study.
The study observation period was January 1, 2004, through December 31, 2010. We first used Vaccine Safety Datalink databases to identify children born between January 1, 2004, and December 31, 2008. For inclusion, each child had to be continuously enrolled in their MCO from at least ages 2 to 12 months. Children were followed up for a maximum of 36 months, and follow-up stopped if a child's enrollment in his or her MCO was discontinued. We excluded children with documented contraindications to some or all vaccinations, including those with hematopoietic stem cell transplant, human immunodeficiency virus, and other immunodeficiencies, or receipt of intravenous immunoglobulin.8 To help ensure that children were receiving primary care services within their MCO, they also had to have at least 1 outpatient visit by age 12 months.9,10 We then ascertained vaccination status on the remaining study population using the recommended ACIP schedule as the standard.
In this large study population, we calculated the average number of days undervaccinated (ADU) for each child across 8 vaccines during the first 24 months of life (Table 1). This calculation was based on a metric developed by Luman et al11 that measures the difference between when the vaccine dose was administered and when the vaccine dose should have been administered according to the ACIP schedule (eTable). For example, the first dose of diphtheria, tetanus, and acellular pertussis vaccine (DTaP) is due at age 2 months (60 days) but is not considered late until age 92 days. Days undervaccinated for this dose would begin accruing on day 93. In the analysis by Luman et al, a day for which a child was undervaccinated was counted as 1 day regardless of how many vaccines a child was undervaccinated for on that day. In our analysis, we first calculated the number of days undervaccinated for each vaccine dose during the first 24 months of life. We then summed the number of days undervaccinated to calculate the total number of days undervaccinated. Last, we divided the total number of days undervaccinated by the number of vaccines that a child should have received according to the ACIP schedule. This calculation of ADU was adjusted for each child's follow-up time during the observation period, since approximately 16% of the children did not have 24 months of continuous enrollment in their MCO.
Our modified version of the metric of Luman et al11 can be further described with the following example. A child in our 2008 birth cohort is due for a total of 8 distinct vaccinations and 23 individual doses during the first 24 months of life. Suppose that the parents of this child chose to follow a commonly cited alternative vaccination schedule12 in which no doses of hepatitis B, polio, varicella, and measles, mumps, and rubella (MMR) vaccines were given in the first 24 months of life, while all other vaccines were received on time. For this child, the days undervaccinated for these respective vaccines were 638, 638, 242, and 242 (eTable), representing a total of 1760 days undervaccinated. Given that the child was to receive a total of 8 distinct immunizations, the ADU across all vaccines was 220 days (1760/8) in the first 24 months of life.
Our algorithm for calculating the ADU also accounted for vaccine shortages and other changes to immunization policy during the 7-year follow-up period (eTable). We accounted for the pneumococcal (PCV) vaccine shortage in early 2004, the Haemophilus influenzae type b (Hib) vaccine shortage from 2007 to 2009, the addition of rotavirus vaccine to the ACIP schedule in 2006, and changes in age recommendations for varicella vaccine in 2007.13-19 In addition, because of the complexities of the changes in the vaccination schedule and shortages during the follow-up period, we conducted 3 subanalyses in which PCV, Hib, and rotavirus vaccination were excluded from the calculations of ADU.
After calculating the ADU, we conducted a sample medical record review to identify a subgroup of children whose parents had intentionally delayed or refused vaccinations for nonmedical reasons. The medical record review was conducted in 2 phases. We used Cochran’s20 formula for calculating the sample size, with an assumed confidence interval width of 0.10 and α = .05.
In phase 1 of the review, we divided the population of undervaccinated children into deciles of ADU, with an equal number of children in each decile. A medical record review was conducted on a random sample of children from each decile.
The second phase of the review focused on children with an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) code for lack of vaccination because of parental choice (V64.05 and V64.06). A medical record review was conducted on a random sample of children with an ICD-9-CM code for parental vaccine refusal and an ADU greater than 0.
For both phases, a medical record abstractor at each MCO examined medical records to identify documentation that a parent had explicitly delayed or refused vaccinations for nonmedical reasons. Results from the review were used to calculate a confirmation rate, measuring the proportion of undervaccinated children with documentation in the medical record that the parent had intentionally selected undervaccination for his or her child. The confirmation rates were used to identify a subgroup of children with a high likelihood of being undervaccinated because of parental choice.
We manually reviewed 1132 medical records from 157 454 children who were undervaccinated at least 1 day (Figure 1). The overall confirmation rate of parental vaccine refusal/delay was 24.3% (95% CI, 22.9%-25.7%) across all 10 deciles (Table 2). Decile-specific confirmation rates ranged from 9.4% to 46.2%, with highest rates in the 8th through 10th deciles (40.2% combined). In most instances, reasons for undervaccination were not documented (60.0%). In 4.5% of the medical records, the vaccines were documented as having been received on time, and in 8.5% of the records, the vaccinations were delayed for medical reasons. Among children with an ICD-9-CM code for parental refusal, the confirmation rate was 93.7% (95% CI, 93.1%-94.2%).
Figure 1. Identifying matched analytic cohorts of undervaccinated children. To be eligible for the study, children had to be born between 2004 and 2008, be continuously enrolled in their managed care site from at least ages 2 to 12 months, and have at least 1 outpatient visit by age 12 months. Children with certain contraindications to vaccinations were excluded from the study. *Children were categorized as having a diagnosis code for vaccine refusal if they had a V64.05 or V64.06 International Classification of Diseases, Ninth Revision, Clinical Modification, code in their electronic medical record before age 24 months.
On the basis of these confirmation rates, we created 2 matched analytic cohort populations: one with children who were undervaccinated without an ICD-9-CM code for vaccine refusal/delay and the other with undervaccinated children with an ICD-9-CM code. The former represented children who were undervaccinated for any reason and the latter represented children who were undervaccinated because of parental choice. To create the analytic cohorts, we matched each undervaccinated child to at least 1 age-appropriately immunized child by birth date (±7 days), MCO, and sex. These children were followed up for a maximum of 36 months after the matched birth date to compare outcomes between the study arms.
Before the matched analytic cohorts were assembled, we conducted a series of descriptive analyses. We first identified the number of distinct patterns of undervaccination in the entire study population. To identify these patterns, we calculated the total number of possible combinations of individual vaccines received. This calculation was based on the 8 vaccines studied during the first 2 years of life. For each vaccine, we created 3 categories: (1) received all doses on time, (2) never received any doses, and (3) certain doses either missing or not received on time. On the basis of these categories, there was a theoretical total of 38 = 6561 possible vaccine combinations, and we identified the actual number of distinct patterns of undervaccination in the study population.
A birth cohort analysis was also conducted in which we calculated the mean and median ADU for children born in each year from 2004 through 2008. In these 5 birth cohorts, we examined the prevalence of undervaccinated children and children with specific patterns of undervaccination over time. These patterns included children with all vaccinations delayed until at least ages 4, 6, 12, and 24 months; children who did not receive DTaP on the same day as PCV and Hib vaccinations; and children who did not receive hepatitis B, polio, varicella, and MMR vaccines in the first 24 months of life. These 6 specific patterns represent commonly cited alternative vaccination schedules.12,21 Time trends were analyzed with linear regression and Cochran-Armitage trend tests.
After the matched analytic cohorts were constructed, they were merged with outpatient, inpatient, and ED data to compare health care utilization rates between the cohort study arms during the first 36 months of life. In one analysis, overall rates, rates of well-child visits, and rates of minor acute illnesses (upper respiratory illness, fever, and pharyngitis) in the outpatient setting were compared. In a second analysis, we compared overall rates of hospitalizations and ED visits between the cohort arms. These data were analyzed with conditional Poisson regression to calculate incidence rate ratios (IRRs) and 95% CIs, controlling for preexisting chronic health conditions.22,23 In the children who were undervaccinated for any reason, the regression analyses were stratified by decile of ADU.
We identified 323 247 children born between 2004 and 2008; 157 454 of these children (48.7%) were undervaccinated (Figure 1). The ADU for each child ranged from 1 to 592, with an overall mean (SD) of 36 (89) days (Table 3). The ICD-9-CM code for parental vaccine refusal was used in 6172 of the undervaccinated children (3.9%). In the overall study population, there were 1399 distinct patterns of undervaccination. Among those with an ICD-9-CM code for vaccine refusal, there were 756 distinct patterns of undervaccination. Based on the medical record confirmation rates (24.3% for the overall undervaccinated population and 93.7% for children with an ICD-9-CM code), the estimated prevalence of undervaccination because of parental choice was 13.0% (95% CI, 11.9%-14.2%).
The prevalence of undervaccinated children increased significantly across the birth cohorts from 2004 to 2008 (Table 3). Across the birth cohorts, the mean of the ADU increased by more than 50%, and the median increased from 0 to 3 days. The trends remained significant in the 3 subanalyses in which PCV, Hib, and rotavirus vaccine were removed from the calculation of ADU.
There was also an increasing trend of children on 1 of 6 specific patterns of undervaccination displayed in Figure 2. A total of 8939 children (2.8%) were identified as being on one of these possible alternative vaccination schedules, and approximately 911 children (10.2%) were up-to-date for all of the following vaccines by age 24 months: 4 DTaP, 3 polio, 1 MMR, 3 Hib, 3 hepatitis B, 1 varicella, and 4 PCV.24
Figure 2. Trends in delayed start to vaccinations and select vaccination patterns by birth cohort before age 2 years. Across all birth cohorts, 8939 children (2.8%) were on specific nonstandard vaccination schedules. All trends were significant at P < .001 using the Cochran-Armitage trend test. DTaP indicates diphtheria, tetanus, and pertussis vaccine; Hib, Haemophilus influenzae type b vaccine; and MMR, measles, mumps, and rubella vaccine.
We created 2 matched analytic cohorts in which undervaccinated children were matched to age-appropriately vaccinated children by birth date, MCO, and sex. Of the 157 454 undervaccinated children, 151 282 did not have an ICD-9-CM code for vaccine delay/refusal and represented children who were undervaccinated for any reason. For the first analytic cohort, we were able to individually match 137 855 of the 151 282 undervaccinated children (91.1%) to an age-appropriately vaccinated child, resulting in a total cohort size of 275 710 children. The average follow-up time per child in the cohort was 956 days.
For the second analytic cohort, we individually matched 6168 of the 6172 undervaccinated children (99.9%) with an ICD-9-CM code for vaccine delay/refusal to a median of 10 age-appropriately vaccinated children. The total cohort size was 75 377 children, and the average follow-up time per child was 940 days.
Rates of health care utilization differed between the cohort study arms (Table 4). Undervaccinated children had lower outpatient visit rates than children who were vaccinated on time (IRR, 0.89; 95% CI, 0.89-0.90). Undervaccinated children also had lower rates of well-child visits and encounters for specific minor acute illnesses in the outpatient setting. The utilization differences generally increased as the decile of undervaccination increased. These decile-specific IRRs ranged from 1.03 to 0.58, and most (88%) were statistically significant (P < .01).
The opposite pattern was observed in the inpatient setting. Overall, undervaccinated children had higher inpatient admission rates than age-appropriately vaccinated children (IRR, 1.21; 95% CI, 1.18-1.23). The decile-specific IRRs ranged from 1.04 to 1.37, with the largest IRRs in the 6th through 10th deciles.
In the ED setting, utilization rates were slightly higher among undervaccinated children (IRR, 1.03; 95% CI, 1.03-1.04). Across the deciles of undervaccination, the IRRs varied, but no pattern was observed.
Children who were undervaccinated because of parental choice had significantly lower utilization rates of the ED and outpatient settings—both overall and for specific acute illnesses—than children who were vaccinated on time (Table 5). The IRRs for these associations ranged from 0.88 to 0.94 and were statistically significant (P < .001). The IRR for inpatient admission rates was not statistically significant (IRR = 0.98; P = .50).
This large multisite cohort study suggests that undervaccination is an increasing trend. We used vaccination and health care utilization data captured from electronic health records to create longitudinal cohorts of children born between 2004 and 2008. Among these birth cohorts, we observed increases in the prevalence and magnitude of undervaccination in children in the first 2 years of life. Our results also suggest that specific patterns of undervaccination have been occurring with greater frequency over time. In addition, our cohort analyses comparing undervaccinated and age-appropriately vaccinated children demonstrated differences in health care utilization, which appear to increase as the magnitude of undervaccination increases. We believe these results have important implications for studying the safety of alternative vaccination schedules in an observational setting.
Approximately 49% of the children in this insured study population were undervaccinated for at least 1 day in the first 24 months of life—a figure that is lower than a published national estimate of 74%.11 In our cohort, we also estimate that approximately 13% of children were undervaccinated because of parental choice, which aligns with other published estimates of 10% to 25% from cross-sectional survey and population-based ecologic studies.1,3,25 However, this likely represents an underestimate of the true prevalence of intentional undervaccination and alternative vaccination schedules. Reasons for undervaccination were not documented in 60% of the records in our review. These insured children were continuously enrolled in their MCO from at least ages 2 to 12 months, and even the most significantly undervaccinated children from the 8th to 10th deciles had a median of 12 clinic visits by age 36 months. This, in turn, suggests that most of these children were using primary care services. It is therefore reasonable to assume that a certain proportion of undervaccinated children without documentation in the medical record represent those who were undervaccinated because of parental choice, thereby underestimating the true prevalence of intentional undervaccination.
The ICD-9-CM codes for parental vaccine refusal/delay had a high confirmation rate (93.7%), suggesting that it correctly classifies children as being undervaccinated because of parental choice. For this reason, we used the code to define our second analytic cohort population to compare health care utilization rates of undervaccinated children with those of children who are age-appropriately immunized. However, this code is infrequently used (3.9% of the unvaccinated cohort) and has a low sensitivity. This implies that the population of children with an ICD-9-CM code for parental vaccine delay/refusal may not be representative of the larger population of children who are undervaccinated because of parental choice. The code should therefore be used cautiously when defining cohorts to study the safety of alternative vaccination schedules.
We identified 1399 distinct patterns of undervaccination in the overall study population and 756 patterns in the cohort who were undervaccinated because of parental choice. Although this variability creates an opportunity to study the safety of alternative vaccination schedules, it also poses significant analytic challenges in trying to identify appropriate comparison groups.26 Moreover, these estimated numbers of patterns of undervaccination are an underestimate of the total number of possible patterns. Our criteria for identifying different combinations of patterns did not consider the number of combinations in which certain doses in a series can be missing or the specific timing of doses that have been delayed. Accounting for these factors could result in billions of possible combinations, thus highlighting the challenges of trying to study the safety of specific alternative vaccination schedules in an observational setting.
In our matched cohort analysis, children who were undervaccinated for any reason had lower rates of outpatient visits and higher rates of ED encounters and inpatient admissions compared with children who were age-appropriately vaccinated. Prior work27-30 has shown similar associations when comparing health care utilization across strata of socioeconomic status. However, in our study, the undervaccinated population likely represents a mix of children, including those who were deliberately undervaccinated for personal reasons, children with vaccine contraindications, children receiving their care outside of the MCO, those who were ill at their well-child visit, children of families experiencing barriers to care, and those whose parents simply missed their scheduled well-child visits. Such heterogeneity implies that there may be several factors influencing differences in health care utilization between the cohort arms.
Children who were undervaccinated because of parental choice had lower rates of outpatient visits, lower rates of ED encounters, and no significant difference in inpatient admission rates compared with age-appropriately vaccinated children. These results suggest inherent health care–seeking behavioral differences between the 2 groups of parents. For example, published survey data31,32 have shown that parents who choose not to have their children vaccinated are less likely to trust health care professionals and more likely to use complementary/alternative medicine providers than are parents who have their children fully vaccinated. It is therefore possible that parents who delay or refuse immunizations are less likely to use the traditional health care system when their children contract minor acute illnesses but will seek medical care when their children become seriously ill. Such differences could create a selection bias in studies that attempt to examine the risk of potential adverse events following vaccination. Future survey work with parents across a range of vaccination concerns could help explain these differences in health care utilization rates.
Although most parents choose to have their children vaccinated according to the ACIP-recommended schedule, an increasing number of parents appear to be delaying or refusing vaccines. These parents express concerns about the safety of vaccines; however, they may be placing their children at increased risk for infectious diseases that are almost 100% preventable with vaccination.33-39 For these reasons, there is national interest in studying the safety of alternative vaccination schedules.40 Our results demonstrate the potential public health impact of alternative vaccination schedules and highlight the obstacles to studying their safety. We therefore believe the findings of this study should be carefully considered when designing and conducting observational studies to examine the safety of alternative vaccination schedules.
Correspondence: Jason M. Glanz, PhD, Institute for Health Research, Kaiser Permanente Colorado, PO Box 378066, Denver, CO 80237 (firstname.lastname@example.org).
Accepted for Publication: August 24, 2012.
Published Online: January 21, 2013. doi:10.1001/jamapediatrics.2013.502
Author Contributions: Dr Glanz had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Glanz, Narwaney, Hambidge, Daley, McClure, Naleway, Nordin, and Weintraub. Acquisition of data: Glanz, Newcomer, Wagner, Lee, Nelson, Donahue, Naleway, Lugg, and Weintraub. Analysis and interpretation of data: Glanz, Newcomer, Narwaney, Hambidge, Daley, McClure, Xu, Rowhani-Rahbar, Nelson, Donahue, Nordin, and Lugg. Drafting of the manuscript: Glanz, Newcomer, and Daley. Critical revision of the manuscript for important intellectual content: Glanz, Newcomer, Narwaney, Hambidge, Daley, Wagner, McClure, Xu, Rowhani-Rahbar, Lee, Nelson, Donahue, Naleway, Nordin, Lugg, and Weintraub. Statistical analysis: Glanz, Newcomer, Narwaney, McClure, Xu, Rowhani-Rahbar, Nelson, and Lugg. Obtained funding: Glanz and Hambidge. Administrative, technical, and material support: Glanz, Daley, Wagner, McClure, Donahue, Naleway, Nordin, Lugg, and Weintraub. Study supervision: Glanz, Daley, Xu, and Lugg.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was funded through a subcontract with America's Health Insurance Plans under contract 200-2002-00732 from the Centers for Disease Control and Prevention (CDC).
Previous Presentations: Preliminary results from this study were presented at the American Public Health Association conference; November 1, 2011; Washington, DC; and at the Institute of Medicine's Committee on the Assessment of Studies of Health Outcomes Related to the Childhood Immunization Schedule meeting; February 9, 2012; Washington, DC. In addition, the preliminary results were presented as a platform presentation at the Pediatric Academic Society Meeting; May 1, 2012; Boston, Massachusetts. A more detailed presentation of these results was presented at an Institute of Medicine meeting; May 29, 2012; Washington, DC.
Disclaimer: Although the CDC played a role in the design and conduct of the study, collection, management, analysis, and interpretation of the data, as well as preparation, review, and approval of the manuscript, the findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC.
Additional Contributions: The data management and medical record abstraction staff at each of the participating sites created the Vaccine Safety Datalink data sets and reviewed the medical records.
Glanz JM, Newcomer SR, Narwaney KJ, et al. A population-based cohort study of undervaccination in 8 managed care organizations across the United States. JAMA Pediatr. Published online January 21, 2013. doi:10.1001/jamapediatrics.2013.502.
eTable 1. Calculating days undervaccinated for specific doses of 8 recommended early childhood vaccines
Glanz JM, Newcomer SR, Narwaney KJ, et al. A Population-Based Cohort Study of Undervaccination in 8 Managed Care Organizations Across the United States. JAMA Pediatr. 2013;167(3):274–281. doi:10.1001/jamapediatrics.2013.502
Create a personal account or sign in to: