Copyright 2006 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2006
To evaluate the use of immunization registry data to supplement missing or incomplete vaccination data reported by immunization providers (referred to as “providers” hereafter) in the National Immunization Survey.
Cross-sectional, random-digit–dialing, telephone survey to measure vaccination coverage among children aged 19 to 35 months in the United States.
Four sites with mature (with >67% of provider participation in the area) immunization registries.
Of the 639 children with complete household interviews, interviewers had consent from the respondents for 569 (89.0%) children to contact their providers and for 556 (87.0%) children to contact both providers and registries.
Main Outcome Measures
Percentages of children up-to-date for vaccines based on data from providers, registries, and both sources combined.
According to provider-reported data, weighted estimates of coverage for the recommended childhood vaccine series 4:3:1:3 at the 4 sites were 65.6%, 78.8%, 81.6%, and 77.0%. According to registry data, these coverage rates were consistently lower: 31.7% (P<.05), 65.4%, 71.9%, and 61.8%, respectively. When all unique vaccine doses were combined from both sources, the pooled 4:3:1:3 coverage rates increased to 72.0%, 92.0%, 88.7%, and 80.2%, respectively. The quality and completeness of vaccination histories from the registries were inconsistent and varied by sites.
Vaccination coverage estimates were the lowest when only registry-reported data were used and were the highest when provider- and registry-reported histories were combined. Although registries enrolled and matched more children, vaccination histories were missing, incomplete, and inconsistent. The quality and completeness of the registry data must be improved and must be comparable across all states before further consideration may be given to supplement or replace the provider-reported National Immunization Survey data.
The National Immunization Survey (NIS), a large, ongoing, nationwide telephone survey conducted by the Centers for Disease Control and Prevention since 1994, monitors the results of efforts to achieve and maintain Healthy People 2010 objectives for childhood vaccination coverage.1 The goal is to achieve 90% or more vaccination coverage among children aged 19 to 35 months for 4 or more doses of diphtheria and tetanus toxoids and pertussis vaccine (DTaP), 3 or more doses of poliovirus vaccine, 1 or more doses of measles-mumps-rubella vaccine, 3 or more doses of Haemophilus influenzae type B vaccine, 3 or more doses of hepatitis B vaccine, and 1 dose of varicella vaccine.
The NIS collects data in 2 phases. In phase 1, a household telephone interview collects immunization histories and sociodemographic information from the respondent and obtains consent to contact the child's immunization provider (referred to as “provider” hereafter). In phase 2, an immunization history questionnaire is mailed to the immunization provider identified in the interview to collect the child's vaccination history. The NIS uses the same method to collect and process vaccination data in 50 states and 28 large urban areas; this enables researchers to compare vaccination coverage rates across 78 immunization action plan areas. Details of the NIS design and estimation methods are published elsewhere.2
In the NIS, approximately 21% of the children with a complete household interview had a missing provider-reported immunization history because the household respondent did not give consent to contact providers; an additional 12% had a missing vaccination history because their immunization provider(s) either did not respond to the mailed questionnaire or did not report an adequate vaccination history to determine the child's up-to-date (UTD) status. Thus, while about 67% of the children had adequate vaccination histories from the providers, this percentage varied across 78 immunization action plan areas (range, 51%-80%).
Another Healthy People 2010 goal is to increase to 95% the proportion of children younger than 6 years in a fully operational population-based immunization registry.1 Immunization registries are confidential, population-based, computerized information systems that collect vaccination data on children younger than 6 years within a state or local area; the age group varies by the registry. Since the measles outbreak in the early 1990s, a concerted effort has been made by federal, state, and local governments and nonprofit foundations to develop immunization registries that track vaccinations received by age-eligible children.3,4 As of 2002, 36 states and the District of Columbia had registries targeting the entire state, 7 states had regional or countywide registries, and the remaining 7 states were planning statewide systems.5 Nationwide, approximately 43% of children younger than 6 years had 2 or more vaccinations recorded in a registry. Methods to collect and process immunization histories by those 43 state and regional registries varied considerably from site to site. Consequently, there are interstate and intrastate differences in the quality and completeness of immunization histories collected from the registries. Several previously published studies6- 8 also reported inconsistencies in data quality and underestimation in coverage rates because of missing dose information (range, 10%-40%) in computerized immunization tracking systems.
The objective of the Centers for Disease Control and Prevention–sponsored registry data query project was to explore whether vaccination data could be collected from mature (with >67% of provider participation in the area) registries in the United States and whether these data would be of sufficient quality so that missing or incomplete vaccination histories in the NIS could be supplemented with the registry data in a state.9,10 Before this study, it was not known how complete or of what quality the data were from the mature registries. It was hoped, however, that these mature registries could consistently demonstrate that they could provide high-quality vaccination data to replace or supplement survey data in order to produce reliable and comparable estimates of vaccination coverage across states in the United States.
The goals of the 2002 NIS registry data query project were to (1) assess whether consent to contact registries can be obtained from the NIS respondents, (2) measure match rates for the NIS children in the registries, (3) evaluate the quality and completeness of the vaccination histories from mature registries, and (4) assess the potential for registry data to replace missing or supplement incomplete vaccination data from the providers.
To achieve these goals, 3 states and 1 urban immunization action plan area with mature registries were selected for this study. Registry sites were selected for geographic diversity and to represent a mix of mandated vs voluntary provider participation. To maintain confidentiality, these 4 sites are referred to as A, B, C, and D in this article.
Quarterly data from the 2002 NIS were used in this study. For 1 of the sites, interview data were used from the first quarter of 2002; and for the other 3 sites, from the third quarter of 2002. After completing the household interview part of the questionnaire, interviewers request consent to contact immunization providers and, if granted, obtain their contact information from the respondents. For this study, household respondents in the 4 areas who consented to contact providers were also requested to grant permission to contact the immunization registry to obtain the child's vaccination history. Following the NIS protocol, immunization history questionnaires with the child's name, sex, and date of birth were mailed to the providers and the same information was also mailed to the registries. Providers were asked to report vaccinations and administration dates for all vaccines given before the date of the household interview. Registries in the 4 areas (A-D) were also searched to locate vaccination records for the NIS-sampled children. This study was approved by the Research Ethics Review Board of the National Center for Health Statistics, Centers for Disease Control and Prevention.
Any registry record or provider response with at least 1 vaccine dose for the matched child was included in this assessment. For comparison, we created 3 subgroups of children who had vaccination data available: (1) only from the providers, (2) only from the registries, and (3) from either providers or registries. For comparability of estimates across the 3 subgroups, simple poststratification was used to adjust the quarterly weights within each subgroup to represent all age-eligible children in the sampling area. (None of the estimates from these 3 subgroups resemble the published NIS estimates because of differences in the weighting procedures.) We compared the weighted proportions of children UTD for the series 4:3:1:3 (with ≥4 DTaP doses, ≥3 poliovirus vaccine doses, ≥1 measles-mumps-rubella vaccine or measles-containing vaccine dose, and ≥3 Haemophilus influenzae type B vaccine doses) according to provider data, registry data, and combined provider and registry data. Counting each unique shot date as a single dose for a vaccine from either source created the total number of combined doses for the vaccine. We also substituted missing dose information from the providers with a valid registry-reported dose (shot dates within vaccine schedule or with at least 14 days' lag between 2 unique shot dates), if available. Thus, for example, using the “combined” pooled-dose approach, the total number of DTaP UTD children with 4 or more doses of DTaP included all children classified as DTaP UTD from the provider(s), all children described as DTaP UTD from the registry, and a subset of not UTD children from either source who had incomplete data from both sources and became UTD with the pooled DTaP doses.
A 2-way distribution of the reported number of doses from both sources was examined to assess agreement in and completeness of the dose information for each vaccine type and vaccine series.10 Generally, multidose vaccines are more likely to have missing doses than a single-dose vaccine when vaccination data are collected from multiple sources; also, medical records or registries are more likely to have missing vaccination data for multidose vaccines when they are administered by multiple providers. In this article, we present the pattern of missing doses for a multidose vaccine, DTaP, and examine the distribution of the number of doses reported by the registries among children who were UTD (ie, having ≥4 doses of DTaP) according to the provider-reported data. We also examine the distribution of the number of doses reported by the providers among those children who were UTD from the registries.
In the 2002 NIS, a total of 639 children from the 4 selected areas had complete household interview data. Of these children, interviewers had consent from the respondents for 569 (89.0%) children to contact their providers and for 556 (87.0%) children to contact both providers and registries (Table 1). In 2002, the overall NIS consent rate was 87.6% across all 78 immunization action plan areas.
For each selected site, Table 1 shows the number and percentage of children with complete household interviews, consent, and registry- or provider-reported vaccination data. Among the 639 children with interview data, 505 (79.0%; range, 74.5%-83.4%) had vaccination data from at least 1 source: 402 (62.9%; range, 59.4%-66.3%) had information from providers, 427 (66.8%; range, 51.0%-73.3%) had information from the registries, and 323 (50.5%; range, 42.9%-53.5%) had information from both sources. Thus, for 79 (402 − 323) children (12.4%; range, 6.0%-23.5%), provider-reported data were the only source of vaccination information; and for 104 (427 − 323) children (16.3%; range, 8.2%-19.8%), registries were the only source of vaccination histories.
From the provider-reported data, the weighted estimates of coverage (and 95% confidence limits) for the vaccine series 4:3:1:3 were as follows: 65.6% (±10.0%), 78.8% (±11.5%), 81.6% (±9.5%), and 77.0% (±13.2%), in sites A through D, respectively. The same weighted estimates based on only registry data were consistently lower, with coverage rates as follows: 31.7% (±8.2%), 65.4% (±13.7%), 71.9% (±9.5%), and 61.8% (±16.2%), respectively (Figure). These estimates were not statistically different from the provider-based estimates, except for the estimate for site A (P<.05). Vaccination coverage rates were the highest, with the pooled estimates at 72.0% (±10.2%), 92.0% (±11.1%), 88.7% (±9.1%), and 80.2% (±16.4%), respectively, and were not statistically different from the provider-based estimates. The pooled estimates included all 4:3:1:3 UTD children according to providers or registries and a subset of the not UTD children who became UTD after combining vaccine doses from both sources.
Comparison of the annual coverage estimate for the 4:3:1:3 series and the estimates based on quarterly data from the immunization providers (eg, physicians, nurses, hospitals, clinics, others), the registries, and the pooled provider and registry data reported: National Immunization Survey, 2002. The 4:3:1:3 series includes 4 or more diphtheria and tetanus toxoids and pertussis vaccine doses, 3 or more poliovirus vaccine doses, 1 or more measles-mumps-rubella vaccine or measles-containing vaccine doses, and 3 or more Haemophilus influenzae type B vaccine doses.
Table 2 compares the registry- and provider-reported number of DTaP doses to assess agreement and completeness of the registry data. It shows that 47.2% to 62.2% of children (across the 4 sites) who were described as DTaP UTD (with ≥4 DTaP doses) by the providers were also DTaP UTD according to the registries. However, 7.7% to 39.6% of children who were described as DTaP UTD from the providers had missing dose information from the registries for 1 to 3 DTaP doses, and 9.9% to 38.5% of children could not be matched in the registry database and, therefore, had all DTaP doses missing from the registries. Similar results were also observed on completeness of other individual vaccines8 (data not shown). These results show that missing records (when children could not be matched) and incomplete dose information (among matched cases) are the major causes of the lower coverage rates from the registries.
Conversely, to assess completeness of the provider data, we examined the distribution of provider-reported DTaP doses among children who were DTaP UTD according to the registries. Table 3 shows that many children (range, 64.6%-83.9%) who were described as DTaP UTD from registries were also DTaP UTD from the provider reports. This implies that if a registry contains complete vaccination history for a child, it is highly likely that the child's provider has the complete vaccination history. However, this agreement varies considerably across the 4 sites. Furthermore, only a few of the DTaP UTD children (range, 3.2%-8.9%) had incomplete histories, with missing information on 1 to 3 doses of DTaP from the providers. One of the key benefits of collecting vaccination data from the registries was that 12.9% to 27.7% of the children who were described as DTaP UTD from the registries had all dose information missing from the providers.
A stable source of comparable vaccination data is the cornerstone of ensuring continuous monitoring of nationwide and statewide coverage. The NIS uses the same sample design and standardized data collection and processing methods that are comparable across all 78 areas to monitor national, state, and urban area vaccination coverage. Although the NIS faces many challenges in collecting immunization histories, it provides the only national source for monitoring many important issues with childhood immunization, including gaps in coverage by socioeconomic status and geography.2 The NIS uses the provider-reported data to publish national and state-specific estimates of vaccination coverage.
The NIS registry data query project was conducted to evaluate the quality and completeness of the vaccination histories collected from mature registries and to assess the potential for registry data to supplement missing and incomplete vaccination data from providers. Most household respondents who gave consent to contact providers also gave consent to contact registries. Match rates for the NIS-sampled children were high (>70%) in all 4 registries.
Missing vaccination records (when children could not be matched) and incomplete or inconsistent dose information (among matched cases) were 2 major causes of lower coverage rates from the registries when compared with the coverage rates from the providers. The analysis also shows that despite incomplete histories reported by registries (or providers), the combined-dose method increased the overall pooled vaccination coverage rates in all 4 sites. Because of large standard errors, none of the pooled estimates were statistically different from the provider-based estimates. However, the increase in coverage estimates shows that supplementing missing or incomplete provider-reported vaccination histories with the data from mature registries has the potential to improve the completeness of vaccination histories.
For regional studies (eg, to identify population subgroups with low vaccination coverage in an area), it may be desirable to use pooled estimates by combining the registry- and provider-reported histories. However, supplementing incomplete or inconsistent registry data to the provider data could jeopardize the comparability of results from state to state (because availability and completeness of registry data vary among states) and over time (because enrollment and completeness of vaccination data in registries may change over time). With many immunization registries operating throughout the United States, several of them have progressed to high levels of enrollment of children in their catchment areas. A recent survey of the Centers for Disease Control and Prevention's 56 grantees in 50 states and 6 cities that receive funding under section 317b of the Public Health Service Act indicated that by 2004 approximately 48% of US children younger than 6 years were enrolled in an immunization registry. Of the 56 grantees, only 10 (18%) had greater than 95% and 7 (13%) had 81% to 94% of children enrolled in the registries.11 Thus, to ensure comparability of vaccination coverage estimates, state and regional registries must be able to sustain high levels of enrollment, data completeness, and accuracy over time.
Finally, there are limitations to both sources of the vaccination data. Because some households with young children move in and out of the state where children were born, it is difficult for registries and for providers to keep track of the vaccination information for these children, leading to incomplete or missing records. Sometimes registries have missing or incomplete records because not all providers have complete data to submit for those children who have previously lived in another state or recently moved out to another state. Conversely, some children who had missing vaccination data from the providers had complete histories from the registry. In addition, sometimes there is a lag between the time when a vaccination is administered to the child and the time the provider submits the vaccination information to the registry and the registry enters the information in the database. Immunization records maintained by providers and by registries are not always of sufficient or consistent quality. Therefore, an adequate vaccination history may not be available for some children from either source because of missing doses, inconsistent quality, transcription errors, or misclassification errors.6,12- 14
In conclusion, vaccination coverage estimates were the lowest when only registry-reported data were used, and increased when all unique vaccine doses from provider- and registry-reported data were combined. Although registries enrolled and matched more children, missing, incomplete, and inconsistent vaccination histories were the major causes for the lower coverage rates. The completeness of dose information varied considerably by registry, and the quality of the information was unknown and inconsistent. The quality and completeness of the registry data must be improved and must be comparable across all states before further consideration may be given to supplement or replace the provider-reported data in the NIS. Immunization information systems that record standardized vaccination data at the time and point of service may greatly enhance the completeness and quality of the registry data.7,8
Correspondence: Meena Khare, MS, National Center for Health Statistics, Centers for Disease Control and Prevention, 3311 Toledo Rd, Room 3218, Hyattsville, MD 20782 (firstname.lastname@example.org).
Accepted for Publication: January 19, 2006.
Author Contributions: Ms Khare has full access to all the data used in the study, and all coauthors take responsibility for the integrity and the accuracy of the data. Study concept and design: Khare, Piccinino, Barker, and Linkins. Acquisition of data: Khare, Piccinino, Barker, and Linkins. Analysis and interpretation of data: Khare, Piccinino, Barker, and Linkins. Drafting of the manuscript: Khare, Piccinino, Barker, and Linkins. Critical revision of the manuscript for important intellectual content: Khare, Piccinino, Barker, and Linkins. Statistical analysis: Khare, Piccinino, and Barker. Administrative, technical, and material support: Khare, Piccinino, and Linkins. Study supervision: Khare, Barker, and Linkins.
Financial Disclosure: None reported.
Disclaimer: The findings and conclusions in this article are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention.
Acknowledgments: We acknowledge Michael P. Battaglia, MS, Abt Associates Inc, for his assistance in designing and implementing the study; Diana Bartlett, MPH, National Immunization Program, Centers for Disease Control and Prevention, for providing consultation and coordinating data collection from registries; and Mary McCauley, MTSC, National Immunization Program, Centers for Disease Control and Prevention, for helping to edit the early drafts of the manuscript.
Khare M, Piccinino L, Barker LE, Linkins RW. Assessment of Immunization Registry Databases as Supplemental Sources of Data to Improve Ascertainment of Vaccination Coverage Estimates in the National Immunization Survey. Arch Pediatr Adolesc Med. 2006;160(8):838-842. doi:10.1001/archpedi.160.8.838