Copyright 2003 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2003
The use of immunization assessment and referral (A/R) in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) has been shown to produce dramatic improvements in vaccination coverage when coupled with parental incentive; however, data are lacking to support the use of A/R alone.
To determine the effectiveness of A/R in increasing immunization coverage among WIC participants.
Participating WIC centers were assigned to1 of 3 interventions that delivered A/R of varying frequency or a control group.
Twenty of the largest Public Health Foundation Enterprises–WIC centers in Los Angeles County.
Children continuously enrolled in participating WIC centers from 6 to 24 months of age.
Assessment of child's vaccination status followed by referral to a health care provider for those lacking indicated vaccinations.
Main Outcome Measure
Up-to-date (UTD) status at 24 months of age for all recommended vaccines.
Baseline coverage rates were similar among all study sites (overall, 77% UTD). After the study period, compared with the controls (88% UTD), we found no differences in immunization coverage among WIC centers that administered A/R at every visit (every 2 months) to all children (90% UTD; adjusted odds ratio [OR], 1.02; 95% confidence interval [CI], 0.54-1.94), every 6 months to all children (89% UTD; OR, 0.98; 95% CI, 0.62-1.56), or every visit to children found to be behind at 8 months of age (89% UTD; OR, 0.89; 95% CI, 0.48-1.68).
In this urban population of WIC children with high baseline immunization coverage, A/R was not effective in increasing immunization coverage.
AS THE name suggests,the Special SupplementalNutrition Program for Women, Infants, and Children (WIC) works to ensure the health of low-income preschoolers by providing food vouchers, breast-feeding promotion and support, and other nutrition and health services.1- 3 WIC serves more than 45% of all US infants.3 Since these infants are at risk for underimmunization and poor nutrition, federal funding agencies recommend that WIC collaborate with local immunization agencies to promote immunization.4,5 The Task Force for Community Preventive Services also recommends immunizationinterventions in WIC.6
WIC centers promote immunization by assessing children's immunization records and referring those in need of vaccination to their health care providers. This assessment and referral (A/R) strategy has been shown to raise immunization rates when it is combined with other strategies.7- 11 One strategy is monthly voucher pickup (MVP). Parents of children who need vaccinations are given a 1-month supply of food vouchers rather than the usual 2- or 3-month supply. The parent must return a month later, have the child's immunization record reassessed, and pick up vouchers. With the MVP strategy, a parent is never denied vouchers, but monthly WIC visits are required until the child is up-to-date (UTD) for vaccines. The second strategy that works with A/R is clinic escort.7 A child who is not UTD is escorted with the parent to a colocated health clinic to be immunized.
However, about 50% of children served by WIC nationwide attend centers that use immunization A/R without MVP, escort, or other strategies.5 Data on the effectiveness of this practice are limited. We conducted this study to investigate whether A/R alone increases immunization coverage among children enrolled in WIC.
The Public Health Foundation Enterprises (PHFE)–WIC program is the largest local-agency WIC program in the United States, serving approximately 316 000 women, infants, and children monthly in Los Angeles County and Orange County, California.12 Based on PHFE-WIC monthly participation reports13 and California birth data,14 we estimated that 42% of the approximately 159 000 children born in Los Angeles County in 1998 were enrolled in PHFE-WIC.
The 20 WIC centers with the largest monthly infant caseloads were identified for participation in the study and systematically assigned to 1 of 3 intervention groups or a control group. The largest center was randomly assigned to a study group, the second largest to another, the third largest to another, and the fourth largest to the last study group. This sequence was repeated until each WIC center was assigned to a study group.
In intervention group 1, children received A/R at their 6-month WIC visit and then every 6 months (at 12, 18, and 24 months of age) during the study period. Parents were reminded to bring their child's immunization card to the 6-month visits.
In intervention group 2, children received A/R at their 6-month WIC visit and at all subsequent WIC visits (generally every 2 months) through 24 months of age during the study period. Parents were requested to bring their child's immunization card to every WIC visit.
In intervention group 3, children received A/R at their 6-month WIC visit and 2 months later at the subsequent WIC visit. Children who were not UTD or did not bring an immunization record to the 8-month visit were considered to be at high risk for underimmunization and continued to receive A/R at each successive visit (generally every 2 months) through 24 months of age. Parents were requested to bring their child's immunization card to every WIC visit. Children who were UTD at the 8-month visit were considered to be at low risk and received no further A/R, but were included in the immunization coverage assessment for this group at the end of the study.
In the control group, children received an immunization assessment at their 6-month WIC visit. Referrals were not made, and no more assessments were conducted during the studyperiod.
The process of A/R was standardized. If the parent provided an immunization record, vaccination dates were entered into PHFE-WIC's computerized immunization registry, which automatically assessed the child's immunization status. If the child was due or overdue for any vaccine doses, WIC staff referred parents to their child's health care provider or a free clinic in the area. Parents of UTD children were congratulated. The intervention period lasted from August 1, 1997, through November 1, 1999.
This study was deemed exempt from institutional review board approval because the A/R intervention was not consistently used throughout Los Angeles County. It was conducted only at those WIC agencies receiving county or state funding for this activity. Not all of the WIC agencies in Los Angeles County were funded, and those that were modified their A/R programs according to their needs and resource limitations.
The study includes the following 3 assessments of children's UTD status by 24 months of age: (1) the baseline survey, which assessed UTD rates before the intervention began; (2) the postintervention survey, which looked at UTD rates after the intervention was conducted; and (3) the validation survey, also performed after the intervention period, which assessed the effect of the WIC registry on participation rates and coverage levels. To be considered UTD for vaccinations by 24 months of age, children had to have received 4 diphtheria and tetanus toxoids and pertussis/acellular pertussis, 3 oral/inactivated poliovirus, 1 measles-mumps-rubella, 3 Haemophilus influenzae type b, and 3 hepatitis B vaccine doses (4:3:1:3:3 series). Additional analyses using the 4:3:1:3 (4 diphtheria and tetanus toxoids and pertussis/acellular pertussis, 3 oral/inactivated poliovirus, 1 measles-mumps-rubella, and 3 Haemophilus influenzae type b vaccine doses) and the 4:3:1 series (4 diphtheria and tetanus toxoids and pertussis/acellular pertussis, 3 oral/inactivated poliovirus, and1 measles-mumps-rubella vaccine dose) were also conducted. Age-appropriate immunization coverage levels were also determined at 6, 12, and 19 months of age.
To determine preintervention coverage at the study sites, we surveyed a total of 1452 children born between April 1 and July 1, 1995, who were continuously enrolled at a WIC center (at a single center or at another in the same study group) from 6 through 24 months of age. These children did not receive any A/R. An error discovered after the study showed that 1 WIC center from intervention group 1 was not sampled from, leaving 19 centers for the baseline survey. We do not believe that this had an effect on the final results.
Data were obtained from parent interviews and health care provider records. Parents of all children were interviewed by WIC staff via telephone. Once they consented to participate, respondents were asked to refer to their child's home immunization record for vaccination dates and for verbal consent to contact their health care provider. If a child was UTD according to the family interview, the child's immunization record was considered complete, and health care providers were not contacted. For children who were not UTD according to the family interview, vaccination history requests were sent to their health care providers. Health care providers who did not respond to the mailing received follow-up by means of telephone and facsimile. Immunization information from health care providers was consolidated with information from the family.
To be eligible for the postintervention survey, children had to be enrolled in the same WIC center or study group continuously throughout the entire intervention period from 6 through 24 months of age.
Data were obtained from the following 3 sources: WIC registry, parent interviews, and health care provider records. After the study period, 450 children aged 24 to 27 months who met eligibility criteria were randomly selected from each study group to have their vaccination status evaluated. Vaccination dates were first obtained from the WIC registry. The registry was developed to conduct computerized A/R for this study with the intent to expand it to all PHFE-WIC centers. As they received the A/R intervention, this group of children was the first to populate the registry. If a child was UTD according to the registry, the child's immunization record was considered complete for study purposes, and data collection ended. For those children who were not UTD according to the WIC registry, data collection continued with follow-up with parents and health care providers, as described for the baseline survey.
Since the WIC registry only contained updated immunization dates for the intervention groups (not the control group, since those children received an assessment only at 6 months of age), a validation survey was conducted to assess the effect of the WIC registry on participation rates and coverage levels for the 3 intervention groups. Registry data were excluded from the analysis so that the data sources for the 3 intervention groups and the control group were identical, thus eliminating potential biases in data collection methods between groups. From each intervention group, 175 children were randomly sampled from the 450 children previously selected for the postintervention survey. For the control group, all 450 children previously sampled for the postintervention survey were included in the validation survey.
Data were obtained from parent interviews and health care provider records in the same manner as the for the baseline and postintervention surveys. Unlike those surveys, health care providers of all children selected for the validation survey were contacted, regardless of the children's UTD status according to the parent-held record.
Analyses were conducted using SAS statistical software.15 We initially conducted descriptive and tabular (categorical data) analyses to examine data distributions, identify outliers, and observe patterns in exposure (intervention)–outcome relationships. Stratified analyses and logistic regression modeling followed these preliminary analyses. The regression models included 3 indicator variables (one for each intervention group), other variables identified a priori as potential confounders, and WIC center baseline coverage rates. Race and education were modeled with series of indicator variables; income and family size were modeled as continuous covariates. Because of correlated observations within WIC centers, the GENMOD procedure in SAS was used to fit the logistic models and to produce SE estimates and confidence intervals that take such clustering into account.16,17 We compared model results with results from tabular analyses, which involved the cross-classification of all model variables (regressors) withthe outcome. We used the Hosmer-Lemeshow goodness-of-fit test to assess model fit.
Children were included in the analysis if they met 1 of the following criteria: (1) The child received all doses of vaccinations according to a single or a consolidation of data sources: for the baseline and validation surveys, parent-held record and health care provider data, and for the postintervention survey, registry, parent-held record, and health care provider data;(2) The child was not UTD, but the parent interview contributed 1 or more vaccinations to the child's record; and (3) The child was not UTD, but 1 or more of the child's health care providers responded to the request for vaccination records.
The children who met the inclusion criteria for analysis were considered study participants. Participation was higher for the postintervention survey (75%) compared with the baseline and validation surveys (70% and 69%, respectively) (Table 1). For the postintervention survey, we found significant differences in participation rates (P<.001), with intervention groups 2 and 3 having the highest rates. Participation rates among the different study groups in the baseline and validation surveys were similar.
For the baseline survey, the population was predominantly Hispanic (Table 2); however, we found significant differences among study groups in race/ethnicity (P<.01). This is probably due to the larger number of African American children enrolled in group 1 and the control group. No other demographic or socioeconomic data were available on the baseline population. The populations for the postintervention and validation surveys were similar as they were both primarily Hispanic (Table 2). In these groups, we also found no substantive differences between their populations in monthly income, family size, or highest level of parental education.
For the postintervention survey, we compared the demographic characteristics of participants with those of nonparticipants. As with the participants, nonparticipants were predominantly Hispanic (80%); however, only 5% of participants were African American, compared with 14% of nonparticipants (P<.001). Parents of nonparticipants were more likely to have completed high school or some college than parents of participant children (P = .03). The mean monthly incomes for the participants and nonparticipants were $1010 and $881, respectively; mean family sizes, 3.9 and 3.7 family members, respectively. Participants were similar in their characteristics across study groups; this was also true for nonparticipants.
Table 3 shows the 4:3:1:3:3 series immunization coverage levels for the baseline, postintervention, and validation surveys by 24 months. For the postintervention survey, immunization coverage levels were similar among the 3 A/R intervention groups, and differences were not statistically significant compared with those of the control group. The validation survey confirmed similar overall immunization coverage rates (87%) with those found in the postintervention survey (89%). Coverage rates did not differ among study groups. There was no difference between the UTD rates of the control group or the intervention groups in the postintervention and validation surveys.
Overall, coverage was higher among the postintervention (89%) than the baseline population (77%; P<.001). In the baseline survey, the higher coverage level for intervention group 3 was of borderline significance compared with that of the control group. Using the 4:3:1 and 4:3:1:3 series as UTD criteria for 24 months of age also did not reveal any substantial or statistically significant differences in immunization coverage among thedifferent study groups.
When logistic regression models were used to control for potential confounding effects of WIC center baseline coverage rates, race/ethnicity, monthly household income, and parents' highest level of education for the postintervention survey, immunization coverage levels remained similar among the different study groups (Table 3). Immunization coverage levels for the postintervention population were also calculated retrospectively (Figure 1). At the 6-, 12-, and 19-month age milestones, intervention group coverage levels were comparable to those of the control group (P = .19, P = .26, and P = .38, respectively). Although not statistically significant, we found a slight increase in UTD rates for groups 2 and 3 at the 12- and 19-month checkpoints that disappeared by 24 months.
Postintervention survey of immunization coverage rates at 6, 12, 19, and 24 months of age for children enrolled in the Special Supplemental Nutrition Program for Women, Infants, and Children in a study of the effectiveness of assessment and referral in raising immunization coverage in Los Angeles County (1997-1999). Groups are described in the "Study Groups" subsection of the "Methods" section. Six-month up-to-date (UTD) criteria consisted of 2 diphtheria and tetanus toxoids and pertussis/acellular pertussis, 2 oral/inactivated poliovirus, 2 Haemophilus influenzae type b, and 2 hepatitis B vaccine doses; 12-month UTD criteria, 3 diphtheria and tetanus toxoids and pertussis/acellular pertussis, 2 oral/inactivated poliovirus, 2 H influenzae type b, and 2 hepatitis B vaccine doses; and 19- and 24-month UTD criteria, 4 diphtheria and tetanus toxoids and pertussis/acellular pertussis, 3 oral/inactivated poliovirus, 1 measles-mumps-rubella, 3 H influenzae type b, and 3 hepatitis B vaccine doses. For χ2 comparison of study group immunization coverage levels, asterisk indicates P= .19; dagger, P= .26; double dagger, P= .38; and section sign, P= .77.
The WIC registry underestimated true coverage by 55%. For the postintervention survey, only 34% of children were UTD in the registry. The addition of follow-up with parent interviews and provider records increased the UTD rate to 89%.
In this study population with high baseline immunization coverage rates and continuous enrollment in WIC through 24 months of age, we found that conducting A/R of WIC clients without other accompanying interventions did not raise vaccination coverage. Although we found an overall increase in 24-month coverage between the baseline and postintervention survey groups from 77% to 89%, the postintervention coverage rates of the intervention groups were not substantively or significantly different from that of the control group. Hence, the results of this study provide no evidence that immunization A/R alone increased immunization coverage in this population.
To our knowledge, this is the first published study to evaluate the effectiveness of immunization A/R alone using a controlled trial design. Other studies have examined the impact of immunization A/R alone on immunization coverage of WIC clients.9,18 In the first of these,18 the results were inconclusive due to high rates of loss to follow-up (ie, unable to locate, changed WIC center, or did not present immunization card), inconsistent results at different age milestones, and the possibility of ascertainment bias owing to the lack of health care provider validation. The second study compared 2 interventions, A/R plus MVP and A/R alone, with historical control groups and found no evidence of an impact of the A/R alone intervention on immunization coverage.9
The findings of 2 other studies of immunization interventions in WIC suggested that A/R used alone has little impact on immunization coverage. One of those studies compared immunization coverage in subjects receiving the A/R intervention every 2 months and once every 6 months and found no difference.19 In another study comparing measles vaccine uptake in WIC subjects receiving A/R alone with those receiving A/R with MVP or clinic escort, rates of measles vaccination were dramatically (3-5 times) lower for A/R alone compared with the other groups, again suggesting that this intervention likely had little impact.7
Immunization histories contained in the WIC registry were found to be incomplete and did not reflect the current UTD status of the children in WIC. Similar findings were seen in studies conducted in Milwaukee, Wis, and Massachusetts, where provider validation confirmed that the WIC registry had vastly underestimated actual UTD status by 45% and 31%, respectively.11,19 WIC registries rely on parents bringing their children's records to the WIC center, so when used alone, such registries may lead to misclassification of fully immunized children as not UTD.
Immunization coverage levels in this low-income, high-risk population were considerably higher than expected, at greater than 75% for all surveys. There areseveral possible reasons for this finding. Even before the implementation of the intervention at the study sites, immunization education had been a counseling topic during WIC visits. WIC has strong linkages with health care providers and, as a condition of participation, requires medical information from them. Therefore, when present in the health care provider's office, children have an increased chance of receiving immunizations. WIC's focus on health care access and the concept of a medical home may help to explain these high immunization rates. Studies have demonstrated that children enrolled in WIC had higher immunization rates than non–WIC-enrolled children in families with comparable income levels.5,20,21
Another possible explanation for the high coverage rates is that the study populations consisted of children who were continuously enrolled in 1 WIC center from their 6-month visit until 24 months of age. Continuity with WIC or 1 site may be a sign of family stability and higher immunization rates, whereas change of residence has been a factor associated with lower immunization coverage.22
In addition to the high coverage levels, we found a substantial increase in immunization coverage from the baseline to postintervention survey. This may be explained, in part, by changes in California's school and child-care entry immunization requirements that occurred in 1997. Starting in August 1997, hepatitis B immunization and a second dose of a measles-containing vaccine were added entry requirements; the accompanying news media campaigns and community outreach efforts may have prompted many parents to have their children immunized. A similar, albeit more modest, increase was also seen in Los Angeles County's coverage rates (from 65% to 71%) between 1997 and 1999.23
Although the participation rate for the postintervention survey was 75% overall, there were significant intergroup differences. The groups that received A/R had study participation rates 6% to 12% higher than the control group (Table 1). These differences were most likely due to the availability of immunization histories in the WIC registry for the intervention groups. The validation survey controlled for the effect of the WIC registry data and confirmed our findings from the postintervention survey.
Since the baseline and postintervention survey groups both consisted primarily of Hispanic children (Table 2), results of this study may not be generalizable to WIC populations with different racial/ethnic characteristics. Although the largely Hispanic populations in this study had high coverage levels, other studies have shown Hispanic and non-Hispanic African American children to have lower immunization rates than other racial or ethnic groups in the United States.24- 26
In addition, the results of this study may not be generalizable to populations with lower immunization coverage rates. Coverage levels at 24 months of age were so high in the control group after the intervention period (88%) that a ceiling effect left only a small margin for immunization coverage improvement in the last and probably most difficult-to-reach 12% of the subjects included in the study population. However, even when an earlier age checkpoint of 19 months was used, although overall coverage rates were lower (78%), only slightly higher coverage rates were found in groups 2 and 3.
Although there were some differences in demographic characteristics between participant and nonparticipant children in the postintervention survey, the participant children were similar in characteristics across study groups, as were the nonparticipant children. Therefore, any bias due to the differences in characteristics between participants and nonparticipants cannot completely explain the lack of effect of the intervention.
It was beyond the scope of the study to assess precisely how well the intervention was implemented; therefore, it is quite possible that the A/R intervention was not implemented per protocol 100% of the time. However, this study evaluated the effectiveness of A/R in a real-world setting of WIC agencies where, because of staff time pressures and other unforeseen circumstances, complete adherence to protocol may not be possible.
Considering the results of our study and previous studies7,9,18,19 and a decade of evaluation research efforts, no study has convincingly demonstrated a true and substantive relationship between the A/R alone intervention and increased immunization coverage of WIC clients. We recommend more research to determine whether this widely used and resource-intensive intervention has a positive impact in improving immunization rates among WIC clients, especially in WIC populations with lower baseline coverage rates.
Our study also emphasizes the importance of using health care provider immunization histories for future studies in WIC. We recommend that future studies evaluating WIC strategies use immunization information obtained by the investigators by directly contacting the clients' families to obtain personal vaccination records and/or those of the health care providers, rather than relying exclusively on WIC registry data obtained by WIC centers when families bring in their children's immunization records.
Corresponding author: Susan H. Ashkar, MA, Department of Health Services, Public Health, Immunization Program, County of Los Angeles, 3530 Wilshire Blvd, Suite 700, Los Angeles, CA 90010 (e-mail: email@example.com).
Accepted for publication November 8, 2002.
This study was supported in part by grant 317 from the Centers for Disease Control and Prevention, Atlanta, Ga.
We thank Cheri Todoroff, MPH, Loren Lieb, MPH, Kimberly J. Holmquist, Kristin Brusuelas, MPH, Deborah A. Starbuck, MPH, RD, and Eloise Jenks, MEd, RD, for their support throughout the project; Otilia Galindo, Ramona D. Meza, and Mary McCauley, MTSC, for their assistance; and all of the dedicated PHFE-WIC staff who participated in this project.
The basic immunization-promoting strategy in WIC is assessing the vaccination status of children and referring those in need of vaccination to their immunization provider. This A/R strategy has been shown to raise immunization coverage when combined with other more intensive interventions in WIC. Although implementation of A/R in WIC is currently a national initiative, studies have not identified whether A/R can be successful alone and at what frequency A/R is most effective.
This study is important because it is among the first to use a control group when investigating the value of the A/R alone intervention. The results demonstrated no substantial relationship between the intervention and increased immunization coverage of WIC clients. More research is needed to guide policy decisions on the most cost-effective strategies to increase immunization rates in the WIC population.
Ashkar SH, Dales LG, Averhoff F, Shefer A, Higa J, Thompson L, Gomez J, Gee DC, Hurwitz EL. The Effectiveness of Assessment and Referral on Immunization Coverage in the Special Supplemental Nutrition Program for Women, Infants, and Children. Arch Pediatr Adolesc Med. 2003;157(5):456-462. doi:10.1001/archpedi.157.5.456