Copyright 1998 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.1998
To evaluate the impact of interventions by a community-based organization on immunization rates.
Controlled community intervention trial.
Setting and Participants
Children aged 3 to 59 months in Fulton County, Georgia, who were patients of 1 of 4 public clinics (clinic based), or residents of 1 of 9 inner-city communities (residence based).
(1) Clinic-based intervention included monthly review of clinic vaccination records to identify undervaccinated children followed by contact with family (reminder-recall strategy); (2) residence-based intervention included door-to-door assessment and education campaigns followed by mobile van vaccinations, temporary on-site vaccination stations, free child care and transportation to providers, incentives of food and baby products, focus groups, and coalitions with local organizations (community saturation with vaccination messages and opportunities).
Change in vaccination rates after 1 year based on clinic record reviews and population surveys.
For clinic-based intervention, series completion rates improved from 43% (87/204) to 58% (99/170) in intervention clinics (P=.003), while rates in control clinics did not change from the baseline of 52% (81/157 to 78/150), for a net difference between intervention and control arms of +15 percentage points (P=.046). For residence-based intervention, age-appropriate vaccination rates improved from 44% (154/347) to 61% (260/429) in intervention communities (+17 percentage points; P<.001) compared with improvement of 44% (78/178) to 58% (129/221) for control communities (+14 percentage points; P=.004), but the difference between arms was not significant (+3 percentage points, P=.78).
Reminder-recall activities by the community-based organization improved vaccination rates in intervention clinics compared with control clinics. A statistically significant impact on vaccination rates could not be detected for residence-based interventions by the community-based organization.
IN 1986, the Georgia Department of Public Health (Ga DPH) began a program of measurement and feedback of vaccination rates in public clinics that over a period of 9 years helped raise clinic coverage from 40% to 89%.1 However, in Fulton County (which includes most of urban Atlanta), public clinic coverage was 48% in 1992 compared with 86% in the rest of the state, where at the time approximately two thirds of children were vaccinated in public clinics.1 A number of indexes suggested the existence of high-risk groups. Fulton County was second among 3141 US counties in crime. Georgia had the second highest rate of infant mortality in the nation,2 and rates among certain African American subgroups were worse than all but 7 nations in the world.3- 5
In 1992, the CDC awarded $467000 of demonstration project funds to a community-based organization to improve vaccination rates among children living in underserved communities in Fulton County. Funding community-based organizations to provide outreach to the urban poor is a common strategy in public health (eg, between 1986 and 1996 more than $400 million was provided by CDC to community-based organizations for a variety of public health projects; CDC unpublished data, 1997), but reports of controlled studies are lacking. This is the first prospective controlled study, to our knowledge, to examine the impact of a community-based organization on vaccination rates.
The organization in receipt of the award had been in existence since 1984 with a mission of providing health-related outreach to disadvantaged populations in the county. It had an office with 28 paid staff, 300 volunteers, and several mobile vans. It was supported by private donations and government grants, had established ties with local groups, particularly those in Hispanic and public housing communities, and had the support of local and state health departments.
Subjects were Fulton County children 3 to 59 months of age who were (1) patients in 1 of 4 public clinics: 2 public clinics serving predominantly African American populations and 2 serving predominantly Hispanic populations, with 1 intervention and 1 control clinic for each ethnic category and (2) residents of 1 of 9 lower socioeconomic communities: 6 public housing communities with predominantly African American populations (3 intervention, 3 control) and 3 private housing communities with predominantly Hispanic populations (2 intervention, 1 control).
Consistent with demonstration project methods, intervention sites were chosen by the organization based on established relationships of the organization to the underserved communities and clinics, with the advice and consent of the Fulton County Department of Health, Ga DPH, and CDC. Control sites were selected prospectively by agreement of the same partners based on racial/ethnic composition of the populations and comparability in access to care. None of the intervention clinics was located in a control community and none of control clinics were located in intervention communities; hence, the 2 groups of study subjects were distinct.
Interventions were chosen by the organization based on prior experience and were applied for a duration of 1 year (September 1, 1992, to August 31, 1993).
In the 2 intervention clinics, workers reviewed vaccination records monthly, identified children in need of vaccinations, and made contact with the family, usually by telephone or mail but, where necessary, by home visit.
The project was begun in the 5 intervention communities with door-to-door campaigns to (1) identify the undervaccinated, (2) provide immunization education to families, (3) provide them with culturally sensitive promotional literature, and (4) introduce them to the services the community-based organization would provide. This was followed by establishment of weekly mobile van vaccinations and/or temporary on-site vaccination stations, provision of free child care and transportation to clinics for vaccinations, and incentives of food and baby products. Focus groups were formed to help identify ways in which the organization could better serve the community, and coalitions were developed with other community organizations, such as local churches and child care centers. The spectrum of interventions was intentionally broad, improvisational, and opportunistic, rather than sharply defined and invariant across populations and study period. The objective was to saturate the at-risk communities in as many ways as possible with a variety of vaccination messages and opportunities, a common method among community-based organizations.6
The Ga DPH has been measuring vaccination rates in every public clinic in the state since 1986 with methods that have been described in detail,1 and no changes in these methods were made for purposes of this study. In brief, the Ga DPH district immunization officer visited each county clinic and reviewed the health records of all children who had been seen in the clinic and were 21 to 23 months of age at the time of the officer's visit. Each child's birthdate and vaccination dates were entered by the health officer into a software application and, after excluding children who were documented to have moved or gone elsewhere, vaccination rates were calculated. Included in study analysis were Ga DPH data for 1992 ("before") and 1993 ("after") from the 4 study clinics, as well as from the 15 other county clinics.
A pilot survey conducted in May 1993 indicated that many residents were reluctant to admit unknown persons to their homes or to provide information to outsiders. As a consequence, for the final survey, a community surveyor strategy was adopted: the community-based organization recruited 29 residents from the study communities to conduct the survey, under the supervision of a professional community survey research organization. These community surveyors were paid to attend a training session, and thereafter were paid $3.75 for each completed household survey.
The pilot survey indicated that public and private landlords were unable to provide reliable lists of occupants; many units listed as occupied were actually vacant, while some units listed as vacant were actually occupied with some of the occupants consisting of families with children younger than <6 years. This suggested that sampling techniques would be difficult to apply since the population denominator could not be reliably estimated beforehand. As a consequence, the final survey consisted of a census of all housing units in all 9 study communities, and was conducted between August 28 and October 31, 1993, immediately after the completion of the intervention phase of the project. Unit maps of each community were used to ensure that all units were surveyed, and for each unit, surveyors recorded whether the unit was occupied and the number of children younger than 6 years in each occupied unit. For units determined to be occupied but for which there was no answer, surveyors returned on at least 3 subsequent days. Where the immunization record was available, birthdate and vaccination dates were recorded for each child younger than 6 years. Where the vaccination card was not available, surveyors obtained the child's birthdate and the name of the provider(s); vaccination dates were then obtained from the provider(s). A brief questionnaire was administered to families in intervention communities to assess the extent to which they had been exposed to the activities of the community-based organization.
Population estimates for the total number of children younger than 6 years in each community were obtained by multiplying the average number of children younger than 6 years in each successfully surveyed unit by the total number of units in the community known to have children as occupants. After ascertainment rates were calculated, only children whose vaccination status had been documented were included in analysis. For the "before" subgroup, only children who were 3 to 59 months of age on August 31, 1992, were included and only vaccines received on or before that date were counted; for the "after" subgroup only children who were 3 to 59 months of age on August 31, 1993, were included and only vaccines received on or before that date were counted.
As a comparison to the population survey, we examined immunization rates for 1991 and 1992 in the 6 public clinics serving the 9 study communities. Since nonstudy communities were also served by these clinics, and the clinic assessment method had a much narrower age range (21-23 months) than the population survey, comparisons were imprecise but did provide information on trends in public clinic coverage for the catchment districts in which the study population lived.
Data were entered into SAS7 and analyzed. Age-appropriate vaccination status was determined using Ga DPH standards: by 3 months of age, 1 dose of diphtheria and tetanus toxoids and pertussis vaccine (DTP) and 1 dose of oral poliovirus vaccine (OPV); by 5 months of age, 2 doses of DTP and 2 doses of OPV; by 7 months of age, 3 doses of DTP and 2 doses of OPV; by 16 months of age, 3 doses of DTP, 2 doses of OPV, and 1 dose of measles, mumps, and rubella vaccine (MMR); and by 19 months of age, 4 doses of DTP, 3 doses of OPV, and 1 dose of MMR ("4-3-1 series complete"). Before-and-after changes were assessed using a 2-tailed χ2 test.
In the 2 intervention clinics, series completion rates increased 15 percentage points from 1992 to 1993, compared with 0% for the 2 control clinics (Table 1). The rate of improvement for each intervention clinic exceeded its matched control, but the difference was only statistically significant for 1 pair. In intervention clinics, the proportion of children who were unvaccinated and had not been seen in intervention clinics for a year was reduced from 40% to 21% (P<.001), while in control clinics the proportion was also reduced but not to a statistically significant extent, 37% to 27% (P=.055). Rates of missed opportunities for simultaneous vaccination were not changed significantly in either intervention or control clinics.
When the 19 county clinics providing complete data were rank-ordered by coverage for 1992 and for 1993, the 2 intervention clinics both rose in rank (intervention clinic A, 14th to 8th; intervention clinic B, 8th to 5th), while both control clinics declined in rank (control clinic A, 6th to 12th; control clinic B, 10th to 13th). The coverage improvement of 15 percentage points in the 2 intervention clinics exceeded the improvement in the county's 17 nonintervention clinics (+8 percentage points), but the difference was not statistically significant (P=.157).
The total number of housing units in the 9 study communities was 2093, of which 1273 (61%) were occupied (range, 25%-85%); of these, 600 (47%) were occupied by families known to have children younger than 6 years; 439 (73%) of these units were successfully surveyed, identifying a study population of 755 children (1.72 children <6 years old per unit), of whom the vaccination status was successfully documented for 674 (89%). The total population of children younger than 6 years living in the 9 communities was estimated at 1040 (600 units with children <6 years × 1.72 children/unit). Thus, overall vaccination ascertainment for the survey was estimated at 65% (674 children <6 years with vaccination status documented/1040 estimated total population <6 years).
In 1992, the overall age-appropriate vaccination rate for the children who were 3 to 59 months of age (median, 27 months) was 44%, with identical 44% rates for the intervention and control arms, as well as for children in Hispanic and public housing communities, although substantial coverage rate variation was present among substrata (Table 2). Age-specific coverage rates for children younger than 24 months are imprecise because of small numbers in the youngest age groups, but data suggest that vaccination rates may have been very low around the first birthday: 3 to 4 months, 69% (9/13); 5 to 6 months, 40% (8/20); 7 to 15 months, 14% (14/102); and 16 to 24 months, 32% (31/96).
For comparison to the population survey, the series completion rate in 1992 for children (median age, 22 months) in the 6 public clinics providing services to the 9 surveyed communities was 48% (238/493), a rate that is not significantly different (P=.192) from the 44% (232/525) obtained by the population survey.
Overall age-appropriate vaccination rates rose from 44% in 1992 to 60% in 1993 (+16 percentage points; P<.001) according to the population survey, and improvements were seen in all subgroups (Table 2). Since the median age of the study population increased from 27 months in 1992 to 36 months in 1993, a proportion of this improvement may be attributable to the progressive vaccination of an aging cohort. However, age-specific coverage improved for all 5 age groups except the 7- to 15-month age group (average percentage point change of +13), suggesting that a true improvement in vaccination coverage occurred as well.
For comparison with the population survey, the series completion rate in 1993 for children (median age, 22 months) in the 6 public clinics providing services to the 9 surveyed communities was 57% (235/414), a rate that is not significantly different from the 60% (389/650) rate obtained by the population survey (P=.320). The improvement of 9 percentage points in these public clinics over the previous year (P=.011) is also consistent with the population survey (P=.144).
The increase in vaccination rates among children in intervention communities was 3 percentage points greater than among children in control communities but the difference was not statistically significant overall or for any stratum (Table 2).
For comparison with the population survey, series completion rates in the 4 clinics serving intervention communities improved from 53% (139/262) in 1992 to 58% (124/215) in 1993 (+5 percentage points; P=.313), while rates in the 2 clinics serving control communities improved from 43% (99/231) in 1992 to 56% (111/199) in 1993 (+13 percentage points; P=.008), for a nonsignificant 8–percentage point difference in improvement rates favoring controls (P=.201).
A total of 341 families with children younger than 6 years were estimated to be residing in the 5 intervention communities, and 264 (77%) were successfully surveyed. Of these, 75% were familiar with the activities of the community-based organization, and the majority found the organization's staff to be friendly and professional. However, 76% preferred to obtain their children's vaccinations from their usual health care provider rather than the organization's mobile van or temporary clinics. The distribution of usual health providers was public clinic, 63%; private physician, 22%; and other provider, 15%. Reported yearly frequency of health care visits was as follows: 0 visits, 11%; 1 visit, 17%; 2 to 5 visits, 63%; and more than 5 visits, 9%.
We report the first controlled study, to our knowledge, of the impact of interventions by a community-based organization on vaccination rates. Results suggest that the organization's clinic-based reminder-recall activities may have produced improvements in coverage compared with controls, but no impact was detected where a broad spectrum of residence-based interventions was aimed by the same organization at particular communities.
Baseline coverage estimates from both the population (44%) and clinic surveys (47%) indicated low coverage among study children. The consistency of results suggests that costly and difficult residence-based surveys may not identify populations in greater need than would be found in a review of records for providers serving the same community. These findings agree with other studies that found that undervaccinated children visit providers, often frequently,8- 12 and thus confirm that door-to-door efforts are not needed to identify and access undervaccinated children.
Generalized improvements in vaccination coverage, independent of study interventions, were found in the population survey, and coverage improvements were also seen in public clinics that were not involved in the study, suggesting general improvements in vaccination rates in the county. The causes of this large-scale improvement cannot be determined by this study. However, a program of measurement and feedback has previously been documented as instrumental in raising coverage 7% to 9% a year in Georgia public clinics during 1988 to 1994, a period that includes the time of this study.1 Since 63% of families surveyed in the intervention communities reported using public clinics as their usual health care provider (consistent with independent data indicating that 65% to 72% of the state birth cohort was vaccinated in public clinics1), one would expect population-based coverage to rise, and the extent of the rise in Fulton County is consistent with what would be expected from public clinic trends.
Intervention clinics improved vaccination rates more than control clinics. Since rates of missed opportunity for simultaneous vaccination did not change in either intervention and control clinics, internal clinic practices do not appear to have improved. In contrast, dropout rates in intervention clinics were reduced almost in half compared with a nonsignificant reduction in control clinics, suggesting that the community-based organization's reminder-recall activities may have been instrumental in improving the vaccination status of children in intervention clinics. The 15–percentage point improvement in coverage in 1 year is also consistent with studies on the effect of provider-based reminders,13- 15 and the organization's methods were similar to those used in such studies: monthly inventory of provider records, followed by contact with the families of unvaccinated children, usually by telephone or mail. However, certain cautions are needed in interpreting the results. Only 4 clinics were involved in the study, the 2 intervention clinics were chosen specifically for their intervention potential, the 2 prospectively selected control clinics underperformed the rest of clinics in the county, and, although intervention clinics improved their standing in a rank-order of county clinics, the difference in coverage improvement between intervention and other county clinics was not statistically significant. On the other hand, using series completion at 21 to 23 months of age as the outcome measure for an intervention lasting only 1 year may underestimate ultimate intervention effect.
For residence-based interventions, impact on vaccination rates was not detected in any strata of the population survey or in the independent data from clinics serving the study communities despite the fact that most of the community-based organization's efforts and resources were directed at this aspect of the project. The majority of children in targeted communities were in need of vaccination, and during the study period vaccination rates improved in both intervention and control communities, so the situation does not appear to have been refractory to change.
Consistent with the demonstration project design, a number of operational factors favored impact: the community-based organization had extensive prior experience with outreach and long-standing ties with the target communities, it was allowed to choose the intervention communities and have input in the selection of control communities, the target populations were relatively small, the project budget was more than $450000, a 6-month preparation period preceded the 12-month intervention, and the size of the surveyed population was adequate to detect an 8–percentage point difference in vaccination rates between intervention and control.
Funding community-based groups to provide interventions to inner-city communities is often justified on the plausible basis that an intermediary organization with established connections to and knowledge of "hard-to-reach" communities may improve the provision of interventions. However, none of the individual interventions used by the organization in the residence-based phase of this study had been tested in previous immunization studies, and their collective effectiveness was not demonstrated in our study. In contrast, provider-based reminder systems have been found effective in multiple studies,13- 15 and there was evidence of impact in our study, though not beyond levels previously reported. These results suggest that in hard-to-reach populations, favorable characteristics on the part of organizations providing interventions may be less important than whether the interventions work.
Interventions using residence-based methods to identify and track unvaccinated children must overcome a number of barriers. In this study, half of all households did not have preschool children, and in other vaccination surveys as many as 17 households had to be contacted to find an eligible child.16 Particularly in high-risk areas, problems can be encountered gaining access to the home, ascertaining the initial vaccination status of the child from records available there,16 monitoring subsequent vaccination status, and identifying children born or migrating into the community after the initial survey. Repeated door-to-door surveys can involve high costs. In contrast, provider-based record reviews can identify, ascertain, and monitor groups of unvaccinated children swiftly, with time-consuming and potentially hazardous home visits saved for those who fail telephone and mail reminders.
Three quarters of surveyed families in the intervention communities were familiar with the activities of the community-based organization and regarded the staff as friendly and professional, but most families preferred to have their children vaccinated by their usual health care provider, which in most cases was a public clinic. Resistance by inner-city families to vaccination outside comprehensive and usual health care has been found in other studies,17,18 and although this resistance may limit the effectiveness of interventions that focus on a single aspect of health, ensuring that children receive nonfragmented, regular care is consistent with public health goals.
Because this study was conducted among a limited number of communities in one city, the results may not apply elsewhere. Selection of intervention sites was not random, but based on the community-based organization's previous ties and perceptions of intervention feasibility; similarly, the organization was permitted input into the selection of control sites. This demonstration project method explicitly favors the detection of impact. In contrast to hypothesis-testing studies in which biases favor the null, the absence of an impact under demonstration project circumstances is less likely to be attributable to study constraints, whereas an impact that is detected requires confirmation with more rigorous study conditions.
Other limitations should be noted. In the population survey, the number of children in each community had to be estimated based on survey findings, and the vaccination ascertainment rate was 65%, with no data available concerning the children not surveyed or ascertained. Although higher ascertainment rates have been reported for vaccination surveys of representative samples of the general population, the rates for this study are consistent with others in which high-risk populations were targeted.8,10,19 The study population was not characterized, risk factors for undervaccination were not identified, the age ranges of the clinic-based and residence-based study populations were not identical, and the 4-3-1 immunization standard has now been replaced by a larger battery of antigens. Since the community-based organization did not provide a few highly defined interventions to target communities for the duration of the study, no conclusions can be drawn concerning the efficacy of any of the many individual interventions they used.
The use of community-based organizations is common in areas of public health where access to high-risk urban populations is desired, but a search of the literature did not suggest that controlled trials of impact on public health outcomes have been reported to date. The one previous study in immunization reported that, among 295 inner-city children targeted by community-based organizations in New York City, age-appropriate vaccination rates improved over time, but there was no population denominator, no ascertainment rate, and no controls, and project costs were not provided.20 Where legislators and other public health decision makers choose to invest heavily, as they have in the past, in providing interventions through community-based organizations, they should consider studies involving controls to evaluate impact on measurable public health outcomes.
Accepted for publication November 3, 1997.
Funding for the demonstration project and its evaluation was provided by a grant from the National Immunization Program, Centers for Disease Control and Prevention, to the Georgia Department of Public Health.
The authors thank the Mercy Mobile Health Program for its industry, dedication, and commitment in providing the interventions evaluated in this article; the Fulton County Health Department for its help and cooperation in the public clinic and other aspects of this study; and Edward Hoekstra, MD, Dennis O'Mara, Lance Rodewald, MD, and Roger Bernier, PhD, for offering their helpful comments on the manuscript.
Reprints: Charles W. LeBaron, MD, Mail MS E-52, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, GA 30333 (e-mail: email@example.com).
LeBaron CW, Starnes D, Dini EF, Chambliss JW, Chaney M. The Impact of Interventions by a Community-Based Organization on Inner-city Vaccination CoverageFulton County, Georgia, 1992-1993. Arch Pediatr Adolesc Med. 1998;152(4):327-332. doi:10.1001/archpedi.152.4.327