Customize your JAMA Network experience by selecting one or more topics from the list below.
Since 1995, states and jurisdictions receiving federal immunization funds have been required to perform annual measurements of vaccination coverage in their public clinics, based on data from Georgia where clinic coverage increased after the institution of a measurement and feedback intervention.
To determine if clinic vaccination coverage improved in localities that used the Georgia intervention model.
Retrospective examination of clinic vaccination coverage data.
Children aged 19 to 35 months enrolled in clinics in localities that had applied the intervention for 4 years or longer.
The Georgia intervention model: assessment of clinic vaccination coverage, feedback of the information to the clinic, incentives to clinics, and promotion of exchange of information among clinics (AFIX).
Main Outcome Measure
Change in median clinic coverage rates, based on the primary (4-3-1) vaccine series, with comparison to results of the National Immunization Survey.
Four states and 2 cities that had applied the AFIX intervention for 4 years or longer were identified. The number of clinic records reviewed annually was 4639 to 18,000 in 73 to 116 clinics for states, and 714 to 5276 in 8 to 25 clinics for cities. Median clinic coverage rose in all localities: Missouri, 44% (1992) to 93% (1997); Louisiana, 61% (1992) to 83% (1997); Colorado, 55% (1993) to 75% (1997); Iowa, 71% (1994) to 89% (1997); Boston, Mass, 41% (1994) to 79% (1997); and Houston, Tex, 28% (1994) to 84% (1997). The increase in clinic coverage exceeded that of the general population in 5 localities and was identical in the sixth. The average annual coverage rise attributable to the intervention was +5 percentage points per year (Georgia, +6 per year). The average crude direct program cost was $49,533 per locality per year.
The Georgia intervention model (AFIX) can be reproduced elsewhere and is associated with improvements in clinic vaccination coverage.
VACCINATION RATES among 2-year-old children enrolled in Georgia public clinics rose from 53% to 89% during a period of 7 years, following implementation of an intervention involving annual measurement of coverage in each clinic, feedback of the data to the clinic, nonmonetary awards to high-performing clinics, and facilitation of exchange of information among the clinics through regular district and statewide meetings.1 This intervention model has been given the acronym of AFIX (assessment, feedback, incentives, and exchange), and in Georgia was associated with relatively low program costs (25% of the work time of 8 district health workers and 1 supervisor).2
Since 1995, all states and jurisdictions receiving federal immunization funds have been required to perform annual assessments of vaccination coverage in all public clinics (Senate appropriation language, 1995-1997), although the other components of the AFIX strategy have not been mandated. State immunization program managers recently reported that coverage had been measured in approximately 75% of the nation's estimated 4400 public clinics in 1996 (Centers for Disease Control and Prevention, unpublished data, 1998).
Only 1 or 2 years of data are available from states that began public clinic assessments in response to the federal requirement. The improvements seen in Georgia were slow and incremental (+6 percentage points per year), so it will be some time before the national impact of this initiative can be reliably assessed. However, several states and cities began to implement components of the AFIX intervention in their localities before the congressional requirement. We examined data from these sources to help address the question of whether the Georgia methods and results could be reproduced elsewhere. This narrow question of reproducibility has large implications—if clinic coverage did not rise in localities that attempted to apply the Georgia model vigorously for several years, does the nationwide effort of clinic vaccination assessments need to be reevaluated?
Materials and methods
Selection criteria for inclusion
A state or city was required to meet 3 criteria: (1) to have performed annual vaccination coverage measurements in its clinics at least since 1994, a duration that permitted comparison of clinic data with published results of the National Immunization Survey (NIS); (2) to have used the resulting information for feedback, incentives, and exchange of information; and (3) to provide us with clinic-specific coverage data.
The 50 states and 14 jurisdictions in receipt of federal immunization funds are required to submit to the Centers for Disease Control and Prevention annual reports that verify that they have met grant requirements, including measurement of public clinic coverage. We inspected these reports to identify candidate states and cities, then contacted program staff to determine if criteria were met.
Estimation of clinic coverage
The Centers for Disease Control and Prevention provides a standard software package for entering and analyzing clinic vaccination data (Clinic Assessment Software Application, or CASA), and the methods by which it estimates coverage are described in detail elsewhere.3 In brief, the records of children aged 19 to 35 months who had been seen in the clinic are randomly sampled and, for those who had not been documented to have moved or gone elsewhere for care, dates of vaccination are entered into the program. Coverage rates are based on documented receipt of the 4-3-1 set of antigens: 4 doses of the diphtheria toxoid, tetanus toxoid, and pertussis vaccine; 3 doses of polio-containing vaccine; and 1 dose of the measles, mumps, and rubella vaccine.
Analysis of trends in clinic coverage
We analyzed coverage data from each state in the same manner as had been previously used for Georgia.1 For each cycle of measurement, we created a distribution plot by rank ordering all clinics in a locality by coverage rate, plotting this distribution in deciles, and determining the median. Change over time in these indicators of clinic coverage was the primary study outcome.
Intervention activities and costs
We administered a questionnaire to determine the clinic measurement activities, how the data were used, the estimated proportion of the birth cohort served by the clinics for 1997 (the last year under analysis), and crude program costs in full-time equivalent employees (FTEs) and current annual expenses. We did not attempt to estimate indirect costs.
Population-based coverage changes
Since 1994, the NIS has provided population-based estimates of series-completion rates for children aged 19 to 35 months living in each of the 50 states and in 27 selected urban areas, with methods that have been described elsewhere.4 For each locality, we compared clinic measurements with published NIS estimates for 2 years: the first full year for which NIS provided data (July 1994 to June 1995)5 and the most recently available comparable full year of data (July 1996 to June 1997).6
Identification of states and cities meeting criteria
From the 64 reports, we identified 14 candidate states and cities that had conducted clinic coverage measurements at least since 1994. On further investigation, 3 were excluded for not having used the measurement data in a way that met study criteria. One had measured clinic coverage but had not shared the information with clinics (coverage rates were reported to be static in the 60% range). A second had measured coverage in its public clinics and provided feedback, but had not offered incentives or opportunities for exchange of information (coverage rates were reported to be static in the 60% range). A third had used a public sector vaccination registry to make clinic-specific coverage data available to clinics and to provide reminder-recall to the patient population, but had not offered incentives or opportunities to exchange information (coverage rates were reported to have risen over time to the 80% range, where they were stable). Five others reported having used the data in a way that met study criteria (all reported coverage improvements), but we were unable to obtain interpretable data: 2 because data were missing for certain years, 1 because the state's software records could not be read by the Centers for Disease Control and Prevention software, and 2 because no data were supplied. Four states and 2 cities were included in the analysis.
Intervention activities, costs, and vaccination coverage changes
Localities are presented in order of the duration and size of their clinic assessment program. All programs are ongoing. The total number of clinics in 4 localities varied from year to year, and the range is shown.
The data for Missouri are shown in Figure 1. Coverage has been measured annually since 1992 in each of the state's 113 to 114 clinics (which provide vaccinations to an estimated 32% of the state birth cohort of 72,804), with 9273 to 16,368 records reviewed each year. At the time of each measurement, state staff assessed each clinic's compliance with the Standards for Pediatric Immunization Practices.7 In a face-to-face feedback session conducted immediately after the measurement, clinic staff were provided with the coverage results and a set of recommendations of how coverage could be improved based on the standards. Rank-order lists of clinics and maps were published yearly in the state publication, Missouri Epidemiologist, with descriptions of the strategies of successful clinics. Nine state employees (with an annual salary of $33,636) devoted 10% of their work time to these activities, for a total of 0.9 FTE. The costs of supervision, clerical support, travel, and supplies were about $10,616, for a total estimated annual program cost of $40,888. Median clinic coverage rose from 44% in 1992 to 93% in 1997, or +49 percentage points over 5 years (+10 percentage points per year).
Distribution of clinics in Missouri by vaccination coverage. The percentage of children aged 19 to 35 months who were series complete based on the 4-3-1 set of antigens (as described in the "Estimation of Clinic Coverage" subsection of the "Materials and Methods" section) is shown on the x-axis. The percentage of clinics among the total number of clinics measured is shown on the y-axis. Each bar represents the proportion of clinics falling into each coverage increment. The arrow indicates the median clinic coverage; the bracketing numbers at the ends of the horizontal line, the 25th (left) and 75th (right) percentiles; and the length of the bracket, the interquartile distance around the median.
The data for Louisiana are shown in Figure 2. Coverage has been measured annually since 1992 in each of the state's 103 to 107 clinics (which provide vaccinations to an estimated 60% of the state birth cohort of 65,574), with approximately 16,000 to 18,000 records reviewed each year. Feedback was conducted immediately after measurement of coverage, with subsequent faxing of a formal report. Rank-order lists of clinics and health districts were distributed during the annual state immunization conference, at which time awards were given for achievement. Ten immunization program specialists (with an annual salary of $32,077) devoted 17% (2 of 12 months) of their time to these activities, for a total of 1.7 FTEs. The costs of supervision, clerical support, travel, and supplies were about $25,000, for a total estimated annual program cost of $78,462. Median clinic coverage rose from 61% in 1992 to 83% in 1997, or +22 percentage points over 5 years (+4 percentage points per year).
Distribution of clinics in Louisiana by vaccination coverage. See legend to Figure 1 for further details.
The data for Colorado are shown in Figure 3. Coverage has been measured annually since 1993 in 73 to 91 of the state's 91 clinics (which provide vaccinations to an estimated 30% of the state birth cohort of 54,050), with 4639 to 14,485 records reviewed each year. Feedback was provided in written form and mailed to the clinic subsequent to the measurement. A detailed annual compilation of every clinic's performance was distributed to all clinics and health units. A quarterly state immunization publication highlighted several individual clinics' achievements. At the annual statewide immunization conference, awards were given and clinics shared successful strategies. One public health nurse (with an annual salary of $63,833) devoted 60% and one administrative assistant (with an annual salary of $28,750) devoted 40% of their work time to these activities. The program is converting to registry-based measurements, involving 10% of the work time of a programmer (with an annual salary of $42,000), 30% of the work time of a data analyst (with an annual salary of $71,333), and 20% of the work time of a customer service representative (with an annual salary of $37,500), for a total of 1.6 FTEs. The costs of supervision, clerical support, travel, and supplies were about $15,000, for an estimated total annual program cost of $97,900. (This excludes other costs associated with the registry, for which the total yearly budget is approximately $250,000.) Median clinic coverage rose from 55% in 1993 to 75% in 1997, or +20 percentage points over 4 years (+5 percentage points per year).
Distribution of clinics in Colorado by vaccination coverage. See legend to Figure 1 for further details.
The data for Iowa are shown in Figure 4. Coverage has been measured annually since 1994 in each of the state's 84 to 116 clinics (which provide vaccinations to an estimated 32% of the state birth cohort of 35,233), with 11,119 to 13,660 records reviewed each year. Feedback was conducted immediately after measurement of coverage, using numerical and graphical output. A yearly report summarizing the results for every clinic in the state was mailed to each clinic and each county board of health, simultaneous with a state media release. At the annual statewide immunization conference, awards were presented to clinics meeting 90% coverage goals, and successful clinics made presentations on their approaches. Six state employees (with an annual salary of $39,396) devoted 12% of their work time (1.5 of 12 months) to these activities, for a total of 0.7 FTE. The costs of supervision, clerical support, travel, and supplies were about $10,000, for an estimated total annual program cost of $38,685. Median clinic coverage rose from 71% in 1994 to 89% in 1997, or +18 percentage points over 3 years (+6 percentage points per year).
Distribution of clinics in Iowa by vaccination coverage. See legend to Figure 1 for further details.
The data for Boston are shown in Figure 5. Coverage has been measured semiannually since 1994 in each of the city's 20 to 25 clinics (which provide vaccinations to an estimated 60% of the city birth cohort of 8020), with 3203 to 5276 records reviewed each measurement. Boston has no clinics under the direct control of a health department, and the clinics assessed were mainly community health centers and hospital-based clinics, as well as one private practice. The source of data was a citywide vaccination registry. The semiannual clinic report included a histogram showing the individual clinic's position in the distribution of other clinics anonymously rank ordered by coverage. A city health department nurse visited each site each month to monitor participation in the registry, to provide in-person feedback on coverage data, to evaluate vaccination practices and strategies, and to share successful practices and strategies from other clinics. An annual ceremony at the Massachusetts Medical Society honored clinics with high coverage. A health department nurse and epidemiologist were dedicated full-time to the registry. The nurse devoted approximately 20% of her time to feedback-related activities, and the epidemiologist devoted approximately 5% of his time to generating coverage reports, for a total of 0.25 FTE. The costs of supervision, clerical support, travel, and supplies were about $8000, for a total annual program cost of $17,400. (This excludes other costs associated with the registry, for which the total yearly budget is approximately $350,000.) Median clinic coverage rose from 41% in 1994 to 79% in 1997, or +38 percentage points over 3 years (+13 percentage points per year).
Distribution of clinics in Boston, Mass, by vaccination coverage. See legend to Figure 1 for further details.
The data for Houston are shown in Figure 6. Coverage has been measured annually or semiannually since 1994 in each of the city's 8 clinics (which provide vaccinations to an estimated 40% of the city birth cohort of 33,145), with 714 to 1277 records reviewed each measurement. Immediately after measurement, one-on-one feedback was conducted with the immunization coordinator, chief nurse, and health care manager for each clinic. Certificates of recognition were awarded to clinics with the most improved coverage and the highest coverage, and these clinics' practices and successes were described in the health department newsletter. Clinics attaining coverage of greater than 80% and greater than 90% were recognized at the state immunization conference. The city immunization bureau paid for managers and immunization nurses from each recognized clinic to attend the state or federal immunization conference. Two registered nurses (with an annual salary of $32,618) and 3 licensed vocational nurses (with an annual salary of $24,465) devoted 10% of their work time to these activities, for a total of 0.5 FTE. The costs of supervision, clerical support, travel, and supplies were about $10,000, for an estimated total annual program cost of $23,863. Median clinic coverage rose from 28% in 1994 to 84% in 1997, or +56 percentage points over 3 years (+19 percentage points per year).
Distribution of clinics in Houston, Tex, by vaccination coverage. See legend to Figure 1 for further details.
Changes in measured clinic coverage compared with the nis
In 1994, median clinic coverage was lower than the population-based coverage estimate in every locality. Between 1994 and 1996, the increase in clinic-based coverage was greater than the change in population-based coverage in 5 localities, and in the sixth it was identical. During the 2 years under examination, the average annual rate of rise of clinic coverage was +11 percentage points, compared with +1.2 for the populations of the 6 localities and +1.5 for the nation as a whole (Table 1).
We report that the Georgia measurement and feedback immunization intervention system (AFIX) was reproduced successfully in 4 states and 2 cities, at relatively low crude direct program cost, and was associated with improvements in clinic vaccination rates that exceeded population-based trends.
Part of the initial rise in clinic coverage is attributable to improvements in clinic record keeping. In 1994, the discrepancy between clinic and population-based coverage was large in localities that had just started measuring clinic coverage that year, eg, Boston (41% vs 86%) and Houston (28% vs 64%). Previous studies8-13 have suggested that vaccination coverage of children enrolled in health department clinics may be lower than that of the general population but not dramatically so (1%-10%). Since large fractions of the birth cohorts of each locality were vaccinated in the surveyed clinics (Boston, 60%; Houston, 40%), such initial clinic measurements are implausibly low. After 2 years of clinic assessments (1996), the discrepancies between clinic and population coverage had been markedly reduced: Boston (77% vs 85%) and Houston (62% vs 65%). These dramatic 2-year percentage point increases in clinic coverage (Boston, +36; Houston, +34) do not resemble the modest yearly effects of measurement and feedback observed elsewhere and strongly suggest documentation artifact.
The sensitivity of clinic-based coverage estimates to record-keeping accuracy was confirmed by studies conducted in 2 of the localities. A 1995 study14 of a large Boston clinic participating in the intervention demonstrated that coverage measured by the registry (which was the source of measurement data in the Boston intervention) would double if all administered vaccinations had been accurately recorded. In Colorado, registry-based measurements were substituted for chart audits in 24 of the 91 clinics during 1996-1997, and median coverage in these clinics fell from 70% to 49%, producing a −4 percentage point decline in overall clinic coverage. A study15 conducted by the state suggested that inaccuracies in data entered into the registry (which had not previously been subject to the influence of the intervention) were responsible. These data are consistent with other studies16,17 of the accuracy of provider-based immunization coverage measurements.
Documentation artifact is unlikely to account for the coverage improvements seen after the first few years of clinic assessments. In 1994, the discrepancy between clinic and population-based coverage was minimal for localities that had more than 1 year of auditing clinic records, eg, Missouri (68% vs 71%) and Louisiana (70% vs 72%). Two years later (1996), clinic coverage in these 2 states had increased to relatively high levels (Missouri, 90%; Louisiana, 81%), and these clinic improvements were associated with population-based increases. In Louisiana, where most children were vaccinated in public clinics, the coverage improvements for clinics and the general population were measured to be the same.
If we exclude as documentation artifact the first 2 years of clinic coverage improvements in each of the 6 localities, the average annual rate of improvement was +5 percentage points. Intensive investigation in Georgia suggested that the +6 percentage point annual coverage rise there was due to increased vaccination, a finding that is consistent with other studies17-21 of the impact of measurement and feedback. As modest as a +5 percentage point annual rise may seem, it is considerably greater than the national population-based increase of +1.5 a year. During a decade, persistent +5 percentage point annual increments can take low immunization rates to high levels.
Estimated average annual costs of implementing the program were reported to be relatively low: 1 FTE (range, 0.3-1.7) implemented the program at an expense of $49,533 (range, $17,400-$97,900). These cost figures assume the existence of a public health infrastructure and do not include expenses incurred by the individual clinics in efforts to raise coverage. Such estimates, while crude, are comparable with those of Georgia (2.3 FTEs; expense, $80,000).
The 4 states and 2 cities that provided data for this study are not representative of the rest of the United States: they began measuring clinic coverage before it was mandated and aggressively applied the other elements of the AFIX strategy in the absence of any requirement. Federal grant language requires only measurement of public clinic coverage, and evidence is lacking that measurement alone will raise immunization rates.
Even fully implemented AFIX may fail in certain contexts, or stall at certain levels. In several of the localities included in this analysis, a number of clinics had poor rates (<50%) despite years of exposure to the intervention. Median vaccination coverage at the end of the study showed considerable variability among the 6 localities (75%-93%), which was not readily explicable by differences in program intensity or fiscal investment. Factors associated with individual clinic response to AFIX have been examined in Georgia22; and, as more data from more localities become available, it may be possible to identify overall ecological factors associated with programmatic success.
In this study, we were explicitly attempting to answer a narrow question: Can the Georgia methods and results be reproduced elsewhere? The answer appears to be yes. The next question is one of generalizability: To what extent can the Georgia experience be reproduced throughout the United States?
Accepted for publication January 4, 1999.
We thank Lance Rodewald, MD, Dennis O'Mara, Igor Bulim, Maureen Dempsey, Marilyn Kemna, Pamela Rice Walker, Ruben A. Tapia, and M. Anita Barry, MD, without whom this article would not have been possible. From September 1997 through March 1998, Mr Mercer was assigned to the National Immunization Program as a prevention specialist in the Public Health Prevention Service by the Division of Applied Public Health Training, Epidemiology Program Office, Centers for Disease Control and Prevention, Atlanta.
Reprints: Charles W. LeBaron, MD, MS E-61, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, GA 30333 (e-mail: firstname.lastname@example.org).
Editor's Note: I guess we can now AFIX vaccine coverage problems.—Catherine D. DeAngelis, MD
LeBaron CW, Mercer JT, Massoudi MS, et al. Changes in Clinic Vaccination Coverage After Institution of Measurement and Feedback in 4 States and 2 Cities. Arch Pediatr Adolesc Med. 1999;153(8):879–886. doi:10.1001/archpedi.153.8.879
Coronavirus Resource Center