Controlling vaccine-preventable diseases by achieving high childhood vaccination coverage levels is a national priority. However, there are few, if any, comprehensive evaluations of state immunization programs in the United States, and little attention has been given to the importance of vaccination clinic management style and staff motivation.
To evaluate the factors associated with the increase in childhood vaccination coverage levels from 53% in 1988 to 89% in 1994 in Georgia's public health clinics.
A 1994 mail survey obtaining information on clinic vaccination policies and practices and management practices.
All 227 public health clinics in Georgia.
Clinic nurses responsible for vaccination services.
The 1994 clinic-specific coverage level for 21- to 23-month-old children for 4 doses of diphtheria and tetanus toxoids and pertussis vaccine, 3 doses of polio vaccine, and 1 dose of a measles-containing vaccine as determined by an independent state assessment of clinic coverage levels.
Univariate analysis showed that higher coverage levels were significantly (P<.05) associated with smaller clinic size, higher proportions of clientele enrolled in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), being a nonurban clinic, and numerous vaccination practices and policies. Multivariable analysis showed that only 8 of greater than 150 factors remained associated with higher coverage levels, including having no waiting time to be seen, having telephone reminder systems, conducting home visits for defaulters, and restricting WIC vouchers when a child was undervaccinated. Motivational factors related to higher coverage included clinic lead nurses receiving an incentive to raise coverage and lead nurses participating in assessments of clinic coverage levels by state immunization staff.
No single factor is responsible for raising vaccination coverage levels. Efforts to improve coverage should include local assessment to provide feedback on performance and identify appropriate local solutions. Coordinating with WIC, conducting recall and reminder activities, motivating clinic staff, and having staff participate in decisions are important in raising vaccination levels.
AS A RESULT of programmatic and policy changes implemented in Georgia by health district and public immunization clinic staff in the late 1980s and early 1990s, vaccination coverage in Georgia's public health clinics increased substantially.1 Median clinic vaccination coverage levels with 4 doses of diphtheria and tetanus toxoids and pertussis vaccine, 3 doses of polio vaccine, and 1 dose of a measles-containing vaccine (4:3:1 series) among children aged 21 to 23 months increased from 53% in 1988 to 89% in 1994.2
Georgia's success in raising coverage levels has been attributed to annual clinic assessments or audits.2,3 Information on clinic vaccination coverage levels obtained from these assessments encouraged local immunization staff to implement strategies to increase coverage. The availability of coverage data for all clinics in Georgia provided the opportunity to evaluate their increase in coverage.
We used clinic vaccination coverage data obtained from 1994 clinic assessments3 of Georgia's 227 public health clinics. The assessments were conducted by staff from the Georgia Immunization Program, who abstracted vaccination data from the clinic records of each child being served by the clinic who was aged 21 to 23 months on the date of the assessment. Data were entered into a portable computer and were evaluated using a standardized clinic audit software package4 that generates a summary of the assessment findings, including information on missed opportunities,5,6 dropout rates, and other indices of clinic performance. Data were reviewed with clinic staff immediately after the assessments. Summaries of clinic and district performance were disseminated via workshops and meetings.
In 1994, we conducted a mail survey of all Georgia public health clinics. To identify variables for inclusion in the mail survey, a conceptual framework was developed that included variables thought to be associated with immunization rates. Information used to develop this framework was gathered from a qualitative presurvey of 9 diverse public health clinics in Georgia and from a focus group with field staff of the Georgia Immunization Program. In our framework, the annual clinic assessments brought about innovations in the vaccination process. Information obtained on clinic performance influenced management and motivated clinic staff to consider alternative interventions (Figure 1). Depending on available resources and the clinic population, staff modified clinic practices, which ultimately led to improvements in coverage levels.
Conceptual framework for evaluation of the Georgia Immunization Program. In this framework, annual clinic assessments brought about innovations in the vaccination process. Information obtained on clinic performance influenced management and motivated clinic staff to consider alternative interventions. Depending on available resources and the clinic population, staff modified clinic practices, which ultimately led to improvements in vaccination coverage levels.
The clinic immunization coordinators or lead nurses in all of Georgia's public health clinics were surveyed. Information was obtained on vaccination policies and practices in the clinic, eg, as outlined in the Standards for Pediatric Immunization Practice7; knowledge of immunization policies and practices; and attitudes toward the clinic, management, and supervision. Before the actual survey, the questionnaires were pretested by 2 state health departments outside of Georgia. Questions were multiple choice or binary. In total, 150 survey questions sought information in 5 analysis categories: (1) clinic population, (2) clinic vaccination practices, (3) management, (4) staff motivation, and (5) resources (Table 1).
A descriptive analysis was first conducted of all question variables. The clinic-specific up-to-date coverage level for 1994 for the 4:3:1 series was obtained for each clinic. Clinics were stratified by their up-to-date levels into 3 categories using arbitrary cutoff points: low-coverage clinics (LCC), with levels of less than 80%; moderate-coverage clinics (MCC), with levels of 80% to 89%; and high-coverage clinics (HCC), with levels of 90% or greater. Survey responses were linked to the 1994 clinic coverage information obtained from the annual assessment. Univariate analysis was performed to determine the association between the clinic-specific 1994 up-to-date coverage levels for the 4:3:1 series and each of the 150 question variables.
Within each category of variables and across categories, we conducted a multivariable analysis using logistic regression to model the probability that a child is up to date. All multivariable models were fit by the GENMOD procedure using a correction for overdispersion.8 For these regression analyses, each question variable found to be significant (P<.05) in univariate analysis, or that was believed to be programmatically relevant (ie, shown in previous studies to be associated with coverage), was included in a within-category multivariable analysis. Variables that remained significantly associated with coverage or that were programmatically relevant were considered candidate variables for building a final integrative model.
We used forward selection and backward elimination procedures to select variables for the final model.9 We determined the proportion of variation in coverage explained by factors in the final model using the squared Pearson product moment correlation (r2) between observed and predicted coverage.10 We assessed interaction between each pair of variables in the final model using a relatively stringent statistical criterion of P<.01.
The overall response rate for both questionnaires for the 227 eligible clinics was 100%. Fifty clinics (22%) were classified as LCCs, with a median coverage level of 72%; 61 (27%) were classified as MCCs, with a median coverage level of 85%; and 116 (51%) were classified as HCCs, with a median coverage level of 97%.
Clinics with smaller populations of children had higher coverage levels than did clinics with larger numbers of children (Table 2). Higher clinic-specific coverage levels were also associated with having higher proportions of assessed children enrolled in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). Twenty-eight clinics (12%) were classified as urban, and they differed from nonurban (rural or suburban) clinics in terms of coverage.
Univariate analyses revealed that many of the 150 question variables were significantly related to coverage levels, eg, charging a fee for vaccines; screening the vaccination status of siblings during health care encounters; and having liberal policies as to who can decide whether a child needs an immunization by allowing nonphysicians, such as nurse practitioners, physician assistants, and other nurses, to administer vaccines (Table 2).
Several WIC-related activities were associated with higher coverage levels. Information collected in the surveys demonstrated that 50% of clinic clientele were enrolled in WIC and that 95% of WIC services were colocated with immunization clinics. When questioned, 99% of WIC staff stated that they screened the immunization status of their clients, and 91% of WIC staff had received training in immunization practices.
All analyses of WIC-related questions were conducted only among the 176 clinics (78%) where at least 10% of assessed children participated in WIC. In addition, the outcome variable for these analyses was restricted to patients receiving WIC services. When asked what actions were taken by WIC staff when a child presented to WIC services and was in need of a vaccine, 44 WIC sites (25%) reported the use of voucher incentive programs that altered the frequency of voucher disbursements or that only distributed partial payment if a vaccination was due, with the remaining voucher given when the immunization was scheduled. Higher clinic-specific coverage levels were associated with the presence of voucher incentive programs, with WIC staff administering needed vaccines, with holding regular meetings between WIC and immunization staff, and with coscheduling appointments for immunization and WIC services.
Several management factors were also associated with high vaccination coverage levels. First, clinic staff in HCCs were more likely than those in MCCs and LCCs to state that district priorities were communicated by upper management in-person rather than by other means (eg, by memorandum). Second, clinic staff in HCCs were more likely to have input into management decisions than were those in MCCs and LCCs. For example, clinic staff in HCCs were more likely than those in MCCs and LCCs to always participate in selecting new staff (75% vs 64% and 50%; P = .08), to approve the selection of new staff (48% vs 36% and 26%; P<.01), and to always participate in the promotion of staff (69% vs 49% and 42%; P = .03).
Two variables related to staff motivation and knowledge of the audits were associated with high coverage levels. Thirty-four percent of lead nurses in HCCs stated that they had been offered an incentive (eg, attendance at conferences, gift coupons, plaques, dinner awards, and financial incentives) to raise coverage levels vs 16% of nurses in MCCs and only 12% in LCCs (P<.01). In HCCs, 76% of clinic lead nurses stated that they had participated in interviews after the clinic assessments, at which time they were informed of their clinic's performance, vs 63% of those in MCCs and only 33% in LCCs (P<.01).
A clinic's ability to receive Medicaid fees and use them as it deems necessary was related to coverage levels. In addition, HCCs were more likely to have sufficient staff and accessible parking space for clientele than were MCCs and LCCs.
A total of 53 variables that were statistically significant or programmatically relevant in the within-category analyses were considered candidate variables for building a final integrative model. After controlling for urban vs nonurban residence, the clinic's coverage level at the time of the first assessment, clinic size, and the proportion of children enrolled in WIC, 8 variables remained significantly, and independently, associated with clinic vaccination coverage (Table 3). Higher clinic coverage levels were associated with not having waiting times to be seen, using telephone reminder systems, conducting home visits for defaulters, using WIC incentive programs, lead nurses having received an incentive to raise coverage, and lead nurses participating in postassessment interviews. Lower coverage levels were associated with conducting special community vaccination sessions and requiring fees for vaccines.
When we considered all 8 variables together, HCCs were more likely than LCCs and MCCs (P<.01) to have greater numbers of the 6 positive factors that were functioning plus the 2 negative factors that were not functioning. For example, only 13% of clinic staff in LCCs and 39% of those in MCCs reported using 3 or more of the 8 practices found by multivariable analysis to be associated with coverage. In comparison, 78% of HCCs had 3 or more practices operational. This final integrative model explained more than half (r2 = 53%) of the entire coverage variability.
Among the 12 variables in the final model (ie, the 4 factors for which we controlled and the 8 variables shown to be significant in the multivariable analysis), only 2 statistically significant interactions were detected. The first interaction suggested that the association between nonurban residence and high coverage was stronger among clinics with greater numbers of children enrolled in WIC (P = .005). The second interaction suggested that the use of home visits to retrieve defaulters was associated with high coverage only among clinics that did not conduct special vaccination sessions (P = .007).
To our knowledge, these data represent the first statewide, comprehensive evaluation of an immunization program in the United States. Our 100% response rate is also unique and lends credibility to the associations detected in this analysis. This evaluation of the Georgia Immunization Program suggests that no single factor, or category of factors, was solely responsible for raising immunization coverage to some of the highest levels in the nation. Rather, these data suggest that a combination of clinic vaccination practices and management styles were related to high coverage levels.
The Georgia Immunization Program involves a team approach. State-level staff conduct clinic audits, which serve as a catalyst to motivate district-level managers and clinic nurses, who then implement activities to improve coverage. This approach gives clinic staff at the local level varying degrees of decision-making authority. Not only are the regular assessments of clinic coverage conducted, which has been demonstrated to be associated with improvements in performance,11,12 but clinic staff are encouraged to participate in these assessments. Most clinic nurses (87%) stated that they believed the assessments had a positive impact on changing policies. In addition, most nurses knew the coverage level in their clinics as a result of these regular assessments. Clinics in which staff participated in the discussions of the assessments had higher levels of coverage than did those in which the clinic staff did not participate.
Encouragement of clinic staff by upper management, however, was not limited to participation in interviews after the regular coverage assessments. Clinic staff were given responsibility and local control for the vaccination process, including defining and implementing changes in the immunization process needed to improve clinic-specific coverage levels. This inclusive management style helped ensure that problem solving was done at the local level and resulted in local, and feasible, solutions. This approach is consistent with management theories that propose that effectiveness is related to possessing knowledge of the problem (eg, information from the assessments), identifying the skills needed to solve the problem (eg, clinic staff ownership), and the desire to effect change (eg, clinic staff motivation).13 A similar approach was used in a health maintenance organization in which findings14 of low immunization coverage among clients prompted managers to evaluate the cause of the low levels and identify interventions to raise coverage in their clinics. In another health maintenance organization evaluation,15 regular audits coupled with peer review, feedback, and incentives to physicians were shown to raise immunization levels.
Data from this study about the importance of coordinating immunization services with WIC are consistent with results of previous studies16- 19 demonstrating the impact on coverage of linking and coordinating with WIC.
This study has several limitations. First, it is an ecological analysis, ie, associations found do not establish causality. Our results might not be representative of other states because most children in Georgia received vaccinations from the public sector, and public health clinics had comparatively high vaccination coverage levels in 1994. Another limitation was the lack of variability of selected clinic practices and policies, which might have prevented detection of their associations with coverage. Some practices or policies that were associated with low vaccination coverage might have been initiated in response to low coverage. This might explain why conducting special community vaccination sessions was associated with lower levels. Last, few data were obtained on changes in staffing over time in the immunization and WIC services. Significant increases in staffing might have accounted for some of the observed increase in coverage.
In summary, no single intervention is sufficient to raise coverage levels. Deciding which interventions to implement at a particular clinic will depend on the results of assessments of the clinic's performance. In addition, these data demonstrate the importance of coordinating with WIC, conducting recall and reminder activities,20 making vaccines financially accessible, and making services convenient. Perhaps most important, our data suggest that a management style that provides clear priorities about the importance of immunization and that fosters staff participation in decisions is needed to improve immunization coverage levels.
Controlling vaccine-preventable diseases by achieving 90% immunization coverage levels is a national health objective.21 Clinic practices and policies should be evaluated regularly at the local level.22 Upper-level management needs to understand the importance of clinic staff control of the vaccination process and staff motivation for achieving high coverage levels. Program managers should ensure that staff at the local level become more involved in making decisions about vaccination policies and procedures. Evaluations similar to this one are needed in other areas, particularly in states with lower coverage levels, in states with less collaboration with WIC, and where clinic practices and policies are more varied.
Editor's Note: Nudging and money seem to be good stimulators for many things.—Catherine D. DeAngelis, MD
Accepted for publication June 16, 1999.
The Georgia Immunization Program Evaluation Team comprises the following: Study conception and design: Felicity T. Cutts, MD; Vance J. Dietz, MD; and Elizabeth Zell, MStat; Qualitative presurvey of 9 public health clinics in Georgia: Roger Bernier, PhD; Dr Cutts; Dr Dietz; Ms Zell; Ken Falter, PhD; Deborah Glik, PhD; and Alacia E. Lyons, RN, MA; Focus group with field staff of Georgia Immunization Program to develop a conceptual framework and to identify key categories or issues for questions: Dr Glik; Conceptual framework of immunization process and questionnaire development: Dr Cutts; Dr Dietz; Dr Falter; Dr Glik; and John F. Newman, Jr, PhD; Pretests of questionnaires: Thomas W. Hicks, Missouri Department of Health, Bureau of Immunization, Jefferson City; and Jesse E. Greene, South Carolina Department of Health and Environmental Control, Division of Immunization and Prevention, Columbia; Questionnaire revision, data collection, and data analysis: Battelle Memorial Institute, Arlington, Va, under Centers for Disease Control and Prevention (CDC), Atlanta, Ga, contract 200-93-0626; Battelle project directors: James C. Hersey, PhD; and Bennett K. Pierce, MS; CDC co–technical monitors: Andrew L. Baughman, MPH; Dr Dietz; John M. Stevenson, MA; and Eugene F. Dini, MPA.
We thank all of the participants in our survey; the staff of the Georgia Immunization Program, Division of Public Health, Department of Human Resources, who helped coordinate survey activities: Michael Chaney, program manager, Helen Conner, assistant program manager, and Del Carvell, public health advisor; Charles W. LeBaron and Edward W. Brink for reviewing the manuscript and providing helpful suggestions; and Robert S. Black for editorial review.
Reprints: National Immunization Program, Centers for Disease Control and Prevention, Mail Stop E34, 1600 Clifton Rd NE, Atlanta, GA 30333.
Dietz VJ, Baughman AL, Dini EF, Stevenson JM, Pierce BK, Hersey JC, for the Georgia Immunization Program Evaluation Team. Vaccination Practices, Policies, and Management Factors Associated With High Vaccination Coverage Levels in Georgia Public Clinics. Arch Pediatr Adolesc Med. 2000;154(2):184–189. doi:10.1001/archpedi.154.2.184