[Skip to Navigation]
Sign In
March 2001

Effectiveness of a Practice-Based Intervention to Increase Vaccination Rates and Reduce Missed Opportunities

Author Affiliations

From the Departments of Population and Family Health Sciences (Dr Minkovitz and Ms Higman) and Health Policy and Management (Ms Belote and Dr Weiner), The Johns Hopkins University School of Hygiene and Public Health, and The Johns Hopkins Children's Center (Drs Minkovitz and Serwint), Baltimore, Md.

Arch Pediatr Adolesc Med. 2001;155(3):382-386. doi:10.1001/archpedi.155.3.382

Background  Although provider feedback and recall/reminder systems have been shown to increase vaccination rates for children, little is known about the effectiveness of less intensive interventions.

Objective  To determine whether provider prompting at acute care visits in an urban hospital-based outpatient clinic can increase vaccination rates and decrease missed opportunities.

Design and Methods  Study participants, 3 years or younger, were identified from a managed care organization as receiving primary care at the clinic. Eligibility criteria included 1 or more visits to the clinic without regard to continuity of enrollment. Patients' vaccination records were generated at nursing triage and attached to the encounter sheet. Vaccination and visit data were abstracted from medical records, and comparisons were made between baseline (n = 521) and postintervention (n = 642) groups for up-to-date vaccination rates, missed opportunity rates, and mean numbers of visits.

Results  Up-to-date rates at the age of 24 months for 4 diphtheria and tetanus toxoids and pertussis, 3 polio, 1 measles-mumps-rubella, 3 hepatitis B, and 3 Haemophilus influenzae type b vaccines changed from 70% to 78% (P = .07). Up-to-date rates increased significantly to 87% among the subset of children continuously enrolled in the managed care organization and the practice (P<.01). Overall, mean numbers of visits were similar. Missed opportunity rates among children not up-to-date for 4 diphtheria and tetanus toxoids and pertussis, 3 polio, 1 measles-mumps-rubella, 3 hepatitis B, and 3 Haemophilus influenzae type b vaccines at the age of 24 months declined from 65% to 45% (P = .04). Similar trends were noted at the age of 10 months.

Conclusions  In the absence of increased funding, minor changes in standard operating procedures may improve vaccination delivery. Further improvements may require efforts to ensure continuity of provider and plan assignment.

INCREASING THE proportion of children with up-to-date (UTD) vaccinations to 90% by the age of 2 years remains a national objective of Healthy People 2010.1 Failure to achieve this goal has been attributed, in part, to ineffective provider practices, which lead to missed opportunities.2-9 Provider recall/reminder and assessment and feedback interventions have been shown to improve childhood vaccination rates10 and have been strongly recommended by the US Task Force on Community Preventive Services.11 However, limitations of these established interventions include high resource intensity and the retrospective nature of assessment and feedback once the patient has left the office.

Variation in immunization status measures also contributes to the difficulty of interpreting results across intervention studies. These variations include differences in selected combination measures, acceptable timing for receipt of vaccinations, ages used to calculate measures, and sources of vaccination histories.12 In addition, prior studies often limit evaluations to active patients with a minimum number of visits and continuous enrollment in a health plan or practice. The degree to which such inclusion criteria impact vaccination rates remains unknown. However, such inclusion criteria may lead to an overestimate of program effectiveness and vaccination rates by excluding children who are at high risk for being not UTD.

This study determines (1) whether low-intensity provider prompting at acute care visits can increase vaccination rates and decrease missed opportunities for young children receiving care at an urban primary care pediatric clinic and (2) the impact of continuous enrollment in a health plan and pediatric practice on vaccination rates and effectiveness of the intervention.

Participants and methods

Study population and setting

This prospective study took place in an urban hospital-based pediatric clinic in which acute care is provided by a team of residents, medical students, nurse practitioners, and preceptors, including general pediatrics fellows and faculty. The residents and students providing acute care rotate on a monthly basis. Patients receiving care at the clinic were predominantly low-income children who qualified for mandatory Medicaid managed care and were enrolled in 1 of several health plans.

The predominant managed care organization (MCO) serving the clinic provided a list of names and dates of birth for the 654 children at baseline who were 3 years or younger as of March 1, 1998, and were assigned to receive care at the Harriet Lane Pediatric Clinic (HLPC), The Johns Hopkins Hospital, Baltimore, Md. The MCO also provided a postintervention list of 930 enrollees who were 3 years or younger as of June 30, 1999, and were assigned to receive care at HLPC. Baseline and postintervention lists were manually examined and compared to identify duplicate entries within and between the lists. Medical record numbers for each child were sought through a systematic search of the computerized patient record that documents all patient encounters.

Since 1995, all vaccinations delivered in the clinic were ordered electronically, with dates and types of vaccines entered into a computerized patient record. Vaccination histories for individual children are printed on the Maryland Vaccination History form, which records vaccination type and date of administration. As part of a city immunization registry initiative, local health department staff entered historical vaccination records into the computerized patient record for children younger than 6 years. This back data entry included vaccinations provided in HLPC before 1996 and those documented (with vaccine type and date) as being given by other providers.

The study was approved by the Committee on Human Research at The Johns Hopkins University School of Hygiene and Public Health.


The intervention took place between May 1998 and July 1999. Triage nurses generated computerized printouts of each child's vaccination record and prominently attached them to the encounter form during each acute care visit. As part of this intervention, the acute care team was instructed to review the vaccination history during the visit and offer age-appropriate vaccinations as appropriate. Study investigators (C.S.M. and J.R.S.) educated the acute care team monthly regarding this policy and the need to assess vaccination status at each visit. During these sessions, clinicians and staff received chocolate bars labeled "Immunize On Time, Every Time" to reinforce compliance. In addition, clinic vaccination rates were reviewed with staff and providers at several monthly staff meetings and at quarterly preceptor meetings.

Data collection

Medical records were abstracted from the outpatient clinic at baseline and postintervention. Paper and electronic medical records were used to gather and verify immunization data. Four trained abstractors recorded demographic information, vaccinations, and visit data, including date, diagnoses, and temperature. Diagnoses and temperature were recorded to identify contraindications to vaccination. Up to 3 attempts were made to locate each missing record.

Data analysis and measures

Medical record data were entered, verified, and checked for internal consistency. Several outcome measures were defined.

Timely Vaccinations by the Ages of 10 and 24 Months

Receipt of timely vaccinations included 3 diphtheria and tetanus toxoids and pertussis, 2 polio, 3 hepatitis B, and 3 Haemophilus influenzae type b vaccines by the age of 10 months, and 4 diphtheria and tetanus toxoids and pertussis, 3 polio, 1 measles-mumps-rubella, 3 hepatitis B, and 3 Haemophilus influenzae type b vaccines by the age of 24 months. Definitions of UTD are based on the Red Book recommendations of the American Academy of Pediatrics.13

Missed Opportunities

A missed opportunity occurred when the child was age eligible for a vaccine and had no contraindication for the vaccine, but the vaccine was not administered. Missed opportunities may occur at any type of visit, including sick visits, and were counted for those children not UTD for immunizations. Age eligibility and contraindications also were based on the Red Book recommendations.6

Mean Visits and Percentage of Well-Child Visits

Well-child visits included visits for health supervision, and acute care visits included visits (first time and follow-up) for injury or illness. Visits for completion of forms or prescription refills were excluded. The percentage of visits for well-child care also was determined.

Children Continuously Enrolled

Children were continuously enrolled in the MCO and clinic if they were included in the baseline and postintervention MCO enrollment files and assigned to the clinic for primary care. Analysis was limited to those children with at least 1 visit to the clinic during the intervention period.

Data analyses were conducted using the Clinical Assessment Software Application and EPI-Info6 (both from the Centers for Disease Control and Prevention, Atlanta, Ga) and commercially available software (Microsoft Access 1997; Microsoft Corporation, Seattle, Wash). Proportions were compared using χ2 analyses, and means were compared using analysis of variance. Comparisons were made between baseline and postintervention groups for 2 groups of children: those aged 10 to 23 months and those aged 24 months and older. The postintervention sample was identified as the total sample and the subset of children continuously enrolled.


Of the 654 baseline and 930 postintervention children identified by the MCO as receiving primary care at the outpatient hospital-based clinic, records were reviewed for 521 baseline and 642 postintervention children (Figure 1). Children excluded from analyses included those with duplicate listings in the MCO enrollment file, those without a documented visit to the institution, those not assigned to HLPC but assigned to an alternate source of care within the institution, and those for whom medical records were unable to be located.

Sample identification. MCO indicates managed care organization; ED, emergency department.

Sample identification. MCO indicates managed care organization; ED, emergency department.

The baseline and postintervention samples were similar with regard to gender, race, number of siblings, and birth weight (Table 1). However, the groups differed regarding age, with the baseline group having a greater proportion of older children than the total postintervention group. Only children aged 12 months or older are eligible for inclusion in the continuously enrolled group.

Table 1. 
Sample Characteristics*
Sample Characteristics*

For children aged 24 months and older and continuously enrolled in the MCO and the clinic, there is an increase in 4 diphtheria and tetanus toxoids and pertussis, 3 polio, 1 measles-mumps-rubella, 3 hepatitis B, and 3 Haemophilus influenzae type b vaccination rates from 70% to 87% (Table 2). As expected, this group experienced a higher number of mean visits, with no difference in the proportion of visits for well-child care and a decrease in missed opportunities among those not UTD. Similar trends of increased vaccination rate and decreased missed opportunities from baseline to postintervention were found for all children in the postintervention group, although the change in vaccination rate was not significant. Among children aged 24 months and older, the proportion completing a visit within the first 2 months of life was higher in the postintervention than in the baseline group. However, there was no difference in UTD rates for the first diphtheria and tetanus toxoids and pertussis vaccine at 3 months between the baseline (79.3%) and postintervention (80.7%) groups (P>.05).

Table 2. 
Vaccination Coverage and Visit Characteristics for Children Aged 24 Months and Older*
Vaccination Coverage and Visit Characteristics for Children Aged 24 Months and Older*

For children aged 10 to 23 months, only missed opportunities were significantly changed in the total postintervention group, with a decrease from 60% at baseline to 42% postintervention. However, trends of increased vaccination rates, without a change in mean visits or percentage well-child visits, were similar to those for the aged 24 months and older group (Table 3).

Table 3. 
Vaccination Coverage and Visit Characteristics for Children Aged 10 to 23 Months*
Vaccination Coverage and Visit Characteristics for Children Aged 10 to 23 Months*


Our study demonstrates the effectiveness of a low-intensity intervention, provider prompting during acute care visits, in an urban hospital-based pediatric clinic among continuously enrolled children. Such an intervention may be adopted more readily than previously described efforts. Rodewald14 and Wood15 and colleagues described hiring of nonclinical personnel to enhance vaccination delivery; however, the use of outreach workers or case managers was associated with high costs. External audits of practice vaccination rates by local health officers also have been shown to be effective,16 but may not be generalizable in localities with minimal public health infrastructure. Several interventions have relied on reallocation of responsibilities of existing clinical staff to assess vaccination status and remind providers about needed vaccinations14,17; foregone tasks and the impact on remaining personnel are not identified. Other investigators18,19 have used financial incentives to improve vaccination delivery among adults; however, such practices require additional resources and have not been shown to be effective in at least 1 pediatric setting.20 Our proposed intervention was implemented easily by existing staff with minimal extra time.

Our study reveals that limiting analyses to those children continuously enrolled in the managed care plan and assigned to the clinic for primary care leads to higher vaccination rates and a larger reported intervention effect. As such, we suspect that performance measures, such as those adopted by the Health Plan Employer Data Information Set, which examines vaccination rates among children without lapses in coverage or receipt of care, may overestimate population-based receipt of vaccinations by excluding those children at highest risk of undervaccination.

We found a substantial proportion of children who had not received any care at HLPC despite MCO administrative records documenting their assignment to the clinic for primary care. Some of these children, in fact, were noted to receive care through other hospital-based facilities, including specialty clinics and other sites for primary care. However, we are unable to account for primary care sites for many of these children. We suspect that many practices and managed care plans lack systems and capacity to verify whether children presumed to be assigned to a particular practice are, in fact, receiving care at that site. While outreach plans are required by many state Medicaid managed care contracts for identifying children not receiving primary care, effective mechanisms for accountability may not be in place.

Several limitations of the study should be noted. First, the use of a computerized immunization data system to generate printouts of vaccination records at nursing triage may limit the generalizability of this study. However, manual health maintenance checklists, including vaccinations, have been shown to increase adult vaccinations.21-25 In addition, computer reminders have been shown to be more effective than manual reminders26 and are likely to be more time efficient and sustained beyond an initial evaluation period. Use of a computerized system also minimizes the likelihood of intervention results being attributable to increased documentation rather than improved performance and actual delivery of vaccinations.

Second, the baseline vs postintervention design does not enable us to determine causal relationships. The proportion of well-child visits did not change during baseline and postintervention assessments, suggesting that increasing numbers of health maintenance visits during which vaccination status was usually assessed did not account for this change. In addition, National Immunization Survey vaccination rates in Baltimore City were not shown to increase during this period.27

Third, the higher proportion of children in the postintervention group seen in the first 2 months of life also may contribute to increased vaccination rates among those aged 24 months and older. Other studies7 have documented the importance of timely receipt of 2-month vaccinations for achieving UTD status at the age of 2 years. However, we found no difference in the timeliness of receipt of 2-month vaccinations among children aged 24 months and older. Moreover, early visits to pediatricians would not account for the decrease in missed opportunities by the age of 24 months observed in this study. Our observation of decreased missed opportunities without a significant increase in vaccination rates among children aged 10 to 23 months may reflect small sample sizes and expectations for fewer total vaccines than for the group aged 24 months and older.

Although previously studied provider-based interventions have been shown to improve vaccination rates, such interventions frequently require excess capacity within practices or public health systems to audit performance. In the absence of new dollars or extensive computerized systems, minor changes in standard operating procedures may improve vaccination delivery, as we found in our study. It is likely that the high degree of motivation of clinicians and staff, reinforced through educational sessions and receipt of chocolate bars, enhanced the effectiveness of these minor changes. Further improvements to increase vaccination delivery to young children may require efforts to ensure continuity of provider and plan assignment.

Accepted for publication November 3, 2000.

This study was funded in part by the Pew Charitable Trusts, Philadelphia, Pa.

Presented at the annual meeting of the Pediatric Academic Societies, Boston, Mass, May 14, 2000.

We thank the managed care organization for providing the lists of study children; Lee Bone, RN, MPH, for her contributions to the development of the intervention; Bernard Guyer, MD, MPH, for his review of an earlier draft of the manuscript; and Jeanne Wehage for her assistance with data abstraction.

Corresponding author and reprints: Cynthia S. Minkovitz, MD, MPP, Department of Population and Family Health Sciences, The Johns Hopkins University School of Hygiene and Public Health, 624 N Broadway, Baltimore, MD 21205.

US Department of Health and Human Services, Healthy People 2010: National Health Promotion and Disease Prevention Objectives.  Washington, DC US Public Health Service2000;
Holt  EGuyer  BHughart  N  et al.  The contribution of missed opportunities to childhood underimmunization in Baltimore.  Pediatrics. 1996;97474- 480Google Scholar
Williams  ITMilton  JDFarrell  JBGraham  NMH Interaction of socioeconomic status and provider practices as predictors of immunization coverage in Virginia children.  Pediatrics. 1995;96439- 446Google Scholar
Wood  DLPereyra  MHalfon  N  et al.  Vaccination levels in public health centers: missed opportunities and other contributing factors.  Am J Public Health. 1995;85850- 853Google ScholarCrossref
McConnochie  KMRoghmann  KJ Immunization opportunities missed among urban poor children.  Pediatrics. 1992;891019- 1026Google Scholar
Wood  DSchuster  MDonald-Sherbourne  CDuan  NMazel  RHalfon  N Reducing missed opportunities to vaccinate during child health visits: how effective are parent education and case management?  Arch Pediatr Adolesc Med. 1998;152238- 243Google Scholar
Guyer  BHughart  NHolt  E  et al.  Immunization coverage and its relationships to preventive health care visits among inner-city children in Baltimore.  Pediatrics. 1994;9453- 58Google Scholar
Szilagyi  PRodewald  LHumiston  S  et al.  Missed opportunities for childhood vaccination in office practices and the effect on vaccination status.  Pediatrics. 1993;911- 7Google Scholar
Ball  TMSerwint  JR Missed opportunities for vaccination and the delivery of preventive care.  Arch Pediatr Adolesc Med. 1996;150858- 861Google ScholarCrossref
Shefer  ABriss  PRodewald  L  et al.  Improving immunization coverage rates: an evidence-based review of the literature.  Epidemiol Rev. 1999;2196- 142Google ScholarCrossref
Not Available, Vaccine-preventable diseases: improving vaccination coverage in children, adolescents, and adults: a report on recommendations from the Task Force on Community Preventive Services.  MMWR Morb Mortal Wkly Rep. 1999;48 (RR-8) 1- 15Google Scholar
Rodewald  LMaes  EStevenson  JLyons  BStokley  SSzilagyi  P Immunization performance measurement in a changing immunization environment.  Pediatrics. 1999;103889- 897Google ScholarCrossref
Committee on Infectious Diseases, Report on the Committee on Infectious Diseases. 21st ed. Elk Grove Village, Ill American Academy of Pediatrics1988;
Rodewald  LESzilagyi  PGHumiston  SGBarth  RKraus  RRaubertas  RF A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care.  Pediatrics. 1999;10331- 38Google ScholarCrossref
Wood  DHalfon  NDonald-Sherbourne  C  et al.  Increasing immunization rates among inner-city, African American children: a randomized trial of case management.  JAMA. 1998;27929- 34Google ScholarCrossref
LeBaron  CWChaney  MBaughman  AL  et al.  Impact of measurement and feedback on vaccination coverage in public clinics, 1988-1994.  JAMA. 1997;277631- 635Google ScholarCrossref
Harper  PGMadlon-Kay  DJLuxenberg  MCTempest  R A clinic system to improve preschool vaccinations in a low socioeconomic status population.  Arch Pediatr Adolesc Med. 1997;1511220- 1223Google ScholarCrossref
Kouides  RWBennett  NMLewis  BCappuccio  MDBarker  WHLaForce  FM Performance-based physician reimbursement and influenza immunization rates in the elderly.  Am J Prev Med. 1998;1489- 95Google ScholarCrossref
Morrow  RWGooding  ADClark  C Improving physician's preventive health care behavior through peer review and financial incentives.  Arch Fam Med. 1995;4165- 169Google ScholarCrossref
Hillman  ALRipley  KGoldfarb  NWeiner  JNuamah  ILusk  E The use of physician financial incentives and feedback to improve pediatric preventive care in Medicaid managed care.  Pediatrics. 1999;104 (pt 1) 931- 935Google ScholarCrossref
Davidson  RAGletcher  SWRetchin  SDuh  S A nurse-initiated reminder system for the periodic health examination: implementation and evaluation.  Arch Intern Med. 1984;1442167- 2170Google ScholarCrossref
Hahn  DLBerger  MG Implementation of a systematic health maintenance protocol in a private practice.  J Fam Pract. 1990;31492- 504Google Scholar
Cohen  DILittenberg  BWetzel  CNeuhasuer  D Improving physician compliance with preventive medicine guidelines.  Med Care. 1982;201040- 1045Google ScholarCrossref
Korn  JESchlossberg  LARich  EC Improved preventive care following an intervention during an ambulatory care rotation: carryover to a second setting.  J Gen Intern Med. 1988;3156- 160Google ScholarCrossref
Cheney  CRamsdell  JW Effect of medical records' checklists on implementation of periodic health measures.  Am J Med. 1987;83129- 136Google ScholarCrossref
Harris  RPO'Malley  MSFletcher  SW  et al.  Prompting physicians for preventive procedures: a five year study of manual and computer reminders.  Am J Prev Med. 1990;6145- 152Google Scholar
Not Available, Rates for 4:3:1:3 for Baltimore City are reported in the National Immunization Surveys. Available at:http://www.cdc.gov/nip/coverage/data.htmAccessed May 30, 2000