Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
November 1999

Impact of Resident Feedback on Immunization Outcomes

Author Affiliations

From the Emory University School of Medicine (Drs Rust, Kuo, Smith, Miller, and Sullivan) and the Mercer University Stetson School of Business and Economics (Dr Sisk), Atlanta, Ga.


Copyright 1999 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.1999

Arch Pediatr Adolesc Med. 1999;153(11):1165-1169. doi:10.1001/archpedi.153.11.1165

Objective  To investigate the effect on immunization levels of retrospective written feedback to residents regarding missed immunization opportunities.

Design  Randomized trial with control group.

Setting  Pediatric resident continuity clinic in an urban hospital-based primary care clinic.

Participants  Thirty-two postgraduate level 2 and postgraduate level 3 pediatric residents.

Intervention  Monthly retrospective written feedback mailed to residents detailing their missed immunization opportunities and appointment failure rates over a 12-month period beginning in February 1997.

Main Outcome Measures  The immunization level of 2-year-old children in the resident clinic was the main outcome of interest. Secondary outcomes included missed immunization opportunity rates and appointment failure rates.

Results  Postintervention immunization levels were 71.4% (95% confidence limits [CLs]: 63.2%, 78.7%) for patients from the intervention group and 68.5% (95% CLs: 60.8%, 75.4%) for patients from the control group. The immunization level for patients of both groups who had fewer than 2 visits during the second year of life was 47.2% (95% CLs: 38.2%, 56.3%). This compares with an immunization level of 78.1% (95% CLs: 66.0%, 87.5%) for patients from both groups who had 2 visits during the second year of life, and with an immunization level of 88.2% (95% CLs: 81.0%, 93.4%) for patients of both groups who had more than 2 visits during the second year of life (P<.001).

Conclusions  In this setting, written retrospective feedback to residents was an ineffective strategy for improving immunization levels. Adequate follow-up during the second year of life is critical in achieving high immunization levels.

IMMUNIZATION LEVELS in the United States have risen dramatically in recent years although many urban localities have not seen their immunization levels rise as high as others.1,2 The standards for pediatric immunization practices developed by the Centers for Disease Control and Prevention,3 Atlanta, Ga, recommend investigating each child's immunization status at every medical encounter to take full advantage of every immunization opportunity. Missed immunization opportunities are known to contribute to lower immunization levels.4,5 Feedback to clinical providers can be an effective tool in reducing missed immunization opportunities and raising immunization levels.6 Given historically low immunization levels in our urban hospital-based resident continuity clinic, we undertook a controlled trial of written feedback to individual residents on their missed immunization opportunities in an attempt to improve immunization levels among our patient population. Because pilot data indicated that many of our underimmunized children were lost to follow-up, we also included information on each resident's appointment failure rate in the feedback he or she received.


Thirty-two postgraduate level (PL) 1 and PL-2 residents were chosen to participate in the study. Three others were ineligible because of plans to leave the residency before the study was complete. Residents were randomized within their PL year to 1 of 2 groups: (1) the control group (no written feedback) and (2) the intervention group (receiving written feedback). Residents were informed by mail at the onset of the study that their continuity clinic patient visits were being monitored for an immunization study and were not given an opportunity to refuse participation. The Emory University School of Medicine, Atlanta, Human Investigations Committee approved the study. Before the study, all residents received formal instruction regarding the immunization schedule, medical contraindications for immunizations, and the influence of missed immunization opportunities on immunization levels. This instruction was presented as one of the routine preclinic conferences, with no mention made of the study.


The immunization level for 2-year-old children in the resident clinic (dependent variable) was the main outcome of interest. The intervention was designed to investigate the effect of written feedback (independent variable) on immunization levels. Secondary outcomes were rates of missed immunization opportunities and appointment failure.

All visits of established patients younger than 2 years scheduled to see residents in the study were monitored. Established patients were defined as those who were visiting the clinic for a second or subsequent visit. A missed immunization opportunity was defined as any visit during which an immunization was due, according to Advisory Committee on Immunization Practices guidelines,7 but not given. Missed immunization opportunities in appointments kept were characterized as one of the following: (1) medical contraindication,8 (2) documentation gaps, (3) parental refusal, or (4) other. Missed immunization opportunities in the "other" category were those for which no medical contraindication, documentation gap, or parental refusal was found. These other missed immunization opportunities were further described as child ill or child well. A documentation gap was noted in cases in which the patient's chart contained evidence that the resident indicated that the immunization had been received, but the exact date of the vaccine was lacking. All visits were monitored by the same PL-4 resident (R.M.), and all missed immunization opportunities were characterized by the same attending physician (C.T.R.). The reviewers were not blinded as to which group, intervention or control, the residents belonged. We chose the commonly used 4:3:1:3 definition for completed immunizations for 2-year-olds, meaning that a child was defined as completely immunized if by age 2 years the following had been received: 4 diphtheria and tetanus toxoids and pertussis vaccines (or diphtheria and tetanus toxoids and acellular pertussis vaccines), 3 oral poliovirus vaccines (or inactivated poliovirus vaccines), 1 measles-mumps-rubella vaccine, and 3 Haemophilus influenzae type b vaccines. Age-appropriate numbers of H influenzae type b vaccinations were also accepted for children who began the series late.


A feedback report was devised and piloted among PL-3 residents who were asked not to share information on the study with their fellow residents. The feedback form included provider- and clinic-specific information on clinic visits of children younger than 2 years, including immunization opportunities, the percentage of immunization opportunities missed, the number of appointment failures, and the number of appointment failures that included potential immunization opportunities. Additionally, each report included 2 suggestions from the English-language literature: one on reducing missed immunization opportunities, and one on reducing appointment failures. The feedback reports were mailed to the intervention group at their homes to avoid contamination since the resident's hospital mailboxes are located together in the continuity clinic. The investigators received informal confirmation that the reports were received and read. The residents' visits were monitored for 2 months prior to receipt of the first feedback report in February 1997. A total of 12 monthly reports were mailed at the end of each month detailing the previous month's relevant statistics.

Data collection consisted of chart review and query of the local immunization registry after each scheduled appointment that met these criteria. Each visit was coded as to whether a particular vaccine was due and/or given. Because of the clinic's known appointment failure and "lost to follow-up" problem, the number of visits kept by study patients was also recorded. The immunization levels for 2-year-old children were determined for 2 points during the study: prestudy and poststudy (March 1998). Immunization rate data collection was modeled after the Health Plan Employer Data Information Set 3.09 and was defined as the proportion of children reaching age 2 years with complete immunizations. Children eligible for each audit were those who reached age 2 years during the calendar year prior to the audit date. All children meeting the above criteria who were also seen by a given study resident at least twice during the first 2 years of life were included. All significance tests were performed using the χ2 test. Confidence limits for proportions were calculated using the exact mid-p method.


Randomization resulted in 7 PL-1 and 9 PL-2 residents in the control group, and 7 PL-1 and 9 PL-2 residents in the intervention group. The median ages of the control and intervention groups at the end of the study were 29.7 years and 30.0 years, respectively. The control group was 38% male (n=6); the intervention group, 25% (n=4).


Prestudy immunization levels for the patients as a whole (including those seen by PL-3 residents who were not in the study) were slightly higher than poststudy immunization levels for patients from both groups. The prestudy 4:3:1:3 immunization level was 127 (77.9%) of 163, compared with the intervention group patients' poststudy immunization level of 100 (71.4%) of 140, and the control group patients' poststudy immunization level of 115 (68.5%) of 168 (Table 1). These differences were not statistically significant.

Table 1. 
Image not available
Immunization Levels of 2-Year-Old Children Before and After Intervention in Both Study Groups

Analysis of poststudy data involving the number of visits during the second year of life revealed interesting findings. Fifty (78%) of 64 children who had the recommended number of visits10 between their first and second birthdays (ie, 2 visits) were up-to-date by age 2 years. In contrast, only 59 (47%) of 125 children who had fewer than 2 visits between their first and second birthdays were up-to-date by age 2, and 105 (88.2%) of 119 children who had more than 2 visits during their second year of life were up-to-date (Table 2). These differences were significant (P<.001).

Table 2. 
Image not available
Proportion of 2-Year-Old Children With Complete Immunizations According to the Number of Clinic Visits During the Second Year of Life*

Missed immunization opportunities in appointments kept (Figure 1) for the intervention group ranged over the course of the study from 5 (13.9%) of 36 to 14 (42.4%) of 33 vs 6 (17.1%) of 35 to 16 (47%) of 34 for the control group. The 95% confidence limits for both groups during all of the months of collection overlap, indicating no significant difference between the groups and no significant before-and-after differences. During the last 10 months of the study, after residents began receiving written feedback reports, the intervention group missed 93 (25.6%) of 363 immunization opportunities in appointments kept vs the control group's rate of 112 (28.7%) of 390, not a statistically significant difference (P=.34). For both groups over the 12-month course of the study, 871 visits made by patients constituted immunization opportunities. In 243 (27.9%) of these visits, at least 1 immunization was missed, making the visit a missed immunization opportunity. Gaps in documentation were found in 223 (45.7%) of 488 missed vaccine doses, parents refused immunizations in 22 (4.5%), and medical contraindications were found in 8 (1.6%). Of the remaining 235 missed vaccine doses (48.2%), 162 (69%) were in well-child visits and 73 (31%) were in sick visits at which the child was ill but not sick enough to require admission to the hospital. Among the 243 visits during which at least 1 vaccine was missed, 131 (54%) of the missed immunization opportunities would have been corrected by the administration of 1 additional vaccine.

Figure 1.
Image not available

Proportion of missed immunization opportunities in appointments kept by month and study group. Residents received first written feedback report at the end of February.


The level of appointments kept remained close to 50% (1520) for the duration of the study and did not vary between the 2 groups (Figure 2). In the total of 2911 visits studied, there were 1732 immunization opportunities. In 861 (49.7%) of these immunization opportunities, the patient failed to keep the appointment.

Figure 2.
Image not available

Proportion of appointments kept by month and study group. Residents received first written feedback report at the end of February.


We failed to demonstrate any effect of written feedback reports used in our resident clinic on immunization levels in 2-year-old children, missed immunization opportunities, or appointment failure rates. There were no before-and-after differences in any of the outcome variables of interest. The proportion of patients with a ratio of 4:3:1:3 immunizations at age 2 years is similar to that found in population-based immunization rate surveys for the entire Atlanta metropolitan area.2 Additionally, the missed immunization opportunity rate of 243 (28%) of 871 appointments kept compares favorably with the rate of 50% found in another recent urban missed immunization opportunity study11 that intervened at the parent level rather than at the provider level. In this study, contamination of the effect from the intervention to control group is a concern, since the intervention was supplied to one subgroup of residents who work closely with members of the control group. The lack of a demonstrable before-after difference in the clinic's immunization levels speaks against a significant contamination effect.

Children who had adequate numbers of visits during the second year of life were significantly more likely to be completely immunized at age 2 years, illustrating the importance of consistent follow-up with the immunization provider. That half of the missed immunization opportunities for scheduled visits were owing to missed appointments highlights the influence of appointment failure on immunization opportunities. Immunization or appointment reminders are known to be effective in improving immunization levels in other locations.12 Another beneficial strategy with demonstrated effectiveness is the use of prompts from nurses given to the resident immediately before or during the visit.13 This strategy will only prove beneficial during appointments kept. The Health Plan Employer Data Information Set 3.0 criteria recognize the importance of consistent follow-up during the second year of life by including only children continuously enrolled between ages 10 months and 2 years in its measure of any given provider's immunization level for 2-year-old children.9 Our residents were surveyed at the end of the study on their attitudes regarding appointment failures. Residents from both groups expressed a sense of powerlessness over their patients' appointment failures. The lack of control residents feel stems from the multitude of clinic and patient factors over which the residents have little control, including scheduling conflicts, third-party payer changes, and transportation difficulties. It is possible that up-to-date immunization status and adequate numbers of visits during the patient's second year are both indicators of a more generalized compliance factor of children's parents or families.

Other potential causes of missed immunization opportunities in appointments kept are overcautious interpretation of immunization contraindications14,15 and a reluctance to give multiple vaccines.16,17 At the time of our study, most residents and faculty recommended diphtheria and tetanus toxoids and acellular pertussis vaccine rather than diphtheria and tetanus toxoids and pertussis vaccine, and inactivated poliovirus vaccine rather than oral poliovirus vaccine. Neither of these vaccines was available in combination with other vaccines, nor was the newer hepatitis B and H influenzae type b vaccine combination vaccine yet available. In a setting that tries to maximize every immunization opportunity, the lack of available combination vaccines can result in up to 4 injections per immunization visit. Continued availability of newer combination vaccines will minimize this potential source of missed immunization opportunities.

In our teaching setting, few immunizations were missed owing to parental refusal or medical contraindications. However, gaps in immunization documentation played a considerable role. In nearly 223 (46%) of 488 cases in which a vaccine dose was missed, the resident believed that the child was indeed up-to-date for the relevant vaccine based on undocumented parental reports. Previous studies18 have indicated that parental reports are not a reliable indicator of immunization status. Other studies have shown reluctance on the part of providers to immunize a child for whom there was incomplete immunization documentation in the face of parental reports that the child's immunization status was up-to-date.19 State laws requiring immunizations prior to school entry are a well-known factor for improved immunization levels. Knowledge of school-entry laws may bolster the residents' case as they attempt to convince reluctant parents, or as they attempt to document immunizations received. Immunization registries are a promising new development that will help to relieve this important source of missed immunization opportunities.


In our urban teaching clinic, retrospective written feedback to residents was an ineffective strategy for reducing missed immunization opportunities or improving immunization levels. Improving documentation of immunizations would potentially reduce missed immunization opportunities to vaccinate. Adequate follow-up during the second year of life is critical in achieving high immunization levels.

Back to top
Article Information

Accepted for publication March 17, 1999.

This study received support from the Ambulatory Pediatrics Association, McLean, Va, and the Emory Medical Care Foundation, Atlanta, Ga.

Presented as a poster at the Ambulatory Pediatrics Association Annual Meeting, New Orleans, La, May 1, 1998.

We are indebted to the residents who so graciously participated in the study.

Editor's Note: Another seemingly good idea bites the dust. Maybe the residents forgot to put reminders in their Palm Pilots.—Catherine D. DeAngelis, MD

Corresponding author: Cynthia T. Rust, MD, Emory University School of Medicine, Department of Pediatrics, 69 Butler St SE, Atlanta, GA 30303 (e-mail:

Centers for Disease Control and Prevention, Retrospective assessment of vaccination coverage among school-aged children—selected US cities, 1991. MMWR Morb Mortal Wkly Rep. 1992;41103- 107
Centers for Disease Control and Prevention, National, state, and urban area vaccination coverage levels among children aged 19-35 months—United States, July 1996-June 1997. MMWR Morb Mortal Wkly Rep. 1998;47108- 116
Centers for Disease Control and Prevention, Standards for pediatric immunization practices. JAMA. 1993;2691817- 1822Article
Centers for Disease Control and Prevention, Impact of missed opportunities to vaccinate preschool-aged children on vaccination coverage levels—selected US sites. MMWR Morb Mortal Wkly Rep. 1994;43709- 718
Szilagyi  PGRodewald  LEHumiston  SGRaubertas  RFCove  LADoane  CB Missed opportunities for childhood vaccinations in office practices and the effect on vaccination status. Pediatrics. 1993;911- 7
LeBaron  CWChaney  MBaughman  AL  et al.  Impact of measurement and feedback on vaccination coverage in public clinics, 1988-1994. JAMA. 1997;277631- 635Article
Centers for Disease Control and Prevention, Recommended childhood immunization schedule—United States, 1997. MMWR Morb Mortal Wkly Rep. 1997;4635- 40
American Academy of Pediatrics, Immunization contraindication guidelines. Peter  Ged.1997 Red Book Report of the Committee on Infectious Diseases 24th ed. Elk Grove Village, Ill American Academy of Pediatrics1997;676- 680
National Committee for Quality Assurance, Health Plan Employer Data Information Set (HEDIS 3.0).  Washington, DC National Committee for Quality Assurance1997;141- 147
Committee on Psychosocial Aspects of Child and Family Health, American Academy of Pediatrics, Recommendations for preventive pediatric health care. Guidelines for Health Supervision, III Elk Grove Village, Ill American Academy of Pediatrics1997;225
Wood  DSchuster  MDonald-Sherbourne  CDuan  NMazel  RHalfon  N Reducing missed opportunities to vaccinate during child health visits: how effective are parent education and case management? Arch Pediatr Adolesc Med. 1998;152238- 243Article
LeBaron  CWStarnes  DDini  EFChambliss  JWChaney  M The impact of interventions by a community-based organization on inner-city vaccination coverage: Fulton County, Georgia, 1992-1993. Arch Pediatr Adolesc Med. 1998;152327- 332Article
Christy  CMcConnochie  KMZernik  NBrzoza  S Impact of an algorithm-guided nurse intervention on the use of immunization opportunities. Arch Pediatr Adolesc Med. 1997;151384- 391Article
England  LShelton  RSchubert  CJ Immunizing preschool children: beliefs and practices of pediatric residents. Clin Pediatr (Phila). 1997;36129- 134Article
Campbell  JRSzilagyi  PGRodewald  LEWinter  NLHumiston  SGRoghmann  KJ Intent to immunize among pediatric and family medicine residents. Arch Pediatr Adolesc Med. 1994;148926- 929Article
Szilagyi  PGRodewald  LEHumiston  SG  et al.  Immunization practices of pediatricians and family physicians in the United States. Pediatrics. 1994;94517- 523
Askew  GLFinelli  LLutz  JDeGraff  JSiegel  BSpitalny  K Beliefs and practices regarding childhood vaccination among urban pediatric providers in New Jersey. Pediatrics. 1995;96889- 892
Kum-Nji  PJames  DHerrod  HG Immunization status of hospitalized preschool children: risk factors associated with inadequate immunization. Pediatrics. 1995;96434- 438
Watson  MAFeldman  KWSugar  NFSommer  CJThomas  ERLin  T Inadequate history as a barrier to immunization. Arch Pediatr Adolesc Med. 1996;150135- 139Article