[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.166.89.187. Please contact the publisher to request reinstatement.
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Download PDF
Figure 1.
CONSORT Diagram
CONSORT Diagram

CC indicates collaborative centralized; PB, practice-based; and UTD, up-to-date.

Figure 2.
Effectiveness of Collaborative Centralized (CC) vs Practice-Based (PB) Reminder/Recall Approaches
Effectiveness of Collaborative Centralized (CC) vs Practice-Based (PB) Reminder/Recall Approaches

A, Percentage of children receiving any vaccination (absolute percentage point difference, 5%). B, Percentage of children achieving up-to-date (UTD) status with vaccinations (absolute percentage point difference, 4%). Data are unadjusted.

aP < .001 compared with the CC reminder/recall approach.

bP = .001 compared with the CC reminder/recall approach.

Table 1.  
County Descriptions at Baseline
County Descriptions at Baseline
Table 2.  
Comparison of Intervention Costs for CC and PB Reminder/Recall Approaches
Comparison of Intervention Costs for CC and PB Reminder/Recall Approaches
Table 3.  
Cost-effectiveness of CC and PB Reminder/Recall Approaches
Cost-effectiveness of CC and PB Reminder/Recall Approaches
1.
Zhou  F, Shefer  A, Wenger  J,  et al.  Economic evaluation of the routine childhood immunization program in the United States, 2009. Pediatrics. 2014;133(4):577-585.
PubMedArticle
2.
Centers for Disease Control and Prevention (CDC).  National, state, and local area vaccination coverage among children aged 19-35 months: United States, 2011. MMWR Morb Mortal Wkly Rep. 2012;61:689-696.
PubMed
3.
US Department of Health and Human Services Office of Disease Prevention and Health Promotion. Healthy People 2020. October 30, 2012. http://www.healthypeople.gov/2020/topicsobjectives2020/default. Accessed February 7, 2013.
4.
Szilagyi  PG, Bordley  C, Vann  JC,  et al.  Effect of patient reminder/recall interventions on immunization rates: a review. JAMA. 2000;284(14):1820-1827.
PubMedArticle
5.
Szilagyi  P, Vann  J, Bordley  C,  et al.  Interventions aimed at improving immunization rates. Cochrane Database Syst Rev. 2002;(4):CD003941.
PubMed
6.
Briss  PA, Rodewald  LE, Hinman  AR,  et al; Task Force on Community Preventive Services.  Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med. 2000;18(1)(suppl):97-140.
PubMedArticle
7.
Task Force on Community Preventive Services.  Recommendations regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med. 2000;18(1)(suppl):92-96.
PubMedArticle
8.
Tierney  CD, Yusuf  H, McMahon  SR,  et al.  Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics. Pediatrics. 2003;112(5):1076-1082.
PubMedArticle
9.
Clark  SJ, Cowan  AE, Bartlett  DL.  Private provider participation in statewide immunization registries. BMC Public Health. 2006;6:33. doi:10.1186/1471-2458-6-33.
PubMedArticle
10.
Dombkowski  KJ, Cowan  AE, Harrington  LB, Allred  NJ, Hudson  E, Clark  SJ.  Feasibility of initiating and sustaining registry-based immunization recall in private practices. Acad Pediatr. 2012;12(2):104-109.
PubMedArticle
11.
Kempe  A, Saville  A, Dickinson  LM,  et al.  Population-based vs practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial. Am J Public Health. 2013;103(6):1116-1123.
PubMedArticle
12.
Thorpe  KE, Zwarenstein  M, Oxman  AD,  et al.  A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009;180(10):E47-E57.
PubMedArticle
13.
Department of Health and Human Services. NIH Health Care Systems Research Collaboratory-Pragmatic Clinical Trials Demonstration Projects (UH2/UH3). January 24, 2012. http://grants.nih.gov/grants/guide/rfa-files/RFA-RM-12-002.html. Accessed July 31, 2012.
14.
Colorado Department of Public Health and Environment CIIS. Colorado Annual Immunization Progress Report for CY2011. Denver, CO: Immunization Information Systems; 2011.
15.
Colorado Rural Health Center. Colorado: county designations, 2014. https://coruralhealth.org/wp-content/uploads/2013/10/2014.Colorado-County-Designations.pdf. Accessed January 23, 2015.
16.
Centers for Disease Control and Prevention. Recommended immunization schedule for persons age 0 through 18 years: United States, 2014.http://www.cdc.gov/vaccines/schedules/hcp/imz/child-adolescent.html. Accessed December 11, 2014.
17.
Raab  GM, Butcher  I.  Balance in cluster randomized trials. Stat Med. 2001;20(3):351-365.
PubMedArticle
18.
Glynn  RJ, Brookhart  MA, Stedman  M, Avorn  J, Solomon  DH.  Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers. Med Care. 2007;45(10)(suppl 2):S38-S43.
PubMedArticle
19.
Ivers  NM, Halperin  IJ, Barnsley  J,  et al.  Allocation techniques for balance at baseline in cluster randomized trials: a methodological review. Trials. 2012;13:120. doi:10.1186/1745-6215-13-120.
PubMedArticle
20.
Moulton  LH.  Covariate-based constrained randomization of group-randomized trials. Clin Trials. 2004;1(3):297-305.
PubMedArticle
21.
US Department of Justice. The Freedom of Information Act (FOIA). September 1, 2014.
22.
Littel  RC, Milliken  GA, Stroup  WW, Wolfinger  RD. SAS System for Mixed Models. Cary, NC: SAS Institute Inc; 1999.
23.
Raudenbush  SW, Bryk  AS. Hierarchical Linear Models: Applications and Data Analysis Methods (Advanced Quantitative Techniques in the Social Sciences).2nd ed. Newbury Park, CA: Sage Publications Inc; 2001.
24.
Bureau of Labor Statistics. Occupational employment statistics.http://www.bls.gov/oes/2011/may/oes_co.htm. Accessed January 12, 2015.
25.
Jacobson Vann  JC, Szilagyi  P.  Patient reminder and patient recall systems to improve immunization rates [update of Cochrane Database Syst Rev. 2002;(4):CD003941]. Cochrane Database Syst Rev. 2005;(3):CD003941.
PubMed
26.
Saville  AW, Albright  K, Nowels  C,  et al.  Getting under the hood: exploring issues that affect provider-based recall using an immunization information system. Acad Pediatr. 2011;11(1):44-49.
PubMedArticle
27.
Albright  K, Saville  A, Lockhart  S, Widmer Racich  K, Beaty  B, Kempe  A.  Provider attitudes toward public-private collaboration to improve immunization reminder/recall: a mixed-methods study. Acad Pediatr. 2014;14(1):62-70.
PubMedArticle
28.
Kempe  A, Daley  MF, Pyrzanowski  J,  et al.  School-located influenza vaccination with third-party billing: what do parents think? Acad Pediatr. 2014;14(3):241-248.
PubMedArticle
29.
Saville  AW, Beaty  B, Dickinson  LM, Lockhart  S, Kempe  A.  Novel immunization reminder/recall approaches: rural and urban differences in parent perceptions. Acad Pediatr. 2014;14(3):249-255.
PubMedArticle
30.
Zhou  F, Santoli  J, Messonnier  ML,  et al.  Economic evaluation of the 7-vaccine routine childhood immunization schedule in the United States, 2001. Arch Pediatr Adolesc Med. 2005;159(12):1136-1144.
PubMedArticle
31.
Zhou  F, Bisgard  KM, Yusuf  HR, Deuson  RR, Bath  SK, Murphy  TV.  Impact of universal Haemophilus influenzae type B vaccination starting at 2 months of age in the United States: an economic analysis. Pediatrics. 2002;110(4):653-661.
PubMedArticle
32.
Lieu  TA, Ray  GT, Black  SB,  et al.  Projected cost-effectiveness of pneumococcal conjugate vaccination of healthy infants and young children. JAMA. 2000;283(11):1460-1468.
PubMedArticle
33.
Ekwueme  DU, Strebel  PM, Hadler  SC, Meltzer  MI, Allen  JW, Livengood  JR.  Economic evaluation of use of diphtheria, tetanus, and acellular pertussis vaccine or diphtheria, tetanus, and whole-cell pertussis vaccine in the United States, 1997. Arch Pediatr Adolesc Med. 2000;154(8):797-803.
PubMedArticle
34.
Fendrick  AM, Lee  JH, LaBarge  C, Glick  HA.  Clinical and economic impact of a combination Haemophilus influenzae and hepatitis B vaccine: estimating cost-effectiveness using decision analysis. Arch Pediatr Adolesc Med. 1999;153(2):126-136.
PubMedArticle
35.
Miller  MA, Sutter  RW, Strebel  PM, Hadler  SC.  Cost-effectiveness of incorporating inactivated poliovirus vaccine into the routine childhood immunization schedule. JAMA. 1996;276(12):967-971.
PubMedArticle
36.
Margolis  HS, Coleman  PJ, Brown  RE, Mast  EE, Sheingold  SH, Arevalo  JA.  Prevention of hepatitis B virus transmission by immunization: an economic analysis of current recommendations. JAMA. 1995;274(15):1201-1208.
PubMedArticle
37.
Lieu  TA, Cochi  SL, Black  SB,  et al.  Cost-effectiveness of a routine varicella vaccination program for US children. JAMA. 1994;271(5):375-381.
PubMedArticle
38.
Huse  DM, Meissner  HC, Lacey  MJ, Oster  G.  Childhood vaccination against chickenpox: an analysis of benefits and costs. J Pediatr. 1994;124(6):869-874.
PubMedArticle
39.
Bloom  BS, Hillman  AL, Fendrick  AM, Schwartz  JS.  A reappraisal of hepatitis B virus vaccination strategies using cost-effectiveness analysis. Ann Intern Med. 1993;118(4):298-306.
PubMedArticle
40.
Hay  JW, Daum  RS.  Cost-benefit analysis of two strategies for prevention of Haemophilus influenzae type B infection. Pediatrics. 1987;80(3):319-329.
PubMed
41.
Cochi  SL, Broome  CV, Hightower  AW.  Immunization of US children with Hemophilus influenzae type B polysaccharide vaccine: a cost-effectiveness model of strategy assessment. JAMA. 1985;253(4):521-529.
PubMedArticle
42.
White  CC, Koplan  JP, Orenstein  WA.  Benefits, risks and costs of immunization for measles, mumps and rubella. Am J Public Health. 1985;75(7):739-744.
PubMedArticle
43.
Hinman  AR, Koplan  JP.  Pertussis and pertussis vaccine: reanalysis of benefits, risks, and costs. JAMA. 1984;251(23):3109-3113.
PubMedArticle
44.
Mulley  AG, Silverstein  MD, Dienstag  JL.  Indications for use of hepatitis B vaccine, based on cost-effectiveness analysis. N Engl J Med. 1982;307(11):644-652.
PubMedArticle
45.
Koplan  JP, Preblud  SR.  A benefit-cost analysis of mumps vaccine. Am J Dis Child. 1982;136(4):362-364.
PubMed
46.
Albritton  RB.  Cost-benefits of measles eradication: effects of a federal intervention. Policy Anal. 1978;4(1):1-21.
PubMed
47.
Schoenbaum  SC, Hyde JN Jr Bartoshesky  L, Crampton  K.  Benefit-cost analysis of rubella vaccination policy. N Engl J Med. 1976;294(6):306-310.
PubMedArticle
Original Investigation
April 2015

Collaborative Centralized Reminder/Recall Notification to Increase Immunization Rates Among Young ChildrenA Comparative Effectiveness Trial

Author Affiliations
  • 1Children’s Outcomes Research Program, Children’s Hospital Colorado, Aurora
  • 2Department of Pediatrics, University of Colorado, Denver
  • 3Colorado Health Outcomes Program, University of Colorado, Denver
  • 4Department of Family Medicine, University of Colorado, Denver
  • 5Department of Health Policy and Management, University of South Florida, Tampa
  • 6Department of Pediatrics, University of South Florida, Tampa
  • 7Colorado Immunization and Information System, Colorado Department of Public Health and Environment, Denver
JAMA Pediatr. 2015;169(4):365-373. doi:10.1001/jamapediatrics.2014.3670
Abstract

Importance  Reminder/recall notifications used by primary care practices increase the rates of childhood immunizations, but fewer than 20% of primary care practitioners nationally deliver such reminders. A reminder/recall notification conducted centrally by health departments in collaboration with primary care practices may reduce practice burden, reach children without a primary care practitioner, and decrease the cost of reminders/recalls.

Objective  To assess the effectiveness and cost-effectiveness of collaborative centralized (CC) vs practice-based (PB) reminder/recall approaches using the Colorado Immunization Information System (CIIS).

Design, Setting, and Participants  We performed a randomized pragmatic trial from September 7, 2012, through March 17, 2013, including 18 235 children aged 19 to 35 months in 15 Colorado counties.

Interventions  In CC counties, children who needed at least 1 immunization were sent as many as 4 reminders/recalls by mail or autodialed telephone calls by the CIIS. Primary care practices in these counties were given the option of endorsing the reminder/recall notification by adding the practice name to the message. In PB counties, primary care practices were invited to web-based reminder/recall training and offered financial support for sending notifications.

Main Outcomes and Measures  Documentation of any new immunization within 6 months constituted the primary outcome; achieving up-to-date (UTD) immunization status was secondary. We assessed the cost and cost-effectiveness of each approach and used a generalized linear mixed-effects model to assess the effect of the intervention on outcomes.

Results  In PB counties, 24 of 308 primary care practices (7.8%) attended reminder/recall training and 2 primary care practices (0.6%) endorsed reminder/recall notifications. Within CC counties, 129 of 229 practices (56.3%) endorsed the reminder/recall letter. Documentation rates for at least 1 immunization were 26.9% for CC vs 21.7% for PB counties (P < .001); 12.8% vs 9.3% of patients, respectively, achieved UTD status (P < .001). The effect of CC counties on children’s UTD status was greater when the reminder/recall notification was endorsed by the primary care practice (19.2% vs 9.8%; P < .001). The total cost of the CC reminder/recall was $28 620 or $11.75 per child for any new immunization and $24.72 per child achieving UTD status; the total cost to the 2 practices that conducted PB reminders/recalls was $74.00 per child for any immunization and $124.45 per child achieving UTD status. The modeling resulted in an adjusted odds ratio of 1.31 (95% CI, 1.16-1.48) for any new immunization in CC vs PB counties.

Conclusions and Relevance  A CC reminder/recall notification was more effective and more cost-effective than a PB system, although the effect size was modest. Endorsement by practices may further increase the effectiveness of CC reminder/recall.

Trial Registration  clinicaltrials.gov Identifier: NCT01557621

Introduction

The present childhood immunization schedule is estimated to prevent 42 000 deaths and 20 million cases of disease and to save $14 billion in direct medical costs per US birth cohort.1 Despite these enormous benefits, full and timely vaccination of young children in the United States remains challenging. National 2011 data demonstrate coverage for the 7 routinely recommended vaccines for children aged 19 to 35 months to be 68.5%,2 which is substantially lower than the national goal of 80% for children in this age group as outlined by Healthy People 2020.3

One method of increasing these rates that is well supported by evidence4,5 and is currently recommended by the Task Force on Community Preventive Services6,7 is the use of reminder/recall notification. A reminder/recall notification by primary care practices has been associated with increases ranging from 5 to 20 percentage points. However, owing to numerous barriers, use of reminder/recall notification by primary care practices nationally has been less than 20%.8 In recent years, state immunization information systems (IISs) have been used to identify children who are in need of immunizations, potentially simplifying reminder/recall efforts. However, even when practices have access to an IIS and support in the use of an IIS-based reminder/recall system, studies suggest that only 14% to 21% of practices report willingness to use a reminder/recall system.9,10 One study11 found that recall efforts conducted centrally by the state health department using the state IIS were more effective and more cost-effective than efforts to instruct and incentivize practices to conduct recall notification for their patients.

The present study expanded the centralized reminder/recall approach to include a collaborative component between the health department and primary care practices. Our primary objectives were to compare the effectiveness and cost-effectiveness of a collaborative centralized (CC) reminder/recall approach with that of a practice-based (PB) reminder/recall approach for increasing immunization rates among preschool-aged children. We hypothesized that, from a population perspective, the CC approach would be more effective and more cost-effective than the PB approach. In addition, within the counties randomized to the CC arm only, we compared the following secondary outcomes: (1) costs per contact and marginal costs for autodialed vs mail contact modalities, (2) effectiveness and cost-effectiveness of 2 protocols for a CC reminder/recall approach, and (3) effectiveness of the CC reminder/recall approach with vs without endorsement by the primary care practice.

Methods

We conducted a stratified (rural and urban strata) cluster-randomized pragmatic trial in 15 counties in Colorado, with randomization at the county level. We followed the criteria established for National Institutes of Health–funded pragmatic trials12,13 in designing the trial and analyzing the data. The study was approved by the Colorado Multiple Institutional Review Board. The expedited protocol did not require informed consent.

Settings and Study Participants
Colorado IIS

The Colorado IIS (CIIS) receives client and vaccination event data through live data entry into the web-enabled application and through electronic transfers from data sources maintained by health care practitioners, state vital statistics, and insurers. The CIIS also includes historical data about immunizations given outside the state if entered by a Colorado practitioner or school. Colorado is not a mandatory reporting state; however, the percentage of children younger than 6 years with at least 2 records in the CIIS was 99% at the time of the study.14 All public health entities, 91% of pediatric practices, and 74% of family medicine practices in Colorado were enrolled in the CIIS in 2013.14

Selection of Participating Counties and Randomization

We originally selected 16 (8 urban and 8 rural) Colorado counties of a total of 64 based on similar prespecified criteria15,16 (Table 1) and because they did not have characteristics that could confound the trial, including (1) existing countywide reminder/recall efforts (6 rural and 4 urban), (2) low CIIS saturation rates (<70% of children aged 3 years with ≥2 immunizations) (10 rural and 4 urban), and (3) frontier counties with fewer than 6 people per square mile (23 counties). Eight counties were randomized to the CC and 8 counties to the PB reminder/recall arms using constrained covariate randomization based on practice and county characteristics that might influence outcome measures (ie, percentage of children in the CIIS, number of children aged 19-35 months, percentage of children whose immunizations were up-to-date (UTD) at baseline, percentages of Hispanic and African American children, mean income, ratio of pediatric to family medicine practices, and number of community health centers) (Table 1).1720 After randomization, 1 county in the CC arm had to be excluded because 32% of children in the county were covered by military insurance, and we discovered that records for these children were not being uploaded into the CIIS.21 This factor would mean excluding approximately 8500 of children aged 19 to 35 months; therefore, this county was excluded.

Denominating Study Population

The study used CIIS county-level reminder/recall functionality to identify children aged 19 to 35 months as of September 7, 2012, through March 17, 2013, with an address in one of the study counties and who appeared to need at least 1 immunization. This method included children born in Colorado and those who moved into a study county if they had received an immunization entered into the CIIS. Because the CIIS is populated from vital statistics, it also included children who were born in the county but whose primary care practitioner did not enter information into the CIIS, thereby providing the best population estimate of all children within the county. The study population was fixed from this point, and children who moved into or out of the counties after this time were not added to or subtracted from the study population. This method meant that children who moved might have been inappropriately assigned during the study period; however, the potential bias was likely to be balanced between the study arms.

CC Reminder/Recall Approach

Collaborations were developed within the first year of the project by completing 37 in-depth interviews with primary care physicians, CIIS leaders, health department leaders, and parents. Based on a summary of these data, an advisory panel of 15 physicians, health department leaders, and parents agreed that practices could collaborate by (1) including their name and telephone number on the centralized reminder/recall notifications along with local health departments (endorsement) and (2) updating addresses or telephone numbers once the reminder/recall effort was under way.

Patients in the CC arm were subsequently randomized at the patient level into the following 2 protocols: (1) an autodial/mail protocol and (2) a mail-only protocol. Using intention to treat, families were randomized regardless of whether the child had a telephone number listed in the CIIS to determine which strategy was most effective on a population level. Patients in the autodial/mail arm received 2 autodialed calls (if they had a telephone number in the CIIS) followed by as many as 2 postcards during a 4-month period. If they had no telephone number, they received as many as 4 mailings but were retained in the autodial/mail protocol. Patients in the mail-only group received 1 letter and as many as 3 postcards approximately every 6 weeks. Families were removed between contacts if the child was determined to have UTD immunization status. Parents could call a toll-free number or send an e-mail to have their child’s name removed from the recall list.

PB Reminder/Recall Approach

All practices in this study arm that participated in the CIIS were sent as many as 8 invitations to attend webinar training on reminder/recall notification using the CIIS. The training recommended that the practices send reminder/recall notifications to children aged 19 to 35 months and follow methods similar to those in the CC approach. Practices were offered financial reimbursement for mailing costs or the costs of generating autodialed reminder/recall messages for children within this age range during the study period. The CIIS ran reports monthly to determine which practices ran a recall report in the CIIS, and, if a recall report was generated, a staff member would telephone the practice to determine whether the practice actually sent out mailings or telephone calls to patients and to assess costs. More details regarding the interventions are included in the eAppendix in the Supplement.

Outcome Measures

The definition of UTD was based on the series of antigens (4:3:1:3:3:1:3) recommended by the National Advisory Committee on Immunization Practices to be received by 19 to 20 months of age.16 The primary study outcome was documentation of any new vaccine within 6 months among children aged 19 to 35 months whose vaccination status was not UTD. This outcome could include receipt of a vaccine or documentation of a previously received vaccine because both might be attributed to the reminder/recall effort. Secondary measures included achieving UTD status, cost-effectiveness comparisons of the 2 interventions, and the effect of practice endorsement on the primary outcomes.

Statistical Analysis

The study was powered to detect an absolute difference of 5 percentage points in documentation of a new immunization among children whose vaccination status was not UTD at baseline between the 2 arms. The α value was set at .05 for the primary outcome. For secondary comparisons, we applied the Bonferroni correction for multiple comparisons. The 3 secondary outcomes were assessed using an α value of .0167. The final eligible cohort was established by matching child identification numbers from the follow-up CIIS database obtained March 17, 2013, to the baseline cohort database from September 7, 2012; 99.7% of children initially identified had a record in the follow-up database. Default site of service was defined at the beginning of the trial as the site to which the child was attributed in the immunization registry and did not change during the trial. All major analyses were intention to treat. We modeled 2 immunization outcomes using generalized linear mixed-effects models (PROC GLIMMIX)22,23 with the logit link function. Default site of service was included as a random intercept to account for the clustering of patients within practices if the number of children was sufficient. However, numerous practices had few patients, and model convergence was problematic in the preliminary analysis. Thus, for smaller practices, patients within a county and single practice type (eg, private) were assigned to a single practice cluster. The fixed-effect variables that were included in all multivariable models were study arm (CC vs PB), county baseline UTD rates, and rural or urban location. We adjusted for baseline UTD rates because the effectiveness of reminder/recall notification could vary based on initial rates owing to a ceiling effect. We also compared the CC reminder/recall approach with and without practice endorsement using generalized linear mixed-effects models. All analyses were performed using commercially available software (SAS, version 9.4; SAS Institute, Inc).

Cost-effectiveness Methods

The cost-effectiveness of the reminder/recall methods was compared between the CC and PB intervention arms and within the CC arm between the 2 reminder/recall protocols from the perspective of the state for the CC approach and the perspective of the practice for the PB approach. The total intervention costs directly related to conducting reminder/recall efforts for each arm were determined and then compared with the primary outcomes to determine the cost-effectiveness of each method.

The activities and associated costs for the CC and PB reminder/recall approaches were identified by contacting personnel through telephone interviews to determine their position title and to estimate the amount of time spent on reminder/recall activities. Personnel wages were estimated using median hourly wages for the relevant position titles reported by the Bureau of Labor Statistics for the State of Colorado in 2011,24 and benefits were estimated at 30%. Nonpersonnel costs were identified through vendor invoices. The activity areas for the CC reminder/recall approach included (1) consensus building, (2) training, (3) data quality, (4) implementation of the notifications, and (5) collaboration. Cost activity areas for the PB reminder/recall approach included (1) training, (2) data quality, and (3) implementation of the notifications. A subanalysis of costs was conducted to determine the cost per contact by contact method (autodialing vs mail) within the CC reminder/recall arm. Costs for the 2 CC reminder/recall protocols were then derived based on the number of contact types in each protocol (intention to treat) and by applying a common factor for other reminder/recall activities, generating cost-effectiveness estimates.

Results

Of the 39 428 children residing in the 15 study counties as of September 7, 2012, 18 235 (46.2%) needed at least 1 immunization according to the CIIS (Figure 1). Baseline characteristics of the counties and patient populations in the CC and PB reminder/recall arms are presented in Table 1. Most of the practices (80.7%) in the CC arm that had patients with incorrect addresses or telephone numbers after the first round of reminder/recall notification (as evidenced by returned mail or indication of a nonworking telephone) assisted with patient contact updates. Initially, 41.3% of contact addresses or numbers were incorrect. Updating by the practices decreased incorrect contact information to 25.5%. No known adverse outcomes were related to the reminder/recall interventions.

Comparative Effectiveness of CC vs PB Reminder/Recall Approaches

The percentage of eligible children who appeared to have received at least 1 reminder/recall contact (telephone or mail) was 7873 of 9049 (87.0%) in the CC reminder/recall arm. Because only 2 practices conducted reminder/recall notifications in the PB arm, the reach in this arm was 75 of 9186 (0.8%). The percentages with any new vaccine or achieving UTD immunization status were significantly higher in the CC counties compared with the PB counties (Figure 2). Adjusted rates of receiving any new vaccine were 25.8% vs 21.0% (unadjusted rates, 26.9% vs 21.7%) and of achieving UTD immunization status were 11.0% vs 8.5% (unadjusted rates, 12.8% vs 9.3%) for the CC vs PB reminder/recall arms, respectively. Among those who had documentation of any new immunization, 54.3% received an immunization and documentation was updated in 45.7%; among those achieving UTD immunization status, 57.5% and 42.5% represented new immunizations or documentation updates, respectively. The estimate of any new vaccine documented in the CC vs PB counties by generalized linear mixed-effects modeling was 1.31 (95% CI, 1.16-1.48); for achieving UTD immunization status, 1.34 (95% CI, 1.09-1.64).

Cost-effectiveness of CC vs PB Reminder/Recall Approaches

The total costs for implementing reminder/recall notifications in each arm are shown in Table 2. Because only 2 practices conducted reminder/recall in the PB arm, the overall implementation costs were much lower here. However, for the purposes of the cost-effectiveness analysis, the number of eligible children represented by these 2 practices was 75 compared with 9049 eligible children in the CC reminder/recall arm. When we compared the costs per primary outcomes, the cost per child receiving any vaccine was more than 6 times higher and the cost per child achieving UTD immunization status was 5 times higher for the PB compared with the CC reminder/recall approaches (Table 3).

Secondary Outcomes for the CC Reminder/Recall Arm
Cost per Contact Type

The mean cost per contact for mail was $0.73 and, for autodialing, $0.53. The marginal costs for mail and autodial contacts were $0.68 and $0.15 cents, respectively.

Effectiveness and Cost-effectiveness of CC Protocols

Children in the CC counties were randomized to the autodial/mail group or to the mail-only group (Figure 1). Of those in the autodial/mail group, 1899 (42.0%) did not have a telephone number in the CIIS. The percentages of children in the autodial/mail arm vs the mail-only arm who received any vaccine (26.9% vs 26.9%) or achieved UTD status (12.9% vs 12.7%) did not differ. The mean number of mail contacts was 2.2 in the autodial/mail group and 3.1 in the mail-only group. The costs per child, per child achieving UTD status, and per vaccine were slightly higher for the mail protocol than the autodial protocol at $3.22 vs $3.11, $25.31 vs $24.13, and $11.97 vs $11.54, respectively.

Effectiveness of CC Reminder/Recall Approach With vs Without Practice Endorsement

In the 7 CC counties, 229 of the total 268 practices (85.4%) used the CIIS and were invited to include their practice name on the reminder/recall notices or autodialed messages; 129 (56.3%) agreed. Among the 3067 children in practices that endorsed the reminder/recall notification, 32.1% received a vaccine compared with 24.2% among the 5982 children in practices that did not endorse the reminder/recall notification (P = .01). Comparable percentages for achieving UTD immunization status were 19.2% vs 9.5% (P < .001).

Discussion

This study demonstrated greater effectiveness and cost-effectiveness of a CC IIS-based reminder/recall approach compared with a PB reminder/recall approach for increasing rates of immunization among populations of preschool-aged children. When a telephone number was found in the CIIS, beginning with autodialed messages and then adding mailings decreased cost without decreasing effectiveness. However, costs would have been much lower if the CIIS were better populated with telephone data and if the accuracy of all contact data could be increased. Endorsement of CC reminder/recall activities by the inclusion of the practice name in reminder/recall messages appeared to increase the effectiveness of a CC reminder/recall approach.

The effect sizes seen in the present trial, although modest, are important from a public health perspective. However, they are lower than those of a previous county-based trial among preschool-aged children,11 which showed a 9% absolute difference in receiving any immunization and a 6.4% difference in the rate of achieving UTD immunization status between centralized and PB approaches. Differences in the effect sizes may result from higher baseline UTD rates in the present trial (14 absolute percentage points), differences in the participating county populations, or immunization delivery interventions of which we were not aware in PB counties. The present and former trials,11 however, were equally strong in demonstrating the substantially higher cost-effectiveness of a CC vs a PB approach to reminder/recall notification.

The present trial demonstrates, more dramatically than the previous one,11 the poor reach of a PB approach from the population perspective. Only 2 of 308 practices in the PB reminder/recall approach conducted reminder/recall notification using the CIIS despite multiple invitations to participate in webinar training and the incentive of covering the costs of reminder/recall notification incurred by the practices. Although multiple previous trials have documented the effectiveness of reminder/recall notification when conducted by practices,4,5,25 the present study and others10,25,26 demonstrate that few practices are willing or able to conduct these efforts without external support, even if financial incentives are provided. Although our study did not examine the reasons for lack of participation in the PB counties, other studies have shown that barriers to an IIS-based reminder/recall approach have included lack of awareness of baseline immunization rates resulting in a lack of perceived need for reminder/recall notification, distrust in the accuracy of IIS-generated data, time and personnel constraints, the need for training and retraining of personnel, and difficulties with execution of reminder/recall notification using the IIS.10,26

We must consider the acceptability of centralized reminder/recall approaches to practitioners and parents. Recent survey data in the counties in which this trial was conducted27 demonstrated that most physicians (56%) preferred that the health department, rather than their practice, conduct reminder/recall notifications centrally, and only 15% reported that they would be bothered if this occurred. Among parents in counties that were not part of a CC reminder/recall appproach,28 at least half of those surveyed had no preference for whether the health department or their child’s physician sent them reminder/recall notifications, and, overall, 89% were supportive of the health department sending the notifications. The inclusion of the practice’s name in a CC reminder/recall notice may garner the support of physicians and patients who prefer the practice’s involvement. Indeed, our study suggests that CC reminder/recall notifications with endorsement by the practice may be the optimal method of a population-based reminder/recall approach. Although endorsement was not randomized in the present trial, allowing for potential selection bias, effectiveness of the CC reminder/recall approach was similar to that of the PB reminder/recall approach if the practice’s name was not included and substantially higher if it was. These results differed from those of an earlier trial that showed higher effectiveness of a CC reminder/recall notification without any practice endorsement,11 suggesting the need for further exploration of the effect of endorsement in a randomized clinical trial.

Our data demonstrate the additional savings that could be realized if more autodialed reminder/recall notifications or other mobile health methods could be used rather than mail. Additional studies are required to find the optimal number of messages needed by various methods to minimize costs while maintaining effectiveness. The costs of updating inaccurate mail and telephone data in our study underline the major cost savings that could be achieved if automatic uploading of electronic contact data within practice administrative databases into the IIS could occur on a routine basis. If multiple telephone numbers and e-mails were routinely collected by practices, and, with the family’s consent, these data could be included with uploads, substantial efficiencies in reminder/recall notification could result.

These data have important strengths and limitations. The trial was population based, and we were able to measure effectiveness and cost-effectiveness for entire counties. Estimates of the denominator for the counties were not exact, although the degree of uncertainty should be similar between the 2 study arms. The number of practices that chose to perform reminder/recall notifications in the PB counties was small; therefore, our cost estimates for the PB reminder/recall approach must be interpreted with caution. However, the costs of conducting reminder/recall notification per practice were similar to those of a previous trial in which cost estimates were based on data from 10 practices,11 suggesting relatively consistent estimates. In the CC reminder/recall arm, practices could choose to endorse the notification and were, therefore, a potentially biased group compared with practices that did not endorse the notification. Our study was also limited to a single-state IIS; results in states using IISs with different forecasting and reminder/recall capabilities could differ. We were unable to track any reminder/recall efforts that did not use the IIS in both arms of the study. As with most pragmatic trials, the practices and the patients were not blinded to the interventions received. Finally, because participation in the CIIS is voluntary, incomplete data entry into the CIIS undoubtedly resulted in artificially low immunizations rates at baseline and after the intervention in all counties, although we would not expect these rates to affect the comparisons reported.

Conclusions

Our findings and those of previous studies11,27,29 support consideration of a CC compared with a PB reminder/recall approach to increase immunization rates during the preschool years. Sustainable funding mechanisms will be needed to support such an approach and may involve a shared investment between practice organizations or accountable care organizations and the public sector. With minimal contributions from each, substantial cost savings should be realized from a societal perspective.3047

Back to top
Article Information

Accepted for Publication: September 29, 2014.

Corresponding Author: Allison Kempe, MD, MPH, Children’s Outcomes Research Program, Children’s Hospital Colorado, 13199 E Montview Blvd, Ste 300, Aurora, CO 80045 (allison.kempe@childrenscolorado.org).

Published Online: February 23, 2015. doi:10.1001/jamapediatrics.2014.3670.

Author Contributions: Dr Kempe had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Kempe, Dickinson, Eisert, Gurfinkel, Herrero, Herlihy.

Acquisition, analysis, or interpretation of data: Kempe, Saville, Dickinson, Beaty, Gurfinkel, Brewer, Shull, Herrero.

Drafting of the manuscript: Kempe, Saville, Dickinson, Eisert.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Dickinson, Beaty, Eisert.

Obtained funding: Kempe, Saville, Eisert.

Administrative, technical, or material support: Saville, Gurfinkel, Brewer, Shull, Herrero, Herlihy.

Study supervision: Kempe, Herlihy.

Conflict of Interest Disclosures: None reported.

Funding/Support: This project was supported by grant P01HS021138 from the Agency for Healthcare Research and Quality.

Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

References
1.
Zhou  F, Shefer  A, Wenger  J,  et al.  Economic evaluation of the routine childhood immunization program in the United States, 2009. Pediatrics. 2014;133(4):577-585.
PubMedArticle
2.
Centers for Disease Control and Prevention (CDC).  National, state, and local area vaccination coverage among children aged 19-35 months: United States, 2011. MMWR Morb Mortal Wkly Rep. 2012;61:689-696.
PubMed
3.
US Department of Health and Human Services Office of Disease Prevention and Health Promotion. Healthy People 2020. October 30, 2012. http://www.healthypeople.gov/2020/topicsobjectives2020/default. Accessed February 7, 2013.
4.
Szilagyi  PG, Bordley  C, Vann  JC,  et al.  Effect of patient reminder/recall interventions on immunization rates: a review. JAMA. 2000;284(14):1820-1827.
PubMedArticle
5.
Szilagyi  P, Vann  J, Bordley  C,  et al.  Interventions aimed at improving immunization rates. Cochrane Database Syst Rev. 2002;(4):CD003941.
PubMed
6.
Briss  PA, Rodewald  LE, Hinman  AR,  et al; Task Force on Community Preventive Services.  Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med. 2000;18(1)(suppl):97-140.
PubMedArticle
7.
Task Force on Community Preventive Services.  Recommendations regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med. 2000;18(1)(suppl):92-96.
PubMedArticle
8.
Tierney  CD, Yusuf  H, McMahon  SR,  et al.  Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics. Pediatrics. 2003;112(5):1076-1082.
PubMedArticle
9.
Clark  SJ, Cowan  AE, Bartlett  DL.  Private provider participation in statewide immunization registries. BMC Public Health. 2006;6:33. doi:10.1186/1471-2458-6-33.
PubMedArticle
10.
Dombkowski  KJ, Cowan  AE, Harrington  LB, Allred  NJ, Hudson  E, Clark  SJ.  Feasibility of initiating and sustaining registry-based immunization recall in private practices. Acad Pediatr. 2012;12(2):104-109.
PubMedArticle
11.
Kempe  A, Saville  A, Dickinson  LM,  et al.  Population-based vs practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial. Am J Public Health. 2013;103(6):1116-1123.
PubMedArticle
12.
Thorpe  KE, Zwarenstein  M, Oxman  AD,  et al.  A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ. 2009;180(10):E47-E57.
PubMedArticle
13.
Department of Health and Human Services. NIH Health Care Systems Research Collaboratory-Pragmatic Clinical Trials Demonstration Projects (UH2/UH3). January 24, 2012. http://grants.nih.gov/grants/guide/rfa-files/RFA-RM-12-002.html. Accessed July 31, 2012.
14.
Colorado Department of Public Health and Environment CIIS. Colorado Annual Immunization Progress Report for CY2011. Denver, CO: Immunization Information Systems; 2011.
15.
Colorado Rural Health Center. Colorado: county designations, 2014. https://coruralhealth.org/wp-content/uploads/2013/10/2014.Colorado-County-Designations.pdf. Accessed January 23, 2015.
16.
Centers for Disease Control and Prevention. Recommended immunization schedule for persons age 0 through 18 years: United States, 2014.http://www.cdc.gov/vaccines/schedules/hcp/imz/child-adolescent.html. Accessed December 11, 2014.
17.
Raab  GM, Butcher  I.  Balance in cluster randomized trials. Stat Med. 2001;20(3):351-365.
PubMedArticle
18.
Glynn  RJ, Brookhart  MA, Stedman  M, Avorn  J, Solomon  DH.  Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers. Med Care. 2007;45(10)(suppl 2):S38-S43.
PubMedArticle
19.
Ivers  NM, Halperin  IJ, Barnsley  J,  et al.  Allocation techniques for balance at baseline in cluster randomized trials: a methodological review. Trials. 2012;13:120. doi:10.1186/1745-6215-13-120.
PubMedArticle
20.
Moulton  LH.  Covariate-based constrained randomization of group-randomized trials. Clin Trials. 2004;1(3):297-305.
PubMedArticle
21.
US Department of Justice. The Freedom of Information Act (FOIA). September 1, 2014.
22.
Littel  RC, Milliken  GA, Stroup  WW, Wolfinger  RD. SAS System for Mixed Models. Cary, NC: SAS Institute Inc; 1999.
23.
Raudenbush  SW, Bryk  AS. Hierarchical Linear Models: Applications and Data Analysis Methods (Advanced Quantitative Techniques in the Social Sciences).2nd ed. Newbury Park, CA: Sage Publications Inc; 2001.
24.
Bureau of Labor Statistics. Occupational employment statistics.http://www.bls.gov/oes/2011/may/oes_co.htm. Accessed January 12, 2015.
25.
Jacobson Vann  JC, Szilagyi  P.  Patient reminder and patient recall systems to improve immunization rates [update of Cochrane Database Syst Rev. 2002;(4):CD003941]. Cochrane Database Syst Rev. 2005;(3):CD003941.
PubMed
26.
Saville  AW, Albright  K, Nowels  C,  et al.  Getting under the hood: exploring issues that affect provider-based recall using an immunization information system. Acad Pediatr. 2011;11(1):44-49.
PubMedArticle
27.
Albright  K, Saville  A, Lockhart  S, Widmer Racich  K, Beaty  B, Kempe  A.  Provider attitudes toward public-private collaboration to improve immunization reminder/recall: a mixed-methods study. Acad Pediatr. 2014;14(1):62-70.
PubMedArticle
28.
Kempe  A, Daley  MF, Pyrzanowski  J,  et al.  School-located influenza vaccination with third-party billing: what do parents think? Acad Pediatr. 2014;14(3):241-248.
PubMedArticle
29.
Saville  AW, Beaty  B, Dickinson  LM, Lockhart  S, Kempe  A.  Novel immunization reminder/recall approaches: rural and urban differences in parent perceptions. Acad Pediatr. 2014;14(3):249-255.
PubMedArticle
30.
Zhou  F, Santoli  J, Messonnier  ML,  et al.  Economic evaluation of the 7-vaccine routine childhood immunization schedule in the United States, 2001. Arch Pediatr Adolesc Med. 2005;159(12):1136-1144.
PubMedArticle
31.
Zhou  F, Bisgard  KM, Yusuf  HR, Deuson  RR, Bath  SK, Murphy  TV.  Impact of universal Haemophilus influenzae type B vaccination starting at 2 months of age in the United States: an economic analysis. Pediatrics. 2002;110(4):653-661.
PubMedArticle
32.
Lieu  TA, Ray  GT, Black  SB,  et al.  Projected cost-effectiveness of pneumococcal conjugate vaccination of healthy infants and young children. JAMA. 2000;283(11):1460-1468.
PubMedArticle
33.
Ekwueme  DU, Strebel  PM, Hadler  SC, Meltzer  MI, Allen  JW, Livengood  JR.  Economic evaluation of use of diphtheria, tetanus, and acellular pertussis vaccine or diphtheria, tetanus, and whole-cell pertussis vaccine in the United States, 1997. Arch Pediatr Adolesc Med. 2000;154(8):797-803.
PubMedArticle
34.
Fendrick  AM, Lee  JH, LaBarge  C, Glick  HA.  Clinical and economic impact of a combination Haemophilus influenzae and hepatitis B vaccine: estimating cost-effectiveness using decision analysis. Arch Pediatr Adolesc Med. 1999;153(2):126-136.
PubMedArticle
35.
Miller  MA, Sutter  RW, Strebel  PM, Hadler  SC.  Cost-effectiveness of incorporating inactivated poliovirus vaccine into the routine childhood immunization schedule. JAMA. 1996;276(12):967-971.
PubMedArticle
36.
Margolis  HS, Coleman  PJ, Brown  RE, Mast  EE, Sheingold  SH, Arevalo  JA.  Prevention of hepatitis B virus transmission by immunization: an economic analysis of current recommendations. JAMA. 1995;274(15):1201-1208.
PubMedArticle
37.
Lieu  TA, Cochi  SL, Black  SB,  et al.  Cost-effectiveness of a routine varicella vaccination program for US children. JAMA. 1994;271(5):375-381.
PubMedArticle
38.
Huse  DM, Meissner  HC, Lacey  MJ, Oster  G.  Childhood vaccination against chickenpox: an analysis of benefits and costs. J Pediatr. 1994;124(6):869-874.
PubMedArticle
39.
Bloom  BS, Hillman  AL, Fendrick  AM, Schwartz  JS.  A reappraisal of hepatitis B virus vaccination strategies using cost-effectiveness analysis. Ann Intern Med. 1993;118(4):298-306.
PubMedArticle
40.
Hay  JW, Daum  RS.  Cost-benefit analysis of two strategies for prevention of Haemophilus influenzae type B infection. Pediatrics. 1987;80(3):319-329.
PubMed
41.
Cochi  SL, Broome  CV, Hightower  AW.  Immunization of US children with Hemophilus influenzae type B polysaccharide vaccine: a cost-effectiveness model of strategy assessment. JAMA. 1985;253(4):521-529.
PubMedArticle
42.
White  CC, Koplan  JP, Orenstein  WA.  Benefits, risks and costs of immunization for measles, mumps and rubella. Am J Public Health. 1985;75(7):739-744.
PubMedArticle
43.
Hinman  AR, Koplan  JP.  Pertussis and pertussis vaccine: reanalysis of benefits, risks, and costs. JAMA. 1984;251(23):3109-3113.
PubMedArticle
44.
Mulley  AG, Silverstein  MD, Dienstag  JL.  Indications for use of hepatitis B vaccine, based on cost-effectiveness analysis. N Engl J Med. 1982;307(11):644-652.
PubMedArticle
45.
Koplan  JP, Preblud  SR.  A benefit-cost analysis of mumps vaccine. Am J Dis Child. 1982;136(4):362-364.
PubMed
46.
Albritton  RB.  Cost-benefits of measles eradication: effects of a federal intervention. Policy Anal. 1978;4(1):1-21.
PubMed
47.
Schoenbaum  SC, Hyde JN Jr Bartoshesky  L, Crampton  K.  Benefit-cost analysis of rubella vaccination policy. N Engl J Med. 1976;294(6):306-310.
PubMedArticle
×