Collaborative Centralized Reminder/Recall Notification to Increase Immunization Rates Among Young Children: A Comparative Effectiveness Trial

Importance  Reminder/recall notifications used by primary care practices increase the rates of childhood immunizations, but fewer than 20% of primary care practitioners nationally deliver such reminders. A reminder/recall notification conducted centrally by health departments in collaboration with primary care practices may reduce practice burden, reach children without a primary care practitioner, and decrease the cost of reminders/recalls.

Objective  To assess the effectiveness and cost-effectiveness of collaborative centralized (CC) vs practice-based (PB) reminder/recall approaches using the Colorado Immunization Information System (CIIS).

Design, Setting, and Participants  We performed a randomized pragmatic trial from September 7, 2012, through March 17, 2013, including 18 235 children aged 19 to 35 months in 15 Colorado counties.

Interventions  In CC counties, children who needed at least 1 immunization were sent as many as 4 reminders/recalls by mail or autodialed telephone calls by the CIIS. Primary care practices in these counties were given the option of endorsing the reminder/recall notification by adding the practice name to the message. In PB counties, primary care practices were invited to web-based reminder/recall training and offered financial support for sending notifications.

Main Outcomes and Measures  Documentation of any new immunization within 6 months constituted the primary outcome; achieving up-to-date (UTD) immunization status was secondary. We assessed the cost and cost-effectiveness of each approach and used a generalized linear mixed-effects model to assess the effect of the intervention on outcomes.

Results  In PB counties, 24 of 308 primary care practices (7.8%) attended reminder/recall training and 2 primary care practices (0.6%) endorsed reminder/recall notifications. Within CC counties, 129 of 229 practices (56.3%) endorsed the reminder/recall letter. Documentation rates for at least 1 immunization were 26.9% for CC vs 21.7% for PB counties (P < .001); 12.8% vs 9.3% of patients, respectively, achieved UTD status (P < .001). The effect of CC counties on children’s UTD status was greater when the reminder/recall notification was endorsed by the primary care practice (19.2% vs 9.8%; P < .001). The total cost of the CC reminder/recall was $28 620 or $11.75 per child for any new immunization and $24.72 per child achieving UTD status; the total cost to the 2 practices that conducted PB reminders/recalls was $74.00 per child for any immunization and $124.45 per child achieving UTD status. The modeling resulted in an adjusted odds ratio of 1.31 (95% CI, 1.16-1.48) for any new immunization in CC vs PB counties.

Conclusions and Relevance  A CC reminder/recall notification was more effective and more cost-effective than a PB system, although the effect size was modest. Endorsement by practices may further increase the effectiveness of CC reminder/recall.

Trial Registration  clinicaltrials.gov Identifier: NCT01557621

The present childhood immunization schedule is estimated to prevent 42 000 deaths and 20 million cases of disease and to save $14 billion in direct medical costs per US birth cohort.¹ Despite these enormous benefits, full and timely vaccination of young children in the United States remains challenging. National 2011 data demonstrate coverage for the 7 routinely recommended vaccines for children aged 19 to 35 months to be 68.5%,² which is substantially lower than the national goal of 80% for children in this age group as outlined by Healthy People 2020.³

One method of increasing these rates that is well supported by evidence^4,5 and is currently recommended by the Task Force on Community Preventive Services^6,7 is the use of reminder/recall notification. A reminder/recall notification by primary care practices has been associated with increases ranging from 5 to 20 percentage points. However, owing to numerous barriers, use of reminder/recall notification by primary care practices nationally has been less than 20%.⁸ In recent years, state immunization information systems (IISs) have been used to identify children who are in need of immunizations, potentially simplifying reminder/recall efforts. However, even when practices have access to an IIS and support in the use of an IIS-based reminder/recall system, studies suggest that only 14% to 21% of practices report willingness to use a reminder/recall system.^9,10 One study¹¹ found that recall efforts conducted centrally by the state health department using the state IIS were more effective and more cost-effective than efforts to instruct and incentivize practices to conduct recall notification for their patients.

The present study expanded the centralized reminder/recall approach to include a collaborative component between the health department and primary care practices. Our primary objectives were to compare the effectiveness and cost-effectiveness of a collaborative centralized (CC) reminder/recall approach with that of a practice-based (PB) reminder/recall approach for increasing immunization rates among preschool-aged children. We hypothesized that, from a population perspective, the CC approach would be more effective and more cost-effective than the PB approach. In addition, within the counties randomized to the CC arm only, we compared the following secondary outcomes: (1) costs per contact and marginal costs for autodialed vs mail contact modalities, (2) effectiveness and cost-effectiveness of 2 protocols for a CC reminder/recall approach, and (3) effectiveness of the CC reminder/recall approach with vs without endorsement by the primary care practice.

We conducted a stratified (rural and urban strata) cluster-randomized pragmatic trial in 15 counties in Colorado, with randomization at the county level. We followed the criteria established for National Institutes of Health–funded pragmatic trials^12,13 in designing the trial and analyzing the data. The study was approved by the Colorado Multiple Institutional Review Board. The expedited protocol did not require informed consent.

The Colorado IIS (CIIS) receives client and vaccination event data through live data entry into the web-enabled application and through electronic transfers from data sources maintained by health care practitioners, state vital statistics, and insurers. The CIIS also includes historical data about immunizations given outside the state if entered by a Colorado practitioner or school. Colorado is not a mandatory reporting state; however, the percentage of children younger than 6 years with at least 2 records in the CIIS was 99% at the time of the study.¹⁴ All public health entities, 91% of pediatric practices, and 74% of family medicine practices in Colorado were enrolled in the CIIS in 2013.¹⁴

We originally selected 16 (8 urban and 8 rural) Colorado counties of a total of 64 based on similar prespecified criteria^15,16 (Table 1) and because they did not have characteristics that could confound the trial, including (1) existing countywide reminder/recall efforts (6 rural and 4 urban), (2) low CIIS saturation rates (<70% of children aged 3 years with ≥2 immunizations) (10 rural and 4 urban), and (3) frontier counties with fewer than 6 people per square mile (23 counties). Eight counties were randomized to the CC and 8 counties to the PB reminder/recall arms using constrained covariate randomization based on practice and county characteristics that might influence outcome measures (ie, percentage of children in the CIIS, number of children aged 19-35 months, percentage of children whose immunizations were up-to-date (UTD) at baseline, percentages of Hispanic and African American children, mean income, ratio of pediatric to family medicine practices, and number of community health centers) (Table 1).^17-20 After randomization, 1 county in the CC arm had to be excluded because 32% of children in the county were covered by military insurance, and we discovered that records for these children were not being uploaded into the CIIS.²¹ This factor would mean excluding approximately 8500 of children aged 19 to 35 months; therefore, this county was excluded.

The study used CIIS county-level reminder/recall functionality to identify children aged 19 to 35 months as of September 7, 2012, through March 17, 2013, with an address in one of the study counties and who appeared to need at least 1 immunization. This method included children born in Colorado and those who moved into a study county if they had received an immunization entered into the CIIS. Because the CIIS is populated from vital statistics, it also included children who were born in the county but whose primary care practitioner did not enter information into the CIIS, thereby providing the best population estimate of all children within the county. The study population was fixed from this point, and children who moved into or out of the counties after this time were not added to or subtracted from the study population. This method meant that children who moved might have been inappropriately assigned during the study period; however, the potential bias was likely to be balanced between the study arms.

Collaborations were developed within the first year of the project by completing 37 in-depth interviews with primary care physicians, CIIS leaders, health department leaders, and parents. Based on a summary of these data, an advisory panel of 15 physicians, health department leaders, and parents agreed that practices could collaborate by (1) including their name and telephone number on the centralized reminder/recall notifications along with local health departments (endorsement) and (2) updating addresses or telephone numbers once the reminder/recall effort was under way.

Patients in the CC arm were subsequently randomized at the patient level into the following 2 protocols: (1) an autodial/mail protocol and (2) a mail-only protocol. Using intention to treat, families were randomized regardless of whether the child had a telephone number listed in the CIIS to determine which strategy was most effective on a population level. Patients in the autodial/mail arm received 2 autodialed calls (if they had a telephone number in the CIIS) followed by as many as 2 postcards during a 4-month period. If they had no telephone number, they received as many as 4 mailings but were retained in the autodial/mail protocol. Patients in the mail-only group received 1 letter and as many as 3 postcards approximately every 6 weeks. Families were removed between contacts if the child was determined to have UTD immunization status. Parents could call a toll-free number or send an e-mail to have their child’s name removed from the recall list.

All practices in this study arm that participated in the CIIS were sent as many as 8 invitations to attend webinar training on reminder/recall notification using the CIIS. The training recommended that the practices send reminder/recall notifications to children aged 19 to 35 months and follow methods similar to those in the CC approach. Practices were offered financial reimbursement for mailing costs or the costs of generating autodialed reminder/recall messages for children within this age range during the study period. The CIIS ran reports monthly to determine which practices ran a recall report in the CIIS, and, if a recall report was generated, a staff member would telephone the practice to determine whether the practice actually sent out mailings or telephone calls to patients and to assess costs. More details regarding the interventions are included in the eAppendix in the Supplement.

The definition of UTD was based on the series of antigens (4:3:1:3:3:1:3) recommended by the National Advisory Committee on Immunization Practices to be received by 19 to 20 months of age.¹⁶ The primary study outcome was documentation of any new vaccine within 6 months among children aged 19 to 35 months whose vaccination status was not UTD. This outcome could include receipt of a vaccine or documentation of a previously received vaccine because both might be attributed to the reminder/recall effort. Secondary measures included achieving UTD status, cost-effectiveness comparisons of the 2 interventions, and the effect of practice endorsement on the primary outcomes.

The study was powered to detect an absolute difference of 5 percentage points in documentation of a new immunization among children whose vaccination status was not UTD at baseline between the 2 arms. The α value was set at .05 for the primary outcome. For secondary comparisons, we applied the Bonferroni correction for multiple comparisons. The 3 secondary outcomes were assessed using an α value of .0167. The final eligible cohort was established by matching child identification numbers from the follow-up CIIS database obtained March 17, 2013, to the baseline cohort database from September 7, 2012; 99.7% of children initially identified had a record in the follow-up database. Default site of service was defined at the beginning of the trial as the site to which the child was attributed in the immunization registry and did not change during the trial. All major analyses were intention to treat. We modeled 2 immunization outcomes using generalized linear mixed-effects models (PROC GLIMMIX)^22,23 with the logit link function. Default site of service was included as a random intercept to account for the clustering of patients within practices if the number of children was sufficient. However, numerous practices had few patients, and model convergence was problematic in the preliminary analysis. Thus, for smaller practices, patients within a county and single practice type (eg, private) were assigned to a single practice cluster. The fixed-effect variables that were included in all multivariable models were study arm (CC vs PB), county baseline UTD rates, and rural or urban location. We adjusted for baseline UTD rates because the effectiveness of reminder/recall notification could vary based on initial rates owing to a ceiling effect. We also compared the CC reminder/recall approach with and without practice endorsement using generalized linear mixed-effects models. All analyses were performed using commercially available software (SAS, version 9.4; SAS Institute, Inc).

The cost-effectiveness of the reminder/recall methods was compared between the CC and PB intervention arms and within the CC arm between the 2 reminder/recall protocols from the perspective of the state for the CC approach and the perspective of the practice for the PB approach. The total intervention costs directly related to conducting reminder/recall efforts for each arm were determined and then compared with the primary outcomes to determine the cost-effectiveness of each method.

The activities and associated costs for the CC and PB reminder/recall approaches were identified by contacting personnel through telephone interviews to determine their position title and to estimate the amount of time spent on reminder/recall activities. Personnel wages were estimated using median hourly wages for the relevant position titles reported by the Bureau of Labor Statistics for the State of Colorado in 2011,²⁴ and benefits were estimated at 30%. Nonpersonnel costs were identified through vendor invoices. The activity areas for the CC reminder/recall approach included (1) consensus building, (2) training, (3) data quality, (4) implementation of the notifications, and (5) collaboration. Cost activity areas for the PB reminder/recall approach included (1) training, (2) data quality, and (3) implementation of the notifications. A subanalysis of costs was conducted to determine the cost per contact by contact method (autodialing vs mail) within the CC reminder/recall arm. Costs for the 2 CC reminder/recall protocols were then derived based on the number of contact types in each protocol (intention to treat) and by applying a common factor for other reminder/recall activities, generating cost-effectiveness estimates.

Of the 39 428 children residing in the 15 study counties as of September 7, 2012, 18 235 (46.2%) needed at least 1 immunization according to the CIIS (Figure 1). Baseline characteristics of the counties and patient populations in the CC and PB reminder/recall arms are presented in Table 1. Most of the practices (80.7%) in the CC arm that had patients with incorrect addresses or telephone numbers after the first round of reminder/recall notification (as evidenced by returned mail or indication of a nonworking telephone) assisted with patient contact updates. Initially, 41.3% of contact addresses or numbers were incorrect. Updating by the practices decreased incorrect contact information to 25.5%. No known adverse outcomes were related to the reminder/recall interventions.

The percentage of eligible children who appeared to have received at least 1 reminder/recall contact (telephone or mail) was 7873 of 9049 (87.0%) in the CC reminder/recall arm. Because only 2 practices conducted reminder/recall notifications in the PB arm, the reach in this arm was 75 of 9186 (0.8%). The percentages with any new vaccine or achieving UTD immunization status were significantly higher in the CC counties compared with the PB counties (Figure 2). Adjusted rates of receiving any new vaccine were 25.8% vs 21.0% (unadjusted rates, 26.9% vs 21.7%) and of achieving UTD immunization status were 11.0% vs 8.5% (unadjusted rates, 12.8% vs 9.3%) for the CC vs PB reminder/recall arms, respectively. Among those who had documentation of any new immunization, 54.3% received an immunization and documentation was updated in 45.7%; among those achieving UTD immunization status, 57.5% and 42.5% represented new immunizations or documentation updates, respectively. The estimate of any new vaccine documented in the CC vs PB counties by generalized linear mixed-effects modeling was 1.31 (95% CI, 1.16-1.48); for achieving UTD immunization status, 1.34 (95% CI, 1.09-1.64).

The total costs for implementing reminder/recall notifications in each arm are shown in Table 2. Because only 2 practices conducted reminder/recall in the PB arm, the overall implementation costs were much lower here. However, for the purposes of the cost-effectiveness analysis, the number of eligible children represented by these 2 practices was 75 compared with 9049 eligible children in the CC reminder/recall arm. When we compared the costs per primary outcomes, the cost per child receiving any vaccine was more than 6 times higher and the cost per child achieving UTD immunization status was 5 times higher for the PB compared with the CC reminder/recall approaches (Table 3).

The mean cost per contact for mail was $0.73 and, for autodialing, $0.53. The marginal costs for mail and autodial contacts were $0.68 and $0.15 cents, respectively.

Children in the CC counties were randomized to the autodial/mail group or to the mail-only group (Figure 1). Of those in the autodial/mail group, 1899 (42.0%) did not have a telephone number in the CIIS. The percentages of children in the autodial/mail arm vs the mail-only arm who received any vaccine (26.9% vs 26.9%) or achieved UTD status (12.9% vs 12.7%) did not differ. The mean number of mail contacts was 2.2 in the autodial/mail group and 3.1 in the mail-only group. The costs per child, per child achieving UTD status, and per vaccine were slightly higher for the mail protocol than the autodial protocol at $3.22 vs $3.11, $25.31 vs $24.13, and $11.97 vs $11.54, respectively.

In the 7 CC counties, 229 of the total 268 practices (85.4%) used the CIIS and were invited to include their practice name on the reminder/recall notices or autodialed messages; 129 (56.3%) agreed. Among the 3067 children in practices that endorsed the reminder/recall notification, 32.1% received a vaccine compared with 24.2% among the 5982 children in practices that did not endorse the reminder/recall notification (P = .01). Comparable percentages for achieving UTD immunization status were 19.2% vs 9.5% (P < .001).

This study demonstrated greater effectiveness and cost-effectiveness of a CC IIS-based reminder/recall approach compared with a PB reminder/recall approach for increasing rates of immunization among populations of preschool-aged children. When a telephone number was found in the CIIS, beginning with autodialed messages and then adding mailings decreased cost without decreasing effectiveness. However, costs would have been much lower if the CIIS were better populated with telephone data and if the accuracy of all contact data could be increased. Endorsement of CC reminder/recall activities by the inclusion of the practice name in reminder/recall messages appeared to increase the effectiveness of a CC reminder/recall approach.

The effect sizes seen in the present trial, although modest, are important from a public health perspective. However, they are lower than those of a previous county-based trial among preschool-aged children,¹¹ which showed a 9% absolute difference in receiving any immunization and a 6.4% difference in the rate of achieving UTD immunization status between centralized and PB approaches. Differences in the effect sizes may result from higher baseline UTD rates in the present trial (14 absolute percentage points), differences in the participating county populations, or immunization delivery interventions of which we were not aware in PB counties. The present and former trials,¹¹ however, were equally strong in demonstrating the substantially higher cost-effectiveness of a CC vs a PB approach to reminder/recall notification.

The present trial demonstrates, more dramatically than the previous one,¹¹ the poor reach of a PB approach from the population perspective. Only 2 of 308 practices in the PB reminder/recall approach conducted reminder/recall notification using the CIIS despite multiple invitations to participate in webinar training and the incentive of covering the costs of reminder/recall notification incurred by the practices. Although multiple previous trials have documented the effectiveness of reminder/recall notification when conducted by practices,^4,5,25 the present study and others^10,25,26 demonstrate that few practices are willing or able to conduct these efforts without external support, even if financial incentives are provided. Although our study did not examine the reasons for lack of participation in the PB counties, other studies have shown that barriers to an IIS-based reminder/recall approach have included lack of awareness of baseline immunization rates resulting in a lack of perceived need for reminder/recall notification, distrust in the accuracy of IIS-generated data, time and personnel constraints, the need for training and retraining of personnel, and difficulties with execution of reminder/recall notification using the IIS.^10,26

We must consider the acceptability of centralized reminder/recall approaches to practitioners and parents. Recent survey data in the counties in which this trial was conducted²⁷ demonstrated that most physicians (56%) preferred that the health department, rather than their practice, conduct reminder/recall notifications centrally, and only 15% reported that they would be bothered if this occurred. Among parents in counties that were not part of a CC reminder/recall appproach,²⁸ at least half of those surveyed had no preference for whether the health department or their child’s physician sent them reminder/recall notifications, and, overall, 89% were supportive of the health department sending the notifications. The inclusion of the practice’s name in a CC reminder/recall notice may garner the support of physicians and patients who prefer the practice’s involvement. Indeed, our study suggests that CC reminder/recall notifications with endorsement by the practice may be the optimal method of a population-based reminder/recall approach. Although endorsement was not randomized in the present trial, allowing for potential selection bias, effectiveness of the CC reminder/recall approach was similar to that of the PB reminder/recall approach if the practice’s name was not included and substantially higher if it was. These results differed from those of an earlier trial that showed higher effectiveness of a CC reminder/recall notification without any practice endorsement,¹¹ suggesting the need for further exploration of the effect of endorsement in a randomized clinical trial.

Our data demonstrate the additional savings that could be realized if more autodialed reminder/recall notifications or other mobile health methods could be used rather than mail. Additional studies are required to find the optimal number of messages needed by various methods to minimize costs while maintaining effectiveness. The costs of updating inaccurate mail and telephone data in our study underline the major cost savings that could be achieved if automatic uploading of electronic contact data within practice administrative databases into the IIS could occur on a routine basis. If multiple telephone numbers and e-mails were routinely collected by practices, and, with the family’s consent, these data could be included with uploads, substantial efficiencies in reminder/recall notification could result.

These data have important strengths and limitations. The trial was population based, and we were able to measure effectiveness and cost-effectiveness for entire counties. Estimates of the denominator for the counties were not exact, although the degree of uncertainty should be similar between the 2 study arms. The number of practices that chose to perform reminder/recall notifications in the PB counties was small; therefore, our cost estimates for the PB reminder/recall approach must be interpreted with caution. However, the costs of conducting reminder/recall notification per practice were similar to those of a previous trial in which cost estimates were based on data from 10 practices,¹¹ suggesting relatively consistent estimates. In the CC reminder/recall arm, practices could choose to endorse the notification and were, therefore, a potentially biased group compared with practices that did not endorse the notification. Our study was also limited to a single-state IIS; results in states using IISs with different forecasting and reminder/recall capabilities could differ. We were unable to track any reminder/recall efforts that did not use the IIS in both arms of the study. As with most pragmatic trials, the practices and the patients were not blinded to the interventions received. Finally, because participation in the CIIS is voluntary, incomplete data entry into the CIIS undoubtedly resulted in artificially low immunizations rates at baseline and after the intervention in all counties, although we would not expect these rates to affect the comparisons reported.

Our findings and those of previous studies^11,27,29 support consideration of a CC compared with a PB reminder/recall approach to increase immunization rates during the preschool years. Sustainable funding mechanisms will be needed to support such an approach and may involve a shared investment between practice organizations or accountable care organizations and the public sector. With minimal contributions from each, substantial cost savings should be realized from a societal perspective.^30-47

Accepted for Publication: September 29, 2014.

Corresponding Author: Allison Kempe, MD, MPH, Children’s Outcomes Research Program, Children’s Hospital Colorado, 13199 E Montview Blvd, Ste 300, Aurora, CO 80045 (allison.kempe@childrenscolorado.org).

Published Online: February 23, 2015. doi:10.1001/jamapediatrics.2014.3670.

Author Contributions: Dr Kempe had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Kempe, Dickinson, Eisert, Gurfinkel, Herrero, Herlihy.

Acquisition, analysis, or interpretation of data: Kempe, Saville, Dickinson, Beaty, Gurfinkel, Brewer, Shull, Herrero.

Drafting of the manuscript: Kempe, Saville, Dickinson, Eisert.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Dickinson, Beaty, Eisert.

Obtained funding: Kempe, Saville, Eisert.

Administrative, technical, or material support: Saville, Gurfinkel, Brewer, Shull, Herrero, Herlihy.

Study supervision: Kempe, Herlihy.

Conflict of Interest Disclosures: None reported.

Funding/Support: This project was supported by grant P01HS021138 from the Agency for Healthcare Research and Quality.

Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.