Effect of Patient Portal Reminders Sent by a Health Care System on Influenza Vaccination Rates: A Randomized Clinical Trial | Infectious Diseases | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Sign In
Figure.  CONSORT Flow Diagram
CONSORT Flow Diagram

Patients were eligible for study participation if they were at least 6 months of age at the start of intervention (ie, eligible for influenza vaccination), a primary care patient within the UCLA Health System according to the health system’s criteria, and an active patient portal user (≥1 log-in during the past 12 months), other than initial account log-in. Patients were ineligible if they were not a primary care patient within the UCLA Health System or not an active patient portal user.

Table 1.  Demographic Characteristics of the Study Samplea
Demographic Characteristics of the Study Samplea
Table 2.  Unadjusted and Adjusted Risk Ratios for Influenza Vaccination for Subgroups vs a Reference Subgroup and for Each Study Group vs the No Reminder Control Group
Unadjusted and Adjusted Risk Ratios for Influenza Vaccination for Subgroups vs a Reference Subgroup and for Each Study Group vs the No Reminder Control Group
Table 3.  Influenza Vaccination Rates by Study Group Overall (Primary Analysis) and by Patient Subgroup at the End of the Study Period Excluding Self-reported Vaccinations
Influenza Vaccination Rates by Study Group Overall (Primary Analysis) and by Patient Subgroup at the End of the Study Period Excluding Self-reported Vaccinations
Table 4.  Influenza Vaccination Rates by Study Group Overall and by Patient Subgroup at the End of the Study Including Self-reported Vaccinations
Influenza Vaccination Rates by Study Group Overall and by Patient Subgroup at the End of the Study Including Self-reported Vaccinations
1.
Grohskopf  LA, Sokolow  LZ, Broder  KR, Walter  EB, Fry  AM, Jernigan  DB.  Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices–United States, 2018-19 influenza season.   MMWR Recomm Rep. 2018;67(3):1-20. doi:10.15585/mmwr.rr6703a1 PubMedGoogle ScholarCrossref
2.
US Department of Health and Human Services. Healthy People 2020. Accessed January 21, 2020. https://www.healthypeople.gov/2020/topicsobjectives2020/objectiveslist.aspx?topicId=23
3.
Centers for Disease Control and Prevention. Flu vaccination coverage, United States, 2018–19 influenza season. Accessed January 21, 2020. https://www.cdc.gov/flu/fluvaxview/coverage-1819estimates.htm
4.
Jacobson Vann  JC, Jacobson  RM, Coyne-Beasley  T, Asafu-Adjei  JK, Szilagyi  PG.  Patient reminder and recall interventions to improve immunization rates.   Cochrane Database Syst Rev. 2018;1:CD003941. doi:10.1002/14651858.CD003941.pub3 PubMedGoogle Scholar
5.
Szilagyi  PG, Bordley  C, Vann  JC,  et al.  Effect of patient reminder/recall interventions on immunization rates: a review.   JAMA. 2000;284(14):1820-1827. doi:10.1001/jama.284.14.1820 PubMedGoogle ScholarCrossref
6.
Briss  PA, Rodewald  LE, Hinman  AR,  et al; Task Force on Community Preventive Services.  Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults.   Am J Prev Med. 2000;18(1)(suppl):97-140. doi:10.1016/S0749-3797(99)00118-X PubMedGoogle ScholarCrossref
7.
Dombkowski  KJ, Harrington  LB, Dong  S, Clark  SJ.  Seasonal influenza vaccination reminders for children with high-risk conditions: a registry-based randomized trial.   Am J Prev Med. 2012;42(1):71-75. doi:10.1016/j.amepre.2011.09.028 PubMedGoogle ScholarCrossref
8.
Saville  AW, Szilagyi  P, Helmkamp  L,  et al.  Potential strategies to achieve universal influenza vaccination for children: provider attitudes in two states.   Acad Pediatr. 2018;18(8):873-881. doi:10.1016/j.acap.2018.07.003 PubMedGoogle ScholarCrossref
9.
Tierney  CD, Yusuf  H, McMahon  SR,  et al.  Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics.   Pediatrics. 2003;112(5):1076-1082. doi:10.1542/peds.112.5.1076 PubMedGoogle ScholarCrossref
10.
Cataldi  JR, O’Leary  ST, Lindley  MC,  et al.  Survey of adult influenza vaccination practices and perspectives among US primary care providers (2016-2017 influenza season).   J Gen Intern Med. 2019;34(10):2167-2175. doi:10.1007/s11606-019-05164-7 PubMedGoogle ScholarCrossref
11.
Saville  AW, Albright  K, Nowels  C,  et al.  Getting under the hood: exploring issues that affect provider-based recall using an immunization information system.   Acad Pediatr. 2011;11(1):44-49. doi:10.1016/j.acap.2010.12.009 PubMedGoogle ScholarCrossref
12.
Kempe  A, Wortley  P, O’Leary  S,  et al.  Pediatricians’ attitudes about collaborations with other community vaccinators in the delivery of seasonal influenza vaccine.   Acad Pediatr. 2012;12(1):26-35. doi:10.1016/j.acap.2011.07.004 PubMedGoogle ScholarCrossref
13.
Fisher  MP, Gurfinkel  D, Szilagyi  PG,  et al.  Supporting and sustaining centralized reminder/recall for immunizations: qualitative insights from stakeholders.   Vaccine. 2019;37(44):6601-6608. doi:10.1016/j.vaccine.2019.09.055 PubMedGoogle ScholarCrossref
14.
Mullooly  JP.  Increasing influenza vaccination among high-risk elderly: a randomized controlled trial of a mail cue in an HMO setting.   Am J Public Health. 1987;77(5):626-627. doi:10.2105/AJPH.77.5.626 PubMedGoogle ScholarCrossref
15.
Baker  AM, McCarthy  B, Gurley  VF, Yood  MU.  Influenza immunization in a managed care organization.   J Gen Intern Med. 1998;13(7):469-475. doi:10.1046/j.1525-1497.1998.00136.x PubMedGoogle ScholarCrossref
16.
Kempe  A, Daley  MF, Barrow  J,  et al.  Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall.   Pediatrics. 2005;115(1):146-154. doi:10.1542/peds.2004-1804 PubMedGoogle ScholarCrossref
17.
Stockwell  MS, Westhoff  C, Kharbanda  EO,  et al.  Influenza vaccine text message reminders for urban, low-income pregnant women: a randomized controlled trial.   Am J Public Health. 2014;104(suppl 1):e7-e12. doi:10.2105/AJPH.2013.301620 PubMedGoogle ScholarCrossref
18.
Nexøe  J, Kragstrup  J, Rønne  T.  Impact of postal invitations and user fee on influenza vaccination rates among the elderly: a randomized controlled trial in general practice.   Scand J Prim Health Care. 1997;15(2):109-112. doi:10.3109/02813439709018497 PubMedGoogle ScholarCrossref
19.
Stockwell  MS, Kharbanda  EO, Martinez  RA, Vargas  CY, Vawdrey  DK, Camargo  S.  Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial.   JAMA. 2012;307(16):1702-1708. doi:10.1001/jama.2012.502 PubMedGoogle ScholarCrossref
20.
Stockwell  MS, Hofstetter  AM, DuRivage  N,  et al.  Text message reminders for second dose of influenza vaccine: a randomized controlled trial.   Pediatrics. 2015;135(1):e83-e91. doi:10.1542/peds.2014-2475 PubMedGoogle ScholarCrossref
21.
Kempe  A, Saville  AW, Albertin  C,  et al.  Centralized reminder/recall to increase influenza vaccination rates: a two-state pragmatic randomized trial.   Acad Pediatr. 2020;20(3):374-383. doi:10.1016/j.acap.2019.10.015 PubMedGoogle ScholarCrossref
22.
Anthony  DL, Campos-Castillo  C, Lim  PS.  Who isn’t using patient portals and why? evidence and implications from a national sample of US adults.   Health Aff (Millwood). 2018;37(12):1948-1954. doi:10.1377/hlthaff.2018.05117 PubMedGoogle ScholarCrossref
23.
Cutrona  SL, Golden  JG, Goff  SL,  et al.  Improving rates of outpatient influenza vaccination through EHR portal messages and interactive automated calls: a randomized controlled trial.   J Gen Intern Med. 2018;33(5):659-667. doi:10.1007/s11606-017-4266-9 PubMedGoogle ScholarCrossref
24.
Jacobson Vann  JC, Szilagyi  P.  Patient reminder and patient recall systems for improving immunization rates.   Cochrane Database Syst Rev. 2018;1(1):CD003941.PubMedGoogle Scholar
25.
Szilagyi  PG, Valderrama  R, Vangala  S,  et al.  Pediatric patient portal use in one health system.   J Am Med Inform Assoc. 2020;27(3):444-448. doi:10.1093/jamia/ocz203 PubMedGoogle ScholarCrossref
26.
Becker  MH, Maiman  LA.  Sociobehavioral determinants of compliance with health and medical care recommendations.   Med Care. 1975;13(1):10-24. doi:10.1097/00005650-197501000-00002 PubMedGoogle ScholarCrossref
27.
Kim  H, Xie  B.  Health literacy in the eHealth era: a systematic review of the literature.   Patient Educ Couns. 2017;100(6):1073-1082. doi:10.1016/j.pec.2017.01.015 PubMedGoogle ScholarCrossref
28.
Jenssen  BP, Buttenheim  AM, Fiks  AG.  Using behavioral economics to encourage parent behavior change: opportunities to improve clinical effectiveness.   Acad Pediatr. 2019;19(1):4-10. doi:10.1016/j.acap.2018.08.010 PubMedGoogle ScholarCrossref
29.
Zou  G.  A modified Poisson regression approach to prospective studies with binary data.   Am J Epidemiol. 2004;159(7):702-706. doi:10.1093/aje/kwh090 PubMedGoogle ScholarCrossref
30.
Zhu  M.  Analyzing Multilevel Models with the GLIMMIX Procedure: Proceedings of the SAS Global Forum 2014 Conference. SAS Institute Inc; 2014. Accessed January 17, 2020. https://support.sas.com/resources/papers/proceedings14/SAS026-2014.pdf
31.
Dempsey  AF, Zimet  GD.  Interventions to improve adolescent vaccination: what may work and what still needs to be tested.   Vaccine. 2015;33(suppl 4):D106-D113. doi:10.1016/j.vaccine.2015.09.032 PubMedGoogle ScholarCrossref
32.
Hofstetter  AM, Vargas  CY, Camargo  S,  et al.  Impacting delayed pediatric influenza vaccination: a randomized controlled trial of text message reminders.   Am J Prev Med. 2015;48(4):392-401. doi:10.1016/j.amepre.2014.10.023 PubMedGoogle ScholarCrossref
33.
Saville  AW, Beaty  B, Dickinson  LM, Lockhart  S, Kempe  A.  Novel immunization reminder/recall approaches: rural and urban differences in parent perceptions.   Acad Pediatr. 2014;14(3):249-255. doi:10.1016/j.acap.2014.02.003 PubMedGoogle ScholarCrossref
34.
Edwards  KM, Hackell  JM; Committee on Infectious Diseases, Committee on Practice and Ambulatory Medicine.  Countering vaccine hesitancy.   Pediatrics. 2016;138(3):e20162146. doi:10.1542/peds.2016-2146 PubMedGoogle Scholar
35.
Jarrett  C, Wilson  R, O’Leary  M, Eckersberger  E, Larson  HJ; SAGE Working Group on Vaccine Hesitancy.  Strategies for addressing vaccine hesitancy: a systematic review.   Vaccine. 2015;33(34):4180-4190. doi:10.1016/j.vaccine.2015.04.040 PubMedGoogle ScholarCrossref
36.
Cunningham  RM, Minard  CG, Guffey  D, Swaim  LS, Opel  DJ, Boom  JA.  Prevalence of vaccine hesitancy among expectant mothers in Houston, Texas.   Acad Pediatr. 2018;18(2):154-160. doi:10.1016/j.acap.2017.08.003 PubMedGoogle ScholarCrossref
37.
Shen  AK, Warnock  R, Selna  W, Chu  S, Kelman  JA.  Patient characteristics of Medicare beneficiaries who report not getting influenza and pneumococcal vaccinations, 2001-2013.   Hum Vaccin Immunother. 2019:1-7. doi:10.1080/21645515.2019.1688033 PubMedGoogle Scholar
38.
Schmid  P, Rauber  D, Betsch  C, Lidolt  G, Denker  ML.  Barriers of influenza vaccination intention and behavior: a systematic review of influenza vaccine hesitancy, 2005-2016.   PLoS One. 2017;12(1):e0170550. doi:10.1371/journal.pone.0170550 PubMedGoogle Scholar
39.
Scott  A, Liu  M, Yong  J.  Financial incentives to encourage value-based health care.   Med Care Res Rev. 2018;75(1):3-32. doi:10.1177/1077558716676594 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    May 18, 2020

    Effect of Patient Portal Reminders Sent by a Health Care System on Influenza Vaccination Rates: A Randomized Clinical Trial

    Author Affiliations
    • 1Department of Pediatrics, UCLA (University of California, Los Angeles) Mattel Children's Hospital, UCLA
    • 2Department of Medicine, David Geffen School of Medicine, UCLA
    • 3Department of Medicine, Veterans Affairs Greater Los Angeles Healthcare System, Los Angeles, California
    • 4Fielding School of Public Health, Department of Health Policy and Management, UCLA
    • 5Department of Medicine Statistics Core, David Geffen School of Medicine, UCLA
    • 6Department of Pediatrics, University of Rochester School of Medicine and Dentistry, Rochester, New York
    • 7Department of Pediatrics, Children’s Mercy, Kansas City, Missouri
    • 8Department of Information Services and Solutions, UCLA Health System, Los Angeles, California
    JAMA Intern Med. 2020;180(7):962-970. doi:10.1001/jamainternmed.2020.1602
    Key Points

    Question  Can reminders sent through a patient portal increase influenza vaccination rates across a health care system?

    Findings  In this randomized clinical trial of 164 205 patients served by 52 primary care practices across a health care system, 1, 2, or 3 influenza vaccine reminders sent through the electronic health record patient portal had a small, statistically significant effect on increasing influenza vaccination rates compared with no reminder.

    Meaning  Generic patient portal reminders may be slightly effective in increasing influenza vaccination rates, but more intensive or more targeted patient motivational strategies appear to be needed.

    Abstract

    Importance  Influenza vaccination rates across the US are low. Because few practices send patient reminders for influenza vaccination, a scalable patient reminder system is needed.

    Objective  To evaluate the effect of patient reminders sent via a health care system’s electronic health record patient portal on influenza vaccination rates.

    Design, Setting, and Participants  This pragmatic, 4-arm randomized clinical trial was performed from October 1, 2018, to March 31, 2019, across the UCLA (University of California, Los Angeles) health care system. A total of 164 205 patients in 52 primary care practices who had used the patient portal within 12 months were included.

    Interventions  Patients due for an influenza vaccine were sent a letter via the patient portal of the health care system reminding them about the importance of influenza vaccination, safety of the vaccine, and morbidity associated with influenza. Patients were randomized within primary care practices to 1 of 4 study groups (no reminder [n = 41 070] vs 1 reminder [n = 41 055], 2 reminders [n = 41 046], or 3 reminders [n = 41 034]).

    Main Outcomes and Measures  The primary outcome was receipt of 1 or more influenza vaccines as documented in the electronic health record, which was supplemented with influenza vaccination data from external sources (eg, pharmacies). Secondary outcomes were influenza vaccination rates among subgroups and influenza vaccinations self-reported by patients in reply to the portal-based query as having been received elsewhere.

    Results  A total of 164 205 patients (mean [SD] age, 46.2 [19.6] years; 95 779 [58.3%] female) were randomly allocated to 1 of the 4 study arms. In the primary analysis across all ages and not including patient self-reported vaccinations in reply to portal reminders, influenza vaccination rates were 37.5% for those receiving no reminders, 38.0% for those receiving 1 reminder (P = .008 vs no reminder), 38.2% for those receiving 2 reminders (P = .03 vs no reminder), and 38.2% for those receiving 3 reminders (P = .02 vs no reminder). In the secondary analysis not including patient self-reported vaccinations, among adults aged 18 to 64 years (vaccination rates: 32.0% in the control group, 32.8% in the 1-reminder group, 32.8% in the 2-reminder group, and 32.8% in the 3-reminder group; P = .001), male patients (vaccination rates: 37.3% vs 38.3%, 38.6%, and 38.8%; P = .001), non-Hispanic patients (vaccination rates: 37.6% vs 38.2%, 38.3%, and 38.2%; P = .004), and those who were not vaccinated in the prior 2 years (vaccination rates: 15.3% vs 15.9%, 16.3%, and 16.1%; P < .001), vaccination rates were higher in the portal reminder groups than in the control group; the findings in these 3 subgroups mirrored the findings in the entire population. When self-reported vaccinations received elsewhere were included, influenza vaccination rates were 1.4 to 2.9 percentage points higher in the portal reminder groups, with a dose-response effect (0 reminders: 15 537 [37.8%]; 1 reminder: 16 097 [39.2%]; 2 reminders: 16 426 [40.0%]; and 3 reminders: 16 714 [40.7%]; P < .001).

    Conclusions and Relevance  Generic patient portal reminders were effective in minimally increasing influenza vaccination rates, but more intensive or more targeted patient motivational strategies appear to be needed.

    Trial Registration  ClinicalTrials.gov Identifier: NCT03666026

    Introduction

    Influenza causes many respiratory illnesses, hospitalizations, ambulatory medical visits, and deaths in the US annually.1 Although the Advisory Committee on Immunization Practices recommends annual influenza vaccination for all US individuals 6 months or older,1 vaccination rates lag behind US Healthy People 2020 vaccination coverage goals of more than 80% for individuals younger than 65 years and more than 90% for those 65 years or older.2 In 2018 to 2019, vaccination coverage rates by age were 62.6% among those aged 6 months to 17 years, 34.9% among those aged 18 to 49 years, 47.3% among those aged 50 to 64 years, and 68.1% among those 65 years or older.3

    Although patient reminders are associated with increased influenza vaccination coverage4,5 and the Task Force on Community Preventive Services recommends reminders,6 few primary care practices send reminders for any vaccine7-10 in part because of limited resources.10-12 Experts have recommended centralized reminders from state immunization information systems or health care systems as a scalable strategy.13 Most studies using centralized reminders for influenza vaccination have focused on children or adults with chronic diseases,7,14 subsets of the population,15,16 pregnant women,17 or small numbers of practices.18-20 A recent study21 that tested reminders for childhood influenza vaccination with state immunization information systems found little effect.

    A scalable method to send centralized reminders involves patient portals, which are secure, web-based applications that provide 2-way communication between patients and health care systems or health care professionals. Most electronic health records (EHRs) have patient portals, and a recent survey22 found that 37% of adults with a health care visit in the past year had used portals in the past year. A randomized clinical trial in a health care system23 that evaluated the patient portal plus interactive voice recording reminders observed 1% improvement in influenza vaccination rates, but that trial23 involved a multispecialty group practice, excluded patients vaccinated before November 10, and had baseline vaccination rates below 15%.23 With Cochrane reviews4,24 reporting that reminder and recall work for influenza vaccination, it is important to study the effect of portal reminders on influenza vaccination across a health care system.

    We performed a 4-arm randomized clinical trial that compared the effect of 1, 2, or 3 vs no portal reminders on influenza vaccination rates across a large health care system. We evaluated the effect of portal reminders on receipt of 1 or more influenza vaccinations in the entire population and in predesignated subgroups. We hypothesized that portal reminders would increase influenza vaccination rates at the population level and in subgroups (age, sex, race/ethnicity, insurance, and influenza vaccination in prior years).

    Methods
    Study Design and Setting

    The intervention was conducted from October 1, 2018, to March 31, 2019. The design was a 4-arm, pragmatic, intention-to-treat randomized clinical trial of 164 205 patients randomized within primary care practices to 1 of 4 study arms (no reminder [n = 41 070] vs 1 reminder [n = 41 055], 2 reminders [n = 41 046], or 3 reminders [n = 41 034]). The setting was all 52 UCLA (University of California, Los Angeles) Health System primary care practices: 38 internal medicine, 5 medicine and pediatrics, 3 family medicine, and 6 pediatrics. The UCLA Institutional Review Board approved this study and provided a waiver of patient consent because (1) the trial was extremely low risk because it was designed to remind patients about a universally recommended vaccine and (2) it was impractical to obtain patient consent from tens of thousands of patients. All data were deidentified. The trial protocol is given in Supplement 1. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.

    Study Participants and Inclusion and Exclusion Criteria

    Patients within UCLA Health System practices can register with the myUCLAHealth (Epic) portal. The Figure shows the flow of the patients through the study. First, we included any UCLA Health System primary care patient 6 months or older at intervention launch (ie, influenza vaccination eligible), defined by 2 or more primary care physician visits (by evaluation and management office codes) within 3 years or 1 or more primary care physician visits with a preventive service code within 1 year. We included managed care patients assigned to the UCLA Health System (even with no health care visits in the prior year) because we were certain about their primary care practice affiliation. Second, we identified the primary care practice most recently visited. Third, we grouped patients into family units using a series of algorithms that matched each patient’s telephone number, address, insurance member number, or patient guarantor identification. Fourth, we identified active portal users as patients or portal proxies (for minors, elderly persons, or disabled persons) who signed up for and logged into the portal 1 or more times during the prior 12 months not including the initial portal log-in; this sample comprised 223 191 (65.0%) of UCLA Health System’s primary care patients.25 Statisticians (S.V., C.-H.T.) randomly selected 1 active portal-using index patient per family for inclusion. This process generated the denominator of potential participants; all other study personnel and health care professionals were blinded to study allocation.

    Exclusion criteria were individuals who were not active portal users and family members of index patients, who were excluded because of adequate sample size with index patients and for consistency with Cochrane criteria for meta-analyses using Cochrane RevMan software.4,24

    Study statisticians randomized index patients to 1 of the 4 study arms, stratifying by their affiliation with a primary care practice. All index patients’ family members (whether proxies or not) received the same portal reminder to avoid confusion, but only index patients’ data were analyzed.

    Intervention

    For all patients randomized to 1, 2, or 3 portal messages, study statisticians (S.V., C.-H.T.) sent files to the health care system’s patient portal team, defining the portal messages to be sent. A secure notification was then sent by email or text (per patients’ standard portal preference) informing patients that “a message from your doctor” had been posted on the portal. Influenza vaccination was not mentioned in the title of the message. Patients had to log in to the portal to read the letter.

    The portal reminder letter content was grounded in the Health Belief Model and principles of health literacy.26,27 We based the content on (1) prior patient interviews used to craft patient reminders for influenza vaccination,21 (2) positive framing principles of behavioral economics,28 and (3) input from external national vaccination experts. The letters (eFigure in Supplement 2) included the following: (1) information that influenza season was coming, the disease can cause substantial morbidity, and the vaccine is the best way to protect against influenza; (2) recommendation to receive an influenza vaccine by calling for an office appointment or going to a pharmacy or other setting; (3) a website link to input influenza vaccinations received elsewhere into the UCLA Health System record; and (4) another website link to a UCLA webpage containing information about influenza vaccine and video testimonials about influenza vaccination. Letters were in English, included the name of the patient’s primary care physician, and had a below seventh grade reading level per Flesch-Kincaid analysis.

    Portal reminders were sent according to randomization arm at the beginning of October (all intervention groups), November (those receiving 2-3 reminders), and December (those receiving 3 reminders). The letter content was identical per round. Before each round, patients who had received an influenza vaccination according to the EHR were removed from the eligible list. For children younger than 8 years and eligible for 2 vaccinations, portal reminders were sent only for the initial vaccination.

    Patient and Practice Characteristics

    Patient and practice characteristics were obtained from the EHR. Patient factors (Table 1) included age, sex, insurance at the last primary care visit, race/ethnicity, and receipt of 1 or more influenza vaccinations during the prior 2 years.

    Influenza Vaccination Data

    Data from the EHR included influenza vaccinations (date and location) administered at any UCLA Health System site. In addition, UCLA health care practitioners could enter vaccination codes manually for vaccines received outside the UCLA Health System. The UCLA Health System merges data into the EHR from external sources, including SureScripts (pharmacy benefits manager), California Immunization Registry (mostly children’s immunizations), claims from select health care plans, and Care Everywhere (Epic’s information exchange application). Patients or proxies can also enter data when they log in to the portal. Data from these external sources (other than from claims) are normally incorporated into the medical record at office visits via a reconciliation process whereby health care professionals add or discard individual data elements. Our study did not modify these standards. However, we integrated data from all these external sources regardless of whether physicians reconciled the immunization record.

    Finally, the portal reminder letters included a link for patients to update influenza vaccinations received elsewhere; these data were also integrated into the EHR. We tracked this method for updating influenza vaccinations received elsewhere and used this flag to distinguish primary vs secondary outcome measures.

    Outcome and Process Measures
    Primary Outcome

    The primary study outcome was EHR-based documentation that included vaccinations obtained elsewhere if inputted into the EHR using the health care system’s standard electronic merging of 1 or more influenza vaccinations between October 1, 2018, and March 31, 2019. The primary analysis excluded vaccinations reported only by patients in response to the portal reminders because the control group did not have this opportunity for self-report, thus eliminating differential outcome ascertainment. This analysis created a conservative bias because portal reminders may have encouraged some patients to receive influenza vaccinations at sites (eg, employers, malls, and some pharmacies) that did not send data to the EHR.

    Secondary Outcomes

    We assessed 5 secondary outcomes related to receipt of 1 or more influenza vaccinations in the relevant season: (1) vaccinations in predetermined subgroups (age, sex, race/ethnicity, insurance, and influenza vaccination in prior 2 years); (2) vaccinations that were self-reported by patients or proxies via the portal in response to portal reminders; (3) influenza vaccinations before December 15, 2018, to determine whether portal reminders increased early vaccination; (4) vaccinations among those who opened 1 or more portal reminders vs those who did not open the reminders; and (5) vaccinations in patients in the upper half vs bottom half of overall portal use.

    Process Metrics

    We assessed whether patients (1) opened the portal reminder letter, (2) updated influenza vaccinations received outside the UCLA Health System in response to the portal reminders, and (3) clicked on the informational website link embedded in the portal letter.

    Sample Size and Power Calculation

    A sample size of approximately 41 000 patients per study arm would provide greater than 90% power to detect a 2–percentage point improvement in the rate of vaccination, which we deemed to be the smallest improvement with clinical effect. This number assumes a χ2 test, a control group rate of 50% (most conservative), and a significance level of P < .017 (3-fold Bonferroni correction for the pairwise comparisons of study arms with 1, 2, and 3 reminders vs control). Because patients were randomized within practices, clustering by practice was ignored in this power calculation for conservatism.

    Statistical Analysis

    We report descriptive statistics for all patient characteristics in terms of frequency distributions for the full patient sample and stratifying by study arm. The primary analysis compared vaccination rates between study arms using mixed-effects Poisson regression with robust SEs.29,30 Models included a fixed effect for study arm (0 vs 1 vs 2 vs 3 reminders), random practice effects, and adjustment for patient characteristics (age, sex, insurance, race/ethnicity, and vaccination history). Secondary subgroup analyses were performed by fitting separate models for each subgroup. In these analyses, the comparison was between the combination of the 3 intervention arms and the control arm. We further performed exploratory analyses to evaluate the effect of opening any reminder letters on vaccination status. In one approach, we replicated the primary analysis strategy but grouped patients by whether they opened any letters rather than by randomization assignment. In a second approach, we used 2-stage least-squares with cluster robust SEs at the practice level to estimate the local mean treatment effect of opening any letters using assignment to intervention as an instrumental variable.

    For the primary analysis comparing each of the individual reminder arms with the control arm, a significance level of P < .017 was used. In all other analyses, we considered P < .05 as statistically significant. All statistical analyses were performed using SAS statistical software, version 9.4 (SAS Institute Inc).

    Results
    Practice and Patient Characteristics

    A total of 164 205 patients (mean [SD] age, 46.2 [19.6] years; 95 779 [58.3%] female) were randomly allocated to 1 of the 4 study arms. Table 1 provides demographic data across study arms. A total of 161 268 patients (or proxies) (98.2%) spoke English as their primary language.

    Primary Outcome: Influenza Vaccination

    Across all patients, the vaccination rates were 37.5% among controls (no reminders), 38.0% in the 1-reminder group (P = .008 vs controls), 38.2% in the 2-reminder group (P = .03 vs controls), and 38.2% in the 3-reminder group (P = .02 vs controls). Table 2 gives the unadjusted and adjusted risk ratios (RRs) for influenza vaccination for each study arm in the entire population. Adjusted RRs were 1.02 (95% CI, 1.00-1.03) for the 1-reminder group, 1.02 (95% CI, 1.00-1.04) for the 2-reminder group, and 1.02 (95% CI, 1.00-1.04) for the 3-reminder group, showing a small effect and no dose response by number of reminders sent.

    Secondary Analyses and Process Metrics
    Subgroups

    For subgroups (Table 3), including age, sex, insurance, race/ethnicity, and prior influenza vaccination, the intervention had only a small (or no) effect on influenza vaccination rates. Some intervention subgroups vs the corresponding control subgroup had vaccination rates that were several percentage points higher, but there was no clear pattern or dose response by the number of reminders sent. As indicated in Table 3, portal reminders had a statistically significant effect among patients who were male (vaccination rates: 37.3% in the control group, 38.3% in the 1-reminder group, 38.6% in the 2-reminder group, and 38.8% in the 3-reminder group; P = .001), covered by private insurance (vaccination rates: 35.3% in the control group, 35.5% in the 1-reminder group, 36.1% in the 2-reminder group, and 36.0% in the 3-reminder group; P = .02), white (vaccination rates: 39.0% in the control group, 39.7% in the 1-reminder group, 39.9% in the 2-reminder group, and 39.6% in the 3-reminder group; P = .004), non-Hispanic (vaccination rates: 37.6% in the control group, 38.2% in the 1-reminder group, 38.3% in the 2-reminder group, and 38.2% in the 3-reminder group; P = .004), and not vaccinated in the prior 2 years (vaccination rates: 15.3% in the control group, 15.9% in the 1-reminder group, 16.3% in the 2-reminder group, and 16.1% in the 3-reminder group; P < .001).

    Vaccination Rates Including Patient Self-reported Vaccinations in Response to Portal Reminders

    Overall, 43 387 vaccinations (69.6%) were administered within UCLA clinics, whereas 18 935 (30.4%) were from external data sources, such as SureScripts. When we included self-reported data about influenza vaccinations received in the community in response to the patient portal reminders beyond external vaccinations that were already merged into the EHR (Table 4), 37.8% of controls (no reminders) received an influenza vaccination vs 39.2% in the 1-reminder group, 40.0% in the 2-reminder group), and 40.7% in the 3-reminder arms (comparison of combined intervention vs control; P < .001). There was a dose-response increase in vaccination rates depending on the number of reminders sent. In analyses of subgroups, statistically significant improvements in vaccination rates were noted among all subgroups except for children, with a small dose response.

    Influenza Vaccines Received Through Mid-December

    For the secondary analysis evaluating early vaccinations, 34.2% of controls (no reminders) received an influenza vaccination by mid-December vs 34.8% in the 1-reminder group (P = .001 vs controls), 35.1% in the 2-reminder group (P = .002 vs controls), and 35.2% on the 3-reminder group (P = .002 vs controls). Adjusted RRs were 1.02 (95% CI, 1.01-1.04) for the 1-reminder group, 1.03 (95% CI, 1.01-1.05) for the 2-reminder group, and 1.03 (95% CI, 1.01-1.05) for the 3-reminder group, showing a small effect and no dose response by number of reminders sent, consistent with the full-season results.

    Patients Who Opened 1 Letter or More

    After adjustment for all variables in Table 1 (including prior influenza vaccination), across all intervention groups, patients who opened 1 letter or more were more likely to be vaccinated than were those who did not open any letters (39.1% vs 25.4%; adjusted RR, 1.41; 95% CI, 1.38-1.44). We also used an instrumental variables approach to adjust the intention-to-treat effect of assignment to intervention for the proportion of intervention patients who opened any letters. Although the intention-to-treat effect of 1, 2, and 3 reminders was estimated to be a 0.66–percentage point increase in rates, the local mean treatment effect of opening a letter, using assignment to intervention as an instrument, was estimated to be 1.18 percentage points and was statistically significant (risk difference, 1.18; 95% CI, 0.29-2.07).

    Intervention Effect by Level of Portal Use

    We compared the intervention effect among those who interacted frequently with the portal (≥15 median log-ins across the population during 12 months) vs less frequently (<15 log-ins). The difference was not statistically significant.

    Process Measures

    A total of 52.9% of patients in the 1-reminder group, 55.9% in the 2-reminder group, and 58.8% in the 3-reminder group read the reminder letters at least once; for the 2-reminder group, 33.2% read both letters, and for the 3-reminder group, 12.8% read 2 letters and 24.5% read 3 letters. Of all influenza vaccinations recorded in the EHR, data for 0.8% of controls came exclusively from self-reporting vaccinations to the portal vs 3.6% of the 1-reminder group, 5.7% of the 2-reminder group, and 7.5% of the 3-reminder group, suggesting that the portal reminders prompted many patients to report vaccinations received elsewhere. Only 416 patients (0.3%) visited the embedded website providing information about influenza vaccination; even fewer viewed the video testimonials. There were no major patient or practitioner concerns from the study.

    Discussion

    This study has several important findings. First, if we exclude patient self-report of vaccinations received elsewhere (triggered by the portal reminders themselves), the portal reminders improved influenza vaccination rates by only 1 percentage point (with an adjusted RR of 1.02), and 2 or 3 reminders did not produce a dose-response effect. Second, portal reminders appeared to have a small, statistically significant effect on increasing influenza vaccination rates among adults aged 18 to 64 years, male patients, non-Hispanic patients, and those not vaccinated in the prior 2 years. Third, when we included self-reported vaccinations indicating a vaccine received elsewhere, the intervention had a larger effect: 1.4% for the 1-reminder group, 2.2% for the 2-reminder group, and 2.9% for the 3-reminder group; however, these findings may be biased by differential outcome ascertainment. Fourth, more than half of patients opened the reminder letters and many after each mailing. Patients who opened at least 1 letter were substantially more likely to be vaccinated than were those who did not. However, because opening a portal letter is a health-seeking behavior, we would expect higher vaccination rates among those who opened a letter even if our intervention did not have any effect. Overall, the primary intention-to-treat analysis revealed a small effect on increasing influenza vaccination rates across this health care system.

    Our findings mirror results from a previous portal reminder study23 that also found only a 1-percentage point increase over a baseline coverage level of less than 15%. We speculate on possible reasons for our small study effect. First, perhaps after receiving a portal reminder, many adult influenza vaccinations were given outside the health care system but not recorded in the EHR despite multiple mechanisms capturing external vaccinations. This possibility would support our secondary analysis that included patient self-reported vaccinations and that found a larger effect. This possibility is supported by our finding that vaccination rates in this population were below national or state coverage estimates, suggesting that data may still be missing about many influenza vaccinations received by this population that has established primary care. Incomplete data on vaccinations received elsewhere may have obscured additional benefit of the intervention.

    A second possibility is that the effectiveness of reminders has started to wane amid many patient-targeted health communications. The 2018 Cochrane Review of reminder and recall4 did not analyze whether the effect of reminders has decreased over time. A recent study21 noted that influenza vaccination reminders sent on behalf of state immunization registries had no effect on child vaccination rates. We had anticipated that portal reminders would have a stronger motivating influence because they come from the physician and from within the patients’ health care system. Although more than 50% of patients opened the reminder letters, the intervention’s effect was small but higher among those who opened the letter. Because only half of patients opened a letter, strategies that enhance reading of portal reminders might be helpful.

    Studies20,31-33 of reminders have suggested that the message content might influence their effect. A postintervention focus group with the health care system’s portal patient representatives noted that patients preferred straightforward, relatively brief, user-friendly messages and positive messages and message titles. Because few study participants (<1%) viewed the linked website with videos, reminder letters themselves should contain critical educational content. Studies are needed to assess whether varying portal message content might improve its effect.

    Another possible reason for the limited effectiveness of the portal reminders in the context of low baseline vaccination rates is vaccine hesitancy.34,35 The intervention was not designed to address hesitancy but rather to nudge patients who desired vaccination. Many studies35-37 have documented frequent influenza vaccination refusals. Vaccination hesitancy is not an all-or-nothing characteristic,35,38 and studies are needed to assess the degree to which reminders and education can affect vaccination uptake among hesitant individuals.

    An important benefit to this intervention was that it captured more complete influenza vaccination data for the health care system because of the additional vaccinations self-reported by patients in response to portal reminders. This approach particularly benefits health care systems facing value-based incentives that incorporate vaccination rates as a quality metric.13,39

    Strengths and Limitations

    This study has strengths and limitations. This was a large, pragmatic trial across an entire health care system that tested a new communication mechanism (patient portal) with randomization within primary care practices to adjust for many unmeasured confounders and ascertainment of vaccinations received elsewhere via linkages with the EHR and patient self-reporting. A limitation is potential lack of generalizability to other health care systems (particularly safety-net populations or those with limited English proficiency) and to portal nonusers. In addition, we could not track all vaccinations received outside the health care system.

    Conclusions

    In this trial, patient portal reminders about influenza vaccination had a small effect on vaccination rates across a health care system. Further research is needed to enhance the effect of portal reminders for influenza vaccination.

    Back to top
    Article Information

    Accepted for Publication: April 3, 2020.

    Corresponding Author: Peter G. Szilagyi, MD, MPH, Department of Pediatrics, UCLA Mattel Children's Hospital, University of California, Los Angeles, 10833 LeConte, Mail Code 175217, Los Angeles, CA 90095 (pszilagyi@mednet.ucla.edu).

    Published Online: May 18, 2020. doi:10.1001/jamainternmed.2020.1602

    Author Contributions: Dr Szilagyi had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Szilagyi, Albertin, Casillas, Valderrama, Vangala, Rand, Evans, Sloyan, Lerner.

    Acquisition, analysis, or interpretation of data: Casillas, Valderrama, Duru, Ong, Vangala, Tseng, Humiston, Sloyan, Lerner.

    Drafting of the manuscript: Szilagyi, Casillas, Valderrama, Ong, Vangala, Tseng, Evans, Lerner.

    Critical revision of the manuscript for important intellectual content: Albertin, Casillas, Valderrama, Duru, Ong, Vangala, Tseng, Rand, Humiston, Evans, Sloyan, Lerner.

    Statistical analysis: Szilagyi, Casillas, Vangala, Tseng, Lerner.

    Obtained funding: Szilagyi, Vangala, Rand.

    Administrative, technical, or material support: Szilagyi, Albertin, Valderrama, Vangala, Humiston, Evans, Sloyan, Lerner.

    Supervision: Szilagyi, Albertin, Casillas.

    Conflict of Interest Disclosures: Drs Szilagyi and Ony and Mss Albertin and Valderrama reported receiving grants from the National Institute of Allergy and Infectious Diseases, National Institutes of Health during the conduct of the study. Dr Rand reported receiving grants from the Centers for Disease Control and Prevention during the conduct of the study. Dr Humiston reported receiving grants from Pfizer Foundation outside the submitted work. No other disclosures were reported.

    Funding/Support: This work was supported by grants 1R01AI135029-01 (Dr Szilagyi, principal investigator) National Institute of Allergy and Infectious Diseases, National Institutes of Health and UL1TR00188 from the the National Center for Advancing Translational Sciences, National Institutes of Health.

    Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Data Sharing Statement: See Supplement 3.

    Additional Contributions: The following individuals work within the UCLA Health System and provided assistance with obtaining data from the electronic medical records and sending the patient portal messages: Alexander E Kurpinski, BS, MyChart/Connected Health; Ashley Elizabeth Turner, BA, clinical decision support analyst; Suzette Ma, BS, senior systems analyst/health informatics analyst, Overlake Medical Center; Hazel H. Oza, BS, clinical decision support lead; Michael A. Pfeffer, MD, assistant vice chancellor and chief information officer; and Deidre Lynn Keeves, BS, PMP, director, MyChart team. None of these people were compensated for their work. L. J. Tan, MS, PhD, chief strategy officer, Immunization Action Coalition, St Paul, Minnesota, and Gregory D. Zimet, PhD, FSAHM, Department of Pediatrics, Indiana University School of Medicine, Indianapolis, provided expert consultation in the development of the content of the portal messages. These individuals were each paid a small stipend. Sarah Friedman, MPH, Department of Pediatrics, UCLA Mattel Children’s Hospital, University of California, Los Angeles, assisted with manuscript preparation. She was not compensated for her work.

    References
    1.
    Grohskopf  LA, Sokolow  LZ, Broder  KR, Walter  EB, Fry  AM, Jernigan  DB.  Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices–United States, 2018-19 influenza season.   MMWR Recomm Rep. 2018;67(3):1-20. doi:10.15585/mmwr.rr6703a1 PubMedGoogle ScholarCrossref
    2.
    US Department of Health and Human Services. Healthy People 2020. Accessed January 21, 2020. https://www.healthypeople.gov/2020/topicsobjectives2020/objectiveslist.aspx?topicId=23
    3.
    Centers for Disease Control and Prevention. Flu vaccination coverage, United States, 2018–19 influenza season. Accessed January 21, 2020. https://www.cdc.gov/flu/fluvaxview/coverage-1819estimates.htm
    4.
    Jacobson Vann  JC, Jacobson  RM, Coyne-Beasley  T, Asafu-Adjei  JK, Szilagyi  PG.  Patient reminder and recall interventions to improve immunization rates.   Cochrane Database Syst Rev. 2018;1:CD003941. doi:10.1002/14651858.CD003941.pub3 PubMedGoogle Scholar
    5.
    Szilagyi  PG, Bordley  C, Vann  JC,  et al.  Effect of patient reminder/recall interventions on immunization rates: a review.   JAMA. 2000;284(14):1820-1827. doi:10.1001/jama.284.14.1820 PubMedGoogle ScholarCrossref
    6.
    Briss  PA, Rodewald  LE, Hinman  AR,  et al; Task Force on Community Preventive Services.  Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults.   Am J Prev Med. 2000;18(1)(suppl):97-140. doi:10.1016/S0749-3797(99)00118-X PubMedGoogle ScholarCrossref
    7.
    Dombkowski  KJ, Harrington  LB, Dong  S, Clark  SJ.  Seasonal influenza vaccination reminders for children with high-risk conditions: a registry-based randomized trial.   Am J Prev Med. 2012;42(1):71-75. doi:10.1016/j.amepre.2011.09.028 PubMedGoogle ScholarCrossref
    8.
    Saville  AW, Szilagyi  P, Helmkamp  L,  et al.  Potential strategies to achieve universal influenza vaccination for children: provider attitudes in two states.   Acad Pediatr. 2018;18(8):873-881. doi:10.1016/j.acap.2018.07.003 PubMedGoogle ScholarCrossref
    9.
    Tierney  CD, Yusuf  H, McMahon  SR,  et al.  Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics.   Pediatrics. 2003;112(5):1076-1082. doi:10.1542/peds.112.5.1076 PubMedGoogle ScholarCrossref
    10.
    Cataldi  JR, O’Leary  ST, Lindley  MC,  et al.  Survey of adult influenza vaccination practices and perspectives among US primary care providers (2016-2017 influenza season).   J Gen Intern Med. 2019;34(10):2167-2175. doi:10.1007/s11606-019-05164-7 PubMedGoogle ScholarCrossref
    11.
    Saville  AW, Albright  K, Nowels  C,  et al.  Getting under the hood: exploring issues that affect provider-based recall using an immunization information system.   Acad Pediatr. 2011;11(1):44-49. doi:10.1016/j.acap.2010.12.009 PubMedGoogle ScholarCrossref
    12.
    Kempe  A, Wortley  P, O’Leary  S,  et al.  Pediatricians’ attitudes about collaborations with other community vaccinators in the delivery of seasonal influenza vaccine.   Acad Pediatr. 2012;12(1):26-35. doi:10.1016/j.acap.2011.07.004 PubMedGoogle ScholarCrossref
    13.
    Fisher  MP, Gurfinkel  D, Szilagyi  PG,  et al.  Supporting and sustaining centralized reminder/recall for immunizations: qualitative insights from stakeholders.   Vaccine. 2019;37(44):6601-6608. doi:10.1016/j.vaccine.2019.09.055 PubMedGoogle ScholarCrossref
    14.
    Mullooly  JP.  Increasing influenza vaccination among high-risk elderly: a randomized controlled trial of a mail cue in an HMO setting.   Am J Public Health. 1987;77(5):626-627. doi:10.2105/AJPH.77.5.626 PubMedGoogle ScholarCrossref
    15.
    Baker  AM, McCarthy  B, Gurley  VF, Yood  MU.  Influenza immunization in a managed care organization.   J Gen Intern Med. 1998;13(7):469-475. doi:10.1046/j.1525-1497.1998.00136.x PubMedGoogle ScholarCrossref
    16.
    Kempe  A, Daley  MF, Barrow  J,  et al.  Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall.   Pediatrics. 2005;115(1):146-154. doi:10.1542/peds.2004-1804 PubMedGoogle ScholarCrossref
    17.
    Stockwell  MS, Westhoff  C, Kharbanda  EO,  et al.  Influenza vaccine text message reminders for urban, low-income pregnant women: a randomized controlled trial.   Am J Public Health. 2014;104(suppl 1):e7-e12. doi:10.2105/AJPH.2013.301620 PubMedGoogle ScholarCrossref
    18.
    Nexøe  J, Kragstrup  J, Rønne  T.  Impact of postal invitations and user fee on influenza vaccination rates among the elderly: a randomized controlled trial in general practice.   Scand J Prim Health Care. 1997;15(2):109-112. doi:10.3109/02813439709018497 PubMedGoogle ScholarCrossref
    19.
    Stockwell  MS, Kharbanda  EO, Martinez  RA, Vargas  CY, Vawdrey  DK, Camargo  S.  Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial.   JAMA. 2012;307(16):1702-1708. doi:10.1001/jama.2012.502 PubMedGoogle ScholarCrossref
    20.
    Stockwell  MS, Hofstetter  AM, DuRivage  N,  et al.  Text message reminders for second dose of influenza vaccine: a randomized controlled trial.   Pediatrics. 2015;135(1):e83-e91. doi:10.1542/peds.2014-2475 PubMedGoogle ScholarCrossref
    21.
    Kempe  A, Saville  AW, Albertin  C,  et al.  Centralized reminder/recall to increase influenza vaccination rates: a two-state pragmatic randomized trial.   Acad Pediatr. 2020;20(3):374-383. doi:10.1016/j.acap.2019.10.015 PubMedGoogle ScholarCrossref
    22.
    Anthony  DL, Campos-Castillo  C, Lim  PS.  Who isn’t using patient portals and why? evidence and implications from a national sample of US adults.   Health Aff (Millwood). 2018;37(12):1948-1954. doi:10.1377/hlthaff.2018.05117 PubMedGoogle ScholarCrossref
    23.
    Cutrona  SL, Golden  JG, Goff  SL,  et al.  Improving rates of outpatient influenza vaccination through EHR portal messages and interactive automated calls: a randomized controlled trial.   J Gen Intern Med. 2018;33(5):659-667. doi:10.1007/s11606-017-4266-9 PubMedGoogle ScholarCrossref
    24.
    Jacobson Vann  JC, Szilagyi  P.  Patient reminder and patient recall systems for improving immunization rates.   Cochrane Database Syst Rev. 2018;1(1):CD003941.PubMedGoogle Scholar
    25.
    Szilagyi  PG, Valderrama  R, Vangala  S,  et al.  Pediatric patient portal use in one health system.   J Am Med Inform Assoc. 2020;27(3):444-448. doi:10.1093/jamia/ocz203 PubMedGoogle ScholarCrossref
    26.
    Becker  MH, Maiman  LA.  Sociobehavioral determinants of compliance with health and medical care recommendations.   Med Care. 1975;13(1):10-24. doi:10.1097/00005650-197501000-00002 PubMedGoogle ScholarCrossref
    27.
    Kim  H, Xie  B.  Health literacy in the eHealth era: a systematic review of the literature.   Patient Educ Couns. 2017;100(6):1073-1082. doi:10.1016/j.pec.2017.01.015 PubMedGoogle ScholarCrossref
    28.
    Jenssen  BP, Buttenheim  AM, Fiks  AG.  Using behavioral economics to encourage parent behavior change: opportunities to improve clinical effectiveness.   Acad Pediatr. 2019;19(1):4-10. doi:10.1016/j.acap.2018.08.010 PubMedGoogle ScholarCrossref
    29.
    Zou  G.  A modified Poisson regression approach to prospective studies with binary data.   Am J Epidemiol. 2004;159(7):702-706. doi:10.1093/aje/kwh090 PubMedGoogle ScholarCrossref
    30.
    Zhu  M.  Analyzing Multilevel Models with the GLIMMIX Procedure: Proceedings of the SAS Global Forum 2014 Conference. SAS Institute Inc; 2014. Accessed January 17, 2020. https://support.sas.com/resources/papers/proceedings14/SAS026-2014.pdf
    31.
    Dempsey  AF, Zimet  GD.  Interventions to improve adolescent vaccination: what may work and what still needs to be tested.   Vaccine. 2015;33(suppl 4):D106-D113. doi:10.1016/j.vaccine.2015.09.032 PubMedGoogle ScholarCrossref
    32.
    Hofstetter  AM, Vargas  CY, Camargo  S,  et al.  Impacting delayed pediatric influenza vaccination: a randomized controlled trial of text message reminders.   Am J Prev Med. 2015;48(4):392-401. doi:10.1016/j.amepre.2014.10.023 PubMedGoogle ScholarCrossref
    33.
    Saville  AW, Beaty  B, Dickinson  LM, Lockhart  S, Kempe  A.  Novel immunization reminder/recall approaches: rural and urban differences in parent perceptions.   Acad Pediatr. 2014;14(3):249-255. doi:10.1016/j.acap.2014.02.003 PubMedGoogle ScholarCrossref
    34.
    Edwards  KM, Hackell  JM; Committee on Infectious Diseases, Committee on Practice and Ambulatory Medicine.  Countering vaccine hesitancy.   Pediatrics. 2016;138(3):e20162146. doi:10.1542/peds.2016-2146 PubMedGoogle Scholar
    35.
    Jarrett  C, Wilson  R, O’Leary  M, Eckersberger  E, Larson  HJ; SAGE Working Group on Vaccine Hesitancy.  Strategies for addressing vaccine hesitancy: a systematic review.   Vaccine. 2015;33(34):4180-4190. doi:10.1016/j.vaccine.2015.04.040 PubMedGoogle ScholarCrossref
    36.
    Cunningham  RM, Minard  CG, Guffey  D, Swaim  LS, Opel  DJ, Boom  JA.  Prevalence of vaccine hesitancy among expectant mothers in Houston, Texas.   Acad Pediatr. 2018;18(2):154-160. doi:10.1016/j.acap.2017.08.003 PubMedGoogle ScholarCrossref
    37.
    Shen  AK, Warnock  R, Selna  W, Chu  S, Kelman  JA.  Patient characteristics of Medicare beneficiaries who report not getting influenza and pneumococcal vaccinations, 2001-2013.   Hum Vaccin Immunother. 2019:1-7. doi:10.1080/21645515.2019.1688033 PubMedGoogle Scholar
    38.
    Schmid  P, Rauber  D, Betsch  C, Lidolt  G, Denker  ML.  Barriers of influenza vaccination intention and behavior: a systematic review of influenza vaccine hesitancy, 2005-2016.   PLoS One. 2017;12(1):e0170550. doi:10.1371/journal.pone.0170550 PubMedGoogle Scholar
    39.
    Scott  A, Liu  M, Yong  J.  Financial incentives to encourage value-based health care.   Med Care Res Rev. 2018;75(1):3-32. doi:10.1177/1077558716676594 PubMedGoogle ScholarCrossref
    ×