Effect of Patient Portal Reminders Sent by a Health Care System on Influenza Vaccination Rates: A Randomized Clinical Trial

Key Points

Question  Can reminders sent through a patient portal increase influenza vaccination rates across a health care system?

Findings  In this randomized clinical trial of 164 205 patients served by 52 primary care practices across a health care system, 1, 2, or 3 influenza vaccine reminders sent through the electronic health record patient portal had a small, statistically significant effect on increasing influenza vaccination rates compared with no reminder.

Meaning  Generic patient portal reminders may be slightly effective in increasing influenza vaccination rates, but more intensive or more targeted patient motivational strategies appear to be needed.

Importance  Influenza vaccination rates across the US are low. Because few practices send patient reminders for influenza vaccination, a scalable patient reminder system is needed.

Objective  To evaluate the effect of patient reminders sent via a health care system’s electronic health record patient portal on influenza vaccination rates.

Design, Setting, and Participants  This pragmatic, 4-arm randomized clinical trial was performed from October 1, 2018, to March 31, 2019, across the UCLA (University of California, Los Angeles) health care system. A total of 164 205 patients in 52 primary care practices who had used the patient portal within 12 months were included.

Interventions  Patients due for an influenza vaccine were sent a letter via the patient portal of the health care system reminding them about the importance of influenza vaccination, safety of the vaccine, and morbidity associated with influenza. Patients were randomized within primary care practices to 1 of 4 study groups (no reminder [n = 41 070] vs 1 reminder [n = 41 055], 2 reminders [n = 41 046], or 3 reminders [n = 41 034]).

Main Outcomes and Measures  The primary outcome was receipt of 1 or more influenza vaccines as documented in the electronic health record, which was supplemented with influenza vaccination data from external sources (eg, pharmacies). Secondary outcomes were influenza vaccination rates among subgroups and influenza vaccinations self-reported by patients in reply to the portal-based query as having been received elsewhere.

Results  A total of 164 205 patients (mean [SD] age, 46.2 [19.6] years; 95 779 [58.3%] female) were randomly allocated to 1 of the 4 study arms. In the primary analysis across all ages and not including patient self-reported vaccinations in reply to portal reminders, influenza vaccination rates were 37.5% for those receiving no reminders, 38.0% for those receiving 1 reminder (P = .008 vs no reminder), 38.2% for those receiving 2 reminders (P = .03 vs no reminder), and 38.2% for those receiving 3 reminders (P = .02 vs no reminder). In the secondary analysis not including patient self-reported vaccinations, among adults aged 18 to 64 years (vaccination rates: 32.0% in the control group, 32.8% in the 1-reminder group, 32.8% in the 2-reminder group, and 32.8% in the 3-reminder group; P = .001), male patients (vaccination rates: 37.3% vs 38.3%, 38.6%, and 38.8%; P = .001), non-Hispanic patients (vaccination rates: 37.6% vs 38.2%, 38.3%, and 38.2%; P = .004), and those who were not vaccinated in the prior 2 years (vaccination rates: 15.3% vs 15.9%, 16.3%, and 16.1%; P < .001), vaccination rates were higher in the portal reminder groups than in the control group; the findings in these 3 subgroups mirrored the findings in the entire population. When self-reported vaccinations received elsewhere were included, influenza vaccination rates were 1.4 to 2.9 percentage points higher in the portal reminder groups, with a dose-response effect (0 reminders: 15 537 [37.8%]; 1 reminder: 16 097 [39.2%]; 2 reminders: 16 426 [40.0%]; and 3 reminders: 16 714 [40.7%]; P < .001).

Conclusions and Relevance  Generic patient portal reminders were effective in minimally increasing influenza vaccination rates, but more intensive or more targeted patient motivational strategies appear to be needed.

Trial Registration  ClinicalTrials.gov Identifier: NCT03666026

Influenza causes many respiratory illnesses, hospitalizations, ambulatory medical visits, and deaths in the US annually.¹ Although the Advisory Committee on Immunization Practices recommends annual influenza vaccination for all US individuals 6 months or older,¹ vaccination rates lag behind US Healthy People 2020 vaccination coverage goals of more than 80% for individuals younger than 65 years and more than 90% for those 65 years or older.² In 2018 to 2019, vaccination coverage rates by age were 62.6% among those aged 6 months to 17 years, 34.9% among those aged 18 to 49 years, 47.3% among those aged 50 to 64 years, and 68.1% among those 65 years or older.³

Although patient reminders are associated with increased influenza vaccination coverage^4,5 and the Task Force on Community Preventive Services recommends reminders,⁶ few primary care practices send reminders for any vaccine^7-10 in part because of limited resources.^10-12 Experts have recommended centralized reminders from state immunization information systems or health care systems as a scalable strategy.¹³ Most studies using centralized reminders for influenza vaccination have focused on children or adults with chronic diseases,^7,14 subsets of the population,^15,16 pregnant women,¹⁷ or small numbers of practices.^18-20 A recent study²¹ that tested reminders for childhood influenza vaccination with state immunization information systems found little effect.

A scalable method to send centralized reminders involves patient portals, which are secure, web-based applications that provide 2-way communication between patients and health care systems or health care professionals. Most electronic health records (EHRs) have patient portals, and a recent survey²² found that 37% of adults with a health care visit in the past year had used portals in the past year. A randomized clinical trial in a health care system²³ that evaluated the patient portal plus interactive voice recording reminders observed 1% improvement in influenza vaccination rates, but that trial²³ involved a multispecialty group practice, excluded patients vaccinated before November 10, and had baseline vaccination rates below 15%.²³ With Cochrane reviews^4,24 reporting that reminder and recall work for influenza vaccination, it is important to study the effect of portal reminders on influenza vaccination across a health care system.

We performed a 4-arm randomized clinical trial that compared the effect of 1, 2, or 3 vs no portal reminders on influenza vaccination rates across a large health care system. We evaluated the effect of portal reminders on receipt of 1 or more influenza vaccinations in the entire population and in predesignated subgroups. We hypothesized that portal reminders would increase influenza vaccination rates at the population level and in subgroups (age, sex, race/ethnicity, insurance, and influenza vaccination in prior years).

The intervention was conducted from October 1, 2018, to March 31, 2019. The design was a 4-arm, pragmatic, intention-to-treat randomized clinical trial of 164 205 patients randomized within primary care practices to 1 of 4 study arms (no reminder [n = 41 070] vs 1 reminder [n = 41 055], 2 reminders [n = 41 046], or 3 reminders [n = 41 034]). The setting was all 52 UCLA (University of California, Los Angeles) Health System primary care practices: 38 internal medicine, 5 medicine and pediatrics, 3 family medicine, and 6 pediatrics. The UCLA Institutional Review Board approved this study and provided a waiver of patient consent because (1) the trial was extremely low risk because it was designed to remind patients about a universally recommended vaccine and (2) it was impractical to obtain patient consent from tens of thousands of patients. All data were deidentified. The trial protocol is given in Supplement 1. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.

Patients within UCLA Health System practices can register with the myUCLAHealth (Epic) portal. The Figure shows the flow of the patients through the study. First, we included any UCLA Health System primary care patient 6 months or older at intervention launch (ie, influenza vaccination eligible), defined by 2 or more primary care physician visits (by evaluation and management office codes) within 3 years or 1 or more primary care physician visits with a preventive service code within 1 year. We included managed care patients assigned to the UCLA Health System (even with no health care visits in the prior year) because we were certain about their primary care practice affiliation. Second, we identified the primary care practice most recently visited. Third, we grouped patients into family units using a series of algorithms that matched each patient’s telephone number, address, insurance member number, or patient guarantor identification. Fourth, we identified active portal users as patients or portal proxies (for minors, elderly persons, or disabled persons) who signed up for and logged into the portal 1 or more times during the prior 12 months not including the initial portal log-in; this sample comprised 223 191 (65.0%) of UCLA Health System’s primary care patients.²⁵ Statisticians (S.V., C.-H.T.) randomly selected 1 active portal-using index patient per family for inclusion. This process generated the denominator of potential participants; all other study personnel and health care professionals were blinded to study allocation.

Exclusion criteria were individuals who were not active portal users and family members of index patients, who were excluded because of adequate sample size with index patients and for consistency with Cochrane criteria for meta-analyses using Cochrane RevMan software.^4,24

Study statisticians randomized index patients to 1 of the 4 study arms, stratifying by their affiliation with a primary care practice. All index patients’ family members (whether proxies or not) received the same portal reminder to avoid confusion, but only index patients’ data were analyzed.

For all patients randomized to 1, 2, or 3 portal messages, study statisticians (S.V., C.-H.T.) sent files to the health care system’s patient portal team, defining the portal messages to be sent. A secure notification was then sent by email or text (per patients’ standard portal preference) informing patients that “a message from your doctor” had been posted on the portal. Influenza vaccination was not mentioned in the title of the message. Patients had to log in to the portal to read the letter.

The portal reminder letter content was grounded in the Health Belief Model and principles of health literacy.^26,27 We based the content on (1) prior patient interviews used to craft patient reminders for influenza vaccination,²¹ (2) positive framing principles of behavioral economics,²⁸ and (3) input from external national vaccination experts. The letters (eFigure in Supplement 2) included the following: (1) information that influenza season was coming, the disease can cause substantial morbidity, and the vaccine is the best way to protect against influenza; (2) recommendation to receive an influenza vaccine by calling for an office appointment or going to a pharmacy or other setting; (3) a website link to input influenza vaccinations received elsewhere into the UCLA Health System record; and (4) another website link to a UCLA webpage containing information about influenza vaccine and video testimonials about influenza vaccination. Letters were in English, included the name of the patient’s primary care physician, and had a below seventh grade reading level per Flesch-Kincaid analysis.

Portal reminders were sent according to randomization arm at the beginning of October (all intervention groups), November (those receiving 2-3 reminders), and December (those receiving 3 reminders). The letter content was identical per round. Before each round, patients who had received an influenza vaccination according to the EHR were removed from the eligible list. For children younger than 8 years and eligible for 2 vaccinations, portal reminders were sent only for the initial vaccination.

Patient and practice characteristics were obtained from the EHR. Patient factors (Table 1) included age, sex, insurance at the last primary care visit, race/ethnicity, and receipt of 1 or more influenza vaccinations during the prior 2 years.

Data from the EHR included influenza vaccinations (date and location) administered at any UCLA Health System site. In addition, UCLA health care practitioners could enter vaccination codes manually for vaccines received outside the UCLA Health System. The UCLA Health System merges data into the EHR from external sources, including SureScripts (pharmacy benefits manager), California Immunization Registry (mostly children’s immunizations), claims from select health care plans, and Care Everywhere (Epic’s information exchange application). Patients or proxies can also enter data when they log in to the portal. Data from these external sources (other than from claims) are normally incorporated into the medical record at office visits via a reconciliation process whereby health care professionals add or discard individual data elements. Our study did not modify these standards. However, we integrated data from all these external sources regardless of whether physicians reconciled the immunization record.

Finally, the portal reminder letters included a link for patients to update influenza vaccinations received elsewhere; these data were also integrated into the EHR. We tracked this method for updating influenza vaccinations received elsewhere and used this flag to distinguish primary vs secondary outcome measures.

The primary study outcome was EHR-based documentation that included vaccinations obtained elsewhere if inputted into the EHR using the health care system’s standard electronic merging of 1 or more influenza vaccinations between October 1, 2018, and March 31, 2019. The primary analysis excluded vaccinations reported only by patients in response to the portal reminders because the control group did not have this opportunity for self-report, thus eliminating differential outcome ascertainment. This analysis created a conservative bias because portal reminders may have encouraged some patients to receive influenza vaccinations at sites (eg, employers, malls, and some pharmacies) that did not send data to the EHR.

We assessed 5 secondary outcomes related to receipt of 1 or more influenza vaccinations in the relevant season: (1) vaccinations in predetermined subgroups (age, sex, race/ethnicity, insurance, and influenza vaccination in prior 2 years); (2) vaccinations that were self-reported by patients or proxies via the portal in response to portal reminders; (3) influenza vaccinations before December 15, 2018, to determine whether portal reminders increased early vaccination; (4) vaccinations among those who opened 1 or more portal reminders vs those who did not open the reminders; and (5) vaccinations in patients in the upper half vs bottom half of overall portal use.

We assessed whether patients (1) opened the portal reminder letter, (2) updated influenza vaccinations received outside the UCLA Health System in response to the portal reminders, and (3) clicked on the informational website link embedded in the portal letter.

A sample size of approximately 41 000 patients per study arm would provide greater than 90% power to detect a 2–percentage point improvement in the rate of vaccination, which we deemed to be the smallest improvement with clinical effect. This number assumes a χ² test, a control group rate of 50% (most conservative), and a significance level of P < .017 (3-fold Bonferroni correction for the pairwise comparisons of study arms with 1, 2, and 3 reminders vs control). Because patients were randomized within practices, clustering by practice was ignored in this power calculation for conservatism.

We report descriptive statistics for all patient characteristics in terms of frequency distributions for the full patient sample and stratifying by study arm. The primary analysis compared vaccination rates between study arms using mixed-effects Poisson regression with robust SEs.^29,30 Models included a fixed effect for study arm (0 vs 1 vs 2 vs 3 reminders), random practice effects, and adjustment for patient characteristics (age, sex, insurance, race/ethnicity, and vaccination history). Secondary subgroup analyses were performed by fitting separate models for each subgroup. In these analyses, the comparison was between the combination of the 3 intervention arms and the control arm. We further performed exploratory analyses to evaluate the effect of opening any reminder letters on vaccination status. In one approach, we replicated the primary analysis strategy but grouped patients by whether they opened any letters rather than by randomization assignment. In a second approach, we used 2-stage least-squares with cluster robust SEs at the practice level to estimate the local mean treatment effect of opening any letters using assignment to intervention as an instrumental variable.

For the primary analysis comparing each of the individual reminder arms with the control arm, a significance level of P < .017 was used. In all other analyses, we considered P < .05 as statistically significant. All statistical analyses were performed using SAS statistical software, version 9.4 (SAS Institute Inc).

A total of 164 205 patients (mean [SD] age, 46.2 [19.6] years; 95 779 [58.3%] female) were randomly allocated to 1 of the 4 study arms. Table 1 provides demographic data across study arms. A total of 161 268 patients (or proxies) (98.2%) spoke English as their primary language.

Across all patients, the vaccination rates were 37.5% among controls (no reminders), 38.0% in the 1-reminder group (P = .008 vs controls), 38.2% in the 2-reminder group (P = .03 vs controls), and 38.2% in the 3-reminder group (P = .02 vs controls). Table 2 gives the unadjusted and adjusted risk ratios (RRs) for influenza vaccination for each study arm in the entire population. Adjusted RRs were 1.02 (95% CI, 1.00-1.03) for the 1-reminder group, 1.02 (95% CI, 1.00-1.04) for the 2-reminder group, and 1.02 (95% CI, 1.00-1.04) for the 3-reminder group, showing a small effect and no dose response by number of reminders sent.

For subgroups (Table 3), including age, sex, insurance, race/ethnicity, and prior influenza vaccination, the intervention had only a small (or no) effect on influenza vaccination rates. Some intervention subgroups vs the corresponding control subgroup had vaccination rates that were several percentage points higher, but there was no clear pattern or dose response by the number of reminders sent. As indicated in Table 3, portal reminders had a statistically significant effect among patients who were male (vaccination rates: 37.3% in the control group, 38.3% in the 1-reminder group, 38.6% in the 2-reminder group, and 38.8% in the 3-reminder group; P = .001), covered by private insurance (vaccination rates: 35.3% in the control group, 35.5% in the 1-reminder group, 36.1% in the 2-reminder group, and 36.0% in the 3-reminder group; P = .02), white (vaccination rates: 39.0% in the control group, 39.7% in the 1-reminder group, 39.9% in the 2-reminder group, and 39.6% in the 3-reminder group; P = .004), non-Hispanic (vaccination rates: 37.6% in the control group, 38.2% in the 1-reminder group, 38.3% in the 2-reminder group, and 38.2% in the 3-reminder group; P = .004), and not vaccinated in the prior 2 years (vaccination rates: 15.3% in the control group, 15.9% in the 1-reminder group, 16.3% in the 2-reminder group, and 16.1% in the 3-reminder group; P < .001).

Overall, 43 387 vaccinations (69.6%) were administered within UCLA clinics, whereas 18 935 (30.4%) were from external data sources, such as SureScripts. When we included self-reported data about influenza vaccinations received in the community in response to the patient portal reminders beyond external vaccinations that were already merged into the EHR (Table 4), 37.8% of controls (no reminders) received an influenza vaccination vs 39.2% in the 1-reminder group, 40.0% in the 2-reminder group), and 40.7% in the 3-reminder arms (comparison of combined intervention vs control; P < .001). There was a dose-response increase in vaccination rates depending on the number of reminders sent. In analyses of subgroups, statistically significant improvements in vaccination rates were noted among all subgroups except for children, with a small dose response.

For the secondary analysis evaluating early vaccinations, 34.2% of controls (no reminders) received an influenza vaccination by mid-December vs 34.8% in the 1-reminder group (P = .001 vs controls), 35.1% in the 2-reminder group (P = .002 vs controls), and 35.2% on the 3-reminder group (P = .002 vs controls). Adjusted RRs were 1.02 (95% CI, 1.01-1.04) for the 1-reminder group, 1.03 (95% CI, 1.01-1.05) for the 2-reminder group, and 1.03 (95% CI, 1.01-1.05) for the 3-reminder group, showing a small effect and no dose response by number of reminders sent, consistent with the full-season results.

After adjustment for all variables in Table 1 (including prior influenza vaccination), across all intervention groups, patients who opened 1 letter or more were more likely to be vaccinated than were those who did not open any letters (39.1% vs 25.4%; adjusted RR, 1.41; 95% CI, 1.38-1.44). We also used an instrumental variables approach to adjust the intention-to-treat effect of assignment to intervention for the proportion of intervention patients who opened any letters. Although the intention-to-treat effect of 1, 2, and 3 reminders was estimated to be a 0.66–percentage point increase in rates, the local mean treatment effect of opening a letter, using assignment to intervention as an instrument, was estimated to be 1.18 percentage points and was statistically significant (risk difference, 1.18; 95% CI, 0.29-2.07).

We compared the intervention effect among those who interacted frequently with the portal (≥15 median log-ins across the population during 12 months) vs less frequently (<15 log-ins). The difference was not statistically significant.

A total of 52.9% of patients in the 1-reminder group, 55.9% in the 2-reminder group, and 58.8% in the 3-reminder group read the reminder letters at least once; for the 2-reminder group, 33.2% read both letters, and for the 3-reminder group, 12.8% read 2 letters and 24.5% read 3 letters. Of all influenza vaccinations recorded in the EHR, data for 0.8% of controls came exclusively from self-reporting vaccinations to the portal vs 3.6% of the 1-reminder group, 5.7% of the 2-reminder group, and 7.5% of the 3-reminder group, suggesting that the portal reminders prompted many patients to report vaccinations received elsewhere. Only 416 patients (0.3%) visited the embedded website providing information about influenza vaccination; even fewer viewed the video testimonials. There were no major patient or practitioner concerns from the study.

This study has several important findings. First, if we exclude patient self-report of vaccinations received elsewhere (triggered by the portal reminders themselves), the portal reminders improved influenza vaccination rates by only 1 percentage point (with an adjusted RR of 1.02), and 2 or 3 reminders did not produce a dose-response effect. Second, portal reminders appeared to have a small, statistically significant effect on increasing influenza vaccination rates among adults aged 18 to 64 years, male patients, non-Hispanic patients, and those not vaccinated in the prior 2 years. Third, when we included self-reported vaccinations indicating a vaccine received elsewhere, the intervention had a larger effect: 1.4% for the 1-reminder group, 2.2% for the 2-reminder group, and 2.9% for the 3-reminder group; however, these findings may be biased by differential outcome ascertainment. Fourth, more than half of patients opened the reminder letters and many after each mailing. Patients who opened at least 1 letter were substantially more likely to be vaccinated than were those who did not. However, because opening a portal letter is a health-seeking behavior, we would expect higher vaccination rates among those who opened a letter even if our intervention did not have any effect. Overall, the primary intention-to-treat analysis revealed a small effect on increasing influenza vaccination rates across this health care system.

Our findings mirror results from a previous portal reminder study²³ that also found only a 1-percentage point increase over a baseline coverage level of less than 15%. We speculate on possible reasons for our small study effect. First, perhaps after receiving a portal reminder, many adult influenza vaccinations were given outside the health care system but not recorded in the EHR despite multiple mechanisms capturing external vaccinations. This possibility would support our secondary analysis that included patient self-reported vaccinations and that found a larger effect. This possibility is supported by our finding that vaccination rates in this population were below national or state coverage estimates, suggesting that data may still be missing about many influenza vaccinations received by this population that has established primary care. Incomplete data on vaccinations received elsewhere may have obscured additional benefit of the intervention.

A second possibility is that the effectiveness of reminders has started to wane amid many patient-targeted health communications. The 2018 Cochrane Review of reminder and recall⁴ did not analyze whether the effect of reminders has decreased over time. A recent study²¹ noted that influenza vaccination reminders sent on behalf of state immunization registries had no effect on child vaccination rates. We had anticipated that portal reminders would have a stronger motivating influence because they come from the physician and from within the patients’ health care system. Although more than 50% of patients opened the reminder letters, the intervention’s effect was small but higher among those who opened the letter. Because only half of patients opened a letter, strategies that enhance reading of portal reminders might be helpful.

Studies^20,31-33 of reminders have suggested that the message content might influence their effect. A postintervention focus group with the health care system’s portal patient representatives noted that patients preferred straightforward, relatively brief, user-friendly messages and positive messages and message titles. Because few study participants (<1%) viewed the linked website with videos, reminder letters themselves should contain critical educational content. Studies are needed to assess whether varying portal message content might improve its effect.

Another possible reason for the limited effectiveness of the portal reminders in the context of low baseline vaccination rates is vaccine hesitancy.^34,35 The intervention was not designed to address hesitancy but rather to nudge patients who desired vaccination. Many studies^35-37 have documented frequent influenza vaccination refusals. Vaccination hesitancy is not an all-or-nothing characteristic,^35,38 and studies are needed to assess the degree to which reminders and education can affect vaccination uptake among hesitant individuals.

An important benefit to this intervention was that it captured more complete influenza vaccination data for the health care system because of the additional vaccinations self-reported by patients in response to portal reminders. This approach particularly benefits health care systems facing value-based incentives that incorporate vaccination rates as a quality metric.^13,39

This study has strengths and limitations. This was a large, pragmatic trial across an entire health care system that tested a new communication mechanism (patient portal) with randomization within primary care practices to adjust for many unmeasured confounders and ascertainment of vaccinations received elsewhere via linkages with the EHR and patient self-reporting. A limitation is potential lack of generalizability to other health care systems (particularly safety-net populations or those with limited English proficiency) and to portal nonusers. In addition, we could not track all vaccinations received outside the health care system.

In this trial, patient portal reminders about influenza vaccination had a small effect on vaccination rates across a health care system. Further research is needed to enhance the effect of portal reminders for influenza vaccination.

Accepted for Publication: April 3, 2020.

Corresponding Author: Peter G. Szilagyi, MD, MPH, Department of Pediatrics, UCLA Mattel Children's Hospital, University of California, Los Angeles, 10833 LeConte, Mail Code 175217, Los Angeles, CA 90095 (pszilagyi@mednet.ucla.edu).

Published Online: May 18, 2020. doi:10.1001/jamainternmed.2020.1602

Author Contributions: Dr Szilagyi had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Szilagyi, Albertin, Casillas, Valderrama, Vangala, Rand, Evans, Sloyan, Lerner.

Acquisition, analysis, or interpretation of data: Casillas, Valderrama, Duru, Ong, Vangala, Tseng, Humiston, Sloyan, Lerner.

Drafting of the manuscript: Szilagyi, Casillas, Valderrama, Ong, Vangala, Tseng, Evans, Lerner.

Critical revision of the manuscript for important intellectual content: Albertin, Casillas, Valderrama, Duru, Ong, Vangala, Tseng, Rand, Humiston, Evans, Sloyan, Lerner.

Statistical analysis: Szilagyi, Casillas, Vangala, Tseng, Lerner.

Obtained funding: Szilagyi, Vangala, Rand.

Administrative, technical, or material support: Szilagyi, Albertin, Valderrama, Vangala, Humiston, Evans, Sloyan, Lerner.

Supervision: Szilagyi, Albertin, Casillas.

Conflict of Interest Disclosures: Drs Szilagyi and Ony and Mss Albertin and Valderrama reported receiving grants from the National Institute of Allergy and Infectious Diseases, National Institutes of Health during the conduct of the study. Dr Rand reported receiving grants from the Centers for Disease Control and Prevention during the conduct of the study. Dr Humiston reported receiving grants from Pfizer Foundation outside the submitted work. No other disclosures were reported.

Funding/Support: This work was supported by grants 1R01AI135029-01 (Dr Szilagyi, principal investigator) National Institute of Allergy and Infectious Diseases, National Institutes of Health and UL1TR00188 from the the National Center for Advancing Translational Sciences, National Institutes of Health.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data Sharing Statement: See Supplement 3.

Additional Contributions: The following individuals work within the UCLA Health System and provided assistance with obtaining data from the electronic medical records and sending the patient portal messages: Alexander E Kurpinski, BS, MyChart/Connected Health; Ashley Elizabeth Turner, BA, clinical decision support analyst; Suzette Ma, BS, senior systems analyst/health informatics analyst, Overlake Medical Center; Hazel H. Oza, BS, clinical decision support lead; Michael A. Pfeffer, MD, assistant vice chancellor and chief information officer; and Deidre Lynn Keeves, BS, PMP, director, MyChart team. None of these people were compensated for their work. L. J. Tan, MS, PhD, chief strategy officer, Immunization Action Coalition, St Paul, Minnesota, and Gregory D. Zimet, PhD, FSAHM, Department of Pediatrics, Indiana University School of Medicine, Indianapolis, provided expert consultation in the development of the content of the portal messages. These individuals were each paid a small stipend. Sarah Friedman, MPH, Department of Pediatrics, UCLA Mattel Children’s Hospital, University of California, Los Angeles, assisted with manuscript preparation. She was not compensated for her work.