Variations in Influenza Vaccination by Clinic Appointment Time and an Active Choice Intervention in the Electronic Health Record to Increase Influenza Vaccination

Key Points

Question  Do influenza vaccination rates vary by clinic appointment time and can an active choice intervention in the electronic health record directed to medical assistants improve vaccination rates in primary care practices?

Findings  In this quality improvement study of 11 primary care practices and 96 291 patients, influenza vaccination rates significantly declined as the clinic day progressed. Primary care practices that implemented an active choice intervention in the electronic health record to prompt medical assistants to ask patients about influenza vaccination and template vaccination orders for clinicians to review were associated with a significant increase in influenza vaccination rates when compared with a control group of primary care practices.

Meaning  Reminders in the electronic health record can improve vaccination rates overall, but other strategies may also be needed to address variations by time of day.

Importance  Influenza vaccination rates in the United States are suboptimal near 40%, but little is known about variations in care based on clinic appointment time.

Objectives  To compare differences in influenza vaccination rates by clinic appointment time and to evaluate the association of an active choice intervention in the electronic health record with changes in vaccination rates.

Design, Setting, and Participants  Retrospective, quality improvement study of 11 primary care practices at the University of Pennsylvania Health System from September 1, 2014, to March 31, 2017. Participants included adults eligible for influenza vaccination. Data analysis was conducted from October 20, 2017, to March 9, 2018.

Interventions  During the 2016 to 2017 influenza season, 3 primary care practices at the University of Pennsylvania Health System implemented an active choice intervention in the electronic health record that prompted medical assistants to ask patients about influenza vaccination during check-in and template vaccination orders for clinicians to review during the visit.

Main Outcomes and Measures  Influenza vaccination rates.

Results  The sample comprised 96 291 patients with a mean (SD) age of 56.2 (17.0) years; 41 865 (43.5%) were men, 61 813 (64.2%) were white, and 23 802 (24.7%) were black. Among all practices across all 3 years, vaccination rates were approximately 44% from 8 am to 10 am, declined to 41.2% by 11 am and 38.3% at noon, increased to 40.2% at 1 pm, and then declined to 34.3% at 3 pm and 32.0% at 4 pm (P < .001 for adjusted linear trend). For the 3 years, vaccination rates were 46.9%, 47.2%, and 45.6% at control practices and 49.7%, 52.2%, and 59.3% at intervention practices, respectively. In adjusted analyses, compared with control practices over time, the active choice intervention was associated with a significant 9.5–percentage point increase in vaccination rates (95% CI, 4.1-14.3; P < .001). Vaccination rates increased similarly across times of the day.

Conclusions and Relevance  Influenza vaccination rates significantly declined as the clinic day progressed. The active choice intervention was associated with a significant increase in influenza vaccination rates that were similar in magnitude throughout the day.

Influenza is a significant cause of illness, hospitalization, and death in the United States.¹ While influenza vaccination has been demonstrated to reduce the burden of disease,^2,3 more than 50% of adults in the United States do not receive the vaccine each year.⁴ The Centers for Disease Control and Prevention recommend that clinicians offer patients the vaccine during clinic visits.⁵ However, primary care physicians (PCPs) often fall behind schedule as the clinic day progresses and this tendency may lead to rushed visits, decision fatigue, and suboptimal care. For example, inappropriate antibiotic prescribing rates have been shown to be greater later in the clinic day.⁶ To our knowledge, there has been no evaluation of variations in vaccination rates by clinic appointment time.

Nudges are changes to choice architecture that can have outsized effects on medical decision making.⁷ For example, in prior work we found that using active choice, a method that requires clinicians to accept or decline an influenza vaccination order in the electronic health record (EHR), significantly increased vaccination rates at 1 primary care practice.⁸ However, this type of approach could lead to clinician alert fatigue.^9-11 So before expanding to 3 other practices at our institution, the active choice intervention was redirected to medical assistants who could ask patients about influenza vaccination and template orders in the EHR for clinicians to review. If variations in care exist by clinic appointment time, these types of nudges could address them. The objective of this study was to evaluate differences in influenza vaccination rates by clinic appointment time and compare changes in vaccination rates among practices receiving an active choice intervention in the EHR with those among control practices.

The University of Pennsylvania institutional review board approved this study and waived informed consent because it was infeasible given the study design, and the study posed minimal risk. This study followed the Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guideline.

The sample comprised adult patients at 11 primary care practices at the University of Pennsylvania Health System who had a new or return patient clinic visit with their PCP during at least 1 of 3 influenza seasons (September 1 to March 31) between 2014 and 2017. The intervention practice sites included 3 internal medicine clinics in Philadelphia and a suburb that chose to adapt evidence from our pilot work⁸ to implement the intervention described in this study. Control practices included 8 internal medicine and family medicine clinics in Philadelphia and the suburbs. These control practices were selected through the following process. First, we identified all primary care practices at our main primary care network at the University of Pennsylvania Health System. We included both internal medicine practice (who belonged to the same division as the intervention practices but decided not to implement the intervention) and family medicine practices (who belonged to a different division and did not discuss the option of implementing the intervention). Second, since preliminary data indicated that the overall vaccination rate during the preintervention period at these 22 control practices were much lower (approximately 30%) than the intervention practices (approximately 50%), we selected practices that were within a 10–percentage point range above or below the average of the intervention sites (approximate range, 40%-60%) and had at least 500 eligible patients in all 3 influenza seasons. This provided a more comparable control group of 8 practices with sufficient sample sizes at each practice. Practice sites’ locations, specialties, and vaccination rates in each of the influenza seasons for the 3 intervention and 8 control practices are available in eTable 1 in the Supplement.

To compare vaccination rates by clinic appointment time, we evaluated patients at their first visit with their PCP in each season. Patients whose first PCP visit was not a new or return visit were excluded since vaccination may not be appropriate for acute, sick, or other visits (n = 15 440). Patients who were already vaccinated based on EHR documentation were also excluded (n = 29 692) (eFigure 1 in the Supplement).

To compare changes in vaccination rates in practices with and without the active choice intervention, we evaluated patients during entire influenza seasons. Patients who switched PCPs within a season (n = 446), did not have a PCP visit (n = 7857), did not have a new or return visit with their PCP at any point during the season (n = 1624), or were already vaccinated based on EHR documentation were excluded (n = 30 024) (eFigure 2 in the Supplement).

Prior to the intervention, PCPs had to remember to manually check if a patient was due for influenza vaccination, discuss it with the patient, and then place an order for it in the EHR. During the 2016 to 2017 influenza season, 3 University of Pennsylvania Health System primary care practices implemented an active choice intervention in the EHR using a best practice alert in EPIC directed to medical assistants (eFigure 3 in the Supplement). Prior to meeting with the clinician, patients met with a medical assistant to check their vitals. At that time, the EHR checked for patient eligibility for the influenza vaccine and prompted medical assistants to accept or cancel an order for the vaccine. If accepted, the order would be templated for the clinician to review and sign during the patient visit. This intervention was similar in design to prior work,^8,12 except that it was delivered only to medical assistants and not to clinicians.

Clarity, an EPIC reporting database, was used to obtain data on patients (demographics, insurance, comorbidities, prior influenza vaccination status, and PCPs) and clinic visits (date, appointment time, practice site, visit type, and presence of an order for influenza vaccination or not). Health insurance claims data were not available for this study. However, in prior work,⁸ more than 99.9% of vaccine orders also had an insurance claim indicating almost all were both ordered and completed. Household income level was obtained using the US Census data on median household income based on zip code.

The primary outcome measure was the percentage of eligible patients who had the influenza vaccination ordered. To evaluate vaccination rates by clinic appointment time, we grouped appointment times by each hour (eg, appointments for 8 am, 8:15 am, 8:30 am, and 8:45 am were all grouped to 8 am). In the adjusted analysis of patient visit level data, we used PROC GENMOD to fit the model based on generalized estimating equations with a logit link and an independence correlation structure using PCPs as the clustering unit.¹³ The model was adjusted for patient demographics (age, sex, race/ethnicity, and household income), insurance, Charlson comorbidity index,¹⁴ practice group (intervention or control site), clinic visit type (new or return), fixed effects by practice site, year and calendar month, and a covariate for each appointment hour. To obtain an adjusted linear trend of vaccination as the clinic day progressed, the same model was fit, but instead of covariates for each appointment hour, we used a continuous variable for appointment time (from 8 am to 4 pm).

To evaluate the association of changes in vaccination rates with the active choice intervention, we used a difference-in-differences approach.^15,16 Similar to prior work,^8,12 we compared changes in vaccination order rates at the intervention vs control practices during the postintervention year relative to the 2 preintervention years. We used the patient as the unit of analysis and included all clinic visits during each influenza season. Since patients were not always represented in all 3 years and some switched PCPs, patients who appeared in the data for multiple years were treated as independent patients for each year as was done in prior work.^8,12 The same generalized estimating equations model was fit for analysis of patient season level data but without visit level covariates (appointment time or visit type) and with interaction terms for practice group (intervention vs control) and year. To obtain the adjusted difference in the percentage of patients vaccinated along with 95% confidence intervals, we used the bootstrap procedure, resampling patients 1000 times.^17,18 Resampling of patients was conducted by PCPs to maintain clustering at the PCP level. A test of controls was performed to test the null hypothesis of parallel trends between the intervention and control practices during the 2 preintervention years.¹⁹ We conducted several sensitivity analyses with different modeling structures. First, we fit the difference-in-differences model, also adjusting for the number of patient visits with their PCP in each season. Second, we fit a generalized linear mixed model using PCP random effects nested under practice random effects. Two-sided hypothesis tests used a significance level of .05; all analyses were conducted in SAS, version 9.4 (SAS Institute Inc).

The sample of the entire influenza seasons comprised 96 291 patients with a mean (SD) age of 56.2 (17.0) years; 41 865 (43.5%) were men, 61 813 (64.2%) were white, and 23 802 (24.7%) were black (Table 1). The sample of first visits with the PCP comprised 72 908 (87.3%) return patients and 10 585 (12.7%) new patients (eTable 2 in the Supplement).

Figure 1 displays the unadjusted influenza vaccination rates by clinic appointment time for all practices during all 3 seasons. Vaccination rates were approximately 44% from 8 am to 10 am and then declined to 41.2% at 11 am and 38.3% at noon. Vaccination rates increased slightly to 40.2% at 1 pm and then declined to 34.3% at 3 pm and 32.0% at 4 pm. Relative to the 8 am appointment time, the adjusted odds ratios of vaccination were significantly lower for each subsequent hour of the day and for the overall linear trend (adjusted odds ratio, 0.95; 95% CI, 0.94-0.96; P < .001) (Table 2). The distribution of patients by clinic appointment time in year 3 was similar among patients vaccinated and not vaccinated in year 2 (eTable 3 in the Supplement), indicating that patients who historically received the vaccine did not make earlier appointments than those who did not receive the vaccine in the past. Regression tables for the models are available in eTables 4 and 5 in the Supplement.

Figure 2 displays the unadjusted influenza vaccination rate by control and intervention practice group by year. For the 3 years, vaccination rates were 46.9%, 47.2%, and 45.6% at control practices and 49.7%, 52.2%, and 59.3% at intervention practices, respectively. Adjusted preintervention trends during the first 2 years did not differ between groups (adjusted odds ratio, 1.08; 95% CI, 0.92-1.27; P = .36). In adjusted analyses, there was a significant 9.5–percentage point increase (95% CI, 4.1-14.3; P < .001) in influenza vaccination rates for the intervention practices relative to the compared practices over time. The regression table for the model is available in eTable 6 in the Supplement. Sensitivity analyses using different modeling structures had similar findings (eTables 7 and 8 in the Supplement).

Figure 3 displays unadjusted influenza vaccination rates for control and intervention during the first 2 years (preintervention) and the third year (postintervention). Among control practices, vaccination trends were similar in both periods but rates were 0 to 5 percentage points higher in the postintervention period (Figure 3A). Among intervention practices, vaccination trends were similar in both periods, but rates were 8 to 14 percentage points greater in the postintervention period (Figure 3B). Among intervention practices, vaccination rates at the end of the day (4 pm) were 33% in the preintervention period and 40% in the postintervention period, which were nearly the same as at 8 am in the preintervention period (41%).

Among a network of primary care practices, we found that influenza vaccination rates significantly declined as the clinic day progressed. An active choice intervention delivered to medical assistants through the EHR was associated with a significant 9.5–percentage point increase in influenza vaccination rates, which is nearly a 20% relative increase compared with the preintervention period. However, even after the intervention, influenza rates declined as the clinic day progressed. To our knowledge, this is one of the first studies of its kind to evaluate variations in influenza vaccination rates by clinic appointment time. Our findings demonstrate the potential of using nudges in the EHR to improve vaccination rates, but further strategies may be needed to address variations in care by time of day.

These findings expand our understanding of how practice environments influence medical decision making in several ways. First, time of day may play an important role in influencing patient care. This could be because as the clinic day progresses, clinicians often fall behind schedule, leading to rushed visits, or because clinicians may be susceptible to decision fatigue, the depletion of self-control and active initiative that results from the cumulative burden of making decisions.²⁰ These tendencies have been demonstrated in other areas of health care. Linder and colleagues⁶ examined behaviors at 23 primary care practices and found that physicians had higher rates of inappropriate antibiotic prescriptions as clinic sessions progressed. Dai and colleagues²¹ evaluated health care workers at 35 hospitals and found that hand hygiene compliance rates declined as time progressed through a work shift. Both Linder et al and Dai et al found that when health care professionals were given a break in the middle of the day or shift, behaviors temporarily improved but then continued to progressively worsen. We also found that influenza vaccination rates increased from noon to 1 pm, presumably after a lunch break for the clinicians and staff, but then continued to decline over the course of the rest of the day. It is important to note that practice- or patient-level factors may also play important roles in medical decision-making behaviors over the course of the clinic day. Future research could further evaluate the relative contribution of clinicians, medical assistants, and other staff, practice workflow, and patient-level factors on vaccination rates. Medical decision-making behaviors over the course of clinic day should also be evaluated in other settings and for other behaviors.

Second, as medical decision making is increasingly occurring within digital environments, the design and framing of information in the EHR can influence clinician behaviors. We found that an active choice intervention that prompted medical assistants to make decisions on whether to template vaccinations orders in the EHR for clinicians to review was associated with a significant increase in vaccination rates. These findings add to the existing literature on using nudges to increase influenza vaccination. Milkman and colleagues²² conducted a randomized clinical trial at a large employer and found that participants who were asked to plan a date and time for the vaccination had a 4.2–percentage point increase in vaccination rates when compared with control. Chapman and colleagues²³ randomly assigned university employees to receive an email with an opt-in vs opt-out framing for influenza vaccination and found that those with the opt-out default setting had a 12–percentage point increase in vaccination rates compared with control. Our study differs from these 2 in that it was directed to clinicians rather than patients, was conducted in primary care practices, and leveraged a scalable approach through the EHR. Future research could compare nudges that are targeted with clinicians, patients, or both in combination.

Third, the design of the active choice intervention differed in this study compared with our previous study and reveals important insights. In prior work, the intervention was delivered to physicians and medical assistants.⁸ While it was associated with a 6.6–percentage point increase in vaccination rates, physicians reported alert fatigue and there was confusion between whether the medical assistant or physician was responsible for addressing the alert. Therefore, before expanding the approach to other practices, the decision was made to deliver the alert only to medical assistants. This design increased clarity in who was responsible for addressing the alert and reduced the burden on physicians. While the 2 studies were conducted in different practices, it is important to highlight that the intervention in this study achieved a greater percentage point increase in vaccination rates (9.5 vs 6.6) on a much higher baseline preintervention level (51% vs 18%). Future research could compare the effectiveness of active choice interventions and other types of nudges such as setting EHR defaults through randomized clinical trials and in other settings.

This study had limitations. First, any observational study is susceptible to unmeasured confounders. However, the active choice intervention was evaluated using a difference-in-differences approach, which reduces potential bias from unmeasured cofounders by comparing changes in vaccination levels over time between the intervention and control practices. Second, this study was conducted within a single health system, which may limit generalizability. However, we included 11 practice sites from 2 different states. Nonetheless, findings should be confirmed in other settings. Third, changes in vaccination rates by clinic appointment time may be owing to physician, practice, and patient factors and we were not able to evaluate their relative contribution. However, our multivariate models do adjust for available physician, practice, and patient characteristics.

Influenza vaccination rates significantly declined as the clinic day progressed. The active choice intervention was associated with a significant increase in vaccination rates that were similar in magnitude throughout the day. These findings indicate that time of day and choice architecture play an important role in influencing clinician behaviors and patient outcomes. Scalable approaches that leverage the EHR to deploy well-designed nudges may be a promising approach to improve medical decision-making behaviors, but further work is needed to address variations in vaccination rates by time of day.

Accepted for Publication: June 13, 2018.

Published: September 14, 2018. doi:10.1001/jamanetworkopen.2018.1770

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2018 Kim RH et al. JAMA Network Open.

Corresponding Author: Mitesh S. Patel, MD, MBA, MS, University of Pennsylvania, 3400 Civic Center Blvd, 14-176 South Pavilion, Philadelphia, PA 19104 (mpatel@pennmedicine.upenn.edu).

Author Contributions: Dr Patel had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Kim, Day, Patel.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Kim, Rareshide, Patel.

Critical revision of the manuscript for important intellectual content: Day, Small, Snider, Patel.

Statistical analysis: Small, Rareshide, Patel.

Obtained funding: Patel.

Administrative, technical, or material support: Kim, Patel.

Supervision: Patel.

Conflict of Interest Disclosures: Dr Patel is supported by the career development awards from the Department of Veterans Affairs Health Services Research and Development Service and the Doris Duke Charitable Foundation. Dr Patel is the founder of Catalyst Health and a technology and behavior change consulting firm. Dr Patel received research funding from Deloitte, which is not related to the work described in this manuscript. Dr Patel reported personal fees from Catalyst Health and is an advisory board member for HealthMine Services, Life.io, and Holistic Industries outside the submitted work. No other disclosures were reported.

Funding/Support: This study was supported by the University of Pennsylvania Health System through the Penn Medicine Nudge Unit.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.