Evaluation of a Clinical Decision Support Strategy to Increase Seasonal Influenza Vaccination Among Hospitalized Children Before Inpatient Discharge | Clinical Decision Support | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Study Design With Sequential Crossover From Control to Intervention
Study Design With Sequential Crossover From Control to Intervention
Figure 2.  Run Chart of Influenza Vaccine Administration Rates Stratified by Intervention, Concurrent Control, and Historical Control Group
Run Chart of Influenza Vaccine Administration Rates Stratified by Intervention, Concurrent Control, and Historical Control Group
Table 1.  Demographic Characteristics and Outcome, Process, and Balance Measures of Patients in the Intervention, Concurrent Control, and Historical Control Groups
Demographic Characteristics and Outcome, Process, and Balance Measures of Patients in the Intervention, Concurrent Control, and Historical Control Groups
Table 2.  Factors Associated With Influenza Vaccination Among Hospitalized Children
Factors Associated With Influenza Vaccination Among Hospitalized Children
1.
Rolfes  MA, Foppa  IM, Garg  S,  et al.  Annual estimates of the burden of seasonal influenza in the United States: a tool for strengthening influenza surveillance and preparedness.   Influenza Other Respir Viruses. 2018;12(1):132-137. doi:10.1111/irv.12486PubMedGoogle ScholarCrossref
2.
Rolfes  MA, Flannery  B, Chung  JR,  et al; US Influenza Vaccine Effectiveness (Flu VE) Network, the Influenza Hospitalization Surveillance Network, and the Assessment Branch, Immunization Services Division, Centers for Disease Control and Prevention.  Effects of influenza vaccination in the United States during the 2017-2018 influenza season.   Clin Infect Dis. 2019;69(11):1845-1853. doi:10.1093/cid/ciz075PubMedGoogle ScholarCrossref
3.
Centers for Disease Control and Prevention. Influenza vaccination coverage for persons 6 months and older. Accessed June 11, 2021. https://www.cdc.gov/flu/fluvaxview/interactive-general-population.htm
4.
Rao  S, Williams  JT, Torok  MR, Cunningham  MA, Glodè  MP, Wilson  KM.  Missed opportunities for influenza vaccination among hospitalized children with influenza at a tertiary care facility.   Hosp Pediatr. 2016;6(9):513-519. doi:10.1542/hpeds.2015-0112PubMedGoogle ScholarCrossref
5.
Pollack  AH, Kronman  MP, Zhou  C, Zerr  DM.  Automated screening of hospitalized children for influenza vaccination.   J Pediatric Infect Dis Soc. 2014;3(1):7-14. doi:10.1093/jpids/pit044PubMedGoogle ScholarCrossref
6.
Foradori  DM, Sampayo  EM, Fanny  SA, Namireddy  MK, Kumar  AM, Lo  HY.  Improving influenza vaccination in hospitalized children with asthma.   Pediatrics. 2020;145(3):145. doi:10.1542/peds.2019-1735PubMedGoogle ScholarCrossref
7.
Howell  EM, Kenney  GM.  The impact of the Medicaid/CHIP expansions on children: a synthesis of the evidence.   Med Care Res Rev. 2012;69(4):372-396. doi:10.1177/1077558712437245PubMedGoogle ScholarCrossref
8.
Fieldston  ES, Zaniletti I, Hall M,  et al. Community household income and resource utilization for common inpatient pediatric conditions.  Pediatrics. 2013;132(6):e1592-e1601. doi:10.1542/peds.2013-0619PubMed
9.
Elia  S, Perrett  K, Newall  F.  Providing opportunistic immunisations for at-risk inpatients in a tertiary paediatric hospital.   J Spec Pediatr Nurs. 2017;22(1):e12167. doi:10.1111/jspn.12167PubMedGoogle Scholar
10.
Shingler  S, Hunter  K, Romano  A, Graham  D.  Opportunities taken: the need for and effectiveness of secondary care opportunistic immunisation.   J Paediatr Child Health. 2012;48(3):242-246. doi:10.1111/j.1440-1754.2011.02231.xPubMedGoogle ScholarCrossref
11.
Grohskopf  LA, Alyanak  E, Broder  KR, Walter  EB, Fry  AM, Jernigan  DB.  Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2019-20 influenza season.   MMWR Recomm Rep. 2019;68(3):1-21. doi:10.15585/mmwr.rr6803a1PubMedGoogle ScholarCrossref
12.
Szilagyi  PG, Rodewald  LE.  Missed opportunities for influenza vaccination among children with asthma.   Pediatr Infect Dis J. 1992;11(9):705-708. doi:10.1097/00006454-199209000-00006PubMedGoogle ScholarCrossref
13.
Sokol  RL, Grummon  AH.  Parental vaccine hesitancy and childhood influenza vaccination.   Pediatrics. 2020;146(6):e2020022871. doi:10.1542/peds.2020-022871PubMedGoogle Scholar
14.
Mihalek  AJ, Kysh  L, Pannaraj  PS.  Pediatric inpatient immunizations: a literature review.   Hosp Pediatr. 2019;9(7):550-559. doi:10.1542/hpeds.2019-0026PubMedGoogle ScholarCrossref
15.
Rao  S, Fischman  V, Kaplan  DW, Wilson  KM, Hyman  D.  Evaluating interventions to increase influenza vaccination rates among pediatric inpatients.   Pediatr Qual Saf. 2018;3(5):e102. doi:10.1097/pq9.0000000000000102PubMedGoogle Scholar
16.
Rao  S, Ziniel  SI, Khan  I, Dempsey  A.  Be influential: evaluation of a multifaceted intervention to increase influenza vaccination rates among pediatric inpatients.   Vaccine. 2020;38(6):1370-1377. doi:10.1016/j.vaccine.2019.12.010PubMedGoogle ScholarCrossref
17.
Freedman  JL, Reilly  AF, Powell  SC, Bailey  LC.  Quality improvement initiative to increase influenza vaccination in pediatric cancer patients.   Pediatrics. 2015;135(2):e540-e546. doi:10.1542/peds.2014-0943PubMedGoogle ScholarCrossref
18.
Opel  DJ, Mangione-Smith  R, Robinson  JD,  et al.  The influence of provider communication behaviors on parental vaccine acceptance and visit experience.   Am J Public Health. 2015;105(10):1998-2004. doi:10.2105/AJPH.2014.302425PubMedGoogle ScholarCrossref
19.
Hofstetter  AM, Robinson  JD, Lepere  K, Cunningham  M, Etsekson  N, Opel  DJ.  Clinician-parent discussions about influenza vaccination of children and their association with vaccine acceptance.   Vaccine. 2017;35(20):2709-2715. doi:10.1016/j.vaccine.2017.03.077PubMedGoogle ScholarCrossref
20.
Fiks  AG, Hunter  KF, Localio  AR,  et al.  Impact of electronic health record-based alerts on influenza vaccination for children with asthma.   Pediatrics. 2009;124(1):159-169. doi:10.1542/peds.2008-2823PubMedGoogle ScholarCrossref
21.
Sittig  DF, Teich  JM, Osheroff  JA, Singh  H.  Improving clinical quality indicators through electronic health records: it takes more than just a reminder.   Pediatrics. 2009;124(1):375-377. doi:10.1542/peds.2009-0339PubMedGoogle ScholarCrossref
22.
Roshanov  PS, Fernandes  N, Wilczynski  JM,  et al.  Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials.   BMJ. 2013;346:f657. doi:10.1136/bmj.f657PubMedGoogle ScholarCrossref
23.
Sahota  N, Lloyd  R, Ramakrishna  A,  et al; CCDSS Systematic Review Team.  Computerized clinical decision support systems for acute care management: a decision-maker–researcher partnership systematic review of effects on process of care and patient outcomes.   Implement Sci. 2011;6(91):91. doi:10.1186/1748-5908-6-91PubMedGoogle Scholar
24.
Souza  NM, Sebaldt  RJ, Mackay  JA,  et al; CCDSS Systematic Review Team.  Computerized clinical decision support systems for primary preventive care: a decision-maker–researcher partnership systematic review of effects on process of care and patient outcomes.   Implement Sci. 2011;6(87):87. doi:10.1186/1748-5908-6-87PubMedGoogle Scholar
25.
Roshanov  PS, Misra  S, Gerstein  HC,  et al; CCDSS Systematic Review Team.  Computerized clinical decision support systems for chronic disease management: a decision-maker–researcher partnership systematic review.   Implement Sci. 2011;6(92):92. doi:10.1186/1748-5908-6-92PubMedGoogle Scholar
26.
Dexter  PR, Perkins  S, Overhage  JM, Maharry  K, Kohler  RB, McDonald  CJ.  A computerized reminder system to increase the use of preventive care for hospitalized patients.   N Engl J Med. 2001;345(13):965-970. doi:10.1056/NEJMsa010181PubMedGoogle ScholarCrossref
27.
Kim  S, Hughes  CA, Sadowski  CA.  A review of acute care interventions to improve inpatient pneumococcal vaccination.   Prev Med. 2014;67:119-127. doi:10.1016/j.ypmed.2014.07.015PubMedGoogle ScholarCrossref
28.
Birmingham  E, Catallozzi  M, Findley  SE, Vawdrey  DK, Kukafka  R, Stockwell  MS.  FluAlert: a qualitative evaluation of providers’ desired characteristics and concerns regarding computerized influenza vaccination alerts.   Prev Med. 2011;52(3-4):274-277. doi:10.1016/j.ypmed.2011.01.008PubMedGoogle ScholarCrossref
29.
Ogrinc  G, Davies L, Goodman D,  et al. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process.  BMJ Qual Saf. 2016;25(12):986-992. doi:10.1136/bmjqs-2015-004411PubMed
30.
Singh  H, Sittig  DF.  A sociotechnical framework for safety-related electronic health record research reporting: the SAFER Reporting Framework.   Ann Intern Med. 2020;172(11)(suppl):S92-S100. doi:10.7326/M19-0879PubMedGoogle Scholar
31.
Zhang  J, Walji  MF.  TURF: toward a unified framework of EHR usability.   J Biomed Inform. 2011;44(6):1056-1067. doi:10.1016/j.jbi.2011.08.005PubMedGoogle ScholarCrossref
32.
Kubiet  MA, Gonzalez-Rothi  RJ, Cottey  R, Bender  BS.  Serum antibody response to influenza vaccine in pulmonary patients receiving corticosteroids.   Chest. 1996;110(2):367-370. doi:10.1378/chest.110.2.367PubMedGoogle ScholarCrossref
33.
Park  CL, Frank  AL, Sullivan  M, Jindal  P, Baxter  BD.  Influenza vaccination of children during acute asthma exacerbation and concurrent prednisone therapy.   Pediatrics. 1996;98(2, pt 1):196-200.PubMedGoogle Scholar
34.
Grohskopf  LA, Alyanak  E, Broder  KR, Walter  EB, Fry  AM, Jernigan  DB.  Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2019-20 influenza season.   MMWR Recomm Rep. 2019;68(3):1-21. doi:10.15585/mmwr.rr6803a1PubMedGoogle ScholarCrossref
35.
Opel  DJ, Zhou  C, Robinson  JD,  et al.  Impact of childhood vaccine discussion format over time on immunization status.   Acad Pediatr. 2018;18(4):430-436. doi:10.1016/j.acap.2017.12.009PubMedGoogle ScholarCrossref
36.
Hofstetter  AM, Simon  TD, Lepere  K,  et al.  Parental vaccine hesitancy and declination of influenza vaccination among hospitalized children.   Hosp Pediatr. 2018;8(10):628-635. doi:10.1542/hpeds.2018-0025PubMedGoogle ScholarCrossref
37.
Feudtner  C, Christakis  DA, Connell  FA. Pediatric deaths attributable to complex chronic conditions: a population-based study of Washington State, 1980-1997.  Pediatrics. 2000;106(1, pt 2):205-209. PubMed
38.
Feudtner  C, Feinstein  JA, Zhong  W, Hall  M, Dai  D.  Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation.   BMC Pediatr. 2014;14:199. doi:10.1186/1471-2431-14-199PubMedGoogle ScholarCrossref
39.
Bates  D, Mächler  M, Bolker  B, Walker  S.  Fitting linear mixed-effects models using lme4.   J Stat Softw. 2015;67:1-48. doi:10.18637/jss.v067.i01Google ScholarCrossref
40.
R Core Team. R: A language and Environment for Statistical Computing. R Group for Statistical Computing; 2019.
41.
Benartzi  S, Peleg  E, Thaler  RH. Choice architecture and retirement saving plans. Accessed June 11, 2021. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.146.1421&rep=rep1&type=pdf
42.
Johnson  EJ, Goldstein  DG. Decisions by default. In: Shafir E, ed. The Behavioral Foundations of Public Policy. Princeton University Press; 2013:417-427.
43.
Rao  S, Fischman V, Moss A,  et al.  Exploring provider and parental perceptions to influenza vaccination in the inpatient setting.   Influenza Other Respir Viruses. 2018;12(3):416-420. doi:10.1111/irv.12482PubMedGoogle Scholar
44.
Shcherbatykh  I, Holbrook  A, Thabane  L, Dolovich  L; COMPETE III investigators.  Methodologic issues in health informatics trials: the complexities of complex interventions.   J Am Med Inform Assoc. 2008;15(5):575-580. doi:10.1197/jamia.M2518PubMedGoogle ScholarCrossref
45.
Stockwell  MS, Kharbanda EO, Martinez RA, Vargas CY, Vawdrey DK, Camargo S. Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial.  JAMA. 2012;307(16):1702-1708. doi:10.1001/jama.2012.502PubMed
46.
Kumar  D, Hemmige  V, Kallen  MA, Street  RL  Jr, Giordano  TP, Arya  M.  The role of text messages in patient-physician communication about the influenza vaccine.   J Mob Technol Med. 2018;7(2):55-59. doi:10.7309/jmtm.7.2.8PubMedGoogle ScholarCrossref
47.
Milkman  KL, Beshears  J, Choi  JJ, Laibson  D, Madrian  BC.  Using implementation intentions prompts to enhance influenza vaccination rates.   Proc Natl Acad Sci U S A. 2011;108(26):10415-10420. doi:10.1073/pnas.1103170108PubMedGoogle ScholarCrossref
48.
Milkman  KL, Patel MS, Gandhi L,  et al  A mega-study of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment.  Rotman School of Management Working Paper No. 3780267. Accessed June 7, 2021. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3780267
49.
Ralston  SL, Brady  PW, Kemper  AR.  Do we really need scholarly quality improvement?   JAMA Pediatr. 2019;173(5):413-414. doi:10.1001/jamapediatrics.2019.0067PubMedGoogle ScholarCrossref
50.
Sun  GH, MacEachern  MP, Perla  RJ, Gaines  JM, Davis  MM, Shrank  WH.  Health care quality improvement publication trends.   Am J Med Qual. 2014;29(5):403-407. doi:10.1177/1062860613503708PubMedGoogle ScholarCrossref
51.
Ellenberg  SS.  The stepped-wedge clinical trial: evaluation by rolling deployment.   JAMA. 2018;319(6):607-608. doi:10.1001/jama.2017.21993PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 1,602
    Citations 0
    Original Investigation
    Health Informatics
    July 22, 2021

    Evaluation of a Clinical Decision Support Strategy to Increase Seasonal Influenza Vaccination Among Hospitalized Children Before Inpatient Discharge

    Author Affiliations
    • 1Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia
    • 2Division of Hospital Medicine, Children’s Healthcare of Atlanta, Atlanta, Georgia
    • 3Information Services and Technology, Children’s Healthcare of Atlanta, Atlanta, Georgia
    • 4Department of Pediatrics, Washington University in St Louis, St Louis, Missouri
    • 5Division of Emergency Medicine, Children’s Healthcare of Atlanta, Atlanta, Georgia
    • 6Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, Georgia
    • 7Division of Infectious Diseases, Children’s Healthcare of Atlanta, Atlanta, Georgia
    • 8Division of Nephrology, Children’s Healthcare of Atlanta, Atlanta, Georgia
    JAMA Netw Open. 2021;4(7):e2117809. doi:10.1001/jamanetworkopen.2021.17809
    Key Points

    Question  Is a clinical decision support (CDS) strategy associated with improved influenza vaccination rates before discharge among eligible hospitalized children?

    Findings  In this quality improvement study, the combinination of a default-checked influenza vaccine order in admission order sets for eligible patients with a nursing script using a presumptive strategy to offer the vaccine was associated with significantly higher odds of the hospitalized child receiving the influenza vaccine compared with concurrent and historical controls.

    Meaning  This study suggests that a user-centered CDS strategy may improve vaccination rates among vulnerable, hospitalized children.

    Abstract

    Importance  Hospitalized children are at increased risk of influenza-related complications, yet influenza vaccine coverage remains low among this group. Evidence-based strategies about vaccination of vulnerable children during all health care visits are especially important during the COVID-19 pandemic.

    Objective  To design and evaluate a clinical decision support (CDS) strategy to increase the proportion of eligible hospitalized children who receive a seasonal influenza vaccine prior to inpatient discharge.

    Design, Setting, and Participants  This quality improvement study was conducted among children eligible for the seasonal influenza vaccine who were hospitalized in a tertiary pediatric health system providing care to more than half a million patients annually in 3 hospitals. The study used a sequential crossover design from control to intervention and compared hospitalizations in the intervention group (2019-2020 season with the use of an intervention order set) with concurrent controls (2019-2020 season without use of an intervention order set) and historical controls (2018-2019 season with use of an order set that underwent intervention during the 2019-2020 season).

    Interventions  A CDS intervention was developed through a user-centered design process, including (1) placing a default influenza vaccine order into admission order sets for eligible patients, (2) a script to offer the vaccine using a presumptive strategy, and (3) just-in-time education for clinicians addressing vaccine eligibility in the influenza order group with links to further reference material. The intervention was rolled out in a stepwise fashion during the 2019-2020 influenza season.

    Main Outcomes and Measures  Proportion of eligible hospitalizations in which 1 or more influenza vaccines were administered prior to discharge.

    Results  Among 17 740 hospitalizations (9295 boys [52%]), the mean (SD) age was 8.0 (6.0) years, and the patients were predominantly Black (n = 8943 [50%]) or White (n = 7559 [43%]) and mostly had public insurance (n = 11 274 [64%]). There were 10 997 hospitalizations eligible for the influenza vaccine in the 2019-2020 season. Of these, 5449 (50%) were in the intervention group, and 5548 (50%) were concurrent controls. There were 6743 eligible hospitalizations in 2018-2019 that served as historical controls. Vaccine administration rates were 31% (n = 1676) in the intervention group, 19% (n = 1051) in concurrent controls, and 14% (n = 912) in historical controls (P < .001). In adjusted analyses, the odds of receiving the influenza vaccine were 3.25 (95% CI, 2.94-3.59) times higher in the intervention group and 1.28 (95% CI, 1.15-1.42) times higher in concurrent controls than in historical controls.

    Conclusions and Relevance  This quality improvement study suggests that user-centered CDS may be associated with significantly improved influenza vaccination rates among hospitalized children. Stepwise implementation of CDS interventions was a practical method that was used to increase quality improvement rigor through comparison with historical and concurrent controls.

    Introduction

    Influenza vaccination prevents millions of health care visits and thousands of influenza-related deaths every year.1,2 Despite its effectiveness, vaccine coverage rates were only 63% in US children in the 2018-2019 influenza season.3 Hospitalized children often have high-risk conditions predisposing them to influenza morbidity and mortality, yet influenza vaccine coverage rates in this group range from 33% to 59% in published studies, lower than the national average.4-6 Children from lower-income households are more likely to be hospitalized and are less likely to have access to medical homes,7,8 which is associated with similar discrepancies in other routine vaccinations.9,10 Most children admitted to a hospital with influenza have had 1 or more missed opportunities for vaccination prior to their hospitalization.4 The Centers for Disease Control and Prevention Advisory Committee on Immunization Practices recommends vaccination during hospitalizations in addition to routine health care visits.11 However, influenza vaccination status is ascertained less often at acute-care visits and is often not considered a high priority.12 The importance of identifying strategies to identify and vaccinate vulnerable children has increased in the setting of the COVID-19 pandemic.13

    Several studies describe strategies to improve influenza vaccine administration among hospitalized children.6,14 Pollack et al5 demonstrated that a nursing-focused screening tool that was triggered for eligible patients and that placed an influenza vaccine order in the background of the patient’s medical record significantly improved influenza vaccine administration rates (adjusted odds ratio [OR], 6.77; 95% CI, 6.14-7.47). However, the absolute vaccination rate increased only from 2.1% to 8.0%, leaving substantial room for improvement. Rao et al15 found that a weekly, manually generated email reminder to clinical teams indicating the influenza vaccination status of each patient improved ordering practices. In a subsequent study, Rao et al16 expanded on this intervention to include educational modules, huddles, and reminders for nurses as well as vaccination lists in the electronic health record (EHR) and financial incentives for residents; this expansion was associated with a 1.39 times higher odds of a child being vaccinated against influenza before discharge. Freedman et al17 incorporated influenza vaccine orders into oncology admission order sets along with family education and clinic staff reports of vaccine uptake in outpatient settings. Their combined efforts were associated with increased influenza immunization rates among oncology patients from 20% to 65%. A quality improvement study of children with asthma admitted to a pediatric hospital medicine service combining a nurse-driven protocol with an alert for physicians, changes to the asthma history and physical note template, and substantial educational efforts was associated with increased influenza immunization rates from 13% to 57% over 4 years.6 However, most of these interventions required ongoing maintenance, and the association of each intervention bundle element with the outcome was unclear.

    Communication about vaccines using presumptive communication strategies is known to improve influenza vaccine uptake in ambulatory settings.18,19 Alerts for influenza vaccine eligibility with a default order have also modestly improved vaccine uptake in primary care settings, although the effect size may not be sufficient owing to workflow issues and problems addressing clinician and patient acceptance.20,21 However, it remains unknown whether these strategies are used in hospital settings, what their effectiveness is at promoting influenza vaccine uptake, or what the necessary adaptations are for the inpatient setting.

    Clinical decision support (CDS) embedded in the EHR can deliver patient-specific recommendations to encourage evidence-based practices. However, the association of CDS with process measures and patient outcomes has been inconsistent,22 particularly in promoting health maintenance interventions in acute care settings.23-26 A review of 35 studies found that the association of CDS with promoting inpatient pneumococcal vaccine in adults varied widely.27 A qualitative study of physicians’ desired characteristics for influenza vaccine alerts noted that CDS needed to show up early during a patient visit, automatically identify patients eligible for the influenza vaccine, facilitate vaccine ordering, and generate appropriate documentation.28 We hypothesize that a CDS system incorporating these characteristics and iteratively designed through formative usability testing with frontline clinicians would improve influenza vaccination rates among eligible hospitalized children. The aim of this study was to increase the proportion of eligible hospitalized children who receive the influenza vaccine prior to discharge.

    Methods
    Ethical Considerations and Reporting Guidelines

    This study was determined to be non–human participants research as a local quality improvement study by the Children’s Healthcare of Atlanta institutional review board. This study is reported according to the Standards for Quality Improvement Reporting Excellence (SQUIRE) 2.0 reporting guideline for quality improvement reports and the Safety-related EHR Research (SAFER) Reporting Framework for safety-related EHR interventions.29,30

    Setting

    This study was performed in an urban pediatric health system including 3 freestanding children’s hospitals: 1 academic tertiary care center (hospital A), 1 community tertiary care center (hospital B), and 1 academic secondary care center (hospital C). Pediatric residents are frequently involved in inpatient care at hospitals A and C but less often at hospital B. All 3 sites are live on a common EHR instance of Epic Systems. Influenza vaccines were available at all 3 sites generally from September through April of each year. At the beginning of influenza season each year, a special grand rounds was devoted to reviewing influenza burden, new guidelines, and promoting the influenza vaccine.

    Prior to the intervention, a noninterruptive alert in the discharge navigator appeared for patients aged 6 months or older with no record of influenza vaccine in the local database or state immunization registry for the current influenza season (eFigure 1 in the Supplement). During the baseline period (September 1, 2018, to May 1, 2019), this alert was ignored 94% of the time that it appeared (23 632 of 25 276), with an influenza vaccine order placed only 2% of the time (409 of 25 276). Of the 1609 influenza vaccine orders placed in the baseline period, 409 (25%) were placed through this noninterruptive alert, with the rest placed as ad hoc orders.

    Interventions

    We performed a user and task analysis31 through informal interviews of key stakeholders in the influenza vaccination process, including general medical ward nurses, pediatric residents, pediatric hospital medicine attending physicians, and pharmacists, to identify barriers to vaccine administration and inform a suite of EHR interventions addressing those barriers (eTable 1 in the Supplement). Patients and families were not interviewed in the development of this intervention.

    Nursing Admission Questionnaire Adjustments

    As part of the admission process, nursing staff members ask a series of questions of all parents. In the immunization section of the nursing administration questionnaire, we added questions with scripting designed to present the influenza vaccine using a presumptive communication strategy (eFigure 2B in the Supplement).

    Influenza Vaccine Order Group

    We developed an order group including a default-checked influenza vaccine order automatically timed for 12 pm the day after admission, just-in-time education regarding influenza vaccine appropriateness for specific populations (eg, patients receiving corticosteroids, patients with asthma exacerbations, patients with egg allergy, patients with cancer, and other immunocompromised patients), as well as links to the state immunization registry, Centers for Disease Control and Prevention guidance, and supporting literature (eFigure 2A in the Supplement).32-34 The order group would dynamically appear if the patient met the following criteria: (1) aged 6 months or older, (2) no influenza vaccine in our local EHR system or the state immunization registry for the current influenza season, (3) no history of anaphylaxis to any influenza vaccine in the local EHR system, and (4) no documentation by nursing staff indicating that the patient has already received influenza vaccine, has had an anaphylactic reaction, or parental refusal (see the Nursing Admission Questionnaire Adjustments subsection). If the nursing admission questionnaire was not filled out when the clinician accessed the orders and all other eligibility criteria were met, then the default-checked influenza vaccine order would show up in the admission order set. The clinician could manually unselect the influenza vaccine order from the order group if they chose. The order group was added sequentially to specific admission order sets as described in the Study of the Interventions subsection. The order group did not appear for patients 8 years of age or younger who had received 1 dose of influenza vaccine in the current season but had not received the influenza vaccine in any prior season and therefore should receive a second dose at least 4 weeks after the first dose.

    Communication Tip Sheet

    Our influenza vaccine working group, including infectious diseases specialists, worked with our marketing and parent and family advocacy groups to develop a tip sheet focused on (1) the benefits associated with a presumptive method for introducing vaccines18,35,36; (2) responses to common issues raised by families, such as “my kids have never had the flu,” “I got the flu shot once and it gave me the flu,” or “[the influenza vaccine] doesn’t work”; and (3) vaccine facts, such as adverse effects and time to protection. We included a link to this tip sheet in the influenza vaccine order, which also appeared in the medication administration activity for nursing staff, but it was not immediately available in the nursing admission questionnaire.

    A description of all 3 interventions was provided at educational sessions for nurses and residents and at an EHR physician oversight committee meeting prior to implementation. It was reviewed again with clinical department leaders prior to each stepwise implementation.

    Interventions

    We implemented the influenza vaccine order group using a sequential crossover design from control to intervention by clustering at the order set level (Figure 1; eTable 2 in the Supplement). On September 19, 2019, the order group was added to the general pediatrics admission order set restricted to hospital A only (phase 1). Two weeks later on October 3, 2019, the same order group was added to the general pediatrics admission order set at hospital B; it was also added to other commonly used disease-specific and general admission order sets on the general pediatrics, critical care, hematology and oncology, gastroenterology, and neurology services at hospitals A and B (phase 2A). On October 29, 2019, the order group was added to additional order sets for the pulmonary, gastroenterology, and hematology and oncology services at hospitals A and B (phase 2B). Finally, on November 19, 2019, the order group was added to general pediatrics and hematology admission order sets at hospital C (phase 3). Although the plan for sequential expansion was devised by the investigators, the actual order of expansion was not randomized or predetermined by the investigators. Rather, the order group was implemented initially on the general pediatrics service at hospital A for convenience (because this was the clinical practice area for one of us [E.W.O.]), and then pilot data were emailed to clinical leaders of each hospital service. Subsequent prioritization of the intervention rollout was based on email responsiveness, the chance to demonstrate the intervention and pilot data to clinical leaders, and consideration of constraints from the information technology department. The nursing admission questionnaire adjustments and communication tip sheet were implemented on September 19, 2019, across the health system.

    We defined eligible hospitalization with the first 3 criteria used for the influenza vaccine order group (see the Interventions subsection). To keep the cohort consistent between the seasons, hospitalizations were included even if the nurse documented that the patient had already received the vaccine elsewhere or the parent refused the vaccine because this documentation was performed inconsistently in the 2018-2019 season. We classified each eligible hospitalization as part of the intervention group if at least 1 order was placed using an order set that included the influenza order group (although this order did not have to be for the influenza vaccine). Concurrent controls were defined as eligible hospitalizations during the 2019-2020 influenza season (September 1, 2019, to May 1, 2020) in which no orders were placed from any order set that included the influenza order group. Historical controls were defined as eligible hospitalizations during the 2018-2019 influenza season (September 1, 2018, to May 1, 2019) in which at least 1 order was placed using an order set that subsequently underwent an intervention during the following influenza season—for example, if a clinician placed admission orders using the general pediatrics admission order set in the 2018-2019 influenza season (which, at the time, did not contain the influenza vaccine order group), then that hospitalized patient would count as a historical control.

    Race/ethnicity was assessed in this study to look for disparities in vaccine uptake. Race/ethnicity was assigned based on what was documented in the EHR, which is generally based on patient and family self-report from a set of discrete choices during the registration process. The number of pediatric complex chronic conditions37,38 was determined from active diagnoses in the problem list in the EHR at the time of discharge.

    Measures

    Our primary outcome measure was the proportion of eligible hospitalizations with at least 1 dose of influenza vaccine administered prior to discharge. Our process measure was the proportion of eligible hospitalizations in which an influenza vaccine order was placed prior to discharge.

    Statistical Analysis

    Demographic characteristics and outcome, process, and balance measures were compared between the intervention group, concurrent controls, and historical controls. Continuous variables with normal distribution were compared across all 3 intervention groups using 1-way analysis of variance and nonnormal variables with the Kruskal-Wallis test. Categorical variables were compared using the χ2 test. The null hypothesis for these tests was that the distribution of the dependent variable was the same across all 3 intervention groups. All P values were from 2-sided tests and results were deemed statistically significant at P < .05.

    In addition, we evaluated the difference in influenza vaccine administration rates among eligible hospitalizations in the intervention group, concurrent control group, and historical control group visually using run charts and analytically using mixed-effects logistic regression adjusting for age, sex, race/ethnicity, insurance status, and week of the influenza season. The unit of analysis was the hospitalization, so patient and hospital were added as random effects to account for individual patients with multiple hospitalizations during the influenza season as well as workflow differences between the hospitals. Statistical analyses were performed using R, version 3.6.1 (R Group for Statistical Computing) with the lme4 package.39,40

    Results

    Among 17 740 hospitalizations (9295 boys [52%]), the mean (SD) age was 8.0 (6.0) years, and the patients were predominantly Black (n = 8943 [50%]) or White (n = 7559 [43%]) and mostly had public insurance (n = 11 274 [64%]) (Table 1). In the 2019-2020 influenza season, there were 10 997 hospitalizations meeting the eligibility criteria. An order set containing the dynamic influenza vaccine order group was used for 5449 patients (50%) composing the intervention group, leaving 5548 hospitalizations (50%) in the concurrent control group. In the 2018-2019 influenza season, there were 6743 eligible hospitalizations after nursing screening in which an order was placed using an order set that subsequently underwent an intervention—these hospitalizations composed the historical control group. The intervention group and the historical control group had a significantly greater proportion of female, Black, and non-Hispanic patients, as well as patients with public insurance, compared with the concurrent control group, although the magnitude of these differences was not large. The intervention group had fewer patients with complex chronic conditions (n = 1699 [31%]) compared with the historical control group (n = 2372 [35%]; P < .001) and the concurrent control group (n = 2038 [37%]; P < .001 compared with the intervention group and P = .08 compared with historical control group).

    The influenza vaccine was ordered and administered most frequently in the intervention group, followed by the concurrent control group and the historical control group (Figure 2). Of the 5449 hospitalizations in the intervention group, an influenza vaccine was ordered for 4199 hospitalizations (77%), and the vaccine was administered during 1676 hospitalizations (31%). In the concurrent control group, the influenza vaccine was ordered for 1488 hospitalizations (27%; P < .001 compared with the intervention group) and administered during 1051 hospitalizations (19%; P < .001 compared with the intervention group). Among historical controls, the vaccine was ordered for 1024 hospitalizations (15%; P < .001 compared with both the intervention and concurrent control groups) and administered during 912 hospitalizations (14%; P < .001 compared with both the intervention and concurrent control groups).

    In univariate analysis, vaccine administration was most strongly associated with the intervention group (crude OR, 3.10; 95% CI, 2.81-3.41) and was also associated with the concurrent control group (crude OR, 1.56; 95% CI, 1.41-1.72) compared with historical controls (Table 2). Vaccine administration was also significantly more common earlier in the influenza season, among age groups outside 13 to 17 years, for patients of unknown race, for Hispanic patients, and for those with public insurance.

    In multivariable analysis, the intervention was associated with a 3.25 (95% CI, 2.94-3.59) times higher odds of having the vaccine administered compared with historical controls (Table 2). Concurrent controls had a slightly higher odds (adjusted OR, 1.28, 95% CI, 1.15-1.42) of vaccine administration than historical controls. The week of the influenza season was also significantly associated with vaccine administration, with each additional week into the season associated with a 3.7% reduction in the odds of receiving the influenza vaccine (adjusted OR, 0.96; 95% CI, 0.96-0.97). Adolescents were less likely to receive the influenza vaccine, with 13- to 17-year-old adolescents having 28% lower odds compared with 6- to 23-month-old children (adjusted OR, 0.72; 95% CI, 0.64-0.81). No other demographic characteristics, including sex, race/ethnicity, insurance, or complex chronic conditions, were significantly associated with receipt of vaccine in the multivariable model.

    SAFER Reporting Framework

    We reviewed the 8 sociotechnical dimensions of patient safety for EHR interventions and described preintervention issues, what sociotechnical changes were made, why they were felt to be effective, and how they could be applied in other settings (eTable 3 in the Supplement). The most noteworthy dimensions likely required for disseminating this intervention included the following: (1) hardware and software with an interface to the state immunization registry to improve the positive predictive value of the CDS, (2) human-computer interface using a default-checked influenza vaccine order in commonly used admission order sets, and (3) workflow and communication, where timing of the influenza vaccine order for 12 pm the day after admission improved nursing flexibility and reduced concern that vaccines may be administered without the physician team having any chance to review appropriateness.

    Discussion

    This quality improvement study demonstrated that CDS was associated with increased influenza vaccine uptake among hospitalized children compared with concurrent and historical controls. The CDS was designed through analysis of a complex sociotechnical system yielding the following key features: (1) automated detection of influenza vaccine eligibility, leading to default-checked vaccine orders at admission; (2) a nursing staff script using a presumptive strategy for offering the vaccine; and (3) just-in-time education regarding appropriate contraindications and evidence-based vaccine communication strategies. These interventions stress the use of defaults as a behavioral economic strategy to influence behavior,41,42 including targeted defaults for clinicians to order the influenza vaccine as well as a nursing staff script emphasizing a default option for families of patients receiving the influenza vaccine during the hospitalization.

    The CDS system was associated with a marked increase in the frequency of influenza vaccine orders for eligible children but with a smaller increase in actual administrations of the vaccine. This gap between orders and administrations may be due to vaccine refusal36,43 or other unknown barriers. Although the effect size was smaller, the influenza vaccination rate among the concurrent controls was significantly higher than among the historical controls. This difference may be due to the presumptive strategy script in the nursing admission questionnaire and communication tip sheet, which were implemented systemwide in September 2019, unlike the influenza vaccine order group, which was implemented in a stepwise fashion. It may also be due to contamination,44 in which the CDS led physicians to think of influenza vaccine more often even without a patient-specific EHR prompt.

    Compared with prior studies of the influenza vaccine for hospitalized children, this approach aimed to reduce clinician burden by (1) automating eligibility screening as much as possible through integration with the state immunization registry, (2) providing CDS early in the workflow to maximize nursing staff flexibility for time of administration, and (3) avoiding interruptive alerts for nurses and physicians. The CDS system also did not depend on operational resources to disseminate reports or provide ongoing education. Additional benefits may be gained by combining an automated CDS approach with other evidence-based implementation science and behavioral economics interventions, such as audit and feedback, text messages to patients and families, explicit markers of vaccine intention, note template changes, and patient and family education.6,15,45-48 Nonetheless, scaling up these interventions would benefit from study designs that identify each bundle element to minimize the resource requirements for new sites to implement similar changes.

    The implementation strategy described in this study using sequential crossover from control to intervention guided by operational constraints is practical and adds scientific rigor to CDS for quality improvement. From an operational perspective, stepwise implementations are associated with a reduced burden of change management and allow learning to be incorporated before expanding an intervention. For example, early in our first pilot on the general pediatrics service, several pharmacists noticed that influenza vaccine orders that were not administered within 12 hours of their due time would fall off the medication administration record for nurses, reducing their administration flexibility. We were able to fix this by changing the frequency setting in the EHR prior to the subsequent rollout of the intervention. From a scientific perspective, multiple comparisons with concurrent and historical controls are associated with improved internal validity of conclusions compared with simple pre-post quality improvement studies.49,50 Stepwise implementation guided by operational needs may still be biased because clusters that adopt an intervention early may be systematically different than later adopters, and learnings incorporated into the intervention may make the comparison of the intervention and control less clean. Stepped-wedge trials that randomize which clusters implement an intervention at each step may reduce this bias, but such trials are more costly and challenging to administer.51 We propose that operationally guided stepwise implementation combined with analysis comparing intervention groups with both concurrent and historical controls balances the goals of scientific rigor with operational feasibility more effectively than common pre-post designs.

    Limitations

    This study has some limitations. This is a single-center study using 1 EHR from Epic Systems. Conclusions from this study may depend on organizational culture prior to the intervention, change management structures, the priority of influenza vaccination among key stakeholders, technical infrastructure, and other factors that reduce its generalizability. Although we have attempted to capture important sociotechnical elements using the SAFER Reporting Framework, these assessments are likely not comprehensive, and other factors may be associated with different outcomes when applied in other health systems. In addition, this study demonstrated the effectiveness of our intervention in only a single influenza season. The stepwise implementation was not randomized; thus, early adopters of the intervention may have had greater enthusiasm for influenza vaccination than later adopters, which could bias our results. The reasons for low influenza vaccine uptake were not systematically evaluated. Finally, in the intervention group, there remained a large fraction of patients who were not vaccinated, leaving substantial room for improvement through other evidence-based interventions.

    Conclusions

    This study suggests that user-centered CDS may improve the rate of influenza vaccine administration during hospitalization for eligible children. Key elements to promote vaccine administration in acute care settings include automated determination of vaccine eligibility early in the hospitalization combined with default-checking the influenza vaccine order, providing a nursing staff script that uses a presumptive strategy to offer the vaccine, and just-in-time education regarding contraindications. Stepwise implementation of CDS interventions using sequential crossover from control to intervention is operationally feasible and improves the scientific rigor of quality improvement studies. Future studies are needed to assess whether the benefits in influenza vaccine administration are sustained across seasons, whether similar results are seen when applied to other institutions, and whether this intervention model can scale to other health maintenance interventions in acute care settings. Despite the improvements in influenza vaccine uptake demonstrated in this study, there remains a substantial gap between patients eligible for influenza vaccine and actual administrations, even in the intervention group. Additional work to understand persistent reasons for low uptake and the potential effect of combining CDS with other behavioral economic and implementation science interventions would likely reduce the burden of influenza in a vulnerable population and provide lessons to improve vaccine coverage for other diseases, such as COVID-19.

    Back to top
    Article Information

    Accepted for Publication: April 30, 2021.

    Published: July 22, 2021. doi:10.1001/jamanetworkopen.2021.17809

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Orenstein EW et al. JAMA Network Open.

    Corresponding Author: Evan W. Orenstein, MD, Department of Pediatrics, Emory University School of Medicine, 1405 Clifton Rd, Atlanta, GA 30322 (evan.orenstein@emory.edu).

    Author Contributions: Dr Orenstein had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Orenstein, ElSayed-Ali, Kandaswamy, Masterson, Blanco, Shah, Lantis, Dawson, Ray, Bryant, Iyer, Shane, Jernigan.

    Acquisition, analysis, or interpretation of data: Orenstein, Kandaswamy, Masterson, Kolwaite, Iyer, Jernigan.

    Drafting of the manuscript: Orenstein, Shah, Kolwaite.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Orenstein, Masterson, Kolwaite.

    Obtained funding: Orenstein.

    Administrative, technical, or material support: ElSayed-Ali, Masterson, Blanco, Shah, Lantis, Dawson, Ray, Bryant, Iyer, Jernigan.

    Supervision: Orenstein, Masterson, Shane, Jernigan.

    Conflict of Interest Disclosures: Dr Orenstein reported being the co-founder of and having equity in Phrase Health, a clinical decision support analytics company, outside the submitted work. Dr Shane reported being a voting member of the US Food and Drug Administration Vaccines and Related Biological Products Advisory Committee. No other disclosures were reported.

    Funding/Support: Drs Orenstein and Kandaswamy’s work was supported by the Agency for Healthcare Research and Quality award R03HS027689-01.

    Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: We would like to acknowledge the pediatric residents who performed usability testing for this intervention, the nurses and physicians who promoted the influenza vaccine, the information technology teams responsible for developing and maintaining clinical decision support, and most importantly our patients and their supportive families.

    References
    1.
    Rolfes  MA, Foppa  IM, Garg  S,  et al.  Annual estimates of the burden of seasonal influenza in the United States: a tool for strengthening influenza surveillance and preparedness.   Influenza Other Respir Viruses. 2018;12(1):132-137. doi:10.1111/irv.12486PubMedGoogle ScholarCrossref
    2.
    Rolfes  MA, Flannery  B, Chung  JR,  et al; US Influenza Vaccine Effectiveness (Flu VE) Network, the Influenza Hospitalization Surveillance Network, and the Assessment Branch, Immunization Services Division, Centers for Disease Control and Prevention.  Effects of influenza vaccination in the United States during the 2017-2018 influenza season.   Clin Infect Dis. 2019;69(11):1845-1853. doi:10.1093/cid/ciz075PubMedGoogle ScholarCrossref
    3.
    Centers for Disease Control and Prevention. Influenza vaccination coverage for persons 6 months and older. Accessed June 11, 2021. https://www.cdc.gov/flu/fluvaxview/interactive-general-population.htm
    4.
    Rao  S, Williams  JT, Torok  MR, Cunningham  MA, Glodè  MP, Wilson  KM.  Missed opportunities for influenza vaccination among hospitalized children with influenza at a tertiary care facility.   Hosp Pediatr. 2016;6(9):513-519. doi:10.1542/hpeds.2015-0112PubMedGoogle ScholarCrossref
    5.
    Pollack  AH, Kronman  MP, Zhou  C, Zerr  DM.  Automated screening of hospitalized children for influenza vaccination.   J Pediatric Infect Dis Soc. 2014;3(1):7-14. doi:10.1093/jpids/pit044PubMedGoogle ScholarCrossref
    6.
    Foradori  DM, Sampayo  EM, Fanny  SA, Namireddy  MK, Kumar  AM, Lo  HY.  Improving influenza vaccination in hospitalized children with asthma.   Pediatrics. 2020;145(3):145. doi:10.1542/peds.2019-1735PubMedGoogle ScholarCrossref
    7.
    Howell  EM, Kenney  GM.  The impact of the Medicaid/CHIP expansions on children: a synthesis of the evidence.   Med Care Res Rev. 2012;69(4):372-396. doi:10.1177/1077558712437245PubMedGoogle ScholarCrossref
    8.
    Fieldston  ES, Zaniletti I, Hall M,  et al. Community household income and resource utilization for common inpatient pediatric conditions.  Pediatrics. 2013;132(6):e1592-e1601. doi:10.1542/peds.2013-0619PubMed
    9.
    Elia  S, Perrett  K, Newall  F.  Providing opportunistic immunisations for at-risk inpatients in a tertiary paediatric hospital.   J Spec Pediatr Nurs. 2017;22(1):e12167. doi:10.1111/jspn.12167PubMedGoogle Scholar
    10.
    Shingler  S, Hunter  K, Romano  A, Graham  D.  Opportunities taken: the need for and effectiveness of secondary care opportunistic immunisation.   J Paediatr Child Health. 2012;48(3):242-246. doi:10.1111/j.1440-1754.2011.02231.xPubMedGoogle ScholarCrossref
    11.
    Grohskopf  LA, Alyanak  E, Broder  KR, Walter  EB, Fry  AM, Jernigan  DB.  Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2019-20 influenza season.   MMWR Recomm Rep. 2019;68(3):1-21. doi:10.15585/mmwr.rr6803a1PubMedGoogle ScholarCrossref
    12.
    Szilagyi  PG, Rodewald  LE.  Missed opportunities for influenza vaccination among children with asthma.   Pediatr Infect Dis J. 1992;11(9):705-708. doi:10.1097/00006454-199209000-00006PubMedGoogle ScholarCrossref
    13.
    Sokol  RL, Grummon  AH.  Parental vaccine hesitancy and childhood influenza vaccination.   Pediatrics. 2020;146(6):e2020022871. doi:10.1542/peds.2020-022871PubMedGoogle Scholar
    14.
    Mihalek  AJ, Kysh  L, Pannaraj  PS.  Pediatric inpatient immunizations: a literature review.   Hosp Pediatr. 2019;9(7):550-559. doi:10.1542/hpeds.2019-0026PubMedGoogle ScholarCrossref
    15.
    Rao  S, Fischman  V, Kaplan  DW, Wilson  KM, Hyman  D.  Evaluating interventions to increase influenza vaccination rates among pediatric inpatients.   Pediatr Qual Saf. 2018;3(5):e102. doi:10.1097/pq9.0000000000000102PubMedGoogle Scholar
    16.
    Rao  S, Ziniel  SI, Khan  I, Dempsey  A.  Be influential: evaluation of a multifaceted intervention to increase influenza vaccination rates among pediatric inpatients.   Vaccine. 2020;38(6):1370-1377. doi:10.1016/j.vaccine.2019.12.010PubMedGoogle ScholarCrossref
    17.
    Freedman  JL, Reilly  AF, Powell  SC, Bailey  LC.  Quality improvement initiative to increase influenza vaccination in pediatric cancer patients.   Pediatrics. 2015;135(2):e540-e546. doi:10.1542/peds.2014-0943PubMedGoogle ScholarCrossref
    18.
    Opel  DJ, Mangione-Smith  R, Robinson  JD,  et al.  The influence of provider communication behaviors on parental vaccine acceptance and visit experience.   Am J Public Health. 2015;105(10):1998-2004. doi:10.2105/AJPH.2014.302425PubMedGoogle ScholarCrossref
    19.
    Hofstetter  AM, Robinson  JD, Lepere  K, Cunningham  M, Etsekson  N, Opel  DJ.  Clinician-parent discussions about influenza vaccination of children and their association with vaccine acceptance.   Vaccine. 2017;35(20):2709-2715. doi:10.1016/j.vaccine.2017.03.077PubMedGoogle ScholarCrossref
    20.
    Fiks  AG, Hunter  KF, Localio  AR,  et al.  Impact of electronic health record-based alerts on influenza vaccination for children with asthma.   Pediatrics. 2009;124(1):159-169. doi:10.1542/peds.2008-2823PubMedGoogle ScholarCrossref
    21.
    Sittig  DF, Teich  JM, Osheroff  JA, Singh  H.  Improving clinical quality indicators through electronic health records: it takes more than just a reminder.   Pediatrics. 2009;124(1):375-377. doi:10.1542/peds.2009-0339PubMedGoogle ScholarCrossref
    22.
    Roshanov  PS, Fernandes  N, Wilczynski  JM,  et al.  Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials.   BMJ. 2013;346:f657. doi:10.1136/bmj.f657PubMedGoogle ScholarCrossref
    23.
    Sahota  N, Lloyd  R, Ramakrishna  A,  et al; CCDSS Systematic Review Team.  Computerized clinical decision support systems for acute care management: a decision-maker–researcher partnership systematic review of effects on process of care and patient outcomes.   Implement Sci. 2011;6(91):91. doi:10.1186/1748-5908-6-91PubMedGoogle Scholar
    24.
    Souza  NM, Sebaldt  RJ, Mackay  JA,  et al; CCDSS Systematic Review Team.  Computerized clinical decision support systems for primary preventive care: a decision-maker–researcher partnership systematic review of effects on process of care and patient outcomes.   Implement Sci. 2011;6(87):87. doi:10.1186/1748-5908-6-87PubMedGoogle Scholar
    25.
    Roshanov  PS, Misra  S, Gerstein  HC,  et al; CCDSS Systematic Review Team.  Computerized clinical decision support systems for chronic disease management: a decision-maker–researcher partnership systematic review.   Implement Sci. 2011;6(92):92. doi:10.1186/1748-5908-6-92PubMedGoogle Scholar
    26.
    Dexter  PR, Perkins  S, Overhage  JM, Maharry  K, Kohler  RB, McDonald  CJ.  A computerized reminder system to increase the use of preventive care for hospitalized patients.   N Engl J Med. 2001;345(13):965-970. doi:10.1056/NEJMsa010181PubMedGoogle ScholarCrossref
    27.
    Kim  S, Hughes  CA, Sadowski  CA.  A review of acute care interventions to improve inpatient pneumococcal vaccination.   Prev Med. 2014;67:119-127. doi:10.1016/j.ypmed.2014.07.015PubMedGoogle ScholarCrossref
    28.
    Birmingham  E, Catallozzi  M, Findley  SE, Vawdrey  DK, Kukafka  R, Stockwell  MS.  FluAlert: a qualitative evaluation of providers’ desired characteristics and concerns regarding computerized influenza vaccination alerts.   Prev Med. 2011;52(3-4):274-277. doi:10.1016/j.ypmed.2011.01.008PubMedGoogle ScholarCrossref
    29.
    Ogrinc  G, Davies L, Goodman D,  et al. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process.  BMJ Qual Saf. 2016;25(12):986-992. doi:10.1136/bmjqs-2015-004411PubMed
    30.
    Singh  H, Sittig  DF.  A sociotechnical framework for safety-related electronic health record research reporting: the SAFER Reporting Framework.   Ann Intern Med. 2020;172(11)(suppl):S92-S100. doi:10.7326/M19-0879PubMedGoogle Scholar
    31.
    Zhang  J, Walji  MF.  TURF: toward a unified framework of EHR usability.   J Biomed Inform. 2011;44(6):1056-1067. doi:10.1016/j.jbi.2011.08.005PubMedGoogle ScholarCrossref
    32.
    Kubiet  MA, Gonzalez-Rothi  RJ, Cottey  R, Bender  BS.  Serum antibody response to influenza vaccine in pulmonary patients receiving corticosteroids.   Chest. 1996;110(2):367-370. doi:10.1378/chest.110.2.367PubMedGoogle ScholarCrossref
    33.
    Park  CL, Frank  AL, Sullivan  M, Jindal  P, Baxter  BD.  Influenza vaccination of children during acute asthma exacerbation and concurrent prednisone therapy.   Pediatrics. 1996;98(2, pt 1):196-200.PubMedGoogle Scholar
    34.
    Grohskopf  LA, Alyanak  E, Broder  KR, Walter  EB, Fry  AM, Jernigan  DB.  Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices—United States, 2019-20 influenza season.   MMWR Recomm Rep. 2019;68(3):1-21. doi:10.15585/mmwr.rr6803a1PubMedGoogle ScholarCrossref
    35.
    Opel  DJ, Zhou  C, Robinson  JD,  et al.  Impact of childhood vaccine discussion format over time on immunization status.   Acad Pediatr. 2018;18(4):430-436. doi:10.1016/j.acap.2017.12.009PubMedGoogle ScholarCrossref
    36.
    Hofstetter  AM, Simon  TD, Lepere  K,  et al.  Parental vaccine hesitancy and declination of influenza vaccination among hospitalized children.   Hosp Pediatr. 2018;8(10):628-635. doi:10.1542/hpeds.2018-0025PubMedGoogle ScholarCrossref
    37.
    Feudtner  C, Christakis  DA, Connell  FA. Pediatric deaths attributable to complex chronic conditions: a population-based study of Washington State, 1980-1997.  Pediatrics. 2000;106(1, pt 2):205-209. PubMed
    38.
    Feudtner  C, Feinstein  JA, Zhong  W, Hall  M, Dai  D.  Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation.   BMC Pediatr. 2014;14:199. doi:10.1186/1471-2431-14-199PubMedGoogle ScholarCrossref
    39.
    Bates  D, Mächler  M, Bolker  B, Walker  S.  Fitting linear mixed-effects models using lme4.   J Stat Softw. 2015;67:1-48. doi:10.18637/jss.v067.i01Google ScholarCrossref
    40.
    R Core Team. R: A language and Environment for Statistical Computing. R Group for Statistical Computing; 2019.
    41.
    Benartzi  S, Peleg  E, Thaler  RH. Choice architecture and retirement saving plans. Accessed June 11, 2021. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.146.1421&rep=rep1&type=pdf
    42.
    Johnson  EJ, Goldstein  DG. Decisions by default. In: Shafir E, ed. The Behavioral Foundations of Public Policy. Princeton University Press; 2013:417-427.
    43.
    Rao  S, Fischman V, Moss A,  et al.  Exploring provider and parental perceptions to influenza vaccination in the inpatient setting.   Influenza Other Respir Viruses. 2018;12(3):416-420. doi:10.1111/irv.12482PubMedGoogle Scholar
    44.
    Shcherbatykh  I, Holbrook  A, Thabane  L, Dolovich  L; COMPETE III investigators.  Methodologic issues in health informatics trials: the complexities of complex interventions.   J Am Med Inform Assoc. 2008;15(5):575-580. doi:10.1197/jamia.M2518PubMedGoogle ScholarCrossref
    45.
    Stockwell  MS, Kharbanda EO, Martinez RA, Vargas CY, Vawdrey DK, Camargo S. Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial.  JAMA. 2012;307(16):1702-1708. doi:10.1001/jama.2012.502PubMed
    46.
    Kumar  D, Hemmige  V, Kallen  MA, Street  RL  Jr, Giordano  TP, Arya  M.  The role of text messages in patient-physician communication about the influenza vaccine.   J Mob Technol Med. 2018;7(2):55-59. doi:10.7309/jmtm.7.2.8PubMedGoogle ScholarCrossref
    47.
    Milkman  KL, Beshears  J, Choi  JJ, Laibson  D, Madrian  BC.  Using implementation intentions prompts to enhance influenza vaccination rates.   Proc Natl Acad Sci U S A. 2011;108(26):10415-10420. doi:10.1073/pnas.1103170108PubMedGoogle ScholarCrossref
    48.
    Milkman  KL, Patel MS, Gandhi L,  et al  A mega-study of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment.  Rotman School of Management Working Paper No. 3780267. Accessed June 7, 2021. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3780267
    49.
    Ralston  SL, Brady  PW, Kemper  AR.  Do we really need scholarly quality improvement?   JAMA Pediatr. 2019;173(5):413-414. doi:10.1001/jamapediatrics.2019.0067PubMedGoogle ScholarCrossref
    50.
    Sun  GH, MacEachern  MP, Perla  RJ, Gaines  JM, Davis  MM, Shrank  WH.  Health care quality improvement publication trends.   Am J Med Qual. 2014;29(5):403-407. doi:10.1177/1062860613503708PubMedGoogle ScholarCrossref
    51.
    Ellenberg  SS.  The stepped-wedge clinical trial: evaluation by rolling deployment.   JAMA. 2018;319(6):607-608. doi:10.1001/jama.2017.21993PubMedGoogle ScholarCrossref
    ×