There were 195 patients who were eligible according to the computer
system for both influenza and pneumococcal vaccines in the physician reminder
group and 168 patients who were eligible for both vaccines in the standing
Customize your JAMA Network experience by selecting one or more topics from the list below.
Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. Inpatient Computer-Based Standing Orders vs Physician Reminders to Increase Influenza and Pneumococcal Vaccination Rates: A Randomized Trial. JAMA. 2004;292(19):2366–2371. doi:https://doi.org/10.1001/jama.292.19.2366
Author Affiliations: Department of Medicine,
Wishard Memorial Hospital and Indiana University School of Medicine, Indianapolis
(Drs Dexter, Perkins, Jones, McDonald, and Ms Maharry); Regenstrief Institute
for Health Care, Indianapolis, Ind (Drs Dexter, Perkins, and McDonald); and
Richard L. Roudebush Veterans Affairs Medical Center, Indianapolis, Ind (Dr
Context Computerized reminder systems increase influenza and pneumococcal vaccination
rates, but computerized standing order systems have not been previously described
Objective To determine the effects of computerized physician standing orders compared
with physician reminders on inpatient vaccination rates.
Design, Setting, and Patients Randomized trial of 3777 general medicine patients discharged from 1
of 6 study wards during a 14-month period (November 1, 1998, through December
31, 1999) composed of 2 overlapping influenza seasons at an urban public teaching
Interventions The hospital’s computerized physician order entry system identified
inpatients eligible for influenza and pneumococcal vaccination. For patients
with standing orders, the system automatically produced vaccine orders directed
to nurses at the time of patient discharge. For patients with reminders, the
computer system provided reminders to physicians that included vaccine orders
during routine order entry sessions.
Main Outcome Measure Vaccine administration.
Results During the approximately 6 months of the influenza season, 50% of all
hospitalized patients were identified as eligible for influenza vaccination.
Twenty-two percent of patients hospitalized during the entire 14 months of
the study were found eligible for pneumococcal vaccination. Patients with
standing orders received an influenza vaccine significantly more often (42%)
than those patients with reminders (30%) (P <.001).
Patients with standing orders received a pneumococcal vaccine significantly
more often (51%) than those with reminders (31%) (P <.001).
Conclusions Computerized standing orders were more effective than computerized reminders
for increasing both influenza and pneumococcal vaccine administration. Our
findings suggest that computerized standing orders should be used more widely
for this purpose.
Few medical interventions rival influenza and pneumococcal vaccines
in their ability to reduce morbidity,1-8 save
costs,1,4 and save lives.1-3,7,8 Yet,
among individuals older than 65 years, as many as 34% have not received their
annual influenza vaccine and 38% have not received their annual pneumococcal
vaccine.9 To improve these vaccination rates,
many organizations recommend the use of both standing orders and computer-based
Manual standing order systems have been effective at increasing rates
among patients identified as eligible for vaccination.13-21 However,
these systems require the investment of personnel time to find eligible patients,
and consequently tend to have both limited reach and persistence. For example,
in one trial, nurses were able to assess fewer than 40% of inpatients for
vaccine eligibility at the beginning of the study, with a decline thereafter.20 In another study, the percentage of patients offered
vaccines across different inpatient wards varied more than 5-fold.22 Yet another trial within an emergency department
achieved a 74% vaccination rate among patients whom nurses screened, but nurses
were able to screen only 13% of the emergency department visits.13
In a previous study,23 we demonstrated
that computer reminders delivered to physicians increased inpatient ordering
rates of influenza and pneumococcal vaccine, but we were limited by physician
acceptance of the reminders. We hypothesized that by using the computer system
to do the screening for eligibility, we could reach more patients and achieve
higher rates of inpatient vaccination than computerized physician reminders.
To test this hypothesis, we compared the effects of computerized standing
orders for influenza and pneumococcal vaccines in a randomized trial using
the computerized physician reminder system as an active control.
We obtained approval for this trial from the institutional review board
at the Indiana University Medical Center, Indianapolis. We included all patients
discharged from the 6 general medicine wards of Wishard Memorial Hospital
from November 1, 1998, through December 31, 1999, a period covering 2 influenza
seasons. General medicine wards are covered by 8 physician teams. Each ward
is composed of 1 medical staff faculty member, 1 resident, and 2 interns.
Individual physicians are assigned to the 8 teams at monthly rotations. Most
vaccine ordering is performed by residents and interns. We randomized the
physicians to 4 standing order teams and 4 reminder teams. The assignment
of physicians to teams has been previously described.23,24 However,
we did not constrain the random assignment of physicians to the same team
status if they returned for more than 1 rotation. Admitted patients were assigned
to teams sequentially by the admitting office and assumed the status of the
team to which they were assigned.
We used the hospital’s Gopher physician order entry system25,26 and Gopher-Care rules27 to
identify patients who were eligible for vaccination and to deliver standing
orders or physician reminders.
Following national recommendations,28,29 the
computer system considered a patient eligible for vaccination if: (1) there
was no evidence of the vaccine being given during the required time frame;
(2) the patient had one of the relevant chronic diseases; or (3) the patient
was older than 65 years. The computer interventions for pneumococcal vaccine
were active throughout the study. For the influenza vaccine, computer system
inteventions were active during the influenza seasons (November 1, 1998, through
January 31, 1999, and October 1, 1999, through December 31, 1999).
For eligible patients in the standing order group, the system automatically
produced vaccine orders at the time of discharge. Different electronic input
forms were required for entering daily orders, admitting orders, and discharge
orders. For eligible patients in the physician reminder group, a pop-up message
appeared with orders for the required vaccines each time a physician began
a daily order-entry session during the first 5 days of hospitalization and
when they began a discharge order session at any time. A physician could accept
a suggested order with 1 keystroke. However, all order sessions eventually
required the physician to save the session with the F8 key and with the user’s
password. The computer system stopped sending pop-up reminder messages once
it received a vaccine order.
Prior to the study, the executive committee at Wishard Memorial Hospital
authorized nurses to administer vaccines in response to computer-generated
standing orders and approved the protocol under which they were dispensed.
All ward nurses were trained by nursing managers and were given printed protocols,
which included questions to ask the patient about egg allergies (for the influenza
vaccine), previous vaccination in the relevant time frame, and the patient’s
willingness to receive the vaccination. Nurses withheld the vaccination when
a patient was unable to answer the screening questions, reported prior vaccinations
or relevant allergies, or refused the vaccination. On the basis of the marked
benefits and safety of the vaccinations, the institutional review board waived
The computer screening rules for finding vaccine-eligible patients and
the questions that the nurses asked before dispensing physician-ordered vaccine
were identical in the reminder group and in the standing order group. In the
physician reminder group, no order was delivered to nurses without the physician’s
acceptance. In the standing order group, an order was produced and delivered
automatically in the system for all eligible patients.
The order entry system produces a cue to nurses about all new order
sessions by displaying a message on workstations located within the nursing
station. Nurses must log on to the computer system and click on each order
session to signify that they have removed a given set of orders. Vaccination
orders were delivered to the nurses via the same mechanisms for both vaccines
in the reminder and standing order groups.
We captured information about whether the nurse actually administered
each ordered vaccination in both branches of the study via a 2-item questionnaire
(Was the vaccination given? If not, why not?). Reminders to complete each
questionnaire also appeared on the nursing workstations. Nurses had to log
in with their user ID and private password to complete these questionnaires.
The computer system required an answer to the first question but not the second.
For the influenza vaccine, the nurse had to choose from 5 response options
for the second question (already received this year, patient refused, egg
allergy, patient unable to answer questions, or other reason). For the pneumococcal
vaccine, the response options were the same, except the time frame was “already
received in the last 10 years” and the “egg allergy” option
We obtained detailed information about the reminders delivered during
each order entry session, the immunizations ordered, the vaccines dispensed,
the reasons ordered vaccines were not dispensed from the Gopher order entry
system, the diagnosis related group (DRG), the length of stay and demographic
information (including race) from the hospital’s registration and case
abstract system, and information about adverse reactions to the 2 vaccines
from the pharmacy’s routine adverse event reporting system.
We used SAS statistical software (version 8.2, SAS Institute Inc, Cary,
NC) to perform all statistical analyses and considered P ≤.05 to be significant. The first eligible hospitalization was
identified for each patient and included for primary analyses. The first eligible
hospitalization was restricted to the specific wards in which the study took
place. Average ages were compared using t tests;
sex, race, and DRG distributions were compared using χ2 tests;
and average lengths of stays were compared using the Wilcoxon rank sum test.
For each of the vaccinations studied, we tabulated the number of patients
who were eligible for vaccination. For patients assigned to the physician
reminder group, we also counted the mean number of reminders displayed per
hospitalization and tabulated the percentages of eligible patients for whom
a physician reminder was responsible for influenza or pneumococcal vaccine
orders. Next, we compared the percentage of vaccinations administered among
eligible patients in the physician reminder group compared with the standing
order group using separate logistic regression models for influenza and for
pneumococcal vaccine (both with and without controlling for patient age, sex,
race, and physician team). Spontaneous orders were
defined as orders entered by physicians during order sessions, which were
not associated with a reminder or standing order. In the physician reminder
group, admission orders and daily orders after the fifth day of hospitalization
were not associated with reminders. Vaccination orders written during such
sessions were classified as spontaneous. In the standing order group, physicians
never received reminders; so all orders written by physicians for the study
vaccines were classified as spontaneous. We credited spontaneous orders to
their respective study groups.
We compared the percentage of vaccinations ordered but not administered
in both study groups using logistic regression models (with and without controlling
for patient age, sex, race, and physician team). Nurses reported the reasons
why they did not deliver ordered vaccines in both study groups. We compared
the reasons for not administering vaccines using χ2 tests.
Because the computer system did not know about all relevant vaccinations
given prior to hospitalization, it misidentified some patients as eligible,
but not as “truly eligible,” and inflated the denominators for
the calculated percentage effects. In the standing order group, we corrected
for the computer system’s overinclusion of patients by subtracting the
number of patients who reported that they had been vaccinated from the denominator
and recalculated all of the relevant percentages. We could not use this same
approach in the physician reminder group because nurses only asked the vaccine-related
questions in the subset of patients for whom physicians accepted the reminder
and ordered the vaccine. Instead, we assumed that the reminder group had the
same proportion of previously vaccinated patients as the standing order group
and adjusted the denominators by the same proportion. The corrected figures
are presented to provide better estimates of the magnitude of the effects
compared with other studies that did not depend on the computer system to
define their study population. We did not apply any significance tests to
the comparison of these adjusted figures between the groups.
Additional statistical analyses were performed that allowed for multiple
hospitalizations per patient using appropriate methods for handling correlated
data (including generalized estimating equation methods30 for
logistic regression). There were no substantive differences in the test of
the intervention effect whether we allowed for multiple hospitalizations,
controlled for covariates, or adjusted for the clustering within physician
team. For the sake of simplicity, we present only the results based on the
first eligible hospitalization without adjusting for covariates or physician
team, except where noted.
During the 14-month study period, a total of 3777 patients accounted
for 5410 discharges from the 6 eligible general medicine wards (Figure). Twenty-four percent of these patients had multiple admissions,
with a mean (SD) of 1.40 (1.05) admissions per patient. The mean (SD) age
of the patients on these wards was 53 (17) years; 52% were women; and 52%
were black. There were neither demographic differences between the 2 study
groups nor DRG distribution differences in the 2 patient groups when the 20
most frequently occurring DRGs were used (45% of the total) and the least
frequently occurring DRGs were merged into 1 category. The median (range)
lengths of stay for the first hospitalization were identical (3.0 [0.5-96.0]
days) between the 2 study groups.
A total of 212 physicians participated in the study during an average
of 1.9 one-month rotations at the hospital. Thirty-six percent of the physicians
(all residents) served on both reminder and standing order rotations. As has
been the case in previous reminder studies dating back to 1976,31,32 the
temporal direction of crossover from physician reminder to nonreminder had
no effect on the physician response rate in this study, and we consequently
ignored this distinction in the analysis.
During the influenza seasons, a total of 1706 patients accounted for
2369 discharges from general medicine wards (45% of all study patients and
44% of all discharges). There were no significant differences in mean age,
ethnicity, or sex between the 2 study groups in the influenza season subset,
or between the subset of patients who were admitted during the influenza season
and those admitted during the other 8 months of the study. No adverse reactions
to the immunizations dispensed during this study were reported.
During the influenza season, the computer system identified 848 patients
(50% of all patients hospitalized during these months) as eligible for influenza
vaccination. Of these, 385 were associated with standing order teams and 463
with physician reminder teams (Figure). Adjusting for the patients who reported
receiving the vaccine prior to hospitalization, 689 patients (313 from standing
order group and 376 from physician reminder group; 40%) hospitalized during
influenza season were truly eligible for influenza vaccination.
In the physician reminder group, physicians received a mean (SE) of
5.3 (0.2) flu shot reminders per patient hospitalization; 4.0 (0.2) reminders
in those cases when the physician eventually ordered the vaccine and 6.9 (0.4)
reminders when the vaccine was not ordered. Physicians tended to wait until
late in the hospital stay or until the discharge order to accept the reminders.
Physicians in the reminder group wrote influenza vaccination orders for 269
(58%) computer-eligible patients, 12 of which were spontaneous orders. In
the standing order group, by design, the computer system generated an influenza
vaccination order for 371 patients. In addition, 14 orders were spontaneous.
Because vaccines only provide benefit when administered, we used the
administration rates as our primary outcome measure. A total of 163 (42%)
eligible patients treated by physicians on the standing order team received
an influenza vaccine compared with 137 (30%) patients treated by physicians
on the reminder team (P <.001). Neither age, race,
nor sex was significant in the covariate-adjusted model. After correcting
the denominator for 19% of patients who reported to a nurse that they had
already received the proposed vaccine prior to hospitalization, the administration
rate became 52% of patients in the standing order group and 36% of patients
in the reminder group.
The computer system identified 829 patients (22% of patients hospitalized
during the 14-month study) as eligible for pneumococcal vaccination. Of these,
406 of those patients were associated with the standing order group and 423
with the reminder group (Figure). Adjusting for the patients who had received
the vaccine prior to hospitalization, 766 patients (375 were associated with
the standing order group and 391 were associated with the reminder group)
(20%) hospitalized during the study were truly eligible for pneumococcal vaccination.
The computer system displayed a pneumococcal vaccine reminder to physicians
a mean (SE) of 5.3 (0.2) times per hospitalization —3.6 (0.3) times
in cases when the physician eventually accepted the order and 6.8 (0.4) times
when he/she did not. Physicians assigned to the reminder group wrote pneumococcal
vaccine orders for 208 (49%) of eligible patients, 21 of which were spontaneous.
All patients in the standing order group received vaccination orders, 29 of
which were spontaneous and 377 of which were generated by standing orders.
Patients in the standing order group received pneumococcal vaccines
more often (209; 51%) than patients in the physician reminder group (132;
31%) (P <.001; Table). Age, race, and sex were not significant in the covariate-adjusted
model. Correcting for the 7% of patients who reported they had already received
the pneumococcal vaccine, the comparison between groups is 56% and 34%, respectively,
which is similar to the corresponding rates among patients truly eligible
for influenza vaccine.
Nurses withheld vaccines in a large percentage of cases based on the
patient’s response to the preadministration questionnaire: 222 (58%)
of influenza vaccine-eligible patients in the standing order group and 132
(49%) patients for whom the physician had ordered an influenza vaccine. This
effect was significant (P = .03) without
adjusting for physician group, but tipped to P = .07
when adjusting for physician group. Nurses withheld vaccines in 197 (49%)
of the pneumococcal vaccine–eligible standing order patients and in
76 (37%) patients for whom the physician had ordered pneumococcal vaccine
(the effect was P = .004 and after adjusting
for physician team was P = .007).
The nurses reported reasons for nonadministration in 98% of influenza
vaccine orders and pneumococcal vaccine orders. There was no significant difference
between the study groups in the distribution of these reasons for withholding
the vaccine. In the standing order group, a total of 32% of the nonadministered
influenza vaccines and 16% for the nonadministered pneumococcal vaccines were
due to the patient stating he/she had already received the vaccine prior to
hospitalization. We used these figures to adjust for the computer system’s
incomplete data about vaccine administration in our corrected estimates of
In the standing order group, patient refusal accounted for 42% of the
nonadministered influenza vaccines and 49% of the nonadministered pneumococcal
vaccines. Correcting for the patients who had already received the vaccine
unbeknownst to the computer system, those percentages become 63% of the nonadministered
influenza vaccines and 58% of the nonadministered pneumococcal vaccines. Five
percent of nonadministration of the influenza vaccine was explained by the
patient’s inability to respond to questions from the nurse, 2% by an
egg allergy, and another 18% by other reasons not further distinguished. The
pattern was similar for pneumococcal vaccine. Eight percent of patients could
not respond to questions from the nurse and another 28% provided other reasons
not further distinguished.
As argued previously,22,33 hospitalization
represents an important opportunity to deliver pneumococcal and influenza
vaccines to persons particularly likely to benefit from them. We found that
such opportunities for inhospital vaccination arise frequently. Adjusting
for those who had already received the vaccine, 40% of general medicine ward
patients hospitalized during the influenza seasons during our study were truly
eligible for the influenza vaccine and 20% hospitalized during our study were
truly eligible for the pneumococcal vaccine. The percentage eligible for influenza
vaccine would be even greater given the recent change in eligibility criterion
from age 65 years to age 50 years.10 We found
that 19% of the patients who were eligible according to the computer system
for influenza vaccine and 7% of those eligible for pneumococcal vaccine had
previously received the vaccine. Because institutional medical record systems
will typically have such gaps in information about the patient’s vaccination
history, additional data collection such as nurse preadministration questions
We have previously demonstrated that computerized physician reminders
can dramatically increase hospital vaccine ordering nearly 50-fold.23 In this study, we used a stricter measure of success:
the rates at which these vaccinations were actually administered. Standing
orders produced a higher rate of vaccine administration than physician reminders.
Of the patients eligible for influenza vaccine per the computer system’s
criteria, 42% received the vaccine in the standing order group compared with
30% in the reminder group (P<.001); for pneumococcal
vaccine, the values were 51% and 31%, respectively (P<.001).
Correcting for patients who received vaccines prior to hospitalization not
recorded in the computer system, 52% of the patients eligible to receive the
influenza vaccine actually received the vaccine in the standing order group
compared with 36% of the patients in the reminder group; for pneumococcal
vaccine, the values were 56% and 34%, respectively (P<.001).
Our study was performed at a single institution so we cannot be certain
that the results will be transferable to other institutions. However, owing
to the success of this study, the Gopher order entry system now generates
standing orders at discharge for eligible adult patients on all but the intensive
care wards. While our data are 5 years old, the fact that standing orders
outperform reminders could lead to a universal, contemporary way to address
many quality-of-care deficiencies.
It should not be surprising that standing orders were more effective
than physician reminders because we used identical computer processes to identify
eligible patients and nurses asked the patients identical preadministration
questions in both groups for both vaccines. The difference in the physician
reminder group was that the physician had to accept the computer system’s
suggestion before the vaccine could be offered to the patient; physicians
did not accept 42% of the reminders for influenza vaccine and 51% of the reminders
for pneumococcal vaccine.
The self-reported rates of influenza vaccine and pneumococcal vaccine
among those aged 65 years or older in the United States are approaching 66%
and 62%, respectively.9 However, these national
rates count any influenza vaccine given during the previous year and any pneumococcal
vaccine given during the patient’s lifetime. Among patients who had
not previously been vaccinated, our standing order intervention achieved rates
of 52% for the influenza vaccine and 56% for the pneumococcal vaccine within
a single hospital stay. In contrast to the results from other studies,34,35 we did not find any significant racial
disparities in the immunization rate in either vaccine or either branch of
Although the physician’s compliance with immunization reminders
increased compared with our previous study,23 compliance
was far from complete. Given that the patient acceptance of ordered vaccines
tended to be higher when immunization was stimulated by a physician order
than by a nurse standing order, some of this noncompliance was likely due
to a physician’s appreciation of patient factors or preferences. Based
on informal discussion with physicians, some of it was likely due to physician
hesitation to do anything that might delay the patient’s hospital discharge.
In the standing order group, 7% of the nonadministration of influenza
vaccine was due to egg allergies or to a patient’s inability (due to
mental state or other impairment) to agree to the vaccine. In these groups,
the nonadministration was appropriate. Information reported by the nurses
explained 18% (influenza vaccine) and 28% (pneumococcal vaccine) of the nonadministration
among truly eligible patients by “other reason” without further
detail. However, patient refusal accounted for the majority of vaccine nondelivery
among 63% of the truly eligible patients for influenza vaccine and 58% for
pneumococcal vaccine, which is likely due to the widespread negative attitudes
and misconceptions about vaccinations. In a 17 000 patient survey, nearly
one third of the patients thought the influenza vaccine caused influenza and/or
caused adverse events.35 To come close to the
Healthy People 2010 targets36 of 90% influenza
and pneumococcal vaccination rates for patients aged 65 years or older in
inner-city populations, the medical community will have to work hard to change
these negative beliefs.
Computer-assisted nurse standing orders improve inpatient immunization
rates more than physician reminders, and would be likely to have the same
advantage when applied to many kinds of preventive screening (eg, cholesterol,
cancer) in many care settings. Furthermore, their adoption faces fewer barriers
because they can be delivered as part of normal nursing workflow through the
order management systems used widely in health care institutions today. However,
physician vaccination reminders require delivery mechanisms (eg, physician
order entry systems) that do not yet exist in most institutions. Computer-assisted
standing orders could reduce the high rate of omissions documented by McGlynn
et al37 for many preventive care interventions.
Corresponding Author: Clement J. McDonald,
MD, Indiana University School of Medicine, 1050 Wishard Blvd, Indianapolis,
IN 46202 (firstname.lastname@example.org).
Author Contributions: Dr McDonald had full
access to all of the data in the study and takes responsibility for the integrity
of the data and the accuracy of the data analysis.
Study concept and design: McDonald, Dexter,
Acquisition of data: Dexter.
Analysis and interpretation of data: Dexter,
Perkins, McDonald, Maharry.
Drafting of the manuscript: Dexter, McDonald,
Critical revision of the manuscript for important
intellectual content: McDonald, Dexter, Perkins, Jones.
Statistical analysis: Perkins, Maharry, Dexter,
Obtained funding: McDonald.
Administrative, technical, or material support:
McDonald, Dexter, Jones.
Study supervision: Dexter.
Funding/Support:This work was supported in
part by grant HS07719 from the Agency for Healthcare Research and Quality,
as well as contracts N01-LM-6-3546 and N01-LM-9-3542 from the National Library
of Medicine, in part from the Regenstrief Foundation, and the Indiana Genomics
Initiative of Indiana University, which is supported in part by Lilly Endowment
Role of the Sponsors: The funding organizations
are federal agencies or nonprofit foundations. None had a role in the design
or conduct of the study. The required guidelines in the conduct of the study
from the National Institutes of Health were followed. The funding agencies
had no role in the collection, management, analysis, interpretation of the
data, or the preparation of the manuscipt. These agencies neither reviewed
nor approved the manuscript at any time.
Acknowledgment: We are indebted to the nurses,
physicians, students, and administration of Wishard Memorial Hospital, Indianapolis,
Ind, for their efforts and patience.
Create a personal account or sign in to: