Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, Vander Vliet M, Seger DL. Effect of Computerized Physician Order Entry and a Team Intervention on Prevention of Serious Medication Errors. JAMA. 1998;280(15):1311–1316. doi:10.1001/jama.280.15.1311
From the Division of General Medicine, Departments of Medicine (Drs Bates and Seger and Ms Vander Vliet), Emergency Medicine (Dr Teich), Quality Management (Drs Hickey and Kleefield), and Pharmacy (Dr Shea), Brigham and Women's Hospital; the Center for Applied Medical Information Systems Research, Partners Healthcare Systems and Harvard Medical School (Drs Bates and Teich); the Department of Anesthesiology, St Elizabeth's Medical Center (Dr Cullen); and the Department of Biostatistics (Dr Laird) and Health Policy and Management (Dr Leap and Ms Burdick), Harvard School of Public Health; and Health Services Research and Development, Brockton/West Roxbury Veterans Affairs Medical Center (Dr Petersen), Boston, Mass.
Context.— Adverse drug events (ADEs) are a significant and costly cause of injury
Objectives.— To evaluate the efficacy of 2 interventions for preventing nonintercepted
serious medication errors, defined as those that either resulted in or had
potential to result in an ADE and were not intercepted before reaching the
Design.— Before-after comparison between phase 1 (baseline) and phase 2 (after
intervention was implemented) and, within phase 2, a randomized comparison
between physican computer order entry (POE) and the combination of POE plus
a team intervention.
Setting.— Large tertiary care hospital.
Participants.— For the comparison of phase 1 and 2, all patients admitted to a stratified
random sample of 6 medical and surgical units in a tertiary care hospital
over a 6-month period, and for the randomized comparison during phase 2, all
patients admitted to the same units and 2 randomly selected additional units
over a subsequent 9-month period.
Interventions.— A physician computer order entry system (POE) for all units and a team-based
intervention that included changing the role of pharmacists, implemented for
half the units.
Main Outcome Measure.— Nonintercepted serious medication errors.
Results.— Comparing identical units between phases 1 and 2, nonintercepted serious
medication errors decreased 55%, from 10.7 events per 1000 patient-days to
4.86 events per 1000 (P=.01). The decline occurred
for all stages of the medication-use process. Preventable ADEs declined 17%
from 4.69 to 3.88 (P=.37), while nonintercepted potential
ADEs declined 84% from 5.99 to 0.98 per 1000 patient-days (P=.002). When POE-only was compared with the POE plus team intervention
combined, the team intervention conferred no additonal benefit over POE.
Conclusions.— Physician computer order entry decreased the rate of nonintercepted
serious medication errors by more than half, although this decrease was larger
for potential ADEs than for errors that actually resulted in an ADE.
HOSPITALS, health care delivery systems, and health care providers all
aim to provide the safest care possible. However, many injuries occur during
hospitalization. Estimates have suggested that 1.3 million injuries may occur
in the United States annually.1 Although many
hospital injuries are unpredictable and unavoidable, 20% to 70% may be preventable.2- 4
Adverse drug events (ADEs) are an important source of injuries. In the
Harvard Medical Practice Study,1,5
ADEs accounted for 19% of injuries in hospitalized patients and represented
the single most common cause of injury. In addition to their human costs,
ADEs are costly to health care systems. Nationally, ADEs occurring after hospitalization
have been projected to cost hospitals $2 billion per year, not including malpractice
costs or the costs of injuries to patients.6,7
Hospitalizations initiated by ADEs appear to be at least as expensive.8
These data suggest that health care organizations should be working
hard to prevent ADEs, and indeed a number of efforts to curb them have begun
recently. However, enthusiasm for these efforts has been hampered by a lack
of data proving that specific strategies can prevent ADEs.
Accordingly, we conducted an ADE prevention study, not only to evaluate
the frequency and types of ADEs9 but also to
analyze the associated errors using a multidisciplinary, systems-oriented
approach to understand their causes.10,11
We evaluated 2 interventions, the first targeting primarily the ordering
stage and the second targeting the administration and dispensing stages. The
first intervention was a physician computer order-entry (POE) system, in which
physicians wrote all orders online. The second intervention was a "team" intervention,
comprising several small interventions developed by teams of nurses, physicians,
and pharmacists. This intervention primarily targeted the administration and
dispensing of drugs. Our primary end point was to reduce the number of preventable
ADEs and nonintercepted potential ADEs, the combination of which we refer
to as nonintercepted serious medication errors. As secondary end points, we
evaluated the numbers of types of events that we expected each intervention
to target, for example, drug allergies for the POE intervention.
All adults admitted to study units at Brigham and Women's Hospital,
Boston, Mass, a 726-bed tertiary care hospital, during the study periods were
included. Phase 1, collection of baseline data for the units, took place over
a 6-month period from February through July 1993,9
while phase 2, the intervention phase, was conducted over a 9-month period
between October 1994 and July 1995. Although the original study was conducted
at 2 hospitals, only Brigham and Women's data are included herein because
staff changes and administrative changes at Massachusetts General Hospital,
Boston, resulted in inadequate data collection. This study was approved by
the institutional review boards of Brigham and Women's Hospital and the Harvard
School of Public Health, Boston.
In phase 1, conducted prior to the implementation of POE, there were
23 adult, nonobstetrical units at Brigham and Women's Hospital. These units
were stratified according to whether they were medical or surgical and whether
they were intensive care or general care units. Six study units were selected
randomly from all units within a stratum using a random number generator.
The study units included 2 intensive care units (ICUs) (1 surgical and 1 medical)
and 4 general care units (2 medical and 2 surgical). We implemented the 2
interventions (POE or POE plus team) in these same units for phase 2 and,
in addition, implemented the intervention in 2 other randomly selected general
care units (1 medical and 1 surgical) to increase power. Thus, for phase 2,
all units received POE, while half the units (randomly selected within stratum
by unit type and level of care) received the team intervention as well (Figure 1).
The 2 interventions evaluated were POE and POE plus team. Two main comparisons
were made. The first assessed the effectiveness of the interventions by comparing
phase 1 and phase 2 for the units assessed in both phases. The second compared
the effect of the team intervention with POE vs POE-only by comparing units
randomized to either arm.
The POE intervention represented a major systems change and included
many features that would be expected to reduce errors. As part of the POE
application, physicians were provided with a menu of medications from the
formulary and default doses and a range of potential doses for each medication.
Physicians were required to enter dosage, route, and frequency for all orders.
Also, computerization ensured that all orders were legible, including the
signatures of the prescribers. Transcription was greatly reduced, although
not eliminated, because the medication administration record was still on
paper. For a number of medications, relevant laboratory results were displayed
on the screen at the time of ordering (eg, potassium levels when furosemide
was ordered). Other features included consequent orders, which are orders
that should follow from other orders (eg, suggestions to perform aminoglycoside
levels when aminoglycosides were ordered), and limited drug-allergy checking,
drug-drug interaction checking, and drug-laboratory checking. This included
checking for the most frequent drug allergies, about 80 of the most serious
drug-drug interactions,12 and several drug-laboratory
combinations (eg, potassium levels in patients receiving potassium). More
comprehensive checking was implemented in 1996.13
Thus, not all features that could reduce the number of ADEs were in place
or present in mature form during the study.
The POE application14,15
functions as part of an internally developed information system, Brigham Integrated
Computing System, which manages the hospital's administrative, financial,
and clinical data.16 All orders are written
using this application, primarily by house officers, although fellows and
attending physicians also write orders. Approximately 16000 orders are written
daily, 40% of them for medications.
The team intervention was a combination of several specific process
changes and was implemented in half the study units selected at random. These
process changes included changing the role of the pharmacist; distributing
a recommended dilutions chart; making available a computerized drip–rate
calculation program; standardizing labeling of intravenous bags, tubes, and
pumps; and implementing a pharmacy communication log so that the nursing staff
could communicate better with the pharmacy staff. Changing the role of the
pharmacists involved changing their work flow so that the pharmacists were
much more often present on the unit and available for questions. The pharmacists
made daily rounds with the team in the study ICU but not in the control ICU.
Our primary end point was the number of nonintercepted serious medication
errors (preventable and nonintercepted potential ADEs). We define preventable
ADEs as those resulting from an error or having been preventable by any means
currently available.9 Potential ADEs are errors
that have potential for harm but do not result in injury. These include errors
that are intercepted before injury occurs and nonintercepted potential ADEs,
which are errors that by chance resulted in no injury (eg, penicillin given
to a patient with a known allergy, but no reaction occurred).9
We excluded intercepted potential ADEs from our primary outcome because these
errors serve to demonstrate that the "safety net" is working, and this rate
could actually increase as error-prevention systems are fine-tuned.
As secondary outcomes, we evaluated the numbers of errors in each stage
and also within specific categories that were targeted by each intervention.
For the POE plus team intervention, the main stages targeted were ordering
and transcription. For the team intervention, the main stages were the administration
and dispensing of drugs. However, we expected some crossover, ie, that ordering
errors would also be diminished by certain team interventions (eg, availability
of the pharmacist on physician rounds) and that the administration error rate
would be decreased by POE because orders would be clearer. Specific categories
of errors targeted by POE included wrong dose, known drug allergy, and drug-drug
interactions. Specific categories targeted by the team intervention included
errors in concentration of intravenous solutions, in calculation, and in labeling
of intravenous tubing, pumps, and bags.
We used the following 3 mechanisms for identifying incidents: (1) nurses
and pharmacists participating in the study and control units were asked to
report incidents to study investigators; (2) a study investigator visited
each unit at least twice daily on weekdays and solicited information from
nurses, pharmacists, and clerical personnel concerning all actual or potential
drug-related incidents; and (3) the study investigator reviewed charts of
all patients at least daily on weekdays.
All incidents were evaluated as previously described9
by 2 physician reviewers blinded to the intervention groups. The reviewers
classified each incident as an ADE, a potential ADE, or an exclusion. The
ADEs and potential ADEs were further classified by severity and by preventability.
If an error was present, the reviewer determined the type of error and at
what stage the error occurred. Categories of preventability were defined as
being definitely preventable, probably preventable, probably not preventable,
and definitely not preventable.17 Results were
collapsed into preventable (definitely + probably) and not preventable (probably
not + definitely not) in the analyses. Categories of severity were defined
as fatal, life-threatening, serious, and significant.18
The stages of the process were ordering, transcribing, dispensing, and administering
the medications. When there were disagreements about how an event should be
classified (eg, 1 reviewer scored it as preventable, but the other did not),
the reviewers met and reached consensus. If consensus could not be reached,
a third reviewer evaluated the incident. Reliability for these judgments has
previously been reported.9 For judgments about
whether an incident was an ADE, κ was 0.81 to 0.98; for preventability, κ
was 0.92; and for severity κ was 0.32 to 0.37.
For the before-after analysis, paired t tests
were used to compare baseline data (phase 1) from the 6 study units with data
after the interventions were implemented in the same units (Figure 1). To compare the effects of the 2 interventions (POE vs
POE plus team), we performed 2 additional analyses. First, to evaluate the
effect of the team intervention, we compared all phase 2 data for POE plus
team with POE-only units, using a multivariate regression model that included
level of care (ICU vs general care) and service (medical vs surgical). Second,
to compare POE vs no POE, we analyzed data from both phases when available
from all 8 units, adjusting for level of care and service status with a multivariate
regression model. All 8 units in phase 2 received the POE intervention, 4
also received the team intervention, and the 6 phase 1 units were controls
in this analysis. To account for correlation between phase 1 and phase 2 results
for specific units, we used a generalized estimating approach to estimate
SEs of the regression coefficients. Because the error rate climbed substantially
with sedatives in phase 2, which could have obscured an effect by stage, we
performed a secondary post hoc analysis removing all sedative errors and determining
how this would affect the results by stage. All analyses were performed using
SAS (SAS Institute Inc, Cary, NC).19
Phase 1 included 2491 admissions to 6 units over a 6-month period. Phase
2 included 4220 admissions to 8 units over a 9-month period (Table 1). Phase 2 patients were older, more often female, and more
often minority, although these differences were small.
In paired analyses comparing phase 1 and phase 2 (Table 2), the rate of nonintercepted serious medication errors fell
55%, from 10.7 events per 1000 patient-days to 4.86 events (P=.01). The preventable ADE rate was 17% lower in phase 2, although
this result did not achieve statistical significance (P=.37). The rate of nonintercepted potential ADEs fell 84% (P=.002), and the rate of intercepted potential ADEs decreased 58% between
the 2 phases, although this result was not statistically significant (P=.15). The rate of nonpreventable ADEs was unchanged at
11.3 events per 1000 patient-days in both phase 1 and phase 2. In unpaired
analyses comparing phases 1 and 2 (Table
3), the results were generally similar. When the team intervention
was controlled for in the analysis, the results did not change (results available
on request). Contemporaneous analyses comparing phase 2 POE plus team intervention
with POE-only units showed no significant differences for any of the event
The results by severity show that decreases were seen across all levels
of severity for nonintercepted serious medication errors (Table 4), and that the proportions of errors considered life-threatening,
serious, and significant remained relatively constant across the 2 phases.
However, for preventable ADEs, while the proportion that were life-threatening
remained similar, the proportion of ADEs that were classified as serious was
higher in phase 2 than in phase 1 (47% vs 22%), and the rate of serious preventable
ADEs was about twice as high (1.96 events vs 0.98 events per 1000 patient-days).
Of these serious ADEs, 16 (33%) of 48 were due to sedatives.
When the results were evaluated by steps in the process from ordering
to administering, the rate of ordering errors decreased 19% overall (P=.03, Table 5).
The number of transcription errors fell by 84% (P<.001).
The rates of dispensing and administration errors also fell between phases
1 and 2, 68% and 59%, respectively. Contemporaneous within-stage comparisons
showed no significant differences between the POE plus team intervention units
and the POE-only units.
Analyses of errors expected to be reduced by the POE intervention (Table 6) showed that dose errors decreased
23% (P=.02) and known allergy errors fell 56% (P=.009). Although drug-drug interaction errors fell 40%,
this result did not reach statistical significance (P=.89).
Analysis by drug group revealed that nonintercepted serious medication
error rates fell for all classes except for sedatives, which increased 99%
(Table 7). Most of these errors
involved use of large doses and multiple sedating agents, particularly in
the ICUs. Neither intervention had specifically addressed these issues during
the study period.
Overall, 134 serious medication errors were not prevented in phase 2.
To understand the reasons that some of these were missed, specifically for
categories that we expected to be virtually eliminated (such as allergy errors),
we evaluated these errors by stage. Analysis of the 81 ordering errors that
were not prevented in phase 2 revealed that 34 appeared to be potentially
preventable through one of a number of systems changes that have been subsequently
developed or that are being developed, another 12 should have been prevented
by systems already in place, and 35 did not appear to be amenable to an automated
Among the 34 errors potentially preventable through new approaches,
the strategies that would yield the best results appear to be from a program
currently being tested that adjusts doses for renal failure (7 events), adjusts
doses for age (7 events), offers additional drug-laboratory checking capabilities
(5 events), includes a dose-checking program (4 events), and makes additional
suggestions about prophylaxis for specific medications (4 events).
However, 12 events occurred in areas we expected the current systems
would have addressed effectively. Seven of these were allergy errors, 4 were
drug-drug interactions, and 1 was an error in electrolyte replacement. Further
investigation showed, in most instances, that clinicians often failed to enter
allergy information into the computer regularly when patients had had allergic
reactions during hospitalization, and the medications were ordered again for
the patients. Of the 114 new reactions that occurred during the study, we
found that allergy information was entered into the computer in only 18 cases
(16%). Sensitivities were entered less often than true allergies.
Among 35 ordering errors not preventable by an automated approach, oversedation
due to multiple sedating drugs accounted for 19 errors. This problem was much
more common in phase 2 than in phase 1.
All 5 of the transcription errors in phase 2 potentially could have
been prevented by placing medication administration information online. Of
the 7 dispensing errors, 3 might have been prevented had a bar code been used,
and 3 might have been prevented had automated delivery devices been used.
We also classified the 41 administration errors into those that could
have been prevented by automation (n=15) and those that were not obviously
amenable to an automated approach—judgment-related and technique-related
errors (n=26). Automated approaches that would address the greatest number
of problems identified were implementing automated delivery devices with a
bar code (n=8) and placing all medication administration information online
We found that using a POE system prevented more than half of the serious
medication errors. We noted a reduction in errors for all stages of the process.
These results suggest that implementing even a modest POE system can result
in important error reduction, if the system includes a dose selection menu,
simple drug-allergy and drug-drug checking, and the requirement that clinicians
indicate the route and frequency of drug doses. Furthermore, a computer system
resolves the difficulty of translating illegible orders and greatly reduces
the need for transcription.
However, potential ADEs were prevented out of proportion to those that
actually resulted in an ADE. While we had not expected the decrease in the
preventable ADE rate to reach statistical significance, we did hope that these
events would be decreased in proportion to the potential ADEs. That this did
not occur suggests that errors that actually cause injuries may be different,
and examination of the individual events bears this out. In particular, 42%
of the preventable ADEs that persisted in phase 2 were due to judgment errors
associated with the use of multiple sedating drugs. The computer program did
not address this issue.
Others have evaluated POE systems20 and
the use of computerized information in reducing the frequency of ADEs21 and improving care.22
Tierney et al20 found that implementation of
a POE system on a medical service resulted in a reduction in the average length-of-stay
days by 0.89 days and a 12.7% reduction in charges. Evans et al21
found that implementation of computerized ADE surveillance, coupled with alerts
to pharmacists about drug allergies, standardization of antibiotic administration
rates, and physician notification about ADEs, reduced ADE rates. In another
study, Evans et al22 found that in a computer-assisted
management program for antibiotics substantially decreased costs and improved
quality of care in an ICU, by reducing the number of times patients experienced
an allergic reaction to drugs and improving the appropriateness of drug dosing.
Our POE program is continuously being improved and, at the time of the
study, did not include all the decision support that could have been beneficial.
Other improvements that we expect will substantially reduce ADEs rates are
guided dose algorithms that suggest appropriate dosing for drugs, such as
aminoglycosides and heparin; drug-laboratory checking; and drug-patient characteristic
checking, which would include adjusting doses for renal failure and age.23 Other features, such as drug-allergy checking and
drug-drug interaction checking, have already been significantly refined since
the study period. Given the potential of these improvements, the point estimate
described in this study represents a lower bound for the efficacy of POE for
reducing ADE rates.
To evaluate the effect of POE, it would have been ideal to randomize
half of the units to POE while maintaining the remainder on a paper system.
However, this was not possible. Implementation of a POE system in which all
orders are written online is a systems change of the first magnitude, which
is very difficult to accomplish.24,25
Therefore, we relied on time-series comparisons.
No effect was seen for the team intervention, and there was actually
a trend toward a higher rate of serious medication errors, primarily at the
ordering stage, which was not the main target of the intervention. Our power
to detect a small effect for the team intervention was limited, and the impact
of the POE intervention was large enough that it could have obscured an incremental
benefit of the team intervention. One part of this intervention was to have
a pharmacist become a member of the clinical team and increase his or her
presence on the unit. In other studies, having pharmacists play a larger clinical
role has proved effective,26 and we think that
this and other components of the team intervention should be investigated
further. Our findings do illustrate the difficulty of linking process improvements
to reduction in ADE rates and suggest that small changes are likely to have
limited overall impact.
An important question is whether introduction of POE is cost-effective.
We previously estimated the annual costs of preventable ADEs at this hospital
to be $2.8 million7. In this study, we observed
a decrease in the preventable ADE rate of 17% (although this decrease was
not significant); if this were the hospital-wide decrease, the annual savings
would be $480000. This figure does not include the costs of injuries borne
by patients, of admissions due to drug errors, of malpractice suits, or of
the extra work generated by the nonserious medication errors. For our institution,
the costs of developing and implementing POE have been estimated to be $1.9
million, with maintenance costs of $500000 per year.27
The net savings have been estimated to be between $5 to $10 million per year.27 While these estimates are crude, they suggest that
POE not only improves the quality of care but it could save money.
We were surprised that the error rate did not fall as much for the ordering
stage as rates for other stages. One reason was the unexpected increase in
errors related to the use of sedating medications. After excluding these errors,
the percent difference from phase 1 to phase 2 was 34% rather than 17% but
was still smaller than the reductions seen for other stages.
The results for individual types of error give further insight into
the potential impact of future prevention strategies. The dosing error rate
fell 23%, probably because prescribers made selections from menus that showed
only appropriate alternatives. While the rate for serious allergy errors decreased
56%, there were still 7 such errors, caused by clinicians failing to enter
new information about allergies when reactions had occurred during the hospitalization.
We currently are pursuing an initiative that encourages physicians to enter
information about reactions that occur in the hospital. The issues regarding
sensitivities, such as a patient's need for specific premedication for amphotericin,
are more complex. We need to enable such information to travel with the patient,
so it is available to all prescribers at the appropriate times.
The number of drug-drug interactions fell with POE, but not to near-zero,
because the system at the time of the study was programmed to catch only the
most severe interactions, which come up infrequently. We have since introduced
a much more complete index of interactions.13
However, we have used many fewer interactions than most commercial databases
report, because most are not significant. An overload of information could
lead to indifference so that health care professionals might ignore serious
interactions, increasing the rate of important errors.28
This study has several limitations, some of which have been mentioned
above. The study took place at only 1 tertiary care hospital, so the results
may not be generalizable to other hospitals or health care settings. Because
the before-after comparisons showed improvement, a temporal trend could have
caused some of the differences seen. However, we found no evidence for an
underlying temporal trend, and the nonpreventable ADE rates remained the same
across the 2 phases. Also, as noted earlier, the POE intervention did not
include all the features that would be expected to make a difference; thus,
our estimate of effect probably represents a lower bound. In addition, the
study did not address omissions, drugs that should have been given but were
not. Finally, the κ between reviewers was lower for severity than for
other determinations in part because of the multiple categories involved.
This has also been the case in prior studies.9,29
We conclude that a POE system decreased the number of medication errors
with potential for harm by more than half. With additional improvements, even
further reductions should be possible. Refinement of this approach and introduction
of other systems changes in which more automation is brought to the drug dispensing
and administration stages should result in further decreases in the numbers
of nonintercepted serious medication errors and, in turn, the injuries they
cause. These data add to the body of knowledge suggesting that POE can reduce
costs and improve quality and suggest that hospitals should consider adopting