Effect of Computerized Physician Order Entry and a Team Intervention on Prevention of Serious Medication Errors

Context.— Adverse drug events (ADEs) are a significant and costly cause of injury during hospitalization.

Objectives.— To evaluate the efficacy of 2 interventions for preventing nonintercepted serious medication errors, defined as those that either resulted in or had potential to result in an ADE and were not intercepted before reaching the patient.

Design.— Before-after comparison between phase 1 (baseline) and phase 2 (after intervention was implemented) and, within phase 2, a randomized comparison between physican computer order entry (POE) and the combination of POE plus a team intervention.

Setting.— Large tertiary care hospital.

Participants.— For the comparison of phase 1 and 2, all patients admitted to a stratified random sample of 6 medical and surgical units in a tertiary care hospital over a 6-month period, and for the randomized comparison during phase 2, all patients admitted to the same units and 2 randomly selected additional units over a subsequent 9-month period.

Interventions.— A physician computer order entry system (POE) for all units and a team-based intervention that included changing the role of pharmacists, implemented for half the units.

Main Outcome Measure.— Nonintercepted serious medication errors.

Results.— Comparing identical units between phases 1 and 2, nonintercepted serious medication errors decreased 55%, from 10.7 events per 1000 patient-days to 4.86 events per 1000 (P=.01). The decline occurred for all stages of the medication-use process. Preventable ADEs declined 17% from 4.69 to 3.88 (P=.37), while nonintercepted potential ADEs declined 84% from 5.99 to 0.98 per 1000 patient-days (P=.002). When POE-only was compared with the POE plus team intervention combined, the team intervention conferred no additonal benefit over POE.

Conclusions.— Physician computer order entry decreased the rate of nonintercepted serious medication errors by more than half, although this decrease was larger for potential ADEs than for errors that actually resulted in an ADE.

HOSPITALS, health care delivery systems, and health care providers all aim to provide the safest care possible. However, many injuries occur during hospitalization. Estimates have suggested that 1.3 million injuries may occur in the United States annually.¹ Although many hospital injuries are unpredictable and unavoidable, 20% to 70% may be preventable.^2-4

Adverse drug events (ADEs) are an important source of injuries. In the Harvard Medical Practice Study,^1,5 ADEs accounted for 19% of injuries in hospitalized patients and represented the single most common cause of injury. In addition to their human costs, ADEs are costly to health care systems. Nationally, ADEs occurring after hospitalization have been projected to cost hospitals $2 billion per year, not including malpractice costs or the costs of injuries to patients.^6,7 Hospitalizations initiated by ADEs appear to be at least as expensive.⁸

These data suggest that health care organizations should be working hard to prevent ADEs, and indeed a number of efforts to curb them have begun recently. However, enthusiasm for these efforts has been hampered by a lack of data proving that specific strategies can prevent ADEs.

Accordingly, we conducted an ADE prevention study, not only to evaluate the frequency and types of ADEs⁹ but also to analyze the associated errors using a multidisciplinary, systems-oriented approach to understand their causes.^10,11

We evaluated 2 interventions, the first targeting primarily the ordering stage and the second targeting the administration and dispensing stages. The first intervention was a physician computer order-entry (POE) system, in which physicians wrote all orders online. The second intervention was a "team" intervention, comprising several small interventions developed by teams of nurses, physicians, and pharmacists. This intervention primarily targeted the administration and dispensing of drugs. Our primary end point was to reduce the number of preventable ADEs and nonintercepted potential ADEs, the combination of which we refer to as nonintercepted serious medication errors. As secondary end points, we evaluated the numbers of types of events that we expected each intervention to target, for example, drug allergies for the POE intervention.

All adults admitted to study units at Brigham and Women's Hospital, Boston, Mass, a 726-bed tertiary care hospital, during the study periods were included. Phase 1, collection of baseline data for the units, took place over a 6-month period from February through July 1993,⁹ while phase 2, the intervention phase, was conducted over a 9-month period between October 1994 and July 1995. Although the original study was conducted at 2 hospitals, only Brigham and Women's data are included herein because staff changes and administrative changes at Massachusetts General Hospital, Boston, resulted in inadequate data collection. This study was approved by the institutional review boards of Brigham and Women's Hospital and the Harvard School of Public Health, Boston.

In phase 1, conducted prior to the implementation of POE, there were 23 adult, nonobstetrical units at Brigham and Women's Hospital. These units were stratified according to whether they were medical or surgical and whether they were intensive care or general care units. Six study units were selected randomly from all units within a stratum using a random number generator. The study units included 2 intensive care units (ICUs) (1 surgical and 1 medical) and 4 general care units (2 medical and 2 surgical). We implemented the 2 interventions (POE or POE plus team) in these same units for phase 2 and, in addition, implemented the intervention in 2 other randomly selected general care units (1 medical and 1 surgical) to increase power. Thus, for phase 2, all units received POE, while half the units (randomly selected within stratum by unit type and level of care) received the team intervention as well (Figure 1).

The 2 interventions evaluated were POE and POE plus team. Two main comparisons were made. The first assessed the effectiveness of the interventions by comparing phase 1 and phase 2 for the units assessed in both phases. The second compared the effect of the team intervention with POE vs POE-only by comparing units randomized to either arm.

The POE intervention represented a major systems change and included many features that would be expected to reduce errors. As part of the POE application, physicians were provided with a menu of medications from the formulary and default doses and a range of potential doses for each medication. Physicians were required to enter dosage, route, and frequency for all orders. Also, computerization ensured that all orders were legible, including the signatures of the prescribers. Transcription was greatly reduced, although not eliminated, because the medication administration record was still on paper. For a number of medications, relevant laboratory results were displayed on the screen at the time of ordering (eg, potassium levels when furosemide was ordered). Other features included consequent orders, which are orders that should follow from other orders (eg, suggestions to perform aminoglycoside levels when aminoglycosides were ordered), and limited drug-allergy checking, drug-drug interaction checking, and drug-laboratory checking. This included checking for the most frequent drug allergies, about 80 of the most serious drug-drug interactions,¹² and several drug-laboratory combinations (eg, potassium levels in patients receiving potassium). More comprehensive checking was implemented in 1996.¹³ Thus, not all features that could reduce the number of ADEs were in place or present in mature form during the study.

The POE application^14,15 functions as part of an internally developed information system, Brigham Integrated Computing System, which manages the hospital's administrative, financial, and clinical data.¹⁶ All orders are written using this application, primarily by house officers, although fellows and attending physicians also write orders. Approximately 16000 orders are written daily, 40% of them for medications.

The team intervention was a combination of several specific process changes and was implemented in half the study units selected at random. These process changes included changing the role of the pharmacist; distributing a recommended dilutions chart; making available a computerized drip–rate calculation program; standardizing labeling of intravenous bags, tubes, and pumps; and implementing a pharmacy communication log so that the nursing staff could communicate better with the pharmacy staff. Changing the role of the pharmacists involved changing their work flow so that the pharmacists were much more often present on the unit and available for questions. The pharmacists made daily rounds with the team in the study ICU but not in the control ICU.

Our primary end point was the number of nonintercepted serious medication errors (preventable and nonintercepted potential ADEs). We define preventable ADEs as those resulting from an error or having been preventable by any means currently available.⁹ Potential ADEs are errors that have potential for harm but do not result in injury. These include errors that are intercepted before injury occurs and nonintercepted potential ADEs, which are errors that by chance resulted in no injury (eg, penicillin given to a patient with a known allergy, but no reaction occurred).⁹ We excluded intercepted potential ADEs from our primary outcome because these errors serve to demonstrate that the "safety net" is working, and this rate could actually increase as error-prevention systems are fine-tuned.

As secondary outcomes, we evaluated the numbers of errors in each stage and also within specific categories that were targeted by each intervention. For the POE plus team intervention, the main stages targeted were ordering and transcription. For the team intervention, the main stages were the administration and dispensing of drugs. However, we expected some crossover, ie, that ordering errors would also be diminished by certain team interventions (eg, availability of the pharmacist on physician rounds) and that the administration error rate would be decreased by POE because orders would be clearer. Specific categories of errors targeted by POE included wrong dose, known drug allergy, and drug-drug interactions. Specific categories targeted by the team intervention included errors in concentration of intravenous solutions, in calculation, and in labeling of intravenous tubing, pumps, and bags.

We used the following 3 mechanisms for identifying incidents: (1) nurses and pharmacists participating in the study and control units were asked to report incidents to study investigators; (2) a study investigator visited each unit at least twice daily on weekdays and solicited information from nurses, pharmacists, and clerical personnel concerning all actual or potential drug-related incidents; and (3) the study investigator reviewed charts of all patients at least daily on weekdays.

All incidents were evaluated as previously described⁹ by 2 physician reviewers blinded to the intervention groups. The reviewers classified each incident as an ADE, a potential ADE, or an exclusion. The ADEs and potential ADEs were further classified by severity and by preventability. If an error was present, the reviewer determined the type of error and at what stage the error occurred. Categories of preventability were defined as being definitely preventable, probably preventable, probably not preventable, and definitely not preventable.¹⁷ Results were collapsed into preventable (definitely + probably) and not preventable (probably not + definitely not) in the analyses. Categories of severity were defined as fatal, life-threatening, serious, and significant.¹⁸ The stages of the process were ordering, transcribing, dispensing, and administering the medications. When there were disagreements about how an event should be classified (eg, 1 reviewer scored it as preventable, but the other did not), the reviewers met and reached consensus. If consensus could not be reached, a third reviewer evaluated the incident. Reliability for these judgments has previously been reported.⁹ For judgments about whether an incident was an ADE, κ was 0.81 to 0.98; for preventability, κ was 0.92; and for severity κ was 0.32 to 0.37.

For the before-after analysis, paired t tests were used to compare baseline data (phase 1) from the 6 study units with data after the interventions were implemented in the same units (Figure 1). To compare the effects of the 2 interventions (POE vs POE plus team), we performed 2 additional analyses. First, to evaluate the effect of the team intervention, we compared all phase 2 data for POE plus team with POE-only units, using a multivariate regression model that included level of care (ICU vs general care) and service (medical vs surgical). Second, to compare POE vs no POE, we analyzed data from both phases when available from all 8 units, adjusting for level of care and service status with a multivariate regression model. All 8 units in phase 2 received the POE intervention, 4 also received the team intervention, and the 6 phase 1 units were controls in this analysis. To account for correlation between phase 1 and phase 2 results for specific units, we used a generalized estimating approach to estimate SEs of the regression coefficients. Because the error rate climbed substantially with sedatives in phase 2, which could have obscured an effect by stage, we performed a secondary post hoc analysis removing all sedative errors and determining how this would affect the results by stage. All analyses were performed using SAS (SAS Institute Inc, Cary, NC).¹⁹

Phase 1 included 2491 admissions to 6 units over a 6-month period. Phase 2 included 4220 admissions to 8 units over a 9-month period (Table 1). Phase 2 patients were older, more often female, and more often minority, although these differences were small.

In paired analyses comparing phase 1 and phase 2 (Table 2), the rate of nonintercepted serious medication errors fell 55%, from 10.7 events per 1000 patient-days to 4.86 events (P=.01). The preventable ADE rate was 17% lower in phase 2, although this result did not achieve statistical significance (P=.37). The rate of nonintercepted potential ADEs fell 84% (P=.002), and the rate of intercepted potential ADEs decreased 58% between the 2 phases, although this result was not statistically significant (P=.15). The rate of nonpreventable ADEs was unchanged at 11.3 events per 1000 patient-days in both phase 1 and phase 2. In unpaired analyses comparing phases 1 and 2 (Table 3), the results were generally similar. When the team intervention was controlled for in the analysis, the results did not change (results available on request). Contemporaneous analyses comparing phase 2 POE plus team intervention with POE-only units showed no significant differences for any of the event types.

The results by severity show that decreases were seen across all levels of severity for nonintercepted serious medication errors (Table 4), and that the proportions of errors considered life-threatening, serious, and significant remained relatively constant across the 2 phases. However, for preventable ADEs, while the proportion that were life-threatening remained similar, the proportion of ADEs that were classified as serious was higher in phase 2 than in phase 1 (47% vs 22%), and the rate of serious preventable ADEs was about twice as high (1.96 events vs 0.98 events per 1000 patient-days). Of these serious ADEs, 16 (33%) of 48 were due to sedatives.

When the results were evaluated by steps in the process from ordering to administering, the rate of ordering errors decreased 19% overall (P=.03, Table 5). The number of transcription errors fell by 84% (P<.001). The rates of dispensing and administration errors also fell between phases 1 and 2, 68% and 59%, respectively. Contemporaneous within-stage comparisons showed no significant differences between the POE plus team intervention units and the POE-only units.

Analyses of errors expected to be reduced by the POE intervention (Table 6) showed that dose errors decreased 23% (P=.02) and known allergy errors fell 56% (P=.009). Although drug-drug interaction errors fell 40%, this result did not reach statistical significance (P=.89).

Analysis by drug group revealed that nonintercepted serious medication error rates fell for all classes except for sedatives, which increased 99% (Table 7). Most of these errors involved use of large doses and multiple sedating agents, particularly in the ICUs. Neither intervention had specifically addressed these issues during the study period.

Overall, 134 serious medication errors were not prevented in phase 2. To understand the reasons that some of these were missed, specifically for categories that we expected to be virtually eliminated (such as allergy errors), we evaluated these errors by stage. Analysis of the 81 ordering errors that were not prevented in phase 2 revealed that 34 appeared to be potentially preventable through one of a number of systems changes that have been subsequently developed or that are being developed, another 12 should have been prevented by systems already in place, and 35 did not appear to be amenable to an automated approach.

Among the 34 errors potentially preventable through new approaches, the strategies that would yield the best results appear to be from a program currently being tested that adjusts doses for renal failure (7 events), adjusts doses for age (7 events), offers additional drug-laboratory checking capabilities (5 events), includes a dose-checking program (4 events), and makes additional suggestions about prophylaxis for specific medications (4 events).

However, 12 events occurred in areas we expected the current systems would have addressed effectively. Seven of these were allergy errors, 4 were drug-drug interactions, and 1 was an error in electrolyte replacement. Further investigation showed, in most instances, that clinicians often failed to enter allergy information into the computer regularly when patients had had allergic reactions during hospitalization, and the medications were ordered again for the patients. Of the 114 new reactions that occurred during the study, we found that allergy information was entered into the computer in only 18 cases (16%). Sensitivities were entered less often than true allergies.

Among 35 ordering errors not preventable by an automated approach, oversedation due to multiple sedating drugs accounted for 19 errors. This problem was much more common in phase 2 than in phase 1.

All 5 of the transcription errors in phase 2 potentially could have been prevented by placing medication administration information online. Of the 7 dispensing errors, 3 might have been prevented had a bar code been used, and 3 might have been prevented had automated delivery devices been used.

We also classified the 41 administration errors into those that could have been prevented by automation (n=15) and those that were not obviously amenable to an automated approach—judgment-related and technique-related errors (n=26). Automated approaches that would address the greatest number of problems identified were implementing automated delivery devices with a bar code (n=8) and placing all medication administration information online (n=5).

We found that using a POE system prevented more than half of the serious medication errors. We noted a reduction in errors for all stages of the process. These results suggest that implementing even a modest POE system can result in important error reduction, if the system includes a dose selection menu, simple drug-allergy and drug-drug checking, and the requirement that clinicians indicate the route and frequency of drug doses. Furthermore, a computer system resolves the difficulty of translating illegible orders and greatly reduces the need for transcription.

However, potential ADEs were prevented out of proportion to those that actually resulted in an ADE. While we had not expected the decrease in the preventable ADE rate to reach statistical significance, we did hope that these events would be decreased in proportion to the potential ADEs. That this did not occur suggests that errors that actually cause injuries may be different, and examination of the individual events bears this out. In particular, 42% of the preventable ADEs that persisted in phase 2 were due to judgment errors associated with the use of multiple sedating drugs. The computer program did not address this issue.

Others have evaluated POE systems²⁰ and the use of computerized information in reducing the frequency of ADEs²¹ and improving care.²² Tierney et al²⁰ found that implementation of a POE system on a medical service resulted in a reduction in the average length-of-stay days by 0.89 days and a 12.7% reduction in charges. Evans et al²¹ found that implementation of computerized ADE surveillance, coupled with alerts to pharmacists about drug allergies, standardization of antibiotic administration rates, and physician notification about ADEs, reduced ADE rates. In another study, Evans et al²² found that in a computer-assisted management program for antibiotics substantially decreased costs and improved quality of care in an ICU, by reducing the number of times patients experienced an allergic reaction to drugs and improving the appropriateness of drug dosing.

Our POE program is continuously being improved and, at the time of the study, did not include all the decision support that could have been beneficial. Other improvements that we expect will substantially reduce ADEs rates are guided dose algorithms that suggest appropriate dosing for drugs, such as aminoglycosides and heparin; drug-laboratory checking; and drug-patient characteristic checking, which would include adjusting doses for renal failure and age.²³ Other features, such as drug-allergy checking and drug-drug interaction checking, have already been significantly refined since the study period. Given the potential of these improvements, the point estimate described in this study represents a lower bound for the efficacy of POE for reducing ADE rates.

To evaluate the effect of POE, it would have been ideal to randomize half of the units to POE while maintaining the remainder on a paper system. However, this was not possible. Implementation of a POE system in which all orders are written online is a systems change of the first magnitude, which is very difficult to accomplish.^24,25 Therefore, we relied on time-series comparisons.

No effect was seen for the team intervention, and there was actually a trend toward a higher rate of serious medication errors, primarily at the ordering stage, which was not the main target of the intervention. Our power to detect a small effect for the team intervention was limited, and the impact of the POE intervention was large enough that it could have obscured an incremental benefit of the team intervention. One part of this intervention was to have a pharmacist become a member of the clinical team and increase his or her presence on the unit. In other studies, having pharmacists play a larger clinical role has proved effective,²⁶ and we think that this and other components of the team intervention should be investigated further. Our findings do illustrate the difficulty of linking process improvements to reduction in ADE rates and suggest that small changes are likely to have limited overall impact.

An important question is whether introduction of POE is cost-effective. We previously estimated the annual costs of preventable ADEs at this hospital to be $2.8 million⁷. In this study, we observed a decrease in the preventable ADE rate of 17% (although this decrease was not significant); if this were the hospital-wide decrease, the annual savings would be $480000. This figure does not include the costs of injuries borne by patients, of admissions due to drug errors, of malpractice suits, or of the extra work generated by the nonserious medication errors. For our institution, the costs of developing and implementing POE have been estimated to be $1.9 million, with maintenance costs of $500000 per year.²⁷ The net savings have been estimated to be between $5 to $10 million per year.²⁷ While these estimates are crude, they suggest that POE not only improves the quality of care but it could save money.

We were surprised that the error rate did not fall as much for the ordering stage as rates for other stages. One reason was the unexpected increase in errors related to the use of sedating medications. After excluding these errors, the percent difference from phase 1 to phase 2 was 34% rather than 17% but was still smaller than the reductions seen for other stages.

The results for individual types of error give further insight into the potential impact of future prevention strategies. The dosing error rate fell 23%, probably because prescribers made selections from menus that showed only appropriate alternatives. While the rate for serious allergy errors decreased 56%, there were still 7 such errors, caused by clinicians failing to enter new information about allergies when reactions had occurred during the hospitalization. We currently are pursuing an initiative that encourages physicians to enter information about reactions that occur in the hospital. The issues regarding sensitivities, such as a patient's need for specific premedication for amphotericin, are more complex. We need to enable such information to travel with the patient, so it is available to all prescribers at the appropriate times.

The number of drug-drug interactions fell with POE, but not to near-zero, because the system at the time of the study was programmed to catch only the most severe interactions, which come up infrequently. We have since introduced a much more complete index of interactions.¹³ However, we have used many fewer interactions than most commercial databases report, because most are not significant. An overload of information could lead to indifference so that health care professionals might ignore serious interactions, increasing the rate of important errors.²⁸

This study has several limitations, some of which have been mentioned above. The study took place at only 1 tertiary care hospital, so the results may not be generalizable to other hospitals or health care settings. Because the before-after comparisons showed improvement, a temporal trend could have caused some of the differences seen. However, we found no evidence for an underlying temporal trend, and the nonpreventable ADE rates remained the same across the 2 phases. Also, as noted earlier, the POE intervention did not include all the features that would be expected to make a difference; thus, our estimate of effect probably represents a lower bound. In addition, the study did not address omissions, drugs that should have been given but were not. Finally, the κ between reviewers was lower for severity than for other determinations in part because of the multiple categories involved. This has also been the case in prior studies.^9,29

We conclude that a POE system decreased the number of medication errors with potential for harm by more than half. With additional improvements, even further reductions should be possible. Refinement of this approach and introduction of other systems changes in which more automation is brought to the drug dispensing and administration stages should result in further decreases in the numbers of nonintercepted serious medication errors and, in turn, the injuries they cause. These data add to the body of knowledge suggesting that POE can reduce costs and improve quality and suggest that hospitals should consider adopting such systems.