Sample drug alert notification using the Online Medical Record. Hcl indicates hydrochloride; Celexa, citalopram hydrobromide; SSRI, selective serotonin reuptake inhibitor; trazodone, trazodone hydrochloride; and 5-HT, serotonin.
Drug allergy and drug interaction alerts in primary care, by physicians' decisions to override. Hcl indicates hydrochloride; SSRI, selective serotonin reuptake inhibitor; trazodone, trazodone hydrochloride; and 5-HT, serotonin.
Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' Decisions to Override Computerized Drug Alerts in Primary Care. Arch Intern Med. 2003;163(21):2625–2631. doi:10.1001/archinte.163.21.2625
Although computerized physician order entry reduces medication errors among inpatients, little is known about the use of this system in primary care.
We calculated the override rate among 3481 consecutive alerts generated at 5 adult primary care practices that use a common computerized physician order entry system for prescription writing. For detailed review, we selected a random sample of 67 alerts in which physicians did not prescribe an alerted medication and 122 alerts that resulted in a written prescription. We identified factors associated with the physicians' decisions to override a medication alert, and determined whether an adverse drug event (ADE) occurred.
Physicians overrode 91.2% of drug allergy and 89.4% of high-severity drug interaction alerts. In the multivariable analysis using the medical chart review sample (n = 189), physicians were less likely to prescribe an alerted medication if the prescriber was a house officer (odds ratio [OR], 0.26; 95% confidence interval [CI], 0.08-0.84) and if the patient had many drug allergies (OR, 0.70; 95% CI, 0.53-0.93). They were more likely to override alerts for renewals compared with new prescriptions (OR, 17.74; 95% CI, 5.60-56.18). We found no ADEs in cases where physicians observed the alert and 3 ADEs among patients with alert overrides, a nonsignificant difference (P = .55). Physician reviewers judged that 36.5% of the alerts were inappropriate.
Few physicians changed their prescription in response to a drug allergy or interaction alert, and there were few ADEs, suggesting that the threshold for alerting was set too low. Computerized physician order entry systems should suppress alerts for renewals of medication combinations that patients currently tolerate.
IN THE Adverse Drug Event Prevention Study, computerized physician order entry (CPOE) prevented up to 84% of medication errors among patients hospitalized at 2 academic medical centers.1 On the strength of this evidence, CPOE was heralded as a hospital "best practice" in medication safety and a litmus test of safe care.2,3 However, little is known about whether CPOE can prevent adverse drug events (ADEs), defined as injuries due to medications, in ambulatory care.
In published studies to date, investigators4,5 reported that rudimentary computerized prescribing at 2 primary care practices resulted in substantially fewer prescription-writing errors than at similar sites without CPOE. Differences were due to legibility and prompting by the automated systems to provide complete prescription information. However, they found no statistically significant difference between ADE rates at computerized and handwritten sites. The lack of a difference was attributed to the small sample size and lack of advanced features such as drug interaction and drug allergy checking at the computerized sites. Many of the preventable ADEs were due to prescription of drugs to which the patient had a known allergy.
Can advanced CPOE features alter physician behavior and reduce ADEs in primary care? The answer depends in part on whether physicians honor the computer's drug alert warning. In inpatient electronic order-writing systems, some clinicians discount, ignore, or circumvent alerts and reminders.6 Clinicians may forfeit the potential benefit of automated decision support if they find the system annoying, unhelpful, or inefficient.7
To assess the potential benefits of computerized prescribing among ambulatory patients, we examined the behavior of general internists with respect to high-severity drug interaction and drug allergy alerts generated by a CPOE system used at adult primary care practices. We hypothesized that most physicians would honor high-severity drug interaction and drug allergy alerts, and that failure to do so would lead to an increase in ADEs. We sought to examine attributes of patient, prescriber, and medication that affected physicians' decisions to override an alert.
This study included physicians at 5 adult primary care practices affiliated with Beth Israel Deaconess Medical Center (BIDMC), a Boston teaching hospital. Two sites are hospital-based clinics together employing 35 full- and part-time academic internists and 158 medical house officers. Three sites are community-based practices employing a total of 19 full-time internists. One community site serves an affluent suburban community; another is a hospital-owned multispecialty group practice; the third is a community health center. Attending physicians and house officers at the 5 study sites care for a diverse population of more than 50 000 patients.
Physician informaticists at BIDMC and the Harvard Center for Clinical Computing, Boston, developed a variety of advanced applications for inpatients and outpatients since the 1970s.8 The Online Medical Record (OMR) was introduced in 1989 in hospital-based primary care practices, but its use spread quickly throughout BIDMC and its community sites.
The OMR provides clinicians with features, including note-writing capability; access to information about laboratory, pathology, and radiology reports and a reminder system for routine screening and preventive health measures. The OMR includes a medication sheet that permits clinicians to submit drug allergy information and to enter and print prescriptions. The system also relies on clinicians to keep medication and allergy information up-to-date. Providers have ready electronic access to the presciption history of their patients. Physicians, nurses, and pharmacists may enter drug allergy and intolerance information into the system; the computer prompts the clinician to record the type of reaction and level of certainty. The prescription-writing feature offers prescribers a list of medications and available strengths, with required fields for drug dose, number of pills or units dispensed, and renewals. Clinicians enter the directions as free text. In 2000, 888 clinicians in 67 practices ordered 296 539 new prescriptions using the OMR.8
In February 2000, the OMR prescription-writing program was enhanced to include drug interaction and drug allergy checks for all prescription orders. Allergy alerts were generated by linking a central database that includes information about each patient's drug allergies and intolerances that was entered by nurses, physicians, and pharmacists in the inpatient, ambulatory, and home care settings. The allergy program triggers an alert if the prescription matches the brand or generic name of the drug or allergen group defined by the National Drug Data File of First Databank, Inc, San Bruno, Calif. Drug interaction alerts were generated by checking patients' electronic medication list using rules in the National Drug Data File of First Databank. First Databank identifies 3 drug interaction severity levels. Level 1 alerts indicate the potential for serious or life-threatening injury and are based on substantial empirical evidence. Level 2 alerts indicate the potential for less serious injury. Level 3 alerts, like level 1 alerts, indicate the potential for serious or life-threatening injury, but the evidence of the interaction is less compelling. A single prescription could generate a drug allergy alert and multiple drug interaction alerts.
When an alert is generated, a note appears on the computer screen that identifies the alerted drug, the drug interaction or allergy, and an indication of the severity of drug interactions (level 1 indicates high; level 2, medium; and level 3, low) (Figure 1 and Figure 2). The system also allows the provider to see a detailed, referenced monograph about the alert. To escape from the alert, the prescriber must select deliberately the "override" menu option; the default is to terminate the order.
The study was a retrospective series. The outcome was the primary care physician's decision whether to write a prescription for a medication flagged for a level 1, 2, or 3 drug interaction or a drug allergy alert.
After obtaining institutional review board approval, we downloaded all 7877 drug interaction and drug allergy alerts generated from October 1 through December 31, 2000. This study period provided a 9-month interval from the implementation of the enhanced CPOE system to allow clinicians (other than some house officers or new hires) to become accustomed to the upgrade. The file included patients' names and hospital identification numbers, name of the medication that generated an alert, date of the alert, type of alert (drug interaction or drug allergy), specific drug-drug interaction(s) associated with a medication, severity of the alert (levels 1-3), and prescriber. The file also included information from the hospital's electronic credentialing system such as physician specialty, sex, and year of medical school graduation.
We reviewed initially all 4751 alerts generated by Department of Medicine internists. We found that physicians frequently tried several times to write a prescription for an alerted medication. They viewed a monograph about the alert, exited from the ordering process using an escape keystroke, then attempted to reorder the medication. This behavior occurred in 1270 of the 4751 alerts (26.7%). To avoid double counting, we selected the last in a sequence of escapes or overrides for a particular physician, drug, patient, and date. This yielded 3481 alerts.
In the initial analysis of all 3481 alerts generated by Department of Medicine internists, we calculated how frequently physicians overrode drug allergy and drug interaction alerts. In the subsequent analysis, we selected a subset of 189 alerts for detailed medical chart review. The medical chart review sample was limited to alerts that triggered drug allergy and level 1 drug interaction alerts, which together represented the greatest potential for harm. We randomly selected 122 alerts (31 drug allergy and 91 drug interaction) from 834 alerts in which the physician chose to override the alert and to complete a written prescription (14.6% of the sample). We randomly selected 67 alerts (15 drug allergy and 52 drug interaction) from 92 alerts where the physician did not prescribe an alerted medication (72.8% of the sample).
An internist (M.T.) reviewed the electronic medical record for the 3-month period from the date an alert was triggered. The reviewer completed a medical chart review instrument and printed the patient's medication sheet and the office note from the date of the index visit. She abstracted demographic information, including the patient's age, sex, and type of insurance. She indicated whether a prescription for the alerted drug was in fact written, the type of encounter (telephone or office visit), and whether the prescribing physician commented in the medical record about the alert. She assessed whether the prescription was for a new medication, a new dose of an existing prescription, or a renewal. She identified the presence of possible ADEs. She documented the name and number of drug allergies, the number of prescription medications, and the number of items on the patient's OMR problem list. A second internist (S.N.W.) reviewed the completed medical chart review instruments and related materials for completeness and accuracy.
Two board-certified internists (D.Z.S. and M.D.A.) independently reviewed each case to determine whether the alert was valid on the basis of scientific data, published drug information, and clinical utility (ie, a good alert), and whether the prescribing physician's decision to override the alert was appropriate given the clinical scenario (ie, a good decision). For events that resulted in an injury to a patient, the reviewers determined whether an ADE was present and whether the ADE resulted from failure to observe a computer alert. Physician reviewers scored each item on a 4-point scale (definite, probable, possible, and unlikely). Differences were resolved by consensus. Interrater agreement regarding the validity of the alert was excellent (κ
= 0.86) and regarding the appropriateness of the physicians' decision to override an alert was good (κ = 0.66).
To evaluate the hypothesis that most physicians accept drug allergy and high-severity drug interaction alerts, we calculated the number and percentage of each type of alert physicians overrode. Next, using the medical chart review sample, we tabulated the drugs, allergies, and types of interaction that generated level 1 interaction or allergy-related computer alerts. We examined the relationship between the decision to prescribe and factors hypothesized to affect physicians' decisions. We included demographic factors, such as the age and sex of patient and physician (using years since graduation from medical school as a proxy for age in the latter group), and insurance type (as a proxy for patients' socioeconomic status). We included the number of medication allergies per patient, number of prescription medications, and number of medical problems present on the patient's problem list. We identified the level and type of physician practice (house officer, community-based physician, or hospital staff physician). We accounted for the type of alert (drug allergy or drug interaction), type of encounter (office visit or telephone contact), and new or renewed prescription.
We hypothesized that house officers would be more receptive to electronic alerts than more experienced physicians. We hypothesized that physicians would be less likely to override alerts in patients with multiple allergies, who were receiving many medications, or who had many comorbid medical illnesses. In these cases, physicians may have fewer available choices. We hypothesized that physicians would override more alerts for new prescriptions than for a changed dose or renewed prescription of a medication that the patient previously tolerated.
We used the χ2 statistic for comparisons involving categorical variables and the Fisher exact test to analyze ADE rates. We used a multivariable logistic regression model with forward selection (P<.20) to identify factors associated with physicians' decision to write a prescription that generated an alert. Analyses used Stata 7.0 software (Stata Corp, College Station, Tex).
During the 3-month study period, primary care physicians at the 5 study sites attempted to write 24 034 prescriptions using the OMR. These prescriptions generated 352 drug allergy and 3129 drug interaction alerts (after correcting for redundant keystrokes) for a total of 3481 alerts (Table 1). Primary care physicians overrode 91.2% of 352 drug allergy alerts, 89.4% of 574 level 1 drug interaction alerts, 96.3% of 2432 level 2 alerts, and 85.4% of 123 level 3 alerts. To understand factors that influenced prescribing behavior, we reviewed a sample of 189 alerts for detailed analysis.
In the medical chart review sample, 96 drugs generated 143 level 1 drug interaction alerts and 35 drugs generated 46 drug allergy alerts. The following 6 drugs accounted for approximately one third of alerts: cyclobenzaprine hydrochloride (16%), azithromycin (6%), atenolol (3%), a combination of trimethoprim and sulfamethoxazole (3%), clarithromycin (3%), and a combination of triamterene and hydrochlorthiazide (3%).
Antibiotics, cardiovascular medications, and analgesics accounted for 42 of 46 drug allergy alerts. Physicians overrode the analgesic allergy alert in 13 of 14 cases. In each case, the alert was triggered when a physician wrote a prescription for a drug that was in the same class as the drug to which the patient had an allergy. For example, hydrocodone bitartrate and oxycodone hydrochloride prescriptions triggered allergy alerts to codeine phosphate. Similarly, ibuprofen triggered an alert to naproxen.
In contrast, physicians overrode only 7 of 17 antibiotic allergy alerts. Prescriptions for amoxicillin and a combination of amoxicillin and clavulanate potassium triggered 6 penicillin allergy alerts; all but one was honored. In contrast, physicians overrode all 3 erythromycin allergy alerts triggered by prescriptions for azithromycin (all involved erythromycin intolerance rather than a true allergy). Overall, physicians overrode 31 of 46 drug allergy alerts included in the medical chart review.
Fifty-three different drug interactions triggered 143 drug interaction alerts. Four drug interactions accounted for 65 of 143 level 1 alerts. The sympathomimetic–tricyclic antidepressant interaction was triggered most often (43 of 143), followed by anticoagulant-macrolide and selective serotonin reuptake inhibitor–tricyclic interactions (8 of 143 for each), then triamterene- or amiloride hydrochloride–nonsteroidal anti-inflammatory drug interactions (6 of 143). Cyclobenzaprine accounted for more than half (24 of 43) of the sympathomimetic–tricyclic antidepressant interaction alerts and 6 of 8 selective serotonin reuptake inhibitor–tricyclic interaction alerts; azithromycin accounted for 6 of 8 anticoagulant-macrolide interaction alerts. Overall, physicians overrode 91 of 143 drug interaction alerts included in the medical chart review.
We compared cases in which the primary care physician did and did not write a prescription for an alerted medication by factors hypothesized to affect physicians' decisions (Table 2). In the univariable analysis, the only factor associated with the decision to write a prescription for an alerted medication was prescription type. New prescriptions were less often written than renewals (50.0% vs 89.6% among alerted medications; P<.001).
In the multivariable logistic regression model (Table 3), physicians were less likely to prescribe (override) an alerted medication if the patient had multiple medication allergies (odds ratio [OR], 0.70; 95% confidence interval [CI], 0.53-0.93), and if the prescriber was a house officer (compared with a staff physician) (OR, 0.26; 95% CI, 0.08-0.84). Physicians were substantially more likely to override an alert for a renewal of a current prescription than for a new prescription (OR, 17.74; 95% CI, 5.60-56.18).
Thirty-two of 185 physicians in the study commented in the electronic medical record about their decision to observe or ignore a medication alert. In 15 of the 22 cases in which a physician had prescribed an alerted medication and commented about the decision, the physician wrote that the patient was not currently taking the medication listed as a potential cause of a level 1 interaction because it had been discontinued or the course had been completed. In 3 cases, the physician wrote that use of the prescribed medication was justified for a limited duration. In 2 more cases, the alert was prompted by a prescription of a medication with a listed allergy, but the drug was reported to be tolerated by the patient in the past. In the remaining 2 cases of allergy alerts, the physician indicated that the medication sheet or allergy information was incorrect.
To assess independently the prescribers' decisions, 2 investigators reviewed each alert. Reviewers judged that 69 (36.5%) of 189 alerts were invalid (including 58 [40.6%] of 143 drug interaction and 11 [23.9%] of 46 drug allergy alerts). They judged that 37 (86.0%) of 43 sympathomimetic–tricyclic antidepressant interaction alerts, none of 8 anticoagulant-macrolide interaction alerts, none of 8 selective serotonin reuptake inhibitor–tricyclic interaction alerts, and none of 6 potassium-sparing diuretic–nonsteroidal anti-inflammatory drug interaction alerts were unjustified on the basis of scientific evidence.
Reviews also examined the prescribers' decisions to override an alert. Reviewers agreed with the prescribers' decisions in 185 (97.9%) of 189 cases, including 65 (95.6%) of 68 cases where the physician chose to override a valid alert. Many of these alerts provided worthwhile warnings. For example, most anticoagulant and antibiotic alerts were judged to be valid but appropriate to override (especially if a dose adjustment was recommended or follow-up laboratory tests were ordered). For each of the 119 cases in which a physician wrote a prescription that the reviewers judged appropriate, the reviewers indicated at least 1 reason to justify the decision (Table 4). Reviewers indicated most often that the patient was no longer taking the medication, the interaction was not clinically significant, the patient tolerated the drug(s), and the benefits of treatment outweighed the disadvantages.
We identified 3 ADEs in the study, all affecting patients whose physicians had prescribed a drug that generated a level 1 drug interaction alert. One involved supratherapeutic anticoagulation (international normalized ratio, 8) in a patient receiving warfarin sodium for an aortic valve replacement and atrial fibrillation; the patient was prescribed an antibiotic (clarithromycin) for a lower respiratory tract infection, but had no bleeding complications. In another case, epistaxis developed that required an emergency department visit. The patient took warfarin for atrial fibrillation and was prescribed aspirin (325 mg/d) 3 weeks earlier for coronary artery disease. The international normalized ratio was in the therapeutic range, and the hematocrit level was stable. Reviewers judged that both ADEs were related to the alerted medications. In a third case, clonidine hydrochloride patch therapy was discontinued 1 month after it was started because the patient complained of itching at the site of the patch. The clonidine prescription had triggered an interaction alert with β-blockers; the ADE was judged unrelated to the interaction.
An ADE affected 3 (2.5%) of 122 patients whose physician had written a prescription for an alerted medication and none of the 67 patients whose physicians had observed the alert, a difference that did not reach statistical significance (P = .55). Reviewers judged only the first event (the warfarin-clarithromycin interaction) as an event that was potentially preventable had the prescribing physician adjusted the anticoagulant dosage in advance.
In this study of medication safety, we set out to examine whether primary care internists honor drug allergy and drug interaction alerts generated by a CPOE system, and to understand the factors that influenced physicians' behavior. Physicians overrode alerts in most cases.
This result is consistent with a study of alerts in the inpatient setting, in which physicians observed fewer than half of potentially life-threatening drug interaction alerts.6 However, clinicians' responses to medication alerts were less frequent than the responses reported in other studies of alerts and reminders in the inpatient setting. In one of the earliest studies, McDonald9 and other investigators10- 12 showed that physicians responded to 51% of computer-generated reminders about condition management and drug toxicities; 22% of physicians in the control group responded to similar conditions without the benefit of a reminder. The effect is also smaller than the compliance rates of 15% to 30% for preventive health reminders in ambulatory care.13- 15
To understand the factors that influenced physicians' decisions to write a prescription for an alerted medication, we sampled from alerts that physician had and had not overridden. We found that certain drugs (eg, cyclobenzaprine) and drug interactions (eg, tricyclic antidepressants and sympathomimetics) accounted for a disproportionate share of alerts. Physicians were more likely to override some alerts than others; this behavior may signal interactions that clinicians view with suspicion. In fact, physician reviewers judged that one third of alerts were inappropriate.
When we examined factors hypothesized to affect physicians' decision to prescribe an alerted medication, we found that physicians less often overrode alerts among patients with multiple drug allergies, suggesting that there were few therapeutic alternatives available. In addition, house officers were less likely than hospital-based faculty internists to override alerts. As novice physicians, house officers may be receptive to new information and to the introduction of technology into their practice. Rejection of alerts by community physicians and faculty physicians may reflect the skepticism of physicians with greater experience about some features of the CPOE system, such as out-of-date information, identification of interactions that were not clinically significant, failure to note patient tolerance of medication combinations, and the inability to balance the risks and benefits of therapy. It may also reflect a deeper-seated resistance among experienced practitioners to the perceived intrusion of information technology into the practice of clinical medicine.
Clinicians' acceptance of alerts and reminders reflects their relevance and validity. For example, Raschke and colleagues16 found that only 53% of 1116 medication alerts generated by a CPOE system at a 650-bed Arizona community teaching hospital accurately identified hazardous situations. A high volume of inappropriate or unhelpful alerts may discourage busy clinicians' willingness to examine, consider, and act on them.17,18
The Adverse Drug Event Prevention study by Bates et al1 showed that a sophisticated CPOE system can prevent medication errors among inpatients. Will CPOE prevent ADEs in primary care? We do not yet know which components of decision support in CPOE systems are necessary to produce improvements in primary care. Checking for drug allergy and interactions may be necessary but not sufficient to improve medication safety. Our results suggest that CPOE designers need to identify and eliminate inappropriate alerts that physicians find incredible, and change the threshold for generating alerts on renewals of medications that patients currently tolerate in combination. In addition, they should design advanced algorithms that ensure up-to-date and clinically relevant information is available to clinicians about laboratory data, dosing schedules, adverse drug reactions, and contraindications.19- 21
This study was subject to several limitations, including its small sample size and retrospective design. It was completed at 5 primary care practices affiliated with a single academic medical center using a single, shared CPOE system. The results may reflect in part features of the patients, clinicians, or information system peculiar to the study site and, hence, may not be generalizable to other practices and other systems. Our data suggest that ADEs occurred more often in cases where the physician failed to observe an alert, but the study was not adequately powered to address this question. We may have underestimated the number and type of ADEs because the medical chart review was limited to 3 months of follow-up. Since patients may fail to report or physicians to record ADE information in the medical record, more aggressive approaches to incident detection might also increase the yield of ADEs. In addition, the relatively short time frame of our study did not allow us to determine whether the override rate increases in a secular fashion. Physicians may increasingly override the system over time, if they perceive the alerts are usually inappropriate.
Nevertheless, the study offers information useful to the design of medication safety programs in primary care. Although a minority of primary care internists changed their prescription in response to an alert, most of these decisions were appropriate. Clinicians' skepticism about some CPOE medication alerts appears justified and indicates the importance of developing alert systems that are clinically useful and produce fewer unnecessary alerts. Specifically, systems should suppress alerts for renewals of previously tolerated medication combinations. Systems should prompt providers for information about the level of certainty associated with drug allergy alerts and to distinguish intolerance of well-recognized side effects from IgE-mediated responses. Systems should improve the signal-to-noise ratio by suppressing alerts with little evidentiary basis or clinical relevance and, in turn, provide access to references that can inform physician decisions.
In addition, systems should prompt providers to review medication lists for out-of-date information. They should also prompt clinicians to explain the basis for the decision to override an alert, so that this information can be used to inform the development of more effective decision support. Failure to document reasons for overrides may conceal sound clinical reasoning and create a liability risk for clinicians. In short, the development of more sophisticated decision support is needed to realize the promise of CPOE in primary care.
Corresponding author: Saul N. Weingart, MD, PhD, at the Division of General Medicine and Primary Care, Beth Israel Deaconess Medical Center, Rose 112, 330 Brookline Ave, Boston, MA 02215 (e-mail: firstname.lastname@example.org).
Accepted for publication January 23, 2003.
This study was funded by a grant from the Stoneman Center for Quality Improvement in General Medicine and Primary Care, Beth Israel Deaconess Medical Center, Boston, Mass, and in part by a K08 Mentored Clinical Scientist Career Development Award (1 K08 HS11644-01) from the US Agency for Healthcare Research and Quality, Rockville, Md (Dr Weingart).
This study was presented as an abstract at the 25th Annual Meeting of the Society of General Internal Medicine; May 3, 2002; Atlanta, Ga.
We thank David W. Bates, MD, MSc, for his comments on an earlier draft of this article.