Conceptual model for estimating the patient safety impact, health care utilization, and cost savings attributable to accepted medication safety alerts. Alerts are generated in an electronic prescribing system. ADE indicates adverse drug event; DDI, drug-drug interaction; and ED, emergency department.
Flow diagram depicting selection of drug-drug interaction (DDI) alerts for analysis and extrapolation of the results of the analysis to all DDI alerts generated by a single electronic prescribing system. Data are from Massachusetts in 2006. ADE indicates adverse drug event; CCI, class-class interaction.
Cumulative number of serious, significant, and minor adverse drug events (ADEs) prevented by safety alerts. Data were obtained from a cohort of electronic prescribers in Massachusetts in 2006. A small percentage of alerts accounted for most of the estimated benefits.
Sensitivity analysis of the cumulative number of prevented adverse drug events (ADEs) owing to medication safety alerts. Varying assumptions about the rate with which patients filled prescribed medications (70% in model 2 vs 90% in the base case), the rate of pharmacy interceptions (10% in model 3 vs panelists' median estimates in the base case), and both factors (70% pharmacy fill rate and 10% pharmacy interception rate in model 4) did not result in drastically different estimates of prevented ADEs. The 75th and 25th percentile curves demonstrate the range of panelists' estimates regarding the frequency with which alerts prevented ADEs, using base-case assumptions.
Sensitivity analysis of the cumulative cost savings owing to medication safety alerts. Varying assumptions about the cost of hospitalization ($9000 for a Massachusetts [MA] admission and $26 555 for the average cost of a hospitalization in the United States), the likelihood of an adverse drug event (ADE) (panelists' median estimates in the base case vs the 25th and 75th percentile estimates in alternate scenarios), and the likelihood of hospitalization given an ADE (panelists' median estimates in the base case vs 25th and 75th percentile estimates in alternate scenarios) affected the estimated cost savings owing to electronic prescribing.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Weingart SN, Simchowitz B, Padolsky H, et al. An Empirical Model to Estimate the Potential Impact of Medication Safety Alerts on Patient Safety, Health Care Utilization, and Cost in Ambulatory Care. Arch Intern Med. 2009;169(16):1465–1473. doi:10.1001/archinternmed.2009.252
Because ambulatory care clinicians override as many as 91% of drug interaction alerts, the potential benefit of electronic prescribing (e-prescribing) with decision support is uncertain.
We studied 279 476 alerted prescriptions written by 2321 Massachusetts ambulatory care clinicians using a single commercial e-prescribing system from January 1 through June 30, 2006. An expert panel reviewed a sample of common drug interaction alerts, estimating the likelihood and severity of adverse drug events (ADEs) associated with each alert, the likely injury to the patient, and the health care utilization required to address each ADE. We estimated the cost savings due to e-prescribing by using third-party–payer and publicly available information.
Based on the expert panel's estimates, electronic drug alerts likely prevented 402 (interquartile range [IQR], 133-846) ADEs in 2006, including 49 (14-130) potentially serious, 125 (34-307) significant, and 228 (85-409) minor ADEs. Accepted alerts may have prevented a death in 3 (IQR, 2-13) cases, permanent disability in 14 (3-18), and temporary disability in 31 (10-97). Alerts potentially resulted in 39 (IQR, 14-100) fewer hospitalizations, 34 (6-74) fewer emergency department visits, and 267 (105-541) fewer office visits, for a cost savings of $402 619 (IQR, $141 012-$1 012 386). Based on the panel's estimates, 331 alerts were required to prevent 1 ADE, and a few alerts (10%) likely accounted for 60% of ADEs and 78% of cost savings.
Electronic prescribing alerts in ambulatory care may prevent a substantial number of injuries and reduce health care costs in Massachusetts. Because a few alerts account for most of the benefit, e-prescribing systems should suppress low-value alerts.
Although the value of electronic prescribing (e-prescribing) is well established in the acute care hospital, its safety benefits in ambulatory care are less well understood. Early studies1,2 showed that rudimentary electronic order entry, without advanced decision support such as drug interaction and allergy alerts, resulted in legible prescriptions but no difference in the rate of adverse drug events (ADEs) compared with paper-based prescribing. In subsequent studies,3-5 however, investigators discovered that clinicians with access to these features overrode as many as 91% of drug interaction and allergy alerts. Although overriding alerts may jeopardize the potential impact of these systems, it is possible that even the small number of accepted alerts may reduce patient harm, decrease unnecessary utilization of health care services, and save money.
To understand the potential benefits of medication safety alerts in ambulatory care, we conducted a multifaceted study of a commercial e-prescribing system serving 2321 Massachusetts ambulatory care providers in 2006. We hypothesized that the alerts that clinicians accepted would, in aggregate, benefit patients, lower health care costs, and help to validate the continued use of these systems.
To examine the impact of medication safety alerts, we modified a conceptual framework that characterizes prescribers' responses to drug safety alerts and the resulting consequences (Figure 1).6,7 When an alert is triggered, a clinician has the following 3 options: (1) override the alert and leave the prescription intact, (2) cancel the prescription, or (3) change the prescription to another medication. We combined canceled prescription orders and those changed to an alternate medication and termed these accepted alerts. Some accepted alerts prevent ADEs from occurring. The number of prevented ADEs is a function of the number of alerts generated, the acceptance rate, and the probability that the alerted interaction would have resulted in an ADE if allowed to reach the patient. The number of ADEs prevented should be discounted by the number of prescriptions that pharmacists do not fill or patients do not collect. The severity of the ADE may affect the type of injury, health care utilization, and cost of care.
We examined medication alerts generated by the users of PocketScript, an e-prescribing application developed by ZixCorp (Dallas, Texas) that allows clinicians to transmit prescriptions electronically to a pharmacy via a desktop computer or a handheld device. The system creates a profile of a patient's active medications based on previously written e-prescriptions. When a prescriber attempts to order a drug, the system checks whether the prescribed medication interacts with any medications on the patient's profile, drawing on a list of medication interactions maintained by Cerner Multum, Inc (Denver, Colorado). If an interaction is detected, a warning banner is displayed showing the severity of the interaction (high, medium, or low), and a description of the interaction is available through a drug reference guide.
We studied all e-prescriptions written and all medication safety alerts generated by 2321 eligible Massachusetts clinicians who used the PocketScript e-prescribing system for at least 1 e-prescription from January 1 through June 30, 2006. ZixCorp provided information on all drug-drug interaction (DDI) alerts generated during the study period, along with the prescribers' action on receiving the alert. Prescribers included physicians (79.2%) and nonphysicians (20.8%), including specialists in family medicine (14.5%), internal medicine (13.1%), pediatrics (13.7%), psychiatry (2.5%), and other specialties (26.5%). Specialty information was unavailable for 29.7% of prescribers.
These clinicians wrote 1 833 254 prescriptions for 60 352 patients and generated 279 476 drug interaction alerts during the study period. Multiple alerts may have appeared for the same prescription attempt if a prescribed drug interacted with more than 1 medication on the patient's profile. To avoid double-counting alerts and multiple prescription attempts for the same drug, we studied the last drug alert that the prescriber triggered sorted by physician, patient, prescribed generic drug name, date, and time.3 This approach eliminated 146 425 alerts, leaving 133 051. Each DDI belongs to a broader class-class interaction (CCI): bupropion hydrochloride with citalopram hydrobromide, for example, is a DDI within the smoking cessation agents and selective serotonin reuptake inhibitor antidepressant CCI. We selected the 100 most frequently accepted CCIs; these alerts represented 56.5% of all accepted alerts. Finally, we selected the most commonly accepted DDI within each CCI.
Because there are no published data estimating the probability and severity of harm caused by most DDIs, we used a modified Delphi technique (a type of consensus method) to characterize the clinical utility, patient safety benefit, and cost savings associated with accepted alerts.8-10 We recruited a 7-member expert panel of 4 Massachusetts physicians and 3 pharmacists (including investigators S.N.W., T.I., and A.C.S.), inviting panelists based on their experience in primary care medicine and patient safety research.
Investigators oriented panelists to the project by means of a conference call and practiced sample cases. Panelists then reviewed the same 100 medication safety alerts and, informed by their experience, made a series of judgments about each alert. We provided information about each DDI taken from pharmaceutical reference texts, including Micromedex,11 Clinical Pharmacology,12 Lexi-Comp,13 Epocrates,14 and a research study by Malone et al.15
For each DDI, panelists estimated the probability that the interaction would result in an ADE—defined as an injury due to medication use—and, if so, whether the most likely ADE would be serious, significant, or minor.2,16,17 Serious ADEs could cause organ system dysfunction, such as a seizure or major gastrointestinal tract bleeding. Significant ADEs could cause symptoms such as a rash or fever and/or laboratory abnormalities such as thrombocytopenia or hyperkalemia. Minor ADEs could cause minimal injury, such as flushing or dyspepsia. Because alerts were generated for initial prescriptions rather than renewals, panelists assumed that the medications were being prescribed together for the first time. Panelists judged the severity of potential ADEs on the basis of the most likely reaction (rather than the worst-case scenario) of the typical patient who would receive the drug combination (in terms of age, comorbidities, and dosage). We asked panelists to classify each incident by likely frequency, injury, and health care utilization using the categories in Table 1. Finally, panelists estimated the probability that a given drug interaction would have been intercepted in a community pharmacy and the prescription not dispensed.
Panelists scored each interaction independently. We selected 15 interactions with the greatest scoring discrepancies and presented them in a second conference call for discussion. Participants rescored these DDIs, and these final scores were used in the analyses.
We used published sources and payer data to estimate the costs to third-party payers associated with specified categories of health care utilization. The average cost of a medical hospitalization was estimated at $9000 (Adrienne Cyrulik, MPH, Blue Cross/Blue Shield of Massachusetts, written communication, June 17, 2008). Based on a study of the cost of emergency department visits,18 the average cost of an emergency department visit was calculated as $427, adjusted to 2006 dollars using the Consumer Price Index for medical services.19 The cost of a physician visit due to an ADE ($111) was derived from the average national charge in 2006 dollars for a 25-minute office visit.20 For physician visits that generated additional prescriptions, we estimated the average cost of a filled prescription at $58, calculated using the sales of prescription drugs in 2006 ($192 024 661 635) divided by the number of prescriptions dispensed (3 308 896 262).21 The cost of telephone calls with clinicians was assumed to be $0 because clinicians are not commonly reimbursed for this service.
We examined the interrater reliability of panelists' first-round judgments regarding the probability that each DDI would produce a serious, significant, or minor ADE; the type of injury to the patient; and the resulting health care utilization. Overall, agreement was satisfactory, with a range of 86% to 94% and κ scores of 0.49 to 0.69 (P < .001).
We used the panelists' median scores to estimate the type of ADE, injury, and health care utilization for each DDI. We then used the cost estimates, the number of CCI alerts, and the mathematical models shown in the eAppendixhttp://www.archinternmed.com to calculate the number of prevented ADEs, number of injuries, extent of health care utilization, and costs. We assumed that nonintercepted prescriptions were filled at a rate of 90% based on a previous study.6
Because our data set included 6 months of prescription information, we annualized the results to provide a more intuitive presentation of the data. We extrapolated the analysis of the 100 most commonly accepted DDIs, each selected from 1 of the top 100 CCIs, to the CCIs as a whole. In doing so, we assumed that each DDI presented to the panelists was representative of other DDIs within the same CCI subset. To estimate the total cost savings and the impact of medication safety alerts on patient safety in Massachusetts attributable to PocketScript prescribers, we then extrapolated from the 100 most common CCIs to the entire universe of alerts, assuming that weighted average rates would apply (Figure 2). A test of our assumptions supported this method: the average acceptance rate for the top 100 DDIs in the top 100 classes was 12.2%, whereas the acceptance rate for all other DDIs in the top 100 classes was 12.9%. The average acceptance rate for the 100 most commonly accepted CCIs was similar to that of the remaining CCIs (10.6% vs 10.0%).
We performed sensitivity analyses to test the stability of our results over a range of assumptions. Given the limited number of reviewers for each DDI and the possibility of a skewed distribution of scores, we calculated upper- and lower-bound estimates of the likelihood of the type of ADE, injury, and health care utilization using the interquartile range (IQR) (25th-75th percentile) for each DDI. We varied our assumptions about the pharmacy interception rate (using the panel's estimate in the base case and 10% in an alternate model) and the rate with which patients filled their prescriptions (90% in the base case and 70% in an alternate model). We also examined the effect on savings of varying assumptions about the cost and likelihood of hospitalization. We used the average cost of a US hospitalization ($26 555) as a high-range estimate, calculated by dividing total hospital revenue from US inpatient hospital admissions in 2006 ($939 459 919 425) by the total number of admissions (35 377 659).22
This study was approved by the institutional review board of the Dana-Farber Harvard Cancer Center. Statistical analyses were performed using commercially available software (SAS [SAS Institute, Inc, Cary, North Carolina] and Stata 9.1 [StataCorp, College Station, Texas]).
During the 6-month study period, Massachusetts clinicians encountered DDI alerts in 7.3% of e-prescribing attempts and overrode 91.1% of the encountered DDI alerts. The crude rate of alert encounters—calculated without removing potential duplicate alerts—was 15.2%.
Table 2 shows the estimated number of ADEs that were prevented in 2006 when 2321 Massachusetts PocketScript users accepted medication safety alerts, based on the expert panel's assessment of risk and harm. In this base-case analysis, electronic alerts prevented an estimated 402 ADEs, of which 49 were serious or life threatening, 125 were significant, and 228 were minor (approximately 0.17 ADEs per year for each prescriber). The IQR (133-846 ADEs per year) shows the variability associated with these judgments.
The panel's estimates showed that many alerts were required to prevent a single ADE because relatively few DDIs posed a risk of harm. Overall, clinicians encountered 331 alerts to prevent a single ADE (of any severity) and 2715 alerts to prevent a single serious ADE. As shown in Figure 3, a small percentage of alerts prevented most ADEs. Ten percent of alerts were estimated to prevent 60% of ADEs, and only 6 of the 100 most commonly accepted DDI alerts were judged to have prevented at least 1 serious ADE per year (Table 3). Seventeen percent of alerts were judged unlikely to prevent ADEs of any severity.
Of the 402 ADEs potentially prevented by medication alerts, the expert panel judged that alerts most commonly prevented symptoms that persisted less than 30 days (n = 272 [67.7%]) (Table 2). Of the 49 serious ADEs, accepted alerts may have prevented a patient death in 3 cases, permanent disability in 14, and temporary disability in 31. Clinicians encountered thousands of alerts (4292-44 350) to prevent a single potential death, disability, or case of prolonged symptoms.
Preventing ADEs with e-prescribing may have resulted in 39 (IQR, 14-100) fewer hospitalizations, 34 (6-74) fewer emergency department visits, 267 (105-541) fewer office visits, and 60 (8-109) fewer telephone calls to clinicians. As shown in Table 4, e-prescribing alerts were judged to yield annual savings of $402 619 (IQR, $141 012-$1 012 386). Divided by the total number of eligible Massachusetts e-prescribers using the system (n = 2321), the average savings per clinician in 2006 was $173 (IQR, $61-$436). The bulk of the savings (86.8% of the total) was attributed to the 39 prevented hospitalizations. Although office visits were the most commonly averted service (n = 267 [66.4% of services]), the cost per hospitalization dominated the calculation.
In a finding similar to the analysis of ADEs, a small number of alerts likely accounted for most of the savings. Ten percent of alerts were estimated to account for 78% of cost savings, derived largely from 32 prevented hospitalizations. Thirty-one percent of alerts resulted in savings of less than $100 annually, and 19% were judged to have no health care utilization impact from the perspective of third-party payers.
Given the inherent uncertainty in estimating the likelihood of ADEs, associated injuries, and health care utilization, we analyzed the impact of alternate assumptions on our results. Compared with the base-case scenario, we found moderate differences (<30%, or <121 ADEs) in the number of ADEs when we varied the rate at which patients filled their prescriptions (70% in model 2 vs 90% in the base case), the likelihood that the pharmacist would intercept a potentially dangerous prescription (10% in model 3 vs the panelists' median estimate in the base case [median, 0.10%; IQR, 0%-1%]), and the combination of these factors (70% fill rate and 10% interception rate in model 4 [Figure 4]). These alternate assumptions resulted in estimates of 313, 367, and 285 prevented ADEs, respectively, relative to the base case (402 ADEs).
Modifying these factors also had a moderate effect (<30%, or <$121 000) on cost savings ($313 000, $368 000, and $286 000 for models 2 through 4, respectively, compared with the base-case savings of $403 000). Savings were highly dependent on assumptions about the cost of hospitalization in Massachusetts. When used as an upper-bound estimate (rather than the base-case assumption of $9000 per hospitalization), the average cost of a US hospitalization ($26 555) yielded savings of $1.08 million (IQR, $383 000-$2.77 million). In addition to varying the cost of hospitalization, we also conducted sensitivity analyses using panelists' 25th and 75th percentile estimates of the likelihood of hospitalization for each DDI. Lower and upper hospitalization estimates resulted in projected cost savings of $146 000 and $487 000, respectively (Figure 5).
Using a modified Delphi technique and data on 1.8 million prescriptions and 135 051 unique alerts generated by ambulatory care clinicians, we estimated that e-prescribing alerts possibly averted 133 to 846 ADEs in Massachusetts in 2006, including 14 to 130 potentially serious ADEs that could have caused 2 to 13 deaths and 13 to 115 disabilities. Alerts may have prevented 125 to 715 hospitalizations, emergency department visits, and office visits, for a total savings to the health care system of $141 012 to $1 012 386. Extrapolating these results to all 38 847 prescribers (both inpatient and outpatient) in Massachusetts suggests that expanding e-prescribing statewide might prevent more than 6700 ADEs per year, including 50 deaths, and result in cost savings of approximately $6.7 million.
Our study provides perspective on earlier investigations of medication safety alerts. In previous studies, computerized prescribing in the hospital reduced nonintercepted serious medication errors by 55%, preventable ADEs by 17%, and hospital costs by $887 to $8958.16,23-28 Decision analytic models similar to the approach used herein have been used to estimate the cost of drug-related morbidity in the United States at more than $177 billion in 2000.7 However, we are aware of no studies that have estimated the safety benefits or cost savings from the use of e-prescribing alerts in the ambulatory environment.
Given the recent passage of the Medicare Improvements for Patients and Providers Act of 2008,29 which provides federal incentives for the adoption of e-prescribing and imposes penalties on providers who fail to switch to the electronic systems, our findings offer timely information to government and private industry. The benefits of medication alerts in ambulatory care are likely derived from a small fraction of alerts encountered by frontline clinicians. Indeed, only 10% of drug interaction alerts were estimated to account for 60% of ADEs and 78% of cost savings in our study. Seventeen percent of alerts had no discernible patient safety benefit, and 31% made a small (<$100/y) contribution to cost savings. Accordingly, clinicians probably reviewed thousands of alerts to prevent a single serious ADE. In fact, our study may underestimate the alert burden by examining only a subset of alerts; we excluded more than 100 000 potentially redundant alerts and did not include formulary adherence or allergy alerts in our analyses.
The phenomenon of alert fatigue has been well described,3,15,30-32 and the disproportionate relationship between the number of alerts and the patient safety and financial benefits of the system makes one wonder whether the juice is, in fact, worth the squeeze. Our findings suggest that the savings attributable to prevented ADEs may be insufficient to cover the costs to third-party payers of investing in e-prescribing systems. However, our estimates do not take into consideration savings that might accrue owing to improved formulary adherence, increased use of generic drugs, legible prescriptions, and the prevention of allergic reactions to prescribed drugs, and they do not account for the effect on the patients and families of lost wages and illness-related morbidity. Most important, we believe that the technology's ability to prevent ADEs makes it worthwhile, and our findings suggest that significant efficiencies could be gained by reducing overalerting. Doing so would mitigate alert fatigue, thereby increasing the percentage of clinically significant alerts accepted and the number of ADEs averted. Previous studies have demonstrated that tiering alerts and interrupting prescribers for only the most serious warnings are effective strategies for increasing alert acceptance rates.4,33
This study has several limitations. First, its generalizability may be restricted by the use of a single e-prescribing system and drug interaction alert database. However, in 2008, the PocketScript system was used by 8% of Massachusetts prescribers and approximately 4000 eligible prescribers in 18 states (Christopher Yu, MPH, ZixCorp, written communication, August 19, 2008). Its features, including required fields, pick lists with available dose forms, and DDI alerts, are common to many commercial and home-grown e-prescribing systems. In addition, Cerner Multum, Inc, which maintains the medication safety alerts for the PocketScript system, is 1 of several major commercial drug interaction databases used throughout North America.
Second, the decisions of Massachusetts clinicians to accept or override electronic alerts may not reflect the behavior of clinicians in other regions. However, in a related study of e-prescribing, we found that the rates and types of drug interaction alerts accepted by Massachusetts clinicians were similar to those of clinicians in New Jersey and Pennsylvania.34 Nevertheless, these studies examined clinicians who have generally opted into the system and therefore may reflect the experience of a relatively committed population of e-prescribers rather than what is expected from novice users.
Third, the structured implicit judgments of the expert panel were dependent on the makeup of the panel and the experience of its members. Although the use of a modified Delphi technique enhanced the reliability of this process, another panel with different members might have offered different estimates.
Finally, our analysis took a conservative approach to calculating the impact of e-prescribing in ambulatory care. The models did not consider the effect of alerts on clinician behavior other than aborting a prescription, such as counseling a patient or monitoring for drug interactions. We did not address benefits from e-prescribing that might derive from standardization of dosing, reduction in duplicate therapy, prevention of allergic reactions, and elimination of illegible prescriptions,35 nor did we attempt to assess the impact of alerts on patient satisfaction and drug adherence. We also focused on the estimated financial benefit to third-party payers, rather than the costs of medical injuries borne by patients and their families. Given the lack of data on these issues, our study used methods that are in line with those of other studies that have attempted to estimate the cost of illness and the benefits of e-prescribing.6,7,10,15 Future research should attempt to quantify the return on investment to medication safety alerts, including reduction in medical liability premiums and losses, to determine when the cost of these systems and the burden of excessive alerts detract from patient care. Studies could also examine differences between drug interaction databases and their effect on patient safety.
Our study suggests that drug alerts have the potential to prevent harm and reduce health care costs. To do so, however, clinicians need relief from alerts with little clinical value.
Correspondence: Saul N. Weingart, MD, PhD, Center for Patient Safety, Dana-Farber Cancer Institute, 44 Binney St, Boston, MA 02115 (email@example.com).
Accepted for Publication: May 15, 2009.
Author Contributions: Dr Weingart had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Weingart, Padolsky, and Weissman. Acquisition of data: Weingart and Seger. Analysis and interpretation of data: Weingart, Simchowitz, Padolsky, Isaac, Massagli, Davis, and Weissman. Drafting of the manuscript: Weingart, Simchowitz, and Padolsky. Critical revision of the manuscript for important intellectual content: Weingart, Padolsky, Isaac, Seger, Massagli, Davis, and Weissman. Statistical analysis: Weingart, Padolsky, Isaac, and Davis. Obtained funding: Weingart and Weissman. Administrative, technical, and material support: Weingart, Simchowitz, Padolsky, and Seger. Study supervision: Weingart.
Financial Disclosure: None reported.
Funding/Support: This study was supported by a grant from Blue Cross Blue Shield of Massachusetts.
Role of the Sponsor: Blue Cross Blue Shield of Massachusetts was not involved in the design and conduct of the study; the collection, management, analysis, and interpretation of data; or the preparation, review, or approval of the final manuscript.
Additional Contributions: Christopher Yu, MPH, Angus MacDonald, and Geoff Bibby, BC, of ZixCorp, provided the electronic prescription and alert data; Adrienne Cyrulik, MPH, and Jessica Fefferman, MPH, of Blue Cross Blue Shield of Massachusetts, assisted in conceptualizing the project and providing third-party-payer information.