Adherence to Black Box Warnings for Prescription Medications in Outpatients

Background  Few data are available regarding the prevalence of potentially dangerous drug-drug, drug-laboratory, and drug-disease interactions among outpatients. Our objectives were to determine how frequently clinicians prescribe drugs in violation of black box warnings for these issues and to determine how frequently such prescribing results in harm.

Methods  In an observational study of 51 outpatient practices using an electronic health record, we measured the frequency with which patients received prescriptions in violation of black box warnings for drug-drug, drug-laboratory, and/or drug-disease interactions. We performed medical record reviews in a sample of patients to detect adverse drug events. Multivariate analysis was conducted to assess the relationship of prescribing in violation of black box warnings to patient and clinician characteristics, adjusting for potential confounders and clustering.

Results  Of 324 548 outpatients who received a medication in 2002, 2354 (0.7%) received a prescription in violation of a black box warning. After adjustment, receipt of medication in violation of a black box warning was more likely when patients were 75 years or older or female. The number of medications taken, the number of medical problems, and the site of care were also associated with violations. Less than 1% of patients who received a drug in violation of a black box warning had an adverse drug event as a result.

Conclusions  About 7 in 1000 outpatients received a prescription violating a black box warning. Few incidents resulted in detectable harm.

Adverse drug events (ADEs) are believed to be among the leading causes of mortality in the United States, with an estimated 100 000 deaths per year.¹ Adverse drug events are common in the outpatient setting,^2-6 and can lead to substantial morbidity.⁷ Lasser et al⁸ have shown that new drugs are particularly high risk, given the number of toxic effects that only emerge once a drug is used outside the setting of premarketing trials. However, old drugs also seem to carry substantial risk; most ADEs in outpatients represent known associations between drugs and complications.³

Prior studies of drugs used in the outpatient setting have examined the adequacy of laboratory monitoring^9-11 and the frequency of harmful drug-drug and drug-disease interactions.^12-14 These studies have all found outpatients to be at substantial risk for preventable ADEs. However, many such studies have been limited by their inclusion of only a few individual drugs or drug classes. One Swedish study¹⁵ calculated that 13.6% of prescriptions dispensed in a single month included at least 1 potential drug interaction. Few US data are available regarding the prevalence of potentially dangerous drug-drug and drug-disease interactions across the universe of drugs in outpatients. Furthermore, few data are available regarding the extent to which prescribers adhere to laboratory monitoring recommendations (drug-laboratory interactions) across a broad spectrum of outpatient drugs.

The Physicians' Desk Reference (PDR)¹⁶ is the most commonly used source of labeling information,¹⁷ and contains black box warnings that are intended to help physicians avoid the most serious ADEs. Black box warnings, developed by the Food and Drug Administration, communicate critical information to providers (physicians, nurse practitioners, and other prescribers). The warnings are separated (and thus highlighted) from other text in the package labeling by a prominent black box border. Analyzing data from 51 ambulatory practices in the greater Boston area, our goal was to determine how frequently physicians and other providers prescribe drugs in violation of black box warnings pertaining to drug-drug, drug-laboratory, and drug-disease interactions. We hypothesized that medical providers may frequently prescribe drugs in violation of these warnings, and that such prescribing may result in patient harm.

We studied ambulatory practices in the greater Boston area that use a common electronic health record (EHR). Such practices included 40 hospital-based clinics, 4 community health centers, and 7 community-based practices. The EHR contains information collected and entered by providers during clinical care. Available information includes patient demographics, lists of medical problems and prescription drugs (with start and stop dates), and results of laboratory tests. Abookire et al¹⁸ have previously documented a high level of accuracy in our EHR electronic problem list, medication list, demographic variables, and laboratory data. For example, when diabetes mellitus or hypertension was listed on the electronic problem list, these problems were found on medical record review 98% of the time. Similarly, if certain drugs (such as statins or hormone therapy agents) were on the electronic medication list, then more than 95% of the time these agents appeared on medical record review. Prescribing was done electronically. Decision support in place during the study included drug-allergy checking and default dose suggestions, but not drug-drug, drug-laboratory, or drug-disease checking. Black box warnings were not part of the decision support; clinicians would only be aware of such warnings if they consulted the PDR, drug package inserts, or other prescribing REFERENCES that contain information about black box warnings. Partners HealthCare System Institutional Review Board approved the study.

We analyzed data from all patients 18 years or older who were seen at ambulatory practices using the EHR and who received (based on order dates in the computer) at least 1 prescription from January 1 to December 31, 2002. We focused our analysis on patients who received a prescription for a drug that contains a black box warning pertaining to drug-drug, drug-laboratory, and/or drug-disease interactions. Staff at ambulatory practice sites routinely record patient race (white, black, or Asian) and ethnicity (Hispanic); we analyzed these variables to determine if they were associated with prescribing patterns.

We obtained a comprehensive list of drugs with black box warnings for drug-drug, drug-laboratory, and drug-disease interactions.¹⁹ Black box warnings are prominently displayed in the PDR¹⁶ and in drug package inserts to alert practitioners about serious risks.²⁰ Two of the study investigators (1 research pharmacist [D.L.S.] and 1 physician [K.E.L.]) independently reviewed the list of drugs. The investigators identified all drug warnings in which the frequency of monitoring was not precisely defined. For example, the drug valproate sodium contains a black box warning to check liver function test results at frequent intervals, but does not specify how often to monitor such tests. The investigators also identified warnings that included imprecise terms, such as advanced renal impairment, active liver disease, and high dose. Providers could interpret such terms in varying ways, leading to different monitoring practices. With 100% agreement, the reviewers identified 55 (52.9%) of 104 drugs for which the black box warning was vague and required clarification. In these cases, 2 of the investigators (K.E.L. and A.S.K.) queried academic medical specialists at 3 Partners HealthCare System–affiliated hospitals to obtain consensus about what is considered to be standard of care laboratory monitoring. When the specialists differed on the optimal frequency of monitoring, we used the most liberal (least frequent) monitoring interval.

We excluded the following drugs from our analysis: (1) drugs for which the black box warning concerned data that are not easily accessible to data analysis in the EHR, such as cumulative drug doses (n = 2); (2) drugs for which our medical specialists were unable to produce a study definition (the only drug class that we excluded for this reason is aminoglycoside drugs, because they contain a black box warning to “avoid with nephrotoxic drugs”; because of the clinical decision making that is different for each patient, our specialists were unable to identify a list of nephrotoxic drugs that are absolutely contraindicated with aminoglycosides); and (3) drugs prescribed exclusively in children (n = 1). We also identified 14 drugs with black box warnings that are not routinely prescribed in the outpatient setting.

For drugs with a black box warning pertaining to safety in pregnancy, we required that baseline pregnancy testing occur within 1 month before the patient started taking a drug. We required that testing for baseline hepatic or renal dysfunction occur within 3 months before the patient started taking a drug. For all other laboratory tests, we defined baseline laboratory testing as receipt of a given laboratory test within 12 months before the patient started taking a drug.

Some of the black box warnings require that a drug be discontinued in the presence of another drug, disease, or laboratory value. It is possible that a prescriber might contact a patient to discontinue a drug, but might not update the EHR until a subsequent visit. In such cases, we scored prescribing as adherent if a prescriber entered a discontinuation date for the drug within 3 months of the appearance of a contraindicated drug, disease, or laboratory value.

To estimate how frequently black box warning violations result in patient harm, we reviewed a random sample of 575 patient records (corresponding to 583 black box warning violations) in which a prescriber violated a black box warning. This sample was drawn from the universe of patient records in which a patient was prescribed a drug in violation of a black box warning (n = 2354). We had initially calculated that we would need to review 400 medical records to obtain a point estimate of ADE incidence with sufficiently narrow confidence intervals (CIs), estimating that about 10% of black box exposures would result in an ADE. We reviewed an additional 175 medical records because the ADE incidence was lower than expected. For each record, we reviewed all visits accessible on the EHR (outpatient, emergency department, and inpatient) and determined whether the black box violation consisted of a medical error (with little or no potential for harm), a potential ADE, or an ADE.²¹ For each patient record, 2 physicians (N.R.S. and J.M.R. or N.R.S. and T.K.G.) independently reviewed all potential ADEs and all ADEs. The reviewers determined the likelihood that the event was related to a medication (thus, meriting classification as an ADE or a potential ADE) and classified the event according to its severity and preventability. Each ADE was classified as fatal or life threatening, serious, or significant. Further details about our ADE review process have been published elsewhere.³ Interrater agreement was high for the classification of events as drug related (κ scores [95% CIs] for the 2 different pairs of reviewers were 0.97 [0.92-1.01] and 0.92 [0.82-1.03], respectively), and was lower for their severity (κ scores [95% CIs] for the 2 different pairs of reviewers were 0.67 [0.38-0.96] and 0.77 [0.57-0.96], respectively).

Among patients who were prescribed a drug with a black box warning for a drug-drug, drug-laboratory, or drug-disease interaction, we calculated the proportion who received a contraindicated drug, who had a contraindicated disease, and who did not receive adequate laboratory monitoring. We used the χ² test to compare differences in groups (patients in whom the black box warning was violated vs patients in whom the black box warning was not violated) according to patient and provider characteristics. The association between the patient and provider characteristics and the presence of a black box warning violation was determined by a logistic regression model, applying the exchangeable covariance structure of the generalized estimating equation approach to adjust for within-patient correlations.²² All variables with P<.10 in the univariate analyses were included in the baseline multivariable model. Interaction terms between patient and provider characteristics were also examined in the baseline model, and were retained using a threshold of P<.05. The baseline model included age group, sex, race, language, and 6 interaction terms between patient characteristics, provider type, site of care, number of medical problems, and number of medications and 6 interaction terms of provider characteristics with these variables. The final model included all single effects and 4 interaction terms that were significant in the baseline model; these included age group × race, age group × language, provider type × site of care, and site of care × number of medical problems. All variables in the final model were statistically significant (P<.05 with the score statistic in the type 3 generalized estimating equation analysis). We computed adjusted odds ratios and 95% CIs based on the multiple logistic regression variable estimates as measures of effect size. All analyses were performed using SAS statistical software for Windows, version 8.2 (SAS Institute Inc, Cary, NC).

In 2002, 324 548 outpatients in the target population received a prescription medication. Of these patients, 33 778 (10.4%) received a medication that contained a black box warning pertaining to drug-drug, drug-laboratory, and/or drug-disease interaction. Of these 33 778 patients, 2354 (7.0%, or 0.7% of all outpatients) received a prescription in violation of the black box warning. The Figure shows the distribution of type of black box warning (drug-drug, drug-laboratory, and drug-disease) and the frequency with which each type of warning was violated. Most patients who received a prescription with a black box warning were at risk for a drug-disease interaction (90.6%), followed by a drug-laboratory interaction (26.6%) and a drug-drug interaction (3.3%) (patients could have >1 type of interaction). Patients who received drugs with drug-drug and drug-laboratory interaction warnings frequently received the drug in violation of the black box warning (36.2% and 19.4%, respectively). Patients who received drugs with drug-disease warnings rarely had contraindicated diseases (0.7%).

Table 1 shows the demographic characteristics of the 33 778 patients. The mean age of the patients was 61 years (SD, 15 years), and most patients were female, white, English speaking, and privately insured. Table 2 shows the 48 863 prescription drug orders corresponding to the 33 778 patients in Table 1. Table 3 shows the results of multivariate analyses, including patient age, sex, race, and language; provider type and site of care; and number of medical problems and medications. Patients who were 75 years and older, white, and female and who took more medications were significantly more likely to receive a drug in violation of a black box warning than were younger, male, nonwhite patients who took fewer medications. Patients who had a moderate number (4-6) of medical problems were less likely to receive a drug in violation of a black box warning than were other patients, while patients seen at community health centers and at hospital-based clinics were more likely to receive drugs in violation of black box warnings than were patients seen at community-based private offices.

A table listing the 69 individual drugs or drug classes in which there was a potential black box warning violation, and the percentage of patients in whom the warning did not seem to be followed is available in an online eTable. Seven drugs (azathioprine, carbamazepine, lithium carbonate or citrate, metformin, propoxyphene, triamterene, and valproate; 10.1% of all drugs) accounted for 1745 (74.1%) of the black box violations.

We reviewed 575 patient records corresponding to 583 black box warning violations. We excluded 92 (15.8%) of the apparent violations for which we discovered that the drug was not actually prescribed in violation of the black box warning. For example, in one record, the physician wrote a note to hold metformin in the setting of acute renal failure, but did not discontinue the metformin from the medication list. In 124 cases (21.3%), there were insufficient data available to determine whether an ADE had occurred. An example is a patient who received lithium and had no provider visits or laboratory tests done in 2002, perhaps indicating care at another facility. In the remaining 367 black box warning violations, there were 4 ADEs related to the black box warning violation (1.1%; 95% CI, 0.03%-2.15%), 4 ADEs unrelated to the black box warning violation (1.1%; 95% CI, 0.03%-2.15%), 92 potential ADEs (25.1%; 95% CI, 20.6%-29.5%), 154 medication errors (42.0%; 95% CI, 36.9%-47.0%), and 115 cases (31.3%) in which propoxyphene was prescribed in violation of its black box warning (2 such cases resulted in an ADE). We present the results for propoxyphene separately because they account for so many cases.

Descriptions of the ADEs related to black box warning violations are available from the authors. Among the 4 ADEs related to a black box warning violation, our reviewers rated 3 as serious and 1 as significant; all were deemed preventable. Among the 92 potential ADEs, 18 were rated as having a potential for a fatal or life-threatening ADE, 71 for a serious ADE, and the remaining 3 for a significant ADE. An example of a fatal or life-threatening potential ADE is a patient taking metformin with a diagnosis of congestive heart failure requiring pharmacologic treatment; an example of a serious potential ADE is a patient who is taking lithium without having levels monitored; and an example of a significant potential ADE is a patient taking anabolic corticosteroids without having lipids levels monitored.

In this study, we found that 1 in 10 outpatients was prescribed 1 or more drugs with a black box warning for drug-drug, drug-laboratory, and/or drug-disease interactions, and that overall 7 in 1000 outpatients received a prescription in violation of these black box warnings. While 2354 patients received a prescription in violation of a black box warning, we performed a detailed record review on a sample of 575 of these records. Based on the rates of ADEs detected in our record review, we estimate that less than 1% of these 2354 patients, or 16 patients, had an ADE resulting from the black box warning violations; about 1 in 6 patients who received a drug in violation of a black box warning had a potential ADE, and about 1 in 4 patients who received a drug in violation of a black box warning had a medication error. A few drugs, including azathioprine, anticonvulsants (carbamazepine and valproate), lithium, metformin, propoxyphene, and potassium-sparing diuretics, accounted for most black box warning violations. This study was done in practices that were using electronic prescribing, but with limited decision support. Limited decision support is characteristic of most prescribing applications at implementation.

Our findings differ from those of previous studies of individual drugs or classes of drugs. Such studies have shown that prescribers fail to adhere to black box warnings much more frequently than was observed in our study. For example, Horlen et al¹² found that almost one quarter of patients with a prescription for metformin had 1 or more absolute contraindications (renal dysfunction and/or congestive heart failure requiring pharmacologic treatment). In our study, fewer patients who received a prescription for metformin (only 5%) had violations of the black box warning regarding absolute contraindications to metformin use. Another study of Medicaid patients with bipolar disorder found that many (36.5% prescribed lithium, 42.2% prescribed carbamazepine, and 42.4% prescribed valproate) of such patients received no therapeutic drug monitoring of mood-stabilizing medications or recommended laboratory tests over 1 year. The researchers⁹ defined inadequate monitoring as monitoring that was not consistent with practice guidelines based on expert consensus. Our study found that a similarly high percentage of patients taking these 3 medications (lithium, 69.1%; carbamazepine, 24.5%; and valproate, 30.1%) did not receive adequate laboratory monitoring as required by the PDR black box warning. Several other studies^10,13,23,24 have also shown that labeling recommendations do not affect prescribing behavior.

Given the potential risk associated with black box warning violations, a better understanding is needed about why health care providers violate such warnings. We speculate that much of the time providers may be unaware of black box warnings, or may not have time to look up information on each drug that they prescribe, especially for their patients with the most complicated conditions. Providing decision support regarding the most frequently violated warnings may be helpful. Older, female, and white patients, and those seen at community health centers and hospital-based clinics, were the most likely to receive medications in violation of a black box warning. Patients taking more medications, and those with fewer than 4 or more than 6 medical problems, were also at risk. While we had no data on socioeconomic status, we suspect that patients seen at community health centers and hospital-based clinics are more likely to be poor than their counterparts seen at community-based private practices. It is possible that poor patients have more social issues that may draw the providers' attention away from issues of prescription drug monitoring. We also hypothesize that patients with few medical problems may be less likely to see their providers regularly, allowing fewer opportunities to review their medication list. Patients with many medical problems, on the other hand, may be so complex that providers do not have time to closely review the prescribing information for each medication.

Providers may also seek alternative sources of guidance for prescribing, such as clinical practice guidelines. The directives of black box warnings may differ from those contained in clinical practice guidelines, and are often difficult to follow. For example, psychiatric guidelines state that blood should be drawn to monitor lithium serum levels every 3 to 6 months.²⁵ The PDR black box warning refers prescribers to the dosage and administration section of the package labeling, which stipulates that lithium levels should be monitored at least every 2 months. Even when providers are aware of black box warnings, they may have difficulty adhering to them. In many cases, the warnings are vague and difficult to interpret. We found that more than half of the black box warnings required clarification from a specialist. Patient failure to complete laboratory testing may result in black box warning violations, although this was a rare occurrence in our record review. Finally, providers may knowingly violate a black box warning because of individual patient circumstances.

We believe these data have implications for the Food and Drug Administration, the developer of black box warnings. The Food and Drug Administration should make these warnings more specific, so that they are readily understandable by providers, and so that such providers can easily take action to avoid violating the warnings. A term like frequently should not be used. While a term like nephrotoxic drugs is undesirable, it may be necessary to make a warning brief. Because of the increasing use of EHRs, the warnings should be mapped to terms that make them computable. A compilation of computable warnings would be a highly useful resource that could be used to design prescribing alerts. Remembering all these warnings is beyond the capability of the human mind. If providers are to consider these warnings, it is essential that at least the most frequently violated warnings be compiled and made available through decision support in EHRs. While such records are only used by about a quarter of physicians nationally,²⁶ their use seems to be increasing rapidly.

Our study was limited by the fact that we did not have access to visit or laboratory data outside of the EHR. Thus, we could not determine whether an ADE occurred in about a fifth of the records reviewed. For example, if a patient saw a provider who listed lithium on the medication list, yet had his or her blood tests done at an outside laboratory, we would not have access to data on that patient's lithium levels. Our study is consistent with a recent report²⁷ documenting a high frequency of missing clinical information during primary care visits. We may overestimate the frequency of nonadherent prescribing, because some tests (such as a purified protein derivative [tuberculin] test) may be done and documented within the text of a note, but may not be entered into the health maintenance section of the EHR, where it would be captured in our analysis. At the same time, we may underestimate the occurrence of ADEs, given that many outpatients do not report symptoms that may be due to an ADE. Furthermore, many providers do not document such symptoms when they are reported.³ Finally, our study was conducted in urban medical practices affiliated with academic teaching centers, and may not be generalizable to other settings.

Our results suggest that although a few outpatients seem to receive prescriptions in violation of black box warnings for drug-drug, drug-laboratory, and/or drug-disease interactions, the absolute number of outpatients at risk is substantial. To increase adherence to black box warnings, such warnings need to be clarified, simplified, and made consistent with commonly used practice guidelines. Future studies should explore the effectiveness of EHR-based alerts for the most commonly violated medication warnings and for warnings that, when violated, have a high potential to cause patient harm.

Correspondence: Karen E. Lasser, MD, MPH, Department of Medicine, Cambridge Health Alliance, 1493 Cambridge St, Cambridge, MA 02139 (klasser@challiance.org).

Accepted for Publication: September 9, 2005.

Author Contributions: Dr Lasser had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Financial Disclosure: Dr Bates is a coinventor on patent 6029138, held by Brigham and Women's Hospital, on the use of decision support software for medical management, licensed to Medicalis. He holds a minority equity position in the privately held company, Medicalis, which develops Web-based decision support for radiology test ordering, and serves as a consultant to Medicalis. He is on the clinical advisory board for Zynx, Inc, which develops evidence-based algorithms, and Voltage Inc, which compiles information on compliance for drug companies.

Funding/Support: This study was supported by grants from the Harvard Risk Management Foundation, and by Partners HealthCare Information Systems, Boston, Mass.

Role of the Sponsor: The funding bodies had no role in data extraction and analyses, in the writing of the manuscript, or in the decision to submit the manuscript for publication.

Previous Presentation: This study was presented as a poster at the National Society for General Internal Medicine Meeting; May 13, 2004; Chicago, Ill.

Additional Information: See the online-only eTable.

Acknowledgment: We thank Melbeth G. Marlang, BA, for her help with data entry and manuscript preparation and Maxim D. Shrayer, PhD, for his constructive comments on earlier drafts of this article.