Communicating Uncertainties About Prescription Drugs to the Public: A National Randomized Trial

Background Many new drugs are aggressively promoted. The public may not realize that even with US Food and Drug Administration (FDA) approval, important uncertainties about the benefits and harms of these drugs remain. We assessed the US public's understanding of the meaning of FDA drug approval and tested how brief explanations communicating drug uncertainties affect consumer choices.

Methods We conducted an Internet-based randomized controlled trial using a national sample of US adults from a research panel of approximately 30 000 households. A total of 2944 participants were randomized to receive 1 of 3 explanations about a pair of cholesterol drugs (1 approved based only on a surrogate outcome [lower cholesterol] and 1 based on a patient outcome [reduced myocardial infarctions]). Participants were randomized a second time to receive 1 of 3 explanations about a pair of heartburn drugs (1 newly approved and 1 approved 8 years earlier). Controls received no explanation; the nondirective group received explanations (for the cholesterol drugs, surrogates do not always translate into patient outcomes; for the heartburn drugs, it takes time to establish the safety of new drugs); the directive group received explanations plus advice to “Ask for a drug shown to reduce heart attacks or ask for one with a longer track record.” The primary outcomes were choice: the cholesterol drug reducing myocardial infarctions, and the older heartburn drug.

Results Thirty-nine percent mistakenly believed that the FDA approves only “extremely effective” drugs; 25% mistakenly believed that the FDA approves only drugs without serious side effects. Explanations affected choices: 71% of those in the directive group, 71% in the nondirective group, and 59% of controls chose the cholesterol drug that reduced myocardial infarctions (absolute difference, 12% [95% confidence interval, 7%-18%] for each explanation vs control). For the heartburn drugs, 53% of the directive group, 53% of the nondirective group, and 34% of controls chose the older drug (absolute difference, 19% [95% confidence interval, 13%-24%] for each explanation vs control).

Conclusions A substantial proportion of the public mistakenly believes that the FDA approves only extremely effective drugs and drugs lacking serious side effects. Brief explanations highlighting uncertainties about the benefit of drugs approved based on surrogate outcomes and the safety of new prescription drugs improved choices. Nondirective explanations worked as well as directive ones.

Trial Registration clinicaltrials.gov Identifiers: NCT00950157, NCT00950131

The US Food and Drug Administration (FDA) approves drugs when it believes that benefits outweigh harms. But approval does not mean that the FDA believes benefits are large (or important) or that all serious side effects are known. The typically small size (a few hundred to a few thousand participants), short duration (less than a year), and limited end points of preapproval studies ensure that important uncertainties remain. Uncertainties are greatest in the first few years after approval and for drugs approved solely on the basis of a surrogate outcome. ^1,2

Despite uncertainties about benefits and harms, many new drugs are aggressively promoted to the US public in no uncertain terms. For example, direct-to-consumer (DTC) advertising for Zetia and Vytorin (approved only on the basis of improving lipid profiles) reached $200 million (sales reached $1.8 billion) in 2007, the year before a randomized trial reported in the New England Journal of Medicine failed to detect any clinical benefit.³ Advertisements only included a statement acknowledging that the clinical benefit was unknown after publication of this trial. In 2000, a year after it was approved by the FDA, DTC advertising for Vioxx surpassed all other drugs,⁴ reaching $500 million (sales reached $2.4 billion) by 2003,⁵ the year before it was withdrawn from the market because it caused myocardial infarction and strokes. The FDA has never required advertisements to acknowledge uncertainties inherent in all new drugs. ¹

While regulators and many clinicians understand these uncertainties, the public may not. Enthusiasm for Zetia and Vioxx (or Avandia, ProCrit, etc) might have been dampened had consumers known to look for drugs approved based on patient outcomes or drugs with a longer safety record. We assessed the US public's understanding of the meaning of FDA drug approval and conducted a randomized controlled trial to test how communicating uncertainties about the benefit of drugs approved on the basis of a surrogate outcome, and about the safety of new drugs affects consumer choices.

This national trial was conducted (and completed) in September 2009 (Figure 1). The protocols were registered with ClinicalTrials.gov prior to recruitment. Participants underwent sequential randomizations: first, to explanations about surrogate outcomes (NCT00950131) and second to explanations about the safety of new drugs (NCT00950157). The committee for the protection of human subjects at Dartmouth Medical School, Hanover, New Hampshire, approved the study.

Participants were randomized to receive 1 of 3 explanations about a pair of cholesterol drugs, 1 approved only based on a surrogate outcome (lower cholesterol), and 1 on a patient outcome (reduced myocardial infarctions). The 3 explanation groups were as follows: controls received no explanation, the nondirective group received an explanation (surrogates do not always translate into patient outcomes), and the directive group received the same explanation plus advice to “Ask for a drug shown to reduce heart attacks.”

Participants were subsequently randomized to 1 of 3 explanations about a pair of heartburn drugs, 1 newly approved and 1 approved 8 years earlier. The 3 explanation groups were as follows: controls received no explanation, the nondirective group received an explanation (“It takes time to establish the safety of new drugs”), and the directive group received the same explanation plus advice to “ask for a drug with a longer track record.” Figure 2 and Figure 3 provide the exact wording of the explanations (which appeared in boxes preceded by the word “CAUTION”).

The trial included a nationally representative sample of Americans 18 years or older recruited from a research panel (created by Knowledge Networks, Menlo Park, California, a professional survey firm). This panel, which consists of approximately 30 000 households recruited by probability methods (random digit dialing and address-based sampling), has been shown to compare favorably with random digit dialing samples in direct comparisons^6-8 and is used by the National Science Foundation for its general population experiments grant program.⁹ In return for free Internet access and cash incentives (or any necessary computer equipment), members agree to complete Internet surveys several times per month.

Among a random sample of 4316 adults 18 years or older invited by e-mail, 2944 agreed to participate (a 68% participation rate) (Figure 1). Randomization to explanation groups was performed using a central computerized random number generator to ensure allocation concealment. The investigators were masked to allocation. Participants were not told about alternative explanations.

The online survey (in which the explanations were presented) asked about the meaning of FDA approval and issues related to surrogate outcomes and new drugs. We conducted 2 online pretests to ensure that questions were reasonable and understandable (the first pretest was for logistics, the second was to elicit open-ended explanations of how participants chose their answers). Pretest data were not included in the analyses. The exact questions are provided with the results and in the online eAppendix and includes screen shots of the survey).

The primary outcome was choice of the better drug (ie, the drug for which there is less uncertainty). For the pair of cholesterol drugs (which were both free and had the same side effects), the better drug was the one with a beneficial effect on a patient outcome (myocardial infarctions) rather than one that only improved a surrogate outcome (cholesterol levels). For the pair of heartburn drugs (which were both free, worked equally well, and had the same side effects), the better drug was the one approved 8 years ago rather than the newly approved one.

We asked a follow-up question after each type of explanation (“What would you do if your doctor recommended the drug with the surrogate outcome? Do you think new—or old—drugs are safer?”). To assess understanding of FDA approval, we adapted 4 true/false questions from the “faith in regulation” scale by Bell et al¹⁰ (eAppendix).

This randomized trial was nested in a larger study that tested the effect of different numeric formats for presenting absolute risks (eg, “1%” vs “1 in 100”) on participants' comprehension of tables summarizing the benefits and harms of a heartburn drug and a cholesterol drug (reported separately¹¹; the eAppendix includes the entire survey). The sample size required for the larger study (2750 participants) resulted in 99% power to detect a 10% difference in this study (ie, 55% vs 65% choosing the better drug) with a 2-sided α of .03 (to allow for multiple comparisons).

We used poststratification weights to account for the sampling strategy (oversampling of African American and Hispanic households into the Knowledge Networks research panel) and nonresponse (ie, nonparticipation in the research panel and survey nonresponse). The weights adjust the demographic characteristics of respondents to match the US population (from the 2009 Current Population Survey¹²) on age, sex, race/ethnicity, education, region, and metropolitan residence. We used Stata statistical software (STATA 11; StataCorp LP, Cary, North Carolina) to perform all analyses using the SVY series of commands to incorporate the poststratification weights. Weighted and unweighted results were very similar.

Item nonresponse was low (range, 2%-5%). Missing answers were considered incorrect for factual questions (ie, the primary outcome: choosing the better drug). For questions about the meaning of FDA approval, missing answers were included as a “don't know” category (results were nearly identical in a complete case sensitivity analysis). We used a postestimation command to calculate weighted confidence intervals (CIs) for absolute differences in proportions and simple logistic regression to generate P values for pairwise comparisons between explanation groups. All comparisons were 2-sided and were considered statistically significant at P <.03. Because the order of questions was fixed (surrogate, new drug, FDA approval), we conducted 3 stratified analyses to see whether the preceding information affected subsequent responses. First, we examined whether the effect of the new drug explanations varied by the preceding surrogate explanation groups (none, nondirective, directive). Second, we examined whether misconceptions about the meaning of FDA approval varied by the preceding explanations (ie, whether respondents received none, 1, or both of the explanations). Third, we examined whether the effect of either the surrogate or new drug explanations varied among the numeric format groups tested in the larger study. There was no evidence that prior information affected subsequent responses in any of the 3 stratified analyses.

Demographic characteristics were similar across the 3 surrogate explanation groups and across the 3 new drug explanation groups (Table). The mean age of participants was 46 years (range, 18-93 years), and 52% were women. Twelve percent had less than a high school education, and 11% had a postgraduate degree.

Thirty-nine percent mistakenly believed that the “FDA only approves prescription drugs that are extremely effective,” and 25% mistakenly believed that “only extremely effective drugs can be advertised to consumers.” One quarter mistakenly believed that the “FDA only approves drugs that do not have serious side effects,” and 17% mistakenly believed that “drugs that have serious side effects cannot be advertised to consumers.” Fifty-six percent held at least 1 of the foregoing misconceptions. The proportion with any misconceptions was inversely related to participant education, ranging from 65% for those with less than high school education to 45% for those with a graduate degree. Misconceptions were not related to age.

For the cholesterol drugs, 71% of the directive group, 71% of the nondirective group, and 59% of controls chose the better of the pair, that is, the drug with a benefit on a patient outcome (absolute difference, 12% [95% CI, 7%-18%] for both comparisons vs control) (Figure 2). When asked what they would do if their physician suggested the drug only known to lower cholesterol, 61%, 58%, and 49% said they would request the drug known to reduce myocardial infarctions (absolute difference, 9% [95% CI, 3%-15%] and (12% [95% CI, 7%-18%], respectively).

For the heartburn drugs, 53% of the directive group, 53% of the nondirective group, and 34% of controls chose the better drug (ie, the older drug): absolute difference (19% [95% CI, 13%-24%] for both comparisons vs control) (Figure 3). When asked which drug is “safer,” 46%, 45%, and 31% chose the older drug (absolute differences 15% [95% CI, 9%-20%] and 14% [95% CI, 8%-19%], respectively). To examine whether the explanation led people to make the correct decision because they understood the safety issue, we compared the proportion in each group who chose the older drug and said it was safer. The proportions in each of the 3 explanation groups were 40%, 41%, and 26% (absolute difference, 15% [95% CI, 10%-20%] and 14% [95% CI, 9%-20%], respectively).

The effects of both explanations were similar at each level of education. For example, the absolute differences in the proportion choosing the better cholesterol drug for the nondirective explanation vs control were 12% (95% CI, 3% to 22%) for respondents with a high school degree or less, 8% for college graduates (95% CI, −1% to 17%), and 17% for those with a graduate degree (95% CI, 4% to 31%). The corresponding numbers for the explanation about new drugs were 17% (95% CI, 8% to 27%), 28% (95% CI, 18% to 39%), and 19% (95% CI, 4% to 34%), respectively.

Many Americans misunderstand what FDA drug approval means. A substantial proportion mistakenly believes that the FDA only approves—and only permits advertising of—drugs that are extremely effective and without serious side effects. Furthermore, many also fail to recognize fundamental uncertainties about the benefits and harms of newer drugs. Almost half chose a drug that was shown only to improve cholesterol levels over one known to reduce myocardial infarctions. And almost two-thirds chose a newly approved drug over an equally effective one approved 8 years earlier.

Our randomized trial demonstrates that simple nondirective explanations about surrogate outcomes and newly approved drugs help. A 23-word explanation resulted in 12% more people correctly choosing a drug that reduced myocardial infarctions over one only known to improve cholesterol levels. A 37-word explanation resulted in 19% more people correctly choosing the drug with a longer track record.

Our study has 3 important limitations. First, participants were making hypothetical choices between drugs. The construct validity of the hypothetical choices is supported by the consistency of responses to the main (eg, choice) and follow-up questions (eg, “What would you do if your doctor recommended the drug with the surrogate outcome?”). For both explanations, over three-quarters of individuals answered the main and follow-up questions consistently. Furthermore, we believe that real patients facing real decisions would probably respond at least as strongly: when the stakes are higher, people pay more attention. Of course, to the extent that patients simply defer to physician recommendations, even the best patient explanations are unlikely to make much of a difference in prescribing.

Second, there may be concerns about generalizability. Knowledge Networks^6-8 (and others¹³) have demonstrated that estimates from their panel are very similar to estimates derived using the gold standard of random digit dialing. Nonparticipation in our study from the panel is unlikely to be an important problem, given the relatively high participation rate (68%) and the similarity between the unweighted and poststratification weighted results.

Finally, there is room for improvement. Even with our most effective explanation, almost one-third of participants still chose a drug with only a surrogate benefit over one with a patient benefit, and nearly half still chose a new drug over an equally effective older drug. Because these ideas may challenge intuitive assumptions (ie, new is better) and be unfamiliar (ie, surrogates do not reliably translate into patient benefit), it may take time—and reinforcement by trusted sources—for more patients to accept them. Ironically, trust in the FDA and the prevalent misconceptions about what FDA approval means may undermine efforts to introduce skepticism about surrogates and new drugs. It is also possible that other wordings might increase the effect of the explanations, and that stronger directive advice might also be more effective (eg, “choose a drug” rather than “ask if there is a drug” with a longer track record). But the explanation alone may be what matters.

We think explanations about surrogate outcomes and new drugs should appear prominently in the professional label, patient information, and DTC drug advertisements. Since we began our study, a statement about the unknown clinical benefit of Zetia was added to its DTC advertisements. While this is encouraging—albeit 6 years after the drug was on the market—there are no FDA requirements to do so for other drugs at the time of approval. And the FDA has yet to implement the Institute of Medicine's 2006 call for a new drug warning on all product labels, an approach adopted by the Medicines and Healthcare products Regulatory Agency¹⁴ (the United Kingdom's FDA), in which a black triangle is added next to the name of all drugs for at least the first 2 years that they are on the market.¹

There are important gaps in what people know about prescription drugs—gaps that undoubtedly contribute to the rapid uptake of drugs despite uncertainty about benefit and harm. Our findings show that simple explanations (ones that are brief enough even for television advertisements) help consumers make better decisions. If the FDA is serious about improving the state of consumer information about prescription drugs, it needs to do a better job routinely communicating what it knows^15-17—and does not know—about how well drugs work.

Correspondence: Steven Woloshin, MD, MS, VA Outcomes Group (111B), White River Junction VA Medical Center, White River Junction, VT 05009 (steven.woloshin@dartmouth.edu).

Accepted for Publication: May 12, 2011.

Author Contributions: Both authors contributed equally to the creation of this manuscript; the order of authorship is entirely arbitrary. Both authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Schwartz and Woloshin. Analysis and interpretation of data: Schwartz and Woloshin. Drafting of the manuscript: Schwartz and Woloshin. Critical revision of the manuscript for important intellectual content: Schwartz and Woloshin. Statistical analysis: Schwartz and Woloshin. Obtained funding: Schwartz and Woloshin.

Funding/Support: This study was supported by a grant from the Attorney General Consumer and Prescriber Education grant program and the Robert Wood Johnson's Pioneer Program.

Role of the Sponsors: Neither of the sponsors had any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.

Disclaimer: The views expressed herein do not necessarily represent the views of the Department of Veterans Affairs or the United States Government.

Additional Contributions: Alastair J. J. Wood, MBChB, from the Department of Pharmacology, Vanderbilt University, encouraged us to test directive explanations; Sir Iain Chalmers, DSc, from The James Lind Library, provided very thoughtful comments on an earlier draft of the paper; and Alice Andrews, PhD, from Dartmouth Institute for Health Policy and Clinical Practice, provided technical assistance. None of these individuals received any compensation.