[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Purchase Options:
[Skip to Content Landing]
Views 21,875
Citations 0
Viewpoint
July 25, 2019

Harms From Uninformative Clinical Trials

Author Affiliations
  • 1Multi-Regional Clinical Trials Center, Harvard Medical School, Cambridge, Massachusetts
  • 2Stanford University School of Medicine, Stanford, California
  • 3Biomedical Ethics Unit, McGill University, Montreal, Quebec, Canada
JAMA. Published online July 25, 2019. doi:10.1001/jama.2019.9892

Individuals who enroll in clinical trials do so with the belief that their participation will help to advance medical science. However, many trials are designed, conducted, and reported in ways that stymie this objective, a problem that can be called “uninformativeness.” From the perspective of researchers, this is a form of research inefficiency.1 But from the perspective of participants, preventable uninformativeness is a serious breach of trust and a violation of research ethics.

An uninformative trial is one that provides results that are not of meaningful use for a patient, clinician, researcher, or policy maker. The following are necessary conditions for a trial to be informative (eTable in the Supplement): (1) the study hypothesis must address an important and unresolved scientific, medical, or policy question; (2) the study must be designed to provide meaningful evidence related to this question; (3) the study must be demonstrably feasible (eg, it must have a realistic plan for recruiting sufficient participants); (4) the study must be conducted and analyzed in a scientifically valid manner; and (5) the study must report methods and results accurately, completely, and promptly. Trials that do not meet all of these conditions are very likely to be uninformative.

Scholars of clinical research have identified many of these problems, labeled them as ethically questionable, and called for change.2,3 Nevertheless, recent studies have shown a continued high prevalence of trials that fail 1 or more of the necessary conditions, despite current funding, review, and reporting processes (eTable in the Supplement). For example, a 2013 systematic review of otitis media had to exclude 24 of 96 eligible trials because of an excess “risk of bias.”4 If a randomized trial is judged to be so biased that it cannot be included in a systematic review, then participants in that trial effectively made no contribution to science. Similarly, a review of the largest unreported trials registered in ClinicalTrials.gov identified 67 trials that had no results reported in the public domain a median of 9 years after completion, effectively minimizing the contributions of more than 87 000 participants.5 Subtler examples are trials that unnecessarily addressed settled questions, which added no new knowledge, or trials that use very short-term outcome measures to assess treatments for long-term illness, rendering the trial findings misleading or irrelevant for clinical decision-making.

Uninformative trials present a challenge to ethics, science, and medical practice. First, such trials do not fulfill the well-established principle of social value that provides justification for asking people to participate in clinical research.6 Second, the spirit of informed consent is not met when patients enroll in trials expecting to contribute to medical progress when it is manifestly unlikely. The coexistence of uninformative trials alongside informative ones, even in the same medical center, makes it very difficult for prospective participants and their advisors to make fully informed decisions about trial participation. Third, uninformative trials divert participants, researchers, and other resources from other endeavors, including more informative trials. Fourth, uninformative trial reports compete for the limited attention of physicians or policy makers, who might not detect the design flaws and may draw invalid inferences.

Strong incentives to conduct research along with inadequate research training promotes the initiation of uninformative trials. These trials can survive the many layers of review and oversight because no oversight mechanism assesses all 5 elements of informative trials. For example, while ethics committees are instructed to assess risk/benefit, they do not typically assess scientific merit beyond that needed to justify risk. As pointed out by Altman,2 assessment of study value requires more expertise than simple assessment of study validity, and many review bodies lack the time, expertise, motivation, or charge to do that. Funders presumably consider scientific merit, but frequently do not review the full trial protocol and thus cannot identify critical design or operational details that could render the study uninformative. Quality of reporting cannot be directly addressed prior to study initiation, and funders typically do not consider past actions or current reporting processes when considering a new study.

Trials without any external funding may be at particular risk for being uninformative. As of March 18, 2019, there were 9484 open clinical trials registered in ClinicalTrials.gov that were enrolling over 5 million US participants and had no evidence of external funding. Unless academic medical centers (AMCs) and other groups that sponsor research, such as independent research organizations, have their own review infrastructure, there is no assurance that these trials had any independent scientific review. Independent scientific review alone is not sufficient, but is one mechanism for identifying correctable design flaws and determining whether the question being posed is unsettled and answerable by the trial.1 AMCs and funders also have a mixed record for assuring results reporting for trials they fund or sponsor. Reviews in the past 5 years have found that only about half of nonindustry trials have publicly reported results, and the US Food and Drug Administration Amendments Act 2007 (FDAAA) trial tracker website shows high rates of noncompliance with the FDAAA reporting from most US AMCs.7

Various incentives within the current research system point in the direction of conducting more trials, with few counterincentives for uninformative trials. Even though the phenomenon and the ethical and policy implications are patent, solutions will not be simple. Reforms need to be evidence based and must avoid adding burdens to researchers that do not confer associated benefits. Changes will need to be sensitive to the risk of deterring informative research.

Four broad recommendations can be made. First, AMCs and other groups that sponsor research should embrace their responsibilities as research sponsors by ensuring that each trial receives meaningful scientific review by a funder or another body identified by the AMC prior to its initiation. This review should ensure that each new trial is informed by the body of relevant completed and ongoing studies. The review should also include scrutiny of the study design to identify and remediate any serious design flaws. Such a process might focus on those trials that are not likely to get reviewed through other processes and those perceived to be at greatest risk for being uninformative (eg, trials with no external funding, trials by less experienced investigators, or trials with design characteristics that carry high risk of bias). Second, in the spirit of “you can’t improve it if you can’t measure it,” priority should be given to the development of metrics that reflect the extent to which the 5 conditions of informative trials are being met. Third, incentives need to be developed that reward researchers, trial sponsors, or AMCs for conducting informative trials, and that more strongly discourage the conduct of uninformative trials. Fourth, funders similarly have a responsibility to ensure that the trials that they fund are likely to be informative. This would require a more detailed review of the trial protocol, in some cases, and a process for holding the researchers they are funding accountable for ensuring that the conduct, analysis, and reporting are done in scientifically appropriate and timely manner.8

The current clinical research enterprise has insufficient safeguards against uninformative clinical trials that do not fulfill the contract between researchers and research participants. Uninformative trials squander scarce resources, divert participants from better research, and dilute the evidence base of medicine. Rather than relying on intensive efforts to screen uninformative trials late in their life cycle, AMCs, funders, and others involved in the conduct and oversight of human research need to devise incentives and plug oversight gaps so that fewer patients are induced to participate in uninformative research. As Altman noted, “As the system encourages poor research it is the system that should be changed. We need less research, better research, and research done for the right reasons.”2

Back to top
Article Information

Corresponding Author: Jonathan Kimmelman, PhD, Biomedical Ethics Unit, Department of Social Studies of Medicine, McGill University, 3647 Peel St, Montreal, QC H3A 1X1, Canada (jonathan.kimmelman@mcgill.ca).

Published Online: July 25, 2019. doi:10.1001/jama.2019.9892

Conflict of Interest Disclosures: Dr Zarin reported consulting for and being the former director of ClinicalTrials.gov. Dr Kimmelman reported receiving personal fees from serving on a data and safety monitoring committee for Ultragenyx Inc. No other disclosures were reported.

References
1.
Yordanov  Y, Dechartres  A, Porcher  R, Boutron  I, Altman  DG, Ravaud  P.  Avoidable waste of research related to inadequate methods in clinical trials.  BMJ. 2015;350:h809. doi:10.1136/bmj.h809PubMedGoogle ScholarCrossref
2.
Altman  DG.  The scandal of poor medical research.  BMJ. 1994;308(6924):283-284. doi:10.1136/bmj.308.6924.283PubMedGoogle ScholarCrossref
3.
Freedman  B.  Scientific value and validity as ethical requirements for research: a proposed explication.  IRB. 1987;9(6):7-10. doi:10.2307/3563623PubMedGoogle ScholarCrossref
4.
Berkman  ND, Wallace  IF, Steiner  MJ,  et al.  Otitis Media With Effusion: Comparative Effectiveness of Treatments. Rockville, MD: Agency for Healthcare Research and Quality; 2013. http://www.ncbi.nlm.nih.gov/books/NBK143306/. Accessed March 5, 2019.
5.
Tatsioni  A, Karassa  FB, Goodman  SN, Zarin  DA, Fanelli  D, Ioannidis  JPA.  Lost evidence from registered large long-unpublished randomized controlled trials: a survey  [published online May 7, 2019].  Ann Intern Med. doi:10.7326/M19-0440PubMedGoogle Scholar
6.
van Delden  JJM, van der Graaf  R.  Revised CIOMS International Ethical Guidelines for Health-Related Research Involving Humans.  JAMA. 2017;317(2):135-136. doi:10.1001/jama.2016.18977PubMedGoogle ScholarCrossref
7.
EBM DataLab. Who’s sharing their clinical trial results? FDAAA TrialsTracker website. http://fdaaa.trialstracker.net/. Published 2018. Accessed May 29, 2019.
8.
Bierer  BE, Strauss  DH, White  SA, Zarin  DA.  Universal funder responsibilities that advance social value.  Am J Bioeth. 2018;18(11):30-32. doi:10.1080/15265161.2018.1523498PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    EXPAND ALL
    Imperfect, Yet Trials Still Have Impact on Disease
    Ron Louie, MD | University of Washington, Clinical Professor of Pediatrics
    This is an excellent piece about clinical trials and the frustrating waste of volunteers' "human capital," investigators' lack of expertise, the waste of money, and the already cumbersome regulatory process being less than competent to ensure that trials are done "right" and provide at least a minimum of information (ethics and the subjects' POV).

    The authors mention external funding issues and bias, but I think they focus on non-industry trials. I wasn't sure from their Viewpoint if industry-sponsored trials, with all their own baggage, are more or less informative? I would guess there would be a reluctance
    to publish the failure of a proprietary agent, not allowing failure analysis which still seems important.

    The reality is that clinical trials are so daunting (and the duration so "non-productive") for investigators that incentives usually have to be assured career advancement or other compensation. Bench results can come so much faster (informative, one hopes), why would a young person bother to work on a clinical trial?

    Altman's BMJ editorial "The Scandal of Poor Medical Research was published 25 years ago (1); his editor Richard Smith wrote a followup blog piece five years ago declaring things were no better (2). So where are we in medicine since then? I know that in my specialty, childhood cancer survival has improved, care in hemophilia and thrombosis has improved, but in other arenas, most notably Alzheimer's and dementia, there has been little to no progress (3). Could the Alzheimer's failures actually be a sign that the cumbersome imperfect system is inefficient but works, as frustrating as it is? The other issue there in dementia might be the lack of an overall therapeutic research strategy.

    References

    1. https://www.bmj.com/content/308/6924/283
    2. https://blogs.bmj.com/bmj/2014/01/31/richard-smith-medical-research-still-a-scandal/
    3. https://alzheimergadfly.net/2018/08/28/spending-2-billion-on-alzheimers-a-fantasy/
    CONFLICT OF INTEREST: None Reported
    READ MORE
    ×