Weighing the Benefits and Risks of Proliferating Observational Treatment Assessments: Observational Cacophony, Randomized Harmony | Research, Methods, Statistics | JAMA | JAMA Network
[Skip to Navigation]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
July 31, 2020

Weighing the Benefits and Risks of Proliferating Observational Treatment Assessments: Observational Cacophony, Randomized Harmony

Author Affiliations
  • 1Verily Life Sciences (Alphabet), South San Francisco, California
  • 2Duke Clinical Research Institute, Durham, North Carolina
  • 3Division of Cardiology, Department of Medicine, Duke University School of Medicine, Durham, North Carolina
  • 4Nuffield Department of Population Health, University of Oxford, Headington, Oxford, United Kingdom
JAMA. 2020;324(7):625-626. doi:10.1001/jama.2020.13319

Amid the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, substantial effort is being directed toward mining databases and publishing case series and reports that may provide insights into the epidemiology and clinical management of coronavirus disease 2019 (COVID-19). However, there is growing concern about whether attempts to infer causation about the benefits and risks of potential therapeutics from nonrandomized studies are providing insights that improve clinical knowledge and accelerate the search for needed answers, or whether these reports just add noise, confusion, and false confidence. Most of these studies include a caveat indicating that “randomized clinical trials are needed.” But disclaimers aside, does this approach help make the case for well-designed randomized clinical trials (RCTs) and accelerate their delivery?1 Or do observational studies reduce the likelihood of a properly designed trial being performed, thereby delaying the discovery of reliable truth?

Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    4 Comments for this article
    Observational Studies and Clinical Trials
    Sylvia Smoller, PhD | Albert Einstein College of Medicine
    I agree with Dr. Califf that clinical trials are the gold standard, but they are not always possible. Let's not forget that the tobacco-lung cancer and the tobacco-CVD evidence of harm came from observational studies and had enormous impact on public health. There are good observational studies and bad observational studies, just as there are well-powered or under-powered clinical trials. Good observational studies can be extremely useful.
    Randomized Clinical Trials. Necessary but with Limitations
    Jeanne Dobrzynski, BA; John B. Kostis, MD, DPhil | Cardiovascular Institute, Rutgers Robert Wood Johnson Medical School
    We recently itemized “Limitations of Randomized Clinical Trials” in the American Journal of Cardiology (1).  Clinical trials are necessary to establish efficacy of interventions, but by themselves do not describe the effectiveness when the intervention comes to general practice. In particular as it pertains to COVID-19, there may be bias in choosing the hypothesis to be examined, especially when randomized clinical trials are sponsored by the industry. Examining 370 randomized drug trials, Als-Nielsen (2) reported that trials funded by for-profit organizations were more likely to report positive results although equipoise dictates that in 50% of the cases an intervention would be beneficial and in 50% not beneficial. In addition, favorable presentation of clinical trials by emphasizing relative risk reduction rather than absolute risk reduction or number needed to treat may be misleading. In our opinion, clinical decisions on individual patients should be made by considering clinical trials as well as from observational studies.


    1. Kostis JB, Dobrzynski JM. Limitations of randomized clinical trials. Am J Cardiol. 2020 May 16:S0002-9149(20)30486-0. doi: 10.1016/j.amjcard.2020.05.011. Online
    2. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials. A reflection of treatment effects or adverse events? JAMA. 2003;290:921-928.

    John B. Kostis, MD, DPhil
    Jeanne M. Dobrzynski, BA
    Cardiovascular Institute
    Rutgers Robert Wood Johnson Medical School
    Pragmatic Realities
    Steven Scott, MD | CORE Physician Resources
    COVID-19 is still racing far ahead of our ability to perform enough research to develop meaningful treatment guidelines. Those guidelines in place rely more on expert opinion than they do on evidence. Scientific studies, regardless of their quality only provide guidance at the population level. Which is why evidence-based medicine produces guidelines rather than cookbooks.

    We all know that observational assessment of treatments has its pitfalls. But until the day that we have a large enough body of randomized trials to offer a viable alternative, wasting our collective energy stating the obvious smacks of parochial protectionism. />
    Our reputation as a profession would be much better served if that energy was spent in coordinating study design between various competing entities. Randomized trials using different designs and criteria produce as much chaos as observational ones do. They lead to constant conclusions of "more study needed." Meanwhile, patients are dying, those in the trenches are flying by the seat of their pants, and politicians and journalists are having a hayday manipulating public opinion.
    Well-Designed Studies Come in all Flavors
    Daniel Waxman, MD, PhD | UCLA David Geffen School of Medicine
    There are well-designed and poorly-designed RCTs and the same holds true for observational studies. With regard to COVID-19, a recent RCT of remdesivir is uninterpretable because it is open-label and underpowered for clinically-important outcomes (1). At the same time, a recent observational study of convalescent plasma made wonderful use of a natural experiment--varying antibody titers measured post-hoc--to provide compelling evidence for a small but significant treatment effect (2). As others have mentioned, RCTs are not always feasible or appropriate. Placebo-controlled trials for patients at high risk of death, where there is strong theoretical basis for treatment efficacy (e.g. convalescent plasma), are particularly problematic. Innovative trial design and critical review are needed all around, but RCTs must not have a monopoly on evidence.


    1. Li, Ling, Wei Zhang, Yu Hu, Xunliang Tong, Shangen Zheng, Juntao Yang, Yujie Kong et al. "Effect of Convalescent Plasma Therapy on Time to Clinical Improvement in Patients With Severe and Life-threatening COVID-19: A Randomized Clinical Trial." JAMA (2020).

    2. Joyner, Michael J., Jonathon W. Senefeld, Stephen A. Klassen, John R. Mills, Patrick W. Johnson, Elitza S. Theel, Chad C. Wiggins et al. "Effect of Convalescent Plasma on Mortality among Hospitalized Patients with COVID-19: Initial Three-Month Experience." medRxiv (2020)