[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Purchase Options:
[Skip to Content Landing]
Views 6,480
Citations 0
July 13, 2017

Multidimensional Evidence Generation and FDA Regulatory Decision MakingDefining and Using “Real-World” Data

Author Affiliations
  • 1Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, Maryland
JAMA. Published online July 13, 2017. doi:10.1001/jama.2017.9991

Evidence linking interventions with health outcomes is the basis for good health care decision making. The widespread use of electronic health records, administrative claims, and social media and the ubiquity of smart devices have created “big data” that heretofore have not been widely utilized. There is substantial enthusiasm for the use of real-world data sources to generate so-called real-world evidence (RWE), but confusion remains about what RWE means. Evidence generation is multidimensional, including data source, study design, and degree of pragmatism. Real-world evidence is defined by the data source and degree of pragmatism independent of study design. Generation of RWE therefore is not limited to observational studies but also includes randomized trials conducted in clinical settings. The US Food and Drug Administration (FDA) currently uses RWE in safety surveillance and development of drugs for rare diseases, but there are other potential applications.

The attraction of RWE is 2-fold. First, the current clinical trial enterprise, based largely on randomized clinical trials (RCTs), is time consuming, burdensome, and expensive.1 Real-world evidence is perceived to be a potential cost saver. Second, RWE is by its nature highly pragmatic and would therefore be expected to be more generalizable. The keys to defining RWE are the data source and degree of pragmatism. Real-world evidence can be generated from any study design as long as the data source is from routine care and the design is highly pragmatic, meaning the trial design and conduct closely approximate the eventual use of the product in clinical practice. Therefore, RCTs performed within the health care system, such as the TASTE trial,2 are considered a source of RWE.

Research studies are either interventional or noninterventional (observational).3 There are numerous variations of each based on the major design factors, including but not limited to research objectives/hypotheses, selection of study participants, allocation of interventions, masking of intervention assignments, data acquisition, and outcome ascertainment. The National Institutes of Health (NIH) has defined a clinical trial as a research study in which 1 or more human research participants are prospectively assigned to 1 or more interventions to evaluate the effects of those interventions on health-related biomedical or behavioral outcomes. This includes both RCTs and single-intervention trials. Studies in which individuals are observed with no attempt to affect the outcome are observational. All of these study designs can have varying degrees of pragmatism. The principal difference between observational and interventional studies is in the allocation of treatment. Many NIH-funded prospective observational studies (eg, the Framingham Heart Study4) have provided a rich source of new data that would not otherwise have been collected in routine clinical care. These data therefore would not be considered real-world data (RWD). Nevertheless, the study may also incorporate RWD elements through, for example, acquisition of hospital records and mortality data. Registries are another observational study design that can be based on patients, diseases, interventions (ie, medical products), or outcomes (eg, the National Death Index). Most registries are based on data collected as part of routine or usual patient care without assignment to a specific intervention and can thus be a rich source of RWD; however, there are registries that generate and collect protocol-mandated data (not otherwise collected as part of the patients’ care) that are not considered RWD.

The study design affects the confidence with which a causal inference can be established, meaning the observed effect was due to the specified intervention. Randomization within the context of an interventional clinical trial is intended to balance confounders, both known and unknown. Single-intervention trials, by their nature, use external controls for interpretation and thus are subject to greater confounding. Observational investigations can demonstrate correlation between an action and an effect but can rarely be used to prove causation.

A literal interpretation of “real world” would include all measured data as opposed to simulated, or modeled, data, but that is not common usage. Real-world data consist of data that are routinely generated or collected in the course of health care delivery or otherwise. Thus, electronic medical records and administrative claims data are excellent sources of RWD. Additional potential sources of RWD that could be utilized for evidence generation include smart devices, social media, meteorological data, census data, and socioeconomic data. This is often a contextual distinction because the same data points, such as blood pressure levels, could be considered RWD if measured as part of routine care or not RWD if those data were collected as part of a clinical trial. There are advantages and disadvantages to the use of RWD for evidence generation. Real-world data are more proximate to the patient and often include primary source data; however, there is greater potential for data elements to be missing or collected in an unstructured fashion because the data are collected for patient care rather than research.

The application of RWD is not limited to generating RWE. Real-world data may be combined with traditional clinical trials in a number of ways to increase their efficiency and reduce costs. Real-world data may be used to aid in the design of a clinical trial by assisting in the selection of study sites that are more likely to enroll study participants, provide a basis for power calculations, provide a prior for a Bayesian statistical analysis, provide an external control group, and guide enrichment. Real-world data may also be used during the conduct of a trial to reduce duplication of data input such as baseline medical history, automated adverse event reporting, and end-point ascertainment. End-point ascertainment may be the most widespread at this time, such as gathering deaths from the National Death Index or tumors from a tumor registry.

The level of pragmatism informs the extent to which the study reflects how an intervention, such as a drug or device, will be used in clinical practice. Factors that determine pragmatism include, among others, elements of study population, intervention/comparator, setting, and outcome measures.5 Traditional clinical trials often exclude large groups of the indicated population through eligibility criteria, disallowed concomitant medications, and restrictions on rescue medications. Moreover, comparators may not be consistent with standard of care and the settings not reflective of routine patient care. These issues could be addressed through less restrictive trial designs, such as pragmatic trials using randomization within the health care system. Nevertheless, the results of RCTs are not always discordant to that of observational studies performed in clinical settings.6 Moreover, observational studies sometimes use strict inclusion criteria that reduce pragmatism, making them less “real world”, and more recent interventional trials are designed with broader inclusion criteria, making them more pragmatic.

It would be easier to define RWE if there was a clear dichotomy between real world and non–real world, but in actuality the 2 exist on a continuum. The 21st Century Cures Act defines RWE as evidence derived from “data regarding the usage, or potential benefits or risks, of a drug derived from sources other than randomized clinical trials.”7 This can be interpreted to mean that the source of the data for RWE cannot be derived from protocol-generated case report forms used in traditional clinical research, rather than that RWE cannot be generated in the context of randomization. The latter interpretation would allow for the use of traditional single-intervention clinical trials to generate RWE, which was not the intent of Congress.

The FDA uses RWE for regulatory decisions, albeit primarily related to safety. Nevertheless, for some drugs, the demonstration of efficacy has been based on RWE from case series or registries.8 These applications have in common the following features; the drug treated a rare disease for which a randomized study may not have been feasible, and both the pathophysiology and natural history of the disease were well enough understood to lend confidence in establishing causality.

Real-world evidence is defined as the analysis of RWD in a study designed with a high degree of pragmatism, regardless of study type. A wide variety of study designs can be used to generate this evidence, including pragmatic RCTs. Many questions about a drug remain unanswered at the time of approval; some of them involve optimal dosing regimen, longer-term outcomes, and outcomes in various subpopulations. It is not feasible to answer all of these questions with traditional RCTs. Using RWE to begin to address these questions is preferable to having no evidence whatsoever.

Back to top
Article Information

Corresponding Author: Jonathan P. Jarow, MD, Center for Drug Evaluation and Research, US Food and Drug Administration, WO51 RM6364, 10903 New Hampshire Ave, Silver Spring, MD 20993-0002 (jonathan.jarow@fda.hhs.gov).

Published Online: July 13, 2017. doi:10.1001/jama.2017.9991

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Sung  NS, Crowley  WF  Jr, Genel  M,  et al.  Central challenges facing the National Clinical Research Enterprise.  JAMA. 2003;289(10):1278-1287.PubMedArticle
Fröbert  O, Lagerqvist  B, Olivecrona  GK,  et al; TASTE Trial.  Thrombus aspiration during ST-segment elevation myocardial infarction.  N Engl J Med. 2013;369(17):1587-1597.PubMedArticle
Feinstein  AR.  The Architecture of Clinical Research. Philadelphia, PA: WB Saunders Co; 1985.
Benjamin  EJ, Levy  D, Vaziri  SM, D’Agostino  RB, Belanger  AJ, Wolf  PA.  Independent risk factors for atrial fibrillation in a population-based cohort: the Framingham Heart Study.  JAMA. 1994;271(11):840-844.PubMedArticle
Thorpe  KE, Zwarenstein  M, Oxman  AD,  et al.  A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.  J Clin Epidemiol. 2009;62(5):464-475.PubMedArticle
Anglemyer  A, Horvath  HT, Bero  L.  Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials.  Cochrane Database Syst Rev. 2014;(4):MR000034.PubMed
21st Century Cures Act, HR 34, 114th Cong, 2nd Sess (2016).
Sasinowski  FJ, Panico  EB, Valentine  JE.  Quantum of effectiveness evidence in FDA’s approval of orphan drugs: update, July 2010 to June 2014.  Ther Innov Regul Sci. 2015;49:680-697. doi:10.1177/2168479015580383Article