[Skip to Navigation]
Sign In
Invited Commentary
July 26, 2021

Assessing Clinical Outcomes in a Data-Rich World—A Reality Check on Real-World Data

Author Affiliations
  • 1Department of Radiation Oncology, University of California, San Francisco
  • 2Bakar Computational Health Sciences Institute, University of California, San Francisco
  • 3Department of Pediatrics, University of California, San Francisco
JAMA Netw Open. 2021;4(7):e2117826. doi:10.1001/jamanetworkopen.2021.17826

Advancements in health information technology and data aggregation have led to the emergence of real-world data and real-world evidence (RWE). The 21st Century Cures Act1 broadly defined RWE as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than randomized clinical trials” and required the US Food and Drug Administration (FDA) to create a program for RWE for expanding indications and postapproval study. In oncology, retrospective RWE generated from electronic health records has had demonstrable impacts in regulatory approval, leading to the FDA-accelerated approval of blinatumomab and expanded indication for palbociclib based on surrogate end points.

Kehl et al2 report their assessment of surrogate end points in a large, multi-institutional cohort of patients with genomically sequenced non–small cell lung cancer and colorectal cancer. Overall, they demonstrated a limited correlation between time to treatment discontinuation and time to next treatment and overall survival (OS). Progression-free survival (PFS) based on imaging reports and medical oncologist assessment was the only end point with a consistently strong correlation with OS, but it is important to note that this cohort undergoing genome sequencing might not reflect all patients with these cancers.

These findings are critical as RWE is increasingly used for drug approval, to inform future clinical trials, and to provide best evidence in gaps in randomized data. Surrogate end points play an important role in the feasibility of clinical trials within a timeframe in which they can impact clinical practice. Ultimately, surrogates are intended to act as a proxy for the true goals of care, such as OS and patient quality of life. Nevertheless, even in clinical trials, which serve as the highest level of clinical evidence, many surrogates do not correlate with their intended outcome. This can yield uncertainty in the results, leading to a constantly evolving landscape as new surrogates are designed and incorporated into clinical trials.3 In observational studies, the value of surrogates is less apparent, as OS is typically ascertainable through sources such as the National Death Index. However, these data elements are still difficult to harmonize with data from claims or electronic health record systems. In many cases, the intent of alternative end points, such as those explored in the study by Kehl et al,2 is to mitigate selection bias of treatments by characterizing cancer-specific outcomes (PFS, treatment duration, and local and distant control).

In many studies, the correlation (or lack of one) between PFS and OS can be attributed to a high rate of noncancer mortality events. In this study, Kehl and colleagues2 performed a sensitivity analysis, finding that very few deaths occurred without an identified progression event. In this case, there appears to be a correlation between cancer control and OS. To define PFS, Kehl et al2 manually curated radiology reports and medical oncology clinical notes. The authors should be commended for their strong, formalized, systematic approach to abstraction. Nevertheless, manual abstraction of these notes is time consuming and, even with formalized approaches, can carry high rates of interrater variability (particularly in clinical notes). These approaches may also not feasibly be generalizable to many commonly used real-world data sources or other outcomes (such as treatment toxicity).

The detection of cancer progression can be further confounded by the introduction of novel imaging approaches over broader study periods, which result in stage migration. Although not a clear limitation of the study by Kehl et al,2 broader conclusions regarding the use of progression or treatment change–related surrogates with respect to RWE must take this possibility into consideration. One such classic example in non–small cell lung cancer was the introduction of positron emission tomography, which resulted in the well-documented Will Rogers phenomenon.4

One of the potentially promising areas of this study was the use of surrogate end points that lend themselves to automated abstraction, such as time to treatment discontinuation and time to next treatment. These end points had previously been evaluated and correlated with OS in an effort advised by the FDA and National Cancer Institute, including major oncology RWE groups, including the American Society of Clinical Oncology CancerLinQ, Kaiser Permanente/Cancer Research Network, and the Surveillance, Epidemiology, and End Results–Medicare Linked Database. These end points may offer potential surrogates for identifying progression based on changes in a patient’s treatment regimen but appear to have limited correlation with OS in the multi-institutional study by Kehl et al.2 Further exploration as to why some of these surrogates are inconsistent proxies is needed, as well as research into whether these correlations have cohort-specific biases.

Overall, these findings further support the need to consider which end points truly define clinical benefit. Cancer progression and its consequent changes in therapy are sometimes thought to have independent value from OS, as they are projected to impact a patients’ longevity and quality of life. It is possible that the increasing electronic capture of clinician- and patient-reported outcomes may allow a shift away from this paradigm.

One frequently cited motivation for cancer-related end points is treatment selection bias (such as surgical candidacy), which is well documented to confound results in observational data. In cancer registries, a well-explored source of observational oncology data, these confounders have made comparative effectiveness studies challenging. A number of causal inference strategies for cohort matching and “emulated” or “synthetic” trials strategies exist; however, these are frequently limited by the available confounders for adjustment, and, at worst, can even increase bias.5 Thus, these studies can frequently produce varying results depending on their methods, which can be discordant with the findings of randomized clinical trials.6 In some cases, this can be challenging to interpret, as one of the benefits of RWE approaches is to investigate areas in which clinical trials may have limited external generalizability owing to stringent eligibility criteria.

Thus, observational data of any form must still be interpreted with caution. As the field of RWE continues to evolve and enable new applications, end point definition will play an important role. Real-world evidence encompasses clinical uses beyond regulatory approval, by building retrospective evidence to inform practice in knowledge gaps and develop hypotheses for future clinical trials. The computational advancements that have enabled the aggregation of data to facilitate RWE analyses have concurrently empowered artificial intelligence approaches; early randomized clinical data demonstrating the clinical benefits of artificial intelligence based on real-world electronic health record data are emerging.7 Strong, objective end points will be needed to develop the coming generation of clinically useful artificial intelligence.

Predictive computational and statistical approaches based on real-world data may also expand the surrogate end points tested in the study from Kehl and colleagues.2 This may facilitate development of complex data-driven surrogates that are better correlated with intended end points for use in future clinical trials. For instance, Project Data Sphere, a nonprofit universal access data-sharing warehouse of clinical trials data, enabled the development of a potential surrogate based on prostate-specific antigen kinetics in metastatic castration-resistant prostate cancer.8 This candidate surrogate based on 8 randomized trials demonstrated strong correlation with OS and could significantly reduce needed statistical power.

Real-world data represent an important data source in the continued development of new clinical knowledge. We look forward to future studies characterizing how to best leverage these data to benefit patients, new technologies to integrate disparate sources of clinical data, especially patients reporting on their own outcomes, and new consortia that put this together. Real-world evidence has the potential to not only provide insights in clinical outcomes but also excitingly supports new opportunities for advanced clinical decision support and novel end points in randomized clinical studies.

Back to top
Article Information

Published: July 26, 2021. doi:10.1001/jamanetworkopen.2021.17826

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Hong JC et al. JAMA Network Open.

Corresponding Author: Julian C. Hong, MD, MS, Department of Radiation Oncology, University of California, San Francisco, 1825 Fourth St, Ste L1101, San Francisco, CA 94158 (julian.hong@ucsf.edu).

Conflict of Interest Disclosures: Dr Hong reported being a coinventor on a pending patent that is broadly related to this manuscript. Dr Butte reported being a cofounder of and serving as a consultant to Personalis and NuMedii; serving as a consultant to Samsung, Mango Tree Corporation, 10x Genomics, Helix, Pathway Genomics, and Verinata (Illumina); serving on paid advisory panels or boards for Geisinger Health, Regenstrief Institute, Gerson Lehman Group, AlphaSights, Covance, Novartis, Genentech, Merck, and Roche; being a shareholder in Personalis and NuMedii; being a minor shareholder in Apple, Facebook, Alphabet (Google), Microsoft, Amazon, Snap, Snowflake, 10x Genomics, Illumina, Nuna Health, Assay Depot (Scientist.com), Vet24seven, Regeneron, Sanofi, Royalty Pharma, Pfizer, BioNTech, AstraZeneca, Moderna, Biogen, Twist Bioscience, Pacific Biosciences, Editas Medicine, Invitae, and Sutro, and several other non–health-related companies and mutual funds; and receiving honoraria and travel reimbursement for invited talks from Johnson and Johnson, Roche, Genentech, Pfizer, Merck, Lilly, Takeda, Varian, Mars, Siemens, Optum, Abbott, Celgene, AstraZeneca, AbbVie, Westat, several investment and venture capital firms, and many academic institutions, medical or disease specific foundations and associations, and health systems; receiving royalty payments through Stanford University for several patents and other disclosures licensed to NuMedii and Personalis; receiving funding from the National Institutes of Health (NIH), Northrop Grumman (as the prime on an NIH contract), Genentech, Johnson and Johnson, the US Food and Drug Administration, the Robert Wood Johnson Foundation, the Leon Lowenstein Foundation, the Intervalien Foundation, Priscilla Chan and Mark Zuckerberg, the Barbara and Gerson Bakar Foundation, the March of Dimes, the Juvenile Diabetes Research Foundation, the California Governor’s Office of Planning and Research, the California Institute for Regenerative Medicine, L’Oreal, and Progenity. No other disclosures were reprted.

21st Century Cures Act, HR 34, 114th Cong (2016). Pub L No. 114-255. Accessed June 25, 2021. https://www.congress.gov/bill/114th-congress/house-bill/34
Kehl  KL, Riely  GJ, Lepisto  EM,  et al; American Association of Cancer Research (AACR) Project Genomics Evidence Neoplasia Information Exchange (GENIE) Consortium. Association between surrogate end points and overall survival in a multi-institutional clinicogenomic cohort of patients with non–small cell lung or colorectal cancer.  JAMA Netw Open. 2021;4(7):e2117547. doi:10.1001/jamanetworkopen.2021.17547
Chen  EY, Haslam  A, Prasad  V.  FDA acceptance of surrogate end points for cancer drug approval: 1992-2019.   JAMA Intern Med. 2020;180(6):912-914. doi:10.1001/jamainternmed.2020.1097 PubMedGoogle ScholarCrossref
Dinan  MA, Curtis  LH, Carpenter  WR,  et al.  Stage migration, selection bias, and survival associated with the adoption of positron emission tomography among Medicare beneficiaries with non–small-cell lung cancer, 1998-2003.   J Clin Oncol. 2012;30(22):2725-2730. doi:10.1200/JCO.2011.40.4392 PubMedGoogle ScholarCrossref
King  G, Nielsen  R.  Why propensity scores should not be used for matching.   Polit Anal. 2019;27(4):435-454. doi:10.1017/pan.2019.11 Google ScholarCrossref
Kumar  A, Guss  ZD, Courtney  PT,  et al.  Evaluation of the use of cancer registry data for comparative effectiveness research.   JAMA Netw Open. 2020;3(7):e2011985-e2011985. doi:10.1001/jamanetworkopen.2020.11985 PubMedGoogle ScholarCrossref
Hong  JC, Eclov  NCW, Dalal  NH,  et al.  System for High-Intensity Evaluation During Radiation Therapy (SHIELD-RT): a prospective randomized study of machine learning-directed clinical evaluations during radiation and chemoradiation.   J Clin Oncol. 2020;38(31):3652-3661. Published online September 4, 2020. doi:10.1200/JCO.20.01688PubMedGoogle ScholarCrossref
Wilkerson  J, Abdallah  K, Hugh-Jones  C,  et al.  Estimation of tumour regression and growth rates during treatment in patients with advanced prostate cancer: a retrospective analysis.   Lancet Oncol. 2017;18(1):143-154. doi:10.1016/S1470-2045(16)30633-7 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words