Pragmatic Clinical Trials—Ready for Prime Time?

Evidence-based medicine has traditionally relied on data generated by large randomized clinical trials (RCTs) performed in a rigorous controlled setting to minimize the effect of unwanted biases and maximize the effect of a treatment. These trials have long been hailed as the reference standard for testing the efficacy of an intervention. Although RCTs have strong internal validity, their external validity may be hampered in real-world settings, where dynamics of clinical care are much different compared with the controlled setting of a RCT.

Pragmatic trials are conducted in everyday clinical settings and can overcome some of the limitations of traditional trials.¹ These trials have been promoted by the National Institutes of Health as a viable path forward, especially given the availability of electronic medical records (EMRs) that permit the adoption of such trials and the human effort and financial constraints associated with the conduct of traditional RCTs. However, the external validity of pragmatic trials may come at the expense of reduced internal validity. It is, therefore, important to evaluate the agreement between results of both trial designs to identify potential obstacles and opportunities for improvement.

In this context, Harper et al² performed post hoc analyses from the ASCEND (A Study of Cardiovascular Events in Diabetes) trial³ to study the sensitivity, specificity, and overall agreement between routinely collected vs directly adjudicated study outcomes. ASCEND was a double-blind, placebo-controlled trial evaluating the efficacy of aspirin therapy and ω-3 fatty acids among 15 480 individuals with diabetes but no evidence of atherosclerotic cardiovascular disease.² The primary outcome was time to a first serious vascular event (composite of nonfatal myocardial infarction, presumed ischemic stroke or transient ischemic attack) and vascular death [excluding intracranial hemorrhage]).

Mail-based questionnaires were predominantly used for participant recruitment and follow-up, whereas nonresponders were contacted by telephone. These (preadjudicated) events were then adjudicated by study clinicians blinded to treatment allocation. Adjudicated events were used in the primary published results of the trial.³ For the current analysis,² the authors ascertained events solely using linkage to hospital admissions data for nonfatal events and the Office for National Statistics death certification data for fatal outcomes (herein referred to as routine data). The differences between adjudicated and routinely collected data included absence of participant contact and lack of adjudication for the routinely collected data.

There were 1401 adjudicated events, of which 1009 were also identified by routinely collected data with a sensitivity of 72.0%.² There was a high overall agreement (κ = 0.78) between routine data and adjudicated data for ascertainment of serious vascular events. A total of 13 961 participants had no events by either approach, so specificity was 99.2%. Although specificity was high (>99%) for all components of the primary outcome, sensitivity varied and was highest for any arterial revascularization (94.6%) and lowest for transient ischemic attack (29.9%). Most events in the ASCEND trial (78.4%) consisted of fatal and hospitalized events. Sensitivity increased after restricting analyses to these outcomes (86.4%), and specificity remained high (98.8%). Importantly, the treatment effect estimates for aspirin for the primary outcome for adjudicated vs routinely collected data were similar when performing randomized comparisons of the trial data (rate ratios, 0.88 [95% CI, 0.79-0.97] vs 0.91 [95% CI, 0.81-1.02]).²

The results of the present analysis² are encouraging as a proof of concept that outcomes obtained via linkage with hospital data and data from death certification can be comprehensive and yield effect estimates in RCTs similar to directly adjudicated outcomes. Although this was most strongly demonstrated for serious events requiring hospitalization or death, other nonfatal outcomes not requiring hospitalization were less common and, therefore, may require additional data sources such as outpatient primary care data to further improve sensitivity.²

Routinely collected data offer the advantage of being readily available and not requiring participant contact or adjudication, which are labor intensive and costly.⁴ Assessment of study outcomes via linkage with EMRs decreases burden on the participants and the research team. This is especially important in the current COVID-19 pandemic. Cutting costs also allows pragmatic trials to be performed on a large scale, which increases statistical power, improves external validity of findings, and may allow a wider breadth of therapies to be evaluated for the same amount of human and financial resources.

There are also limitations of using routinely collected data in the conduct of clinical trials. Pragmatic trials may require a larger sample size or longer duration of follow-up if there are fewer events identified through routine data compared with adjudicated events. Second, serious vascular events in ASCEND were first ascertained by mail-based surveys to participants followed by adjudication of those events by clinicians.² Because self-reported outcomes may be less reliable among patients with cognitive dysfunction or poor medical literacy, further analyses are needed in trials where direct participant follow-up is performed by telephone or in-person visits as opposed to mailed surveys. This is to ensure that the results do not reflect missed events using traditional approaches, which are then picked up by a review of medical records. It is important to note that although there were 392 events picked up only by adjudicated data in ASCEND trial, there were also 118 events picked up by routinely collected data but not adjudicated data.² Third, forgoing in-person visits in pragmatic trials would not afford the ability to collect laboratory measurements; therefore, evaluating these outcomes may prove difficult if they are not routinely performed in clinical care. Finally, the ASCEND trial was conducted in England, Scotland, and Wales, where there are robust but limited numbers of EMR systems. In contrast, the US has approximately 18 different EMR vendors with limited interoperability between vendors,⁵ which could complicate data collection using this approach. Although interoperability may pose challenges if multiple EMRs coexist, multinational clinical trials could pose additional challenges. Furthermore, low- and middle-income countries do not typically have widespread EMR availability, which can result in unintended consequences, including bias in collection of outcomes for patients included from those countries. Importantly, potential exclusion of participants from some countries due to unavailability of EMRs can limit the external validity of a clinical trial findings at a global level.

The current study² is a welcome step toward adopting pragmatic trials. Future and ongoing clinical trials should perform similar direct comparisons between routinely collected and adjudicated data in their country- and region-specific context and assess the reasons for discrepancies between the two. For example, the Aspirin Dosing–A Patient-Centric Trial Assessing Benefits and Long-term Effectiveness (ADAPTABLE) trial was a pragmatic, open-labeled, patient-centric trial comparing the long-term effectiveness of 81 mg vs 325 mg of aspirin among patients with cardiovascular disease.⁶ The ADAPTABLE Trial had an innovative study design that randomized and allocated participants entirely via a web platform in addition to assessing outcomes by linkage queries of the National Patient-Centered Clinical Research Network in the US.⁶

The authors of the current analysis² should be congratulated for this important work. Similar studies are needed before widespread implementation of pragmatic trials to understand how this study design can be optimized while maintaining scientific integrity of results. Widespread adoption of strategies like these represent the pillars of a learning health system. This strategy could facilitate adoption of pragmatic trials in any health care system not only to track health outcomes in real time but also to provide the evidence base that is necessary for accurate decision-making.

Published: December 28, 2021. doi:10.1001/jamanetworkopen.2021.40212

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Al Rifai M et al. JAMA Network Open.

Corresponding Author: Salim S. Virani, MD, PhD, Section of Cardiology, Department of Medicine, Baylor College of Medicine, 2002 Holcombe Blvd, Houston, TX 77030 (virani@bcm.edu).

Conflict of Interest Disclosures: Dr Virani reported receiving grants from the Department of Veterans Affairs, World Heart Federation, and the Tahir and Jooma Family and honoraria from the American College of Cardiology (Associate Editor for Innovations for acc.org) outside the submitted work. No other disclosures were reported.