Feasibility, Process, and Outcomes of Cardiovascular Clinical Trial Data Sharing: A Reproduction Analysis of the SMART-AF Trial

Key Points

Question  Is it feasible to reproduce the results of a major cardiovascular clinical trial using publicly available, shared clinical trial data?

Findings  The total time to complete the reproduction was 11 months. The primary effectiveness and safety outcomes were similar between the 2 studies; however, there were differences in the secondary analyses, and some important findings could not be replicated.

Meaning  Reproduction analyses are feasible. Incentivizing collaboration for data sharing, reporting statistical code of primary studies, and decentralizing data sharing platforms may improve clinical trial data sharing.

Importance  Data sharing is as an expanding initiative for enhancing trust in the clinical research enterprise.

Objective  To evaluate the feasibility, process, and outcomes of a reproduction analysis of the THERMOCOOL SMARTTOUCH Catheter for the Treatment of Symptomatic Paroxysmal Atrial Fibrillation (SMART-AF) trial using shared clinical trial data.

Design, Setting, and Participants  A reproduction analysis of the SMART-AF trial was performed using the data sets, data dictionary, case report file, and statistical analysis plan from the original trial accessed through the Yale Open Data Access Project using the SAS Clinical Trials Data Transparency platform. SMART-AF was a multicenter, single-arm trial evaluating the effectiveness and safety of an irrigated, contact force-sensing catheter for ablation of drug refractory, symptomatic paroxysmal atrial fibrillation in 172 participants recruited from 21 sites between June 2011 and December 2011. Analysis of the data was conducted between December 2016 and April 2017.

Main Outcomes and Measures  Effectiveness outcomes included freedom from atrial arrhythmias after ablation and proportion of participants without any arrhythmia recurrence over the 12 months of follow-up after a 3-month blanking period. Safety outcomes included major adverse device- or procedure-related events.

Results  The SMART AF trial participants’ mean age was 58.7 (10.8) years, and 72% were men. The time from initial proposal submission to final analysis was 11 months. Freedom from atrial arrhythmias at 12 months postprocedure was similar compared with the primary study report (74.0%; 95% CI, 66.0-82.0 vs 76.4%; 95% CI, 68.7-84.1). The reproduction analysis success rate was higher than the primary study report (65.8%; 95% CI 56.5-74.2 vs 75.6%; 95% CI, 67.2-82.5). Adverse events were minimal and similar between the 2 analyses, but contact force range or regression models could not be reproduced.

Conclusions and Relevance  The feasibility of a reproduction analysis of the SMART-AF trial was demonstrated through an academic data-sharing platform. Data sharing can be facilitated through incentivizing collaboration, sharing statistical code, and creating more decentralized data sharing platforms with fewer restrictions to data access.

The National Academy of Medicine, International Committee of Medical Journal Editors, and other leading organizations have highlighted clinical trial data sharing as an important initiative for enhancing trust in the clinical research enterprise.^1,2 Data-sharing platforms have been created including government platforms such as the National Heart, Lung, and Blood Institute’s Biologic Specimen and Data Repository Information Coordinating Center, commercial platforms such as ClinicalStudyDataRequest.com, and academic platforms such as the Yale Open Data Access (YODA) Project.³

Replication is an essential component of any scientific process, and study reproducibility can bolster clinical and public confidence in trial results.¹ Reproduction analyses represent a fundamental approach for and outcome from data sharing but are uncommonly performed.⁴ For example, among 172 data requests to the National Heart, Lung, and Blood Institute’s Biologic Specimen and Data Repository Information Coordinating Center between 2000 and 2016, only 2 (1%) were for reproduction analyses.⁴ However, 35% of reproduction analyses lead to differences in the interpretation of the primary outcome results (most commonly as a result of differing statistical approaches or analytical definitions) related to direction or magnitude of treatment effects.⁵ Less commonly, there are cases where reanalyses have uncovered fundamentally flawed results, which is perhaps the most compelling indication for increased data transparency.⁶ Because there are limited reports describing the feasibility, process, and outcomes of reanalyzing cardiovascular clinical trial data, we sought to perform a reproduction analysis of the Thermocool Smarttouch Catheter for Treatment of Symptomatic Paroxysmal Atrial Fibrillation (SMART-AF) trial, the only available cardiovascular clinical trial in the YODA Project database.

We submitted a research proposal through the YODA Project website to gain access to the original data sets from the SMART-AF trial performed by Natale et al.⁷ SMART-AF was a multicenter, single-arm trial evaluating the effectiveness and safety of an irrigated, contact force-sensing catheter for ablation of drug refractory, symptomatic paroxysmal atrial fibrillation in 172 participants recruited from 21 sites between June 2011 and December 2011. The primary study was published in 2014, and the device was approved by the US Food and Drug Administration in 2014.⁸

The primary effectiveness outcome of the SMART-AF trial was freedom from documented symptomatic atrial fibrillation, atrial tachycardia, or atrial flutter during the evaluation period. All participants were given a 90-day postprocedure blanking period to allow for atrial healing and remodeling, during which time participants were allowed 2 repeated ablations that would not be considered treatment failures. After the blanking period, participants were evaluated through 12 months of follow-up for recurrence, defined as any documented atrial fibrillation, atrial tachycardia, or atrial flutter episodes lasting at least 30 seconds. Treatment failure also included absence of entrance block confirmed in all pulmonary veins at the end of the ablation procedure, repeated ablation after day 90, or new antiarrhythmic drug requirement during the evaluation period. Secondary outcomes included the overall success rate (proportion) of participants free from arrhythmia recurrence after initial ablation and remaining off antiarrhythmic medications at the 12-month follow-up. Safety outcomes were defined as major adverse procedure- or device-related events occurring within 7 days after the ablation procedure.

After we received access to the analysis data sets, data dictionary, case report file, and statistical analysis plan through a SAS-based data sharing platform, we sought to reproduce the effectiveness and safety outcomes described in the primary study report.

We sought and received approval for our analysis of deidentified trial data through the Northwestern University institutional review board. Participant consent was obtained in the primary study but was not required for this reproduction analysis using deidentfied data.

We followed the methods in the statistical analysis plan and original publication. When we identified discrepancies in our results with the primary study report, we contacted the study authors, the YODA Project team, and the trial sponsor for clarification.

We calculated descriptive statistics for all participants in the effectiveness and safety cohorts. We created Kaplan-Meier survival curves for the primary effectiveness outcome of time to first recurrence procedure after ablation and blanking period for participants in the effectiveness cohort. We attempted to further stratify our analysis of this outcome by proportion of time that investigators spent operating in their selected contact force range, dichotomized at the 80% level as described in the primary study report. We calculated the success rate for the effectiveness cohort as the proportion of participants without any recurrence during the 12-month follow-up period. We attempted to create a logistic regression model with the primary effectiveness outcome as the independent variable but were unable to maintain fidelity to the original analysis owing to these data not being available. We calculated the rate of major adverse events from the safety cohort as the proportion of participants experiencing such an event during the study period.

We performed all statistical analyses using SAS Drug Development, version 4.3 (SAS Institute) through the SAS Clinical Trials Data Transparency platform.

The Figure demonstrates the project timeline. We submitted our data request and research proposal through the YODA Project website in May 2016. We received approval in 1 month, and the final data use agreement was finalized in October 2016, after which time we received access to the SAS Clinical Trials Data Transparency platform. We completed our initial reproduction analysis by November 2016 and identified and worked through clarifications with the SMART-AF authors, YODA Project team, and trial sponsor between December 2016 and April 2017.

Using the trial data sets and metadata, we confirmed that 172 participants were enrolled into the study. We calculated that 162 participants had the study catheter inserted to comprise the safety cohort compared with 161 participants reported by the study authors. After removing 38 participants selected as roll-in participants (a study requirement by which investigators calibrated their experience with the catheter on 1-2 participants to configure the operating reference ranges for future cases), we identified 124 participants in the effectiveness cohort compared with 122 participants reported by the study authors. We matched the 5 participants from the effectiveness cohort who were noted as lost to follow-up during the 12-month study period in the publication analysis.

Table 1 demonstrates demographic, clinical history, and imaging data reported by the study authors and in our reproduction analysis. We identified minor differences from the primary study that were driven by the differences in the cohorts but do not materially influence the baseline characteristics.

Table 2 demonstrates the effectiveness and safety outcomes reported by the study authors and in our reproduction analysis. We calculated a similar rate for the primary effectiveness outcome of freedom from symptomatic atrial arrhythmia at 12-month postprocedure compared with the primary study (74.0%; 95% CI, 66.0-82.0 vs 76.4%; 95% CI, 68.7-84.1). We calculated a higher success rate than the primary study (65.8%; 95% CI, 56.7-74.3 vs 75.6%; 95% CI 67.2-82.5). The mean (SD) contact force calculated in the SMART-AF trial of 17.9 (9.4) g was lower compared with our finding of 29.2 (14.1) g. Our analysis of the safety cohort revealed a similar number of major adverse events as reported by the study authors: vascular access complications (n = 5), cardiac tamponade (n = 4), pericarditis (n = 4), pericardial effusion (n = 3), heart block (n = 1), pulmonary vein stenosis (n = 0), and atrioesophageal fistula (n = 0). We were unable to reproduce the analyses predicated on proportion of time that investigators spent operating in their selected contact force range because not all variables or information to derive them were available in the data set. Similarly, we could not reproduce the multivariable logistic regression model owing to missing dependent variables. We attempted to resolve these discrepancies with requests for clarification about discordant results and missing or unavailable data through communication with the study authors, YODA team, and trial sponsor analysts. While some of our queries were clarified through this process, other discrepancies remained unresolved.

This study demonstrates the feasibility, process, and outcomes of a reproduction analysis of the SMART-AF trial and identifies potentially valuable lessons as clinical trial data sharing increases. First, the SMART-AF study authors, YODA Project team, and trial sponsors are vanguards in cardiovascular clinical trial data sharing and were under no obligation to share these data or respond to our requests. Clinical trial data sharing requires time and effort that many investigators may be reluctant to spend but is likely necessary for effective clinical trial data sharing.

We closely reviewed the data sets, data dictionary, case report files, and statistical analysis plan and worked to minimize queries to the SMART-AF authors, YODA Project team, and trial sponsor. However, questions invariably arose, even from this relatively small and straightforward, single-arm trial. One potential strategy to minimize the need for specific technical questions may be to include the statistical code in data sharing packages. This would not necessitate that replicating authors use such code as the lone method for reproduction but would provide a valuable reference and further advance transparency in reporting. Journals, such as Biostatistics, encourage its contributing authors to submit all supporting data and code with their original publications as supplemental material or through links to online code-sharing repositories such as GitHub.⁹ As data sharing evolves, software programs that facilitate replication of analytical working environments may become more prevalent. While these tools may improve transparency in the analytical process, analytical independence also remains important for enhancing trust through reproducible research. There may be an inherent tension between primary study teams, who best understand the study procedures and data, and independent research teams seeking information for hypothesis-generating analyses or evidence synthesis activities. This tension is evident when considering questions of fairness in and incentivization of attribution and contribution for clinical trial data sharing.

Methods for optimal clinical trial data sharing remain under development. Commercial and academic platforms, such as ClinicalStudyDataRequest.com and YODA, require centralized review and approval for access to secure online portals. On the other hand, government platforms, such as the National Heart, Lung, and Blood Institute’s Biologic Specimen and Data Repository Information Coordinating Center, only require a local institutional review board letter of approval for the proposed analysis to gain access to data files, which may be a decentralized approach for more efficient data sharing; these methods were recently demonstrated by the Systolic Blood Pressure Intervention Trial (SPRINT) Data Analysis Challenge.¹⁰ Investigators will need to use rigorous deidentification methods to ensure participant confidentiality for this type of data sharing, but limiting barriers to access will likely enhance trust in the clinical research enterprise, which is a key goal for data sharing.

Our reproduction was limited by an inability to attempt certain reanalyses, such as the logistic regression models, owing to the limited availabilty of some data elements through the data-sharing platform. Second, the process of accessing and analyzing data through an academic data-sharing platform is feasible, although the time spent executing data use agreements represented approximately half of the overall project time. The SAS Clinical Trials Data Transparency platform was effective for this reproduction analysis, but other investigators may prefer to download the analyzable data sets and use different statistical packages with which they might be more familiar. Third, we demonstrated the ability to largely reproduce the direction and magnitude of effectiveness and safety of the intervention, although we did demonstrate discrepancies related to success rate, which may represent a true yet imprecise difference of effect that clinicians and patients may judge to be significant. Further, we could not reproduce all reported analyses, including inferential analyses and those related to contact force range. Our inability to exactly replicate the analysis was likely based on a number of factors (eg, data availability, interpretation of the inclusion and exclusion criteria, assumptions during data cleaning, differences in the analytic methods and statistical package, or combination thereof) but also acknowledge these discrepancies may have been related to our own errors.

We demonstrate the feasibility, process, and outcomes of a reproduction analysis of the SMART-AF clinical trial using the YODA Project academic data-sharing platform. Our findings were largely similar to the results reported by the study authors. We identified and described areas for potentially improving the growing clinical trial data-sharing process and enterprise based on our experience. Such proposals include incentivizing greater collaboration between original investigators and independent researchers, sharing statistical code, and more decentralized data-sharing platforms, with fewer restrictions to data access.

Corresponding Author: Mark D. Huffman, MD, MPH, Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, 680 N Lake Shore Dr, Ste 1400, Chicago, IL 60611 (m-huffman@northwestern.edu).

Accepted for Publication: August 30, 2017.

Published Online: October 18, 2017. doi:10.1001/jamacardio.2017.3808

Author Contributions: Drs Gay and Huffman had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Gay, Huffman.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Gay.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Gay, Baldridge.

Administrative, technical, or material support: Baldridge, Huffman.

Supervision: Huffman.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Huffman receives support from the World Heart Federation to serve as its senior program advisor for the Emerging Leaders program, which is supported by unrestricted educational grants from Boehringer-Ingelheim and Novartis and has been previously supported by BUPA and AstraZeneca.

Disclaimer: Dr Huffman is Associate Editor of JAMA Cardiology but was not involved in any of the decisions regarding review of the manuscript or its acceptance.

Additional Contributions: We thank original study authors and sponsor for sharing their data through the Yale Open Data AccessProject platform, and for working with the Yale Open Data Access Project to respond to our queries. We also thank the Yale Open Data Access Project and their representatives for their assistance and collaboration throughout this process. The code used for this analysis is provided in a repository at https://github.com/abigailbaldridge/SMART-AF-Reproduction.