Evaluation of the Inclusion of Studies Identified by the FDA as Having Falsified Data in the Results of Meta-analyses: The Example of the Apixaban Trials | Research, Methods, Statistics | JAMA Internal Medicine | JAMA Network
[Skip to Content]
[Skip to Content Landing]
Figure.  Study Inclusion Flow Diagram
Study Inclusion Flow Diagram
Table.  Sensitivity Analysis Estimate Changes
Sensitivity Analysis Estimate Changes
1.
Garmendia  CA, Bhansali  N, Madhivanan  P.  Research misconduct in FDA-regulated clinical trials: a cross-sectional analysis of warning letters and disqualification proceedings.  Ther Innov Regul Sci. 2018;52(5):592-605.PubMedGoogle Scholar
2.
Seife  C.  Research misconduct identified by the US Food and Drug Administration: out of sight, out of mind, out of the peer-reviewed literature.  JAMA Intern Med. 2015;175(4):567-577. doi:10.1001/jamainternmed.2014.7774PubMedGoogle ScholarCrossref
3.
Slavin  RE.  Best evidence synthesis: an intelligent alternative to meta-analysis.  J Clin Epidemiol. 1995;48(1):9-18. doi:10.1016/0895-4356(94)00097-APubMedGoogle ScholarCrossref
4.
The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). https://training.cochrane.org/handbook. Accessed September 5, 2018.
5.
Office of Research Integrity, US Department of Health and Human Services. Definition of Research Misconduct (April 25, 2011). https://ori.hhs.gov/definition-misconduct. Accessed September 5, 2018.
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Research Letter
    March 4, 2019

    Evaluation of the Inclusion of Studies Identified by the FDA as Having Falsified Data in the Results of Meta-analyses: The Example of the Apixaban Trials

    Author Affiliations
    • 1Office of Bioresearch Monitoring Operations, Office of Regulatory Affairs, US Food and Drug Administration, Miami, Florida
    • 2Department of Epidemiology, Robert Stempel College of Public Health and Social Work, Florida International University, Miami
    • 3Department of Psychology, School of Integrated Science and Humanity, Florida International University, Miami
    • 4Department of Biostatistics, Robert Stempel College of Public Health and Social Work, Florida International University, Miami
    JAMA Intern Med. 2019;179(4):582-584. doi:10.1001/jamainternmed.2018.7661

    Clinical trials under the purview of the US Food and Drug Administration (FDA) have been shown to report falsified data.1 The FDA warns researchers when falsified data are discovered, but these data have made it into the medical literature.1,2 The desire to include all available data in a meta-analysis to obtain the “best estimate” of effect size may result in the inclusion of falsified data, which may compromise future research, policy decisions, and patient care.3 The present study evaluates the inclusion of studies with falsified data in meta-analyses.

    Methods

    As identified by Seife,2 the ARISTOTLE clinical trial, which studied the pharmaceutical agent apixaban, had the most publications containing falsified data. Therefore, we conducted a systematic review, per the Cochran Handbook for Systematic Reviews of Interventions (PROSPERO Identifier: CRD42017055627),4 to identify meta-analyses that contained at least 1 ARISTOTLE clinical trial publication with falsified data. The institutional review board of Florida International University determined that board approval was not required because the study involved no human participants.

    Sensitivity analyses were conducted to evaluate the inclusion of the ARISTOTLE trial in meta-analyses by determining the odds ratios (ORs) and associated 95% confidence intervals (95% CIs). Estimates were calculated using either a fixed- or random-effects model outlined in each meta-analysis publication using the Mantel-Hanszel method.

    Results

    The ARISTOTLE clinical trial was found in 22 meta-analyses (Figure). All meta-analyses found were in English and published between 2012 and 2017. The median number of publications contributing to the analyses was 9 (range, 2-28), and the median meta-analysis publication InCite journal impact factor was 5.658 (range, 3.154-17.202). The median weight of the publication with falsified data toward each meta-analysis was 37.3% (range, 7%-100%).

    In our reanalysis of the 22 meta-analyses, we found that 10 (46%) yielded results that would change the initial meta-analysis findings. Each affected meta-analysis had a median of 9.5 publications (range, 2-17), and the median meta-analysis publication InCite journal impact factor was 5.830 (range, 3.154-17.202). The median weight of publications with falsified data was 55.7% (range, 13.1%-99.6%).

    From our reanalysis of the 22 meta-analyses, we found that 32 of 99 analyses (32%) yielded results that would change the conclusions of the initial analysis (Table). Of the 32 affected estimates, 31 (97%) no longer favored apixaban for the prevention of serious medical issues, and 1 (3%) favored the control.

    Of the 99 analyses, 32 (32%) were subgroup analyses, while 67 (68%) were full analyses (Table). Of the 32 subgroup analyses, 14 (44%) yielded results that would change the conclusions of the initial subgroup analyses. For the full analyses, 18 of 67 estimates (27%) had results that were affected.

    Discussion

    This study found that 46% of all meta-analysis publications had conclusions altered by publications with falsified data, and 32% of all the analyses had a considerable change in the outcome. Overall analyses were more robust than subgroup analyses against the effects of publications with falsified data. For ORs not statistically affected, the estimates generally moved toward the null when more than 1 publication remained.

    This study was limited to only meta-analyses that contained ARISTOTLE publications identified by Seife to contain falsified data.2 Not all the data within the ARISTOTLE publications were falsified. However, because the researchers knowingly published falsified data, a form of research misconduct, we removed all ARISTOTLE data.5

    Our sensitivity analysis results showed that conclusions may be altered in meta-analyses by the inclusion of publications with falsified data. This study should add impetus for robust sensitivity analyses and stronger protections against falsified data. Falsified data can affect not only the original publication, but also any subsequent meta-analyses and any resulting clinical or policy changes resulting from the findings of these studies.

    Back to top
    Article Information

    Accepted for Publication: November 8, 2018.

    Corresponding Author: Craig A. Garmendia, MS, PhD, Department of Epidemiology, Robert Stempel College of Public Health and Social Work, Florida International University, 11200 SW Eighth St, AHC5, 4th Floor, Miami, FL 33199 (cgarm003@fiu.edu).

    Published Online: March 4, 2019. doi:10.1001/jamainternmed.2018.7661

    Author Contributions: Dr Garmendia had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Study concept and design: Garmendia, Trepka, Veledar, Madhivanan.

    Acquisition, analysis, or interpretation of data: Garmendia, Nassar Gorra, Rodriguez.

    Drafting of the manuscript: Garmendia, Nassar Gorra.

    Critical revision of the manuscript for important intellectual content: Garmendia, Rodriguez, Trepka, Veledar, Madhivanan.

    Statistical analysis: Garmendia, Nassar Gorra, Rodriguez, Veledar, Madhivanan.

    Administrative, technical, or material support: Garmendia.

    Study supervision: Garmendia, Trepka, Veledar, Madhivanan.

    Conflict of Interest Disclosures: Dr Garmendia reported grants from Federal Employee Education and Assistance Fund/National Treasury Employee Union during the conduct of the study. Ms Rodriguez reported funding from a predoctoral training program by The Fogarty Institute (Global Health Equity Scholars Program) National Institutes of Health (NIH) FIC D43TW010540. No other disclosures were reported.

    Funding/Support: Dr Garmendia received funding through the Federal Employee Education and Assistance Fund/National Treasury Employee Union. Ms Rodriguez was funded by The Global Health Equity Scholars Program, National Institutes of Health (NIH) (FIC D43TW010540).

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: This article reflects the view of the authors and should not be construed to represent the FDA’s views or policies.

    Additional Contributions: We thank Stanislaw F. Wnuk, PhD, of Florida International University, for his collaboration on the design of this study, Barbara M. Sorondo, MS, MLIS, of Florida International University, for her collaboration in the systematic review method, and Katrina Epnere, MPH, of Statistics Collaborative, for her statistical analysis guidance. They received no compensation for their contributions.

    References
    1.
    Garmendia  CA, Bhansali  N, Madhivanan  P.  Research misconduct in FDA-regulated clinical trials: a cross-sectional analysis of warning letters and disqualification proceedings.  Ther Innov Regul Sci. 2018;52(5):592-605.PubMedGoogle Scholar
    2.
    Seife  C.  Research misconduct identified by the US Food and Drug Administration: out of sight, out of mind, out of the peer-reviewed literature.  JAMA Intern Med. 2015;175(4):567-577. doi:10.1001/jamainternmed.2014.7774PubMedGoogle ScholarCrossref
    3.
    Slavin  RE.  Best evidence synthesis: an intelligent alternative to meta-analysis.  J Clin Epidemiol. 1995;48(1):9-18. doi:10.1016/0895-4356(94)00097-APubMedGoogle ScholarCrossref
    4.
    The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). https://training.cochrane.org/handbook. Accessed September 5, 2018.
    5.
    Office of Research Integrity, US Department of Health and Human Services. Definition of Research Misconduct (April 25, 2011). https://ori.hhs.gov/definition-misconduct. Accessed September 5, 2018.
    ×