Assessment of Concordance and Discordance Among Clinical Studies Posted as Preprints and Subsequently Published in High-Impact Journals | Medical Journals and Publishing | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Table 1.  Characteristics of Clinical Study Preprints on medRxiv With a Corresponding Peer-Reviewed Journal Article in a High-Impact Journala
Characteristics of Clinical Study Preprints on medRxiv With a Corresponding Peer-Reviewed Journal Article in a High-Impact Journala
Table 2.  Concordance Between Clinical Study Preprints and Articles Published in High-Impact Journals
Concordance Between Clinical Study Preprints and Articles Published in High-Impact Journals
1.
Krumholz  HM, Bloom  T, Sever  R, Rawlinson  C, Inglis  JR, Ross  JS.  Submissions and downloads of preprints in the first year of medRxiv.   JAMA. 2020;324(18):1903-1905. doi:10.1001/jama.2020.17529PubMedGoogle ScholarCrossref
2.
Massey  DS, Opare  MA, Wallach  JD, Ross  JS, Krumholz  HM.  Assessment of preprint policies of top-ranked clinical journals.   JAMA Netw Open. 2020;3(7):e2011127. doi:10.1001/jamanetworkopen.2020.11127PubMedGoogle Scholar
3.
Maslove  DM.  Medical preprints: a debate worth having.   JAMA. 2018;319(5):443-444. doi:10.1001/jama.2017.17566PubMedGoogle ScholarCrossref
4.
Becker  JE, Krumholz  HM, Ben-Josef  G, Ross  JS.  Reporting of results in ClinicalTrials.gov and high-impact journals.   JAMA. 2014;311(10):1063-1065. doi:10.1001/jama.2013.285634PubMedGoogle ScholarCrossref
5.
Klein  M, Broadwell  P, Farb  SE, Grappone  T.  Comparing published scientific journal articles to their pre-print versions.   Int J Digit Libr. 2019;20(4):335-350. doi:10.1007/s00799-018-0234-1Google ScholarCrossref
6.
Flanagin  A, Fontanarosa  PB, Bauchner  H.  Preprints involving medical research: do the benefits outweigh the challenges?   JAMA. 2020;324(18):1840-1843. doi:10.1001/jama.2020.20674PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 2,312
    Citations 0
    Research Letter
    Medical Journals and Publishing
    March 18, 2021

    Assessment of Concordance and Discordance Among Clinical Studies Posted as Preprints and Subsequently Published in High-Impact Journals

    Author Affiliations
    • 1Department of Environmental Health Sciences, Yale School of Public Health, New Haven, Connecticut
    • 2Section of General Internal Medicine, Yale School of Medicine, New Haven, Connecticut
    • 3Yale College, New Haven, Connecticut
    • 4Department of Emergency Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland
    • 5Section of Cardiovascular Medicine, Yale School of Medicine, New Haven, Connecticut
    JAMA Netw Open. 2021;4(3):e212110. doi:10.1001/jamanetworkopen.2021.2110
    Introduction

    Dissemination of clinical and health science research as preprints—that is, preliminary reports of studies that have not yet undergone peer review—has grown rapidly since the launch of the medRxiv preprint server in 2019.1,2 Although not all preprints will subsequently be published in peer-reviewed journals, among those that are, the extent of the changes to studies’ reporting of design, conduct, and results remains unknown.3 Accordingly, in this cross-sectional study, we examined the concordance between clinical studies posted as preprints and subsequently published in high-impact journals (preprint–journal article pairs), including key study characteristics, reported results, and study interpretations.

    Methods

    This study did not require institutional review board approval because it was based on publicly available information, in accordance with 45 CFR §46. Informed consent was not needed because no patient data were used.

    We used the bioRxiv/medRxiv Application Programming Interface to identify all preprint–journal article pairs from medRxiv’s inception on June 11, 2019, through August 13, 2020. We limited our sample to clinical trials, prospective and retrospective observational studies, and systematic reviews with meta-analyses reporting health-related outcomes that were published in peer-reviewed journals with a 2019 impact factor greater than 10 according to InCites Journal Citation Reports. For preprint–journal article pairs, we abstracted the number of authors; first and senior authors; funding, conflict of interest, and institutional review board disclosure statements; and sample size (number of patients for trials and observational studies or number of studies for meta-analyses). In addition, we identified the prespecified primary end points, corresponding results from inferential analyses, and overarching abstract-level conclusions and study interpretations; for studies without clearly defined end points, we recorded all abstract-level end points and results or categorized the study as purely descriptive. Building on previously published methods,4 disclosure statements, sample size, number of authors, and end points were considered concordant if they contained the same information or had numerical equivalence. Results from inferential analyses were considered discordant if effect estimates and/or 95% CIs or P values changed. For preprint–journal article pairs with descriptive results, we did not determine whether results were concordant or discordant. Descriptive analyses were conducted using Excel software version 16.0 (Microsoft) from November 2020 to January 2021.

    Results

    Among 8941 preprints posted on medRxiv, 944 had been subsequently published in a peer-reviewed journal as of our search date. Of these, 47 were clinical studies published in a high-impact journal (Table 1). Most preprint–journal article pairs were retrospective studies (35 of 47 pairs [74%]) and were related to the coronavirus disease 2019 pandemic (28 of 47 pairs [60%]).

    Among the 47 preprint–journal article pairs, 46 (98%) reported the same first and senior authors and 36 (77%) had the same total number of authors (Table 2). Most of the pairs had concordant funding statements (39 pairs [83%]) and institutional review board disclosures (44 pairs [94%]). Of the 17 pairs (36%) with discordant conflict of interest statements, there were 8 where only the journal articles contained a conflict of interest disclosure. Among the 44 pairs reporting sample sizes in both sources, 38 were concordant. Sample sizes were larger in the journal articles for nearly all discordant pairs (5 of 6 pairs [83%]).

    Overall, 44 preprint–journal article pairs (94%) reported concordant primary end points (Table 2). Ten of 47 pairs (21%) were descriptive (noninferential) studies without clearly comparable numerical results. Among the remaining 37 pairs reporting primary results from inferential analyses, 12 (32%) reported discordant results; 7 had effect estimates that were in the same direction and were statistically consistent. Nearly all pairs (45 of 46 pairs [98%]) with clearly stated study interpretations were concordant, including all pairs with discordant results.

    Discussion

    Although preprints are preliminary reports that should be critically assessed, similar to prior studies of text content,5 our findings suggest that the overarching study interpretations, as well as other study design details described in preprints of clinical studies that are subsequently published in high-impact journals, did not change markedly. Our study was limited to medRxiv preprints published in high-impact journals and may represent the best-case scenario; we did not account for preprints never submitted to or rejected by journals, nor could we determine whether changes improved research reporting accuracy. Given concerns about the role of preprints in clinical research,3,6 future evaluations should monitor the concordance of preprint–journal article pairs of clinical studies.

    Back to top
    Article Information

    Accepted for Publication: January 26, 2021.

    Published: March 18, 2021. doi:10.1001/jamanetworkopen.2021.2110

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Shi X et al. JAMA Network Open.

    Corresponding Author: Joshua D. Wallach, PhD, MS, Department of Environmental Health Sciences, Yale School of Public Health, 60 College St, 4th Flr, Rm 411, New Haven, CT 06510 (joshua.wallach@yale.edu).

    Author Contributions: Ms Shi and Dr Wallach had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Shi, Ross, Niforatos, Krumholz, Wallach.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Shi, Ross, Krumholz, Wallach.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Shi, Wallach.

    Supervision: Wallach.

    Conflict of Interest Disclosures: Dr Ross reported receiving grants from the US Food and Drug Administration, Johnson and Johnson, Medical Devices Innovation Consortium, Agency for Healthcare Research and Quality, National Institutes of Health and National Heart, Lung, and Blood Institute, and the Laura and John Arnold Foundation outside the submitted work; and reported being a founder of medRxiv but not receiving remuneration for the role. Dr Krumholz reported receiving personal fees from UnitedHealth, IBM Watson Health, Element Science, Aetna, Facebook, Siegfried & Jensen Law Firm, Arnold & Porter Law Firm, Martin/Baughman Law Firm, F-Prime, and National Center for Cardiovascular Diseases, Beijing; being a cofounder of HugoHealth and Refactor Health; receiving Centers for Medicare & Medicaid Services contracts through Yale New Haven Hospital to develop and maintain measures of hospital performance; and receiving grants from Medtronic, the US Food and Drug Administration, Johnson and Johnson, and Shenzhen Center for Health Information outside the submitted work. Drs Ross and Krumholz reported being cofounders of medRxiv, the preprint server for health sciences research, and Dr Wallach reported being a medRxiv affiliate; none of these individuals received any compensation for their roles. Dr Wallach reported receiving grants from the Laura and John Arnold Foundation and the Yale–Mayo Clinic Center for Excellence in Regulatory Science and Innovation outside the submitted work. No other disclosures were reported.

    Funding/Support: Ms Shi is supported by the China Scholarship Council. Dr Wallach is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health and under award K01AA028258.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Additional Information: The bioRxiv/medRxiv Application Programming Interface can be found at https://api.biorxiv.org/pubs/help.

    References
    1.
    Krumholz  HM, Bloom  T, Sever  R, Rawlinson  C, Inglis  JR, Ross  JS.  Submissions and downloads of preprints in the first year of medRxiv.   JAMA. 2020;324(18):1903-1905. doi:10.1001/jama.2020.17529PubMedGoogle ScholarCrossref
    2.
    Massey  DS, Opare  MA, Wallach  JD, Ross  JS, Krumholz  HM.  Assessment of preprint policies of top-ranked clinical journals.   JAMA Netw Open. 2020;3(7):e2011127. doi:10.1001/jamanetworkopen.2020.11127PubMedGoogle Scholar
    3.
    Maslove  DM.  Medical preprints: a debate worth having.   JAMA. 2018;319(5):443-444. doi:10.1001/jama.2017.17566PubMedGoogle ScholarCrossref
    4.
    Becker  JE, Krumholz  HM, Ben-Josef  G, Ross  JS.  Reporting of results in ClinicalTrials.gov and high-impact journals.   JAMA. 2014;311(10):1063-1065. doi:10.1001/jama.2013.285634PubMedGoogle ScholarCrossref
    5.
    Klein  M, Broadwell  P, Farb  SE, Grappone  T.  Comparing published scientific journal articles to their pre-print versions.   Int J Digit Libr. 2019;20(4):335-350. doi:10.1007/s00799-018-0234-1Google ScholarCrossref
    6.
    Flanagin  A, Fontanarosa  PB, Bauchner  H.  Preprints involving medical research: do the benefits outweigh the challenges?   JAMA. 2020;324(18):1840-1843. doi:10.1001/jama.2020.20674PubMedGoogle ScholarCrossref
    ×