Effect and Reach of Medical Articles Posted on Preprint Servers During the COVID-19 Pandemic | Medical Journals and Publishing | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Table 1.  Attention and Online Engagement Metrics for Preprints and Publications About COVID-19 Therapiesa
Attention and Online Engagement Metrics for Preprints and Publications About COVID-19 Therapiesa
Table 2.  Comparison of Attention and Online Engagement Metrics of MedXriv Preprints About COVID-19 Therapies That Were Published and Not Published During the Study Perioda
Comparison of Attention and Online Engagement Metrics of MedXriv Preprints About COVID-19 Therapies That Were Published and Not Published During the Study Perioda
1.
Iredale  J, Al-Shahi Salman  R.  Peer review: a flawed but essential process.   J R Coll Physicians Edinb. 2012;42(4):290-291. doi:10.4997/JRCPE.2012.401PubMedGoogle ScholarCrossref
2.
Smith  R.  Peer review: a flawed process at the heart of science and journals.   J R Soc Med. 2006;99(4):178-182. doi:10.1177/014107680609900414PubMedGoogle ScholarCrossref
3.
Lauer  MS, Krumholz  HM, Topol  EJ.  Time for a prepublication culture in clinical research?   Lancet. 2015;386(10012):2447-2449. doi:10.1016/S0140-6736(15)01177-0PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Research Letter
    November 9, 2020

    Effect and Reach of Medical Articles Posted on Preprint Servers During the COVID-19 Pandemic

    Author Affiliations
    • 1Medical School for International Health, Ben-Gurion University of the Negev, Beersheva, Israel
    • 2Mailman School of Public Health, Columbia University, New York, New York
    • 3New York Medical College, Valhalla, New York
    JAMA Intern Med. 2021;181(3):395-397. doi:10.1001/jamainternmed.2020.6629

    There are ongoing controversies about the evaluation and dissemination of medical and scientific research.1-3 Preprint servers offer researchers the opportunity to post manuscripts before publication in peer-reviewed journals and regardless of whether they have been submitted to journals for review. The global coronavirus disease 2019 (COVID-19) pandemic has focused attention on the timely reporting and dissemination of research findings, and the respective roles of preprint servers and traditional peer-reviewed journals. We compared the effect and reach of studies about therapies for COVID-19 posted on the medRxiv preprint server, subsequent publications in medical journals of some of these studies, and journal articles that were not posted on either medRxiv or another preprint server.

    Methods

    We identified preprints and peer-reviewed publications of studies about clinical outcomes of proposed COVID-19 treatments, including chloroquine and hydroxychloroquine, lopinavir-ritonavir, angiotensin receptor blockers, angiotensin-converting enzyme inhibitors, convalescent plasma, and corticosteroids. The institutional review board at New York Medical College deemed the study as non–human participants research. We included articles in English posted at medrxiv.org or published between February 1 and May 10, 2020, as well as publications from these articles in medical journals between February 1 and July 10, 2020. We obtained data on attention and online engagement for each article as a preprint, and if published as a journal article, from February 1 to July 10, 2020. The data included the Altmetric attention score and the number of news outlets citing the article, individuals posting about the article on Twitter, page views, and scientific citations. Full details of the methods and search strategy are provided in the Supplement. To compare data between preprints and publications, we used bootstrapping to conduct nonparametric tests on the difference in medians. Statistical analyses were conducted using SPSS, version 26 (IBM), and statistical significance was set at P < .05.

    Results

    Between February 1 and May 10, 2020, we identified 45 publications in peer-reviewed journals that met the search criteria and had not been posted as preprints on medRxiv and 17 that had previously been posted; 1 preprint led to 2 publications. In addition, we identified 18 preprints on medRxiv that had not been published in a journal by the cutoff date of July 10, 2020. Table 1 shows the attention and online engagement metrics for the 34 preprints and the 62 publications. Although the publications had significantly more citations than the preprints (median [interquartile range] of 22 [4.3-52.5] vs 5.5 [1.3-20.3] citations; P = .01), there were no significant differences for the other metrics. There were also wide variations in the metrics between individual studies. The mean impact factor of the journals in which COVID-19–related articles were published was similar to that for articles that had been posted as preprints and those that had not (12.3 [95% CI, 6.2-18.4] vs 13.8 [95% CI, 7.6-20.0]; P = .20).

    Table 2 shows the attention and online engagement metrics of the 34 medRxiv preprints that were published and not published during the study period. Compared with the 18 preprint articles that were not published, the median Altmetric attention scores, number of news outlets citing, and the number of page views were significantly higher for the 16 preprint articles that were published, and there were trends toward greater numbers of individuals posting on Twitter and citations.

    Discussion

    In a small study comparing articles about therapies for COVID-19 posted on the medRxiv preprint server, subsequent publications in medical journals of some of these articles, and journal articles that were not posted on either medRxiv or another preprint server, we found widespread dissemination of reports that had not undergone traditional peer review. We also found that articles posted as preprints but that were not published during the study period received less attention that those that did across multiple metrics.

    This study has limitations in addition to the relatively small number of reports considered and the short duration of the analysis. Attention and online engagement metrics are inexact and shed little light on how the traffic is being generated. We did not compare the metrics for the published and preprint versions of the same study, as the available follow-up time for the studies and versions varied widely. We also did not study the associations of any of the articles with medical practice, regardless of whether they were first posted on medRxiv or published in a medical journal.

    Back to top
    Article Information

    Accepted for Publication: September 22, 2020.

    Published Online: November 9, 2020. doi:10.1001/jamainternmed.2020.6629

    Corresponding Author: Neil W. Schluger, MD, Department of Medicine, New York Medical College, 40 Sunshine Cottage Rd, Valhalla, NY 10595 (nschluge@nymc.edu).

    Author Contributions: Dr Schluger had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Jung, Schluger.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Jung, Schluger.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: All authors.

    Administrative, technical, or material support: Jung, Schluger.

    Supervision: Schluger.

    Conflict of Interest Disclosures: None reported.

    References
    1.
    Iredale  J, Al-Shahi Salman  R.  Peer review: a flawed but essential process.   J R Coll Physicians Edinb. 2012;42(4):290-291. doi:10.4997/JRCPE.2012.401PubMedGoogle ScholarCrossref
    2.
    Smith  R.  Peer review: a flawed process at the heart of science and journals.   J R Soc Med. 2006;99(4):178-182. doi:10.1177/014107680609900414PubMedGoogle ScholarCrossref
    3.
    Lauer  MS, Krumholz  HM, Topol  EJ.  Time for a prepublication culture in clinical research?   Lancet. 2015;386(10012):2447-2449. doi:10.1016/S0140-6736(15)01177-0PubMedGoogle ScholarCrossref
    ×