eFigure. Search strategy and selection of papers for the analysis
eTable 1. References for COVID-19 preprints
eTable 2. References for COVID-19 publications
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Jung YE, Sun Y, Schluger NW. Effect and Reach of Medical Articles Posted on Preprint Servers During the COVID-19 Pandemic. JAMA Intern Med. 2021;181(3):395–397. doi:10.1001/jamainternmed.2020.6629
There are ongoing controversies about the evaluation and dissemination of medical and scientific research.1-3 Preprint servers offer researchers the opportunity to post manuscripts before publication in peer-reviewed journals and regardless of whether they have been submitted to journals for review. The global coronavirus disease 2019 (COVID-19) pandemic has focused attention on the timely reporting and dissemination of research findings, and the respective roles of preprint servers and traditional peer-reviewed journals. We compared the effect and reach of studies about therapies for COVID-19 posted on the medRxiv preprint server, subsequent publications in medical journals of some of these studies, and journal articles that were not posted on either medRxiv or another preprint server.
We identified preprints and peer-reviewed publications of studies about clinical outcomes of proposed COVID-19 treatments, including chloroquine and hydroxychloroquine, lopinavir-ritonavir, angiotensin receptor blockers, angiotensin-converting enzyme inhibitors, convalescent plasma, and corticosteroids. The institutional review board at New York Medical College deemed the study as non–human participants research. We included articles in English posted at medrxiv.org or published between February 1 and May 10, 2020, as well as publications from these articles in medical journals between February 1 and July 10, 2020. We obtained data on attention and online engagement for each article as a preprint, and if published as a journal article, from February 1 to July 10, 2020. The data included the Altmetric attention score and the number of news outlets citing the article, individuals posting about the article on Twitter, page views, and scientific citations. Full details of the methods and search strategy are provided in the Supplement. To compare data between preprints and publications, we used bootstrapping to conduct nonparametric tests on the difference in medians. Statistical analyses were conducted using SPSS, version 26 (IBM), and statistical significance was set at P < .05.
Between February 1 and May 10, 2020, we identified 45 publications in peer-reviewed journals that met the search criteria and had not been posted as preprints on medRxiv and 17 that had previously been posted; 1 preprint led to 2 publications. In addition, we identified 18 preprints on medRxiv that had not been published in a journal by the cutoff date of July 10, 2020. Table 1 shows the attention and online engagement metrics for the 34 preprints and the 62 publications. Although the publications had significantly more citations than the preprints (median [interquartile range] of 22 [4.3-52.5] vs 5.5 [1.3-20.3] citations; P = .01), there were no significant differences for the other metrics. There were also wide variations in the metrics between individual studies. The mean impact factor of the journals in which COVID-19–related articles were published was similar to that for articles that had been posted as preprints and those that had not (12.3 [95% CI, 6.2-18.4] vs 13.8 [95% CI, 7.6-20.0]; P = .20).
Table 2 shows the attention and online engagement metrics of the 34 medRxiv preprints that were published and not published during the study period. Compared with the 18 preprint articles that were not published, the median Altmetric attention scores, number of news outlets citing, and the number of page views were significantly higher for the 16 preprint articles that were published, and there were trends toward greater numbers of individuals posting on Twitter and citations.
In a small study comparing articles about therapies for COVID-19 posted on the medRxiv preprint server, subsequent publications in medical journals of some of these articles, and journal articles that were not posted on either medRxiv or another preprint server, we found widespread dissemination of reports that had not undergone traditional peer review. We also found that articles posted as preprints but that were not published during the study period received less attention that those that did across multiple metrics.
This study has limitations in addition to the relatively small number of reports considered and the short duration of the analysis. Attention and online engagement metrics are inexact and shed little light on how the traffic is being generated. We did not compare the metrics for the published and preprint versions of the same study, as the available follow-up time for the studies and versions varied widely. We also did not study the associations of any of the articles with medical practice, regardless of whether they were first posted on medRxiv or published in a medical journal.
Accepted for Publication: September 22, 2020.
Published Online: November 9, 2020. doi:10.1001/jamainternmed.2020.6629
Corresponding Author: Neil W. Schluger, MD, Department of Medicine, New York Medical College, 40 Sunshine Cottage Rd, Valhalla, NY 10595 (email@example.com).
Author Contributions: Dr Schluger had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Jung, Schluger.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Jung, Schluger.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: All authors.
Administrative, technical, or material support: Jung, Schluger.
Conflict of Interest Disclosures: None reported.