[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Purchase Options:
[Skip to Content Landing]
Research Letter
January 23/30, 2018

Altmetric Scores, Citations, and Publication of Studies Posted as Preprints

Author Affiliations
  • 1Department of Health Research and Policy, Stanford University School of Medicine, Stanford, California
  • 2Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California
JAMA. 2018;319(4):402-404. doi:10.1001/jama.2017.21168

Preprints are versions of articles that are made publicly available prior to peer-reviewed publication, are widely used in physical sciences,1 and are now emerging in life sciences.2 Preprints provide immediate access to new information; however, articles not formally peer reviewed may contain errors in methods, results, or interpretation.3,4

As preprints in medicine are debated, data on how preprints are used, cited, and published are needed. We evaluated views and downloads and Altmetric scores and citations of preprints and their publications. We also assessed whether Altmetric scores and citations of published articles correlated with prior preprint posting.


We downloaded all information from the preprint repository bioRxiv on all preprints posted between November 7, 2013, and January 17, 2017 (including publication status), and all data from Altmetric and CrossRef. Altmetric records mentions of articles in the media and creates an “attention score,” with a score of more than 20 corresponding to articles in the top 5%. CrossRef records citation counts. We similarly downloaded all data from PubMed, Altmetric, and CrossRef on September 16, 2017, for all article publications of preprints. The probability of publication was analyzed with Kaplan-Meier estimates. Published articles were compared with their preprints using the sign test.

We also randomly selected 30% of the articles with a preprint on bioRxiv; 30% was chosen to balance power vs computational processing time. PubMed was then searched for up to 5 research articles without a preprint matched for being published in the same journal during the same period as each of the selected articles. Matched articles were compared for Altmetric scores and CrossRef citations using the Friedman test. Statistical significance for 2-tailed P values was claimed at P < .005 based on prior recommendations.4 Data search, extraction, and analyses used R version 3.4.1 (R Foundation for Statistical Computing).


Of 7760 preprints, 7750 were unique. Preprint availability on bioRxiv increased over time (from a median of 54/month in 2013 to median of 392/month in 2016; Table). The bioRxiv-defined disciplines with the most preprints were bioinformatics (15.8%), evolutionary biology (13.7%), neuroscience (12.6%), and genomics (11.8%); only 3 preprints were labeled as clinical trials.

Descriptive Characteristics of Preprints in bioRxiv
Descriptive Characteristics of Preprints in bioRxiv

The median number of preprint abstracts views was 924 (range, 6-192 570) and the median number of preprint PDF downloads was 321 (range, 2-151 520). The median Altmetric score was 7.3 (range, 0-2506) and the median CrossRef citation count was 0 (range, 0-55); 18.2% (1414/7750) of preprints achieved an Altmetric score of more than 20. Of 7750 preprints, 2628 articles (34%) were published in a peer-reviewed journal. The probability of publication in the peer-reviewed literature was 48% within 12 months and 55.5% within 24 months.

The median Altmetric score of the published articles was 8.8, and of their respective preprint, 8.4 (median pairwise difference, −0.3; interquartile range [IQR], −5.4 to 7.7; P = .17). The median number of citations for published articles was 5, and of their respective preprint, 0 (median pairwise difference, 5 [IQR, 2 to 12]; P < .001).

The sample of 776 published articles with preprints was matched to 3647 published articles without preprints. Published articles with preprints had significantly higher Altmetric scores than published articles without preprints (median, 9.5 [IQR, 3.1 to 35.3] vs 3.5 [IQR, 0.8 to 12.2], respectively; between-group difference, 4 [IQR, 0 to 15]; P < .001) and received more citations (median, 4 [IQR, 1 to 10] vs 3 [IQR, 1 to 7]; between-group difference, 1 [IQR, −1 to 5]; P < .001).


The number of preprints posted on bioRxiv rapidly increased between 2013 and 2016, much more than the increase in MEDLINE-indexed publications during the same period (1.2%).5 Although preprints were not well cited, 18% had Altmetric scores in the top 5% and 48% were estimated to reach peer-reviewed publication within 1 year. Articles with a preprint received higher Altmetric scores and more citations than articles without a preprint. These results add to a previous report from bioRxiv6 by also quantifying social media attention and citations received by preprints and published articles and comparing articles with and without preprints.

This analysis has limitations. First, it was limited to a few years during which preprint posting has rapidly evolved; patterns may change over time. Second, only a short time was available for preprints to be published; the rate of publication is therefore an underestimate. Third, the association between Altmetric scores and citations in articles with and without preprints may not be causal because differences between authors choosing to post or not to post a preprint were not considered.

Section Editor: Jody W. Zylke, MD, Deputy Editor.
Back to top
Article Information

Accepted for Publication: December 14, 2017.

Corresponding Author: John P. A. Ioannidis, MD, DSc, Meta-Research Innovation Center at Stanford (METRICS), Stanford University, 1265 Welch Rd, Stanford, CA 94305 (jioannid@stanford.edu).

Author Contributions: Drs Serghiou and Ioannidis had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Both authors.

Acquisition, analysis, or interpretation of data: Both authors.

Drafting of the manuscript: Serghiou.

Critical revision of the manuscript for important intellectual content: Both authors.

Statistical analysis: Both authors.

Administrative, technical, or material support: Serghiou.

Supervision: Ioannidis.

Conflict of Interest Disclosures: Both authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: METRICS is funded by a grant from the Laura and John Arnold Foundation. The work of Dr Ioannidis is supported by an unrestricted gift from Sue and Bob O’Donnell.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Meeting Presentation: The results of this study were presented at the Eighth International Congress on Peer Review and Scientific Publication; September 12, 2017; Chicago, Illinois.

Additional Contributions: We acknowledge assistance received by Altmetric in using its application program interface. Our code and data can be found at https://github.com/serghiou.

Butler  D.  Biologists join physics preprint club.  Nature. 2003;425(6958):548.PubMedGoogle ScholarCrossref
Bourne  PE, Polka  JK, Vale  RD, Kiley  R.  Ten simple rules to consider regarding preprint submission.  PLoS Comput Biol. 2017;13(5):e1005473.PubMedGoogle ScholarCrossref
Annesley  T, Scott  M, Bastian  H,  et al.  Biomedical journals and preprint services: friends or foes?  Clin Chem. 2017;63(2):453-458.PubMedGoogle ScholarCrossref
Benjamin  DJ, Berger  JO, Johannesson  M,  et al.  Redefine statistical significance [published online September 1, 2017].  Nat Hum Behav. doi:10.1038/s41562-017-0189-zGoogle Scholar
US National Library of Medicine. Yearly citation totals from 2017 MEDLINE/PubMed baseline: 26,759,399 citations found. https://www.nlm.nih.gov/bsd/licensee/2017_stats/2017_Totals.html. Accessed November 29, 2017.
Inglis  JR, Sever  R. bioRxiv: a progress report. http://asapbio.org/biorxiv. Accessed September 18, 2017.