Editorial Evaluation, Peer Review, and Publication of Research Reports With and Without Supplementary Online Content

Journal article supplementary online content is an option for publication of data and materials that may not fit in the space allotted to a scientific article. The use of online supplements is increasing,¹ and standards have been published to guide their format, length, and appropriateness.^2,3 However, the value of supplements has been questioned.^4-6 We assessed whether research manuscripts with supplements have a higher likelihood of peer review, acceptance, and online use after publication than research manuscripts without supplements.

This cohort study assessed the numbers and types of supplements with research manuscripts submitted between January 1 and December 31, 2016, to the 3 journals in The JAMA Network with the highest volume of research submissions: JAMA (general medical journal) and 2 specialty journals, JAMA Internal Medicine (JIM) and JAMA Pediatrics (JPED). Research manuscripts included randomized clinical trials (RCTs), meta-analyses, and other observational studies. Research Letters, which do not permit supplements, were excluded.

We compared the rates of obtaining peer review and acceptance for manuscripts with and without supplements and analyzed the peer reviewer comments about the supplements. In addition, we evaluated all research articles and the numbers and types of supplements published in these journals between January 1 and December 31, 2016. We compared online use (views and downloads) of articles vs the supplements between January 1, 2016, and March 31, 2017 (allowing time for viewing articles published later in 2016). Given that supplements are required for RCTs (for trial protocols) and most meta-analyses have supplements, we conducted a second analysis excluding these study types.

We calculated rate differences and 95% CIs and the χ² test for 2-sided P values using Stata version 14.1 (StataCorp). P<.05 defined statistical significance.

Overall, 7985 research manuscripts were submitted to the 3 journals (61.0% [n = 4868] with supplements and 39.0% [n = 3117] without supplements; Table 1). Manuscripts with supplements were more likely to be peer reviewed than those without supplements in JAMA (34.1% vs 16.5%, respectively; between-group difference of 17.6%, 95% CI, 15.1%-20.1%), JIM (17.0% vs 9.0%; between-group difference of 8.0%, 95% CI, 4.8%-11.1%), and JPED (32.9% vs 22.9%; between-group difference of 10.0%, 95% CI, 5.8%-14.3%). Of 1421 manuscripts with supplements that were sent for peer review, 484 (34.1%) were commented on by 538 reviewers and 155 (32.0%) of these comments were substantive (eg, addressed specific content in the supplement or suggested moving supplemental content to the main manuscript).

Manuscripts with supplements were more likely to be accepted than those without supplements in JAMA (3.9% vs 0.3%, respectively; between-group difference of 3.6%, 95% CI, 2.9%-4.4%), JIM (8.4% vs 1.3%; between-group difference of 7.1%, 95% CI, 5.4%-8.9%), and JPED (8.7% vs 0.3%; between-group difference of 8.4%, 95% CI, 6.6%-10.2%). Similar results were seen in the analysis of the 6894 observational studies, which excluded 569 RCTs and 522 meta-analyses (Table 2).

Of 374 published research articles (162 in JAMA, 117 in JIM, and 95 in JPED), 372 had supplements. The medians (interquartile ranges) for online views and downloads of articles were higher compared with those of supplements in JAMA, 10 918 (1324-35 877) vs 37 (1-96), respectively; JIM, 3573 (568-17 654) vs 12 (1-53); and JPED, 1386 (407-5947) vs 5 (1-13). The majority of online views and downloads of articles (76%) occurred within 2 months of publication; for supplements, the majority (77%) took 7 months to occur.

Research manuscripts with online supplements were more likely to be peer reviewed and accepted for publication vs manuscripts without supplements. These findings were shown for all research manuscripts and in the analysis that excluded RCTs and meta-analyses. However, peer reviewers only commented on about one-third of supplements. Online views and downloads of supplements were low compared with views and downloads of the associated articles. Although possibly less valuable to peer reviewers and readers, online supplements may be a marker for complexity and quality of research manuscripts submitted and published, facilitate complete and transparent reporting for studies, and may provide information for replication and future research.

This study was limited by its retrospective nature, the number of journals, and lack of detailed assessment of the methodological characteristics of the included studies and content analysis of the supplements. Future research could include assessment of the content and perceived usefulness of online supplements by authors and readers.

Accepted for Publication: December 8, 2017.

Corresponding Author: Annette Flanagin, RN, MA, JAMA and The JAMA Network, 330 N Wabash Ave, Chicago, IL 60610 (annette.flanagin@jamanetwork.org).

Author Contributions: Mss Flanagin and Christiansen had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Flanagin, Borden, Kyriacou.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Flanagin.

Critical revision of the manuscript for important intellectual content: Christiansen, Borden, Kyriacou, Sietmann, Williams, Bryant.

Statistical analysis: Borden, Kyriacou, Sietmann.

Administrative, technical, or material support: Christiansen, Borden, Bryant.

Supervision: Flanagin.

Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Disclaimer: Mss Flanagin and Christiansen and Dr Kyriacou serve as editors for JAMA but they had no role in the editorial review of or decision on acceptance of the manuscript.

Meeting Presentation: The results of this study were presented at the Eighth International Congress on Peer Review and Scientific Publication; September 11, 2017; Chicago, Illinois.

Additional Contributions: We thank Jaime Reickel, BS, and Allison Newton, BS, of eJournalPress for assistance with data collection, for which no additional compensation was received.