Malički M, von Elm E, Marušić A. Study Design, Publication Outcome, and Funding of Research Presented at International Congresses on Peer Review and Biomedical Publication. JAMA. 2014;311(10):1065-1067. doi:10.1001/jama.2014.143
The first International Congress on Peer Review and Biomedical Publication (PRC) was organized in 1989 to “subject the editorial review process to some of the rigorous scrutiny that editors and reviewers demand of the scientists whose work they are assessing.”1 Since then, peer review research was introduced as a Medical Subject Heading (MeSH),2 and the number of indexed articles has been continuously increasing. To evaluate the development of peer review research in biomedicine, we analyzed research presented at all PRCs since 1989.
We established a retrospective cohort of PRC poster and podium abstracts and used author names to search the PRC’s website, Web of Science (WoS), and PubMed for full articles through August 20, 2013. We collected data on authorship, time to publication, declared funding sources, article availability, and citation counts in WoS. Two authors classified study design independently, with acceptable agreement (κ = 0.78). Data were analyzed using MedCalc; 2-sided significance testing included χ2 tests and Kruskal-Wallis test (type I error was set to P < .05).
Of 614 presented abstracts, 75% described observational studies; 18%, intervention studies, and 7%, opinion pieces (Table 1). Over time, the number of opinion pieces decreased from 17 in 1989 to 1 in 2013 (χ21 for trend = 47.3, P < .001). The number of cohort studies increased from 0 in 1989 to 8 in 2013 (χ21 = 10.7, P = .002). Feasibility studies increased from 1 in 1989 to 20 in 2013 (χ21 = 11.3, P < .001). The median number of abstract authors increased from 1 (95% CI, 1-1) in 1989 to 4 (95% CI, 4-5) in 2013 (P < .001). Of the 504 abstract presentations from the first 6 PRCs, 305 (61%) led to 294 published articles (Table 2). From abstract presentation to publication, there were no changes in the byline order or number of byline authors in 166 abstracts (56%), whereas 83 abstracts (28%) had changes in the number of authors listed and 45 (15%) had changes in the byline order. One hundred fourteen articles (38%) were published in JAMA, 21 (7%) in BMJ, 12 (4%) in Annals of Emergency Medicine, and 8 (3%) each in the Journal of Clinical Epidemiology and in PLoS ONE (Table 2). The median time to publication was 14 months (95% CI, 12-16), when excluding 110 articles in JAMA theme issues. One hundred articles (63%) were freely available online.
Funding was reported in 106 (36%) published articles that had been presented as abstracts at the 1989-2009 PRCs and in 45 abstracts (41%) presented at the 2013 PRC, most commonly from public or charity sources (Table 1). The absolute number and proportion of articles with declared funding increased over time, with a peak in 2005 (Table 2).
Two hundred eighty-four published articles (97%) were indexed in WoS; 265 (93%) of them received at least 1 citation, with a median of 20 (95% CI, 17-27) citations per article. Articles with the most citations were on a reporting guideline for health research3 (published in 17 journals; n = 1798 citations), synthesis of evidence4 (n = 1016), and publication bias5 (n = 547).
Peer review research uses various study designs and is published in a broad spectrum of journals. However, experimental studies aimed at improving methods of peer review and reporting of biomedical research are still underrepresented. Although the peer review research community is aware of the consequences of nonpublication of research,6 39% of studies presented at PRCs have not been fully published. In our cohort, we were unable to determine whether the underreporting was selective (eg, favoring positive results) and were not able to determine its causes. Lack of suitable journal outlets is an unlikely explanation because there was no decrease in publication output after JAMA ceased its PRC theme issues in 2005. Because our cohort represents research presented for more than 20 years at the discipline’s major meeting, it may have limited generalizability to research presented elsewhere.
Peer review and other editorial procedures have the potential to significantly influence the knowledge base of health care. Despite their critical role in biomedical publishing, methods of peer review are still underresearched and lack dedicated funding. Systematic and competitive funding schemes are needed to build and sustain excellence, innovation, and methodological rigor in peer review research.
Corresponding Author: Ana Marušić, MD, PhD, Department of Research in Biomedicine and Health, University of Split School of Medicine, Šoltanska 2, 21000 Split, Croatia (firstname.lastname@example.org).
Author Contributions: Dr Marušić had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: von Elm, Marušić.
Acquisition of data: all authors.
Analysis and interpretation of data: all authors.
Drafting of the manuscript: all authors.
Critical revision of the manuscript for important intellectual content: all authors.
Statistical analysis: Malički.
Administrative, technical, and material support: Malički, Marušić.
Study supervision: von Elm, Marušić.
Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Drs Marušić and von Elm are members of the Advisory Board of the Seventh International Congress on Peer Review and Biomedical Publication.
Previous Presentation: This study was presented at the Seventh International Congress on Peer Review and Biomedical Publication, Chicago, Illinois, September 8-10, 2013.
Additional Contributions: We thank the Esteve Foundation for organizing a Discussion Group on Editorial Research in Barcelona (Catalonia, Spain) on December 12-13, 2012, which gave us the idea for the study. We thank Annette Flanagin, RN, MA, and Drummond Rennie, MD, at JAMA for providing access to the abstracts and JAMA articles from the early Peer Review Congresses, for which they were not compensated.