Peer Review in a General Medical Research Journal Before and During the COVID-19 Pandemic

IMPORTANCE Although peer review is an important component of publication for new research, the viability of this process has been questioned, particularly with the added stressors of the COVID-19 pandemic. OBJECTIVE To characterize rates of peer reviewer acceptance of invitations to review manuscripts, reviewer turnaround times, and editor-assessed quality of reviews before and after the start of the COVID-19 pandemic at a large, open-access general medical journal. DESIGN, SETTING, AND PARTICIPANTS This retrospective, pre-post cohort study examined all research manuscripts submitted to JAMA Network Open between January 1, 2019, and June 29, 2021, either directly or via transfer from other JAMA Network journals, for which at least 1 peer review of manuscript content was solicited. Measures were compared between the period before the World Health Organization declaration of a COVID-19 pandemic on March 11, 2020 (14.3 months), and the period during the pandemic (15.6 months) among all reviewed manuscripts and between pandemic-period manuscripts that did or did not address COVID-19. MAIN OUTCOMES AND MEASURES For each reviewed manuscript, the number of invitations sent to reviewers, proportions of reviewers accepting invitations, time in days to return reviews, and editor-assessed quality ratings of reviews were determined. RESULTS In total, the journal sought review for 5013 manuscripts, including 4295 Original Investigations (85.7%) and 718 Research Letters (14.3%); 1860 manuscripts were submitted during the prepandemic period and 3153 during the pandemic period. Comparing the prepandemic with the pandemic period, the mean (SD) number of reviews rated as high quality (very good or excellent) per manuscript increased slightly from 1.3 (0.7) to 1.5 (0.7) (P < .001), and the mean (SD) time for reviewers to return reviews was modestly shorter (from 15.8 [7.6] days to 14.4 [7.0] days; P < .001), a difference that persisted in linear regression models accounting for manuscript type, study design, and whether the manuscript addressed COVID-19. CONCLUSIONS AND RELEVANCE In this cohort study, the speed and editor-reported quality of peer reviews in an open-access general medical journal improved modestly during the initial year of the pandemic. Additional study will be necessary to understand how the pandemic has affected reviewer burden and fatigue.


Introduction
The effect of COVID-19 on academic publishing has been the subject of substantial discussion.In particular, the pandemic has reinvigorated conversations about the growing role and variable quality of preprints that do not undergo peer review, 1 the burden of peer review on the academic community, 2 concerns about reviewer fatigue, 3 and how best to ensure the rigor and value of peer-reviewed medical literature. 4,5For medical publishing specifically, increasing volumes of manuscripts related to COVID-19 [6][7][8] and expectations for rapid publication and dissemination have further stressed a system that some in medicine already believed was broken. 9w empirical observations about the ways in which the peer review process may have changed during the pandemic have been reported.A recent study 7 of manuscripts and reviews submitted to 2329 journals before and during the pandemic found a modest decrease in rates of reviewer acceptance of invitations to review among health and medical journals between February and May 2020, with a more pronounced decrease among potential reviewers who were women, but not men.
To understand how peer review changed with the onset of the COVID-19 pandemic, we examined peer review data from JAMA Network Open, an open-access general medical journal launched in 2018 with a 2020 impact factor of 8.5. 10 Specifically, we aimed to quantify changes in rates of peer reviewer acceptance of invitations to review manuscripts and review quality from the period before to the period during the COVID-19 pandemic.

Manuscript Review Process and Data Set Generation
Manuscripts submitted to JAMA Network Open first undergo technical quality assessment, then close evaluation by an editor.For manuscripts that are deemed of sufficient quality and priority to undergo peer review, editors seek review from 1 or more content reviewers and 1 statistical reviewer.We extracted data from databases used by the JAMA Network to track manuscript submissions and peer reviews.We included all manuscripts received at JAMA Network Open from January 1, 2019, through June 30, 2021, that were categorized as Original Investigations or Research Letters for which at least 1 content review was sought.These manuscripts could be submitted directly to the journal or transferred from other journals within the JAMA Network.The study used deidentified administrative data with no participant contact and followed the journal's policy for such research, which indicates that information may be systematically collected and analyzed as part of research to improve the quality of the editorial or peer review process.This study was reviewed by the Massachusetts General-Brigham Institutional Review Board and considered to be exempt from informed consent because it uses deidentified data and posed minimal risk.This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies. 11

Measures
For each manuscript, we determined the total number of individuals who were invited to review the manuscript and the number and proportion of reviewers accepting, declining, or failing to respond to invitations.For the accepted invitations, we determined the mean time (across reviewers) to return the review and the number and proportion of reviews rated by editors as very good or excellent on a 5-point anchored scale: poor, fair, good, very good, or excellent.This process was completed for all manuscripts as part of routine editorial practice.For purposes of analysis, the last 2 categories (very good and excellent) were aggregated to reflect high-quality reviews.For descriptive purposes and because some individuals served as peer reviewers for multiple manuscripts, we also report acceptance rate for reviewer invitations, time to return review, and quality of review at the level of individual reviews.
Additional manuscript features collected at submission included study design 12 (eg, randomized trial, cohort study, or economic evaluation), whether a study was submitted directly or transferred from another JAMA Network journal, and whether a study referenced or addressed COVID-19.The last of these is determined via an automated process implemented to identify any manuscript with any of the following terms in the manuscript text: acute respiratory virus, personal protective equipment, N95, COVID, COVID-19, coronavirus, SARS, or novel virus.

Statistical Analysis
We compared peer review characteristics for submitted manuscripts with an index date before or after March 11, 2020, the date that the World Health Organization declared a COVID-19 pandemic. 13Thus, data were divided into 2 groups: the prepandemic period from January 1, 2019, to March 10, 2020 (14.3 months), and the pandemic period from March 11, 2020, to June 29, 2021 (15.6 months).We similarly compared characteristics of reviews provided for manuscripts submitted after March 11, 2020, that did or did not address COVID-19.We used multivariable linear regression to calculate effect sizes and 95% CIs for the association between prepandemic and pandemic status for reviewer turnaround time, adjusted for other manuscript characteristics (study design, direct submission vs transfer from other JAMA Network journals, article type, and whether the manuscript addressed COVID-19).(Incorporating clustering by subject area, as a proxy for a handling editor, did not meaningfully change results.) To visualize changes in peer review characteristics over time-and, in particular, whether secular trends in these characteristics might have preceded the pandemic-we also described manuscript and review features on a weekly basis, with each manuscript assigned to the date of first reviewer invitation.For graphic presentation, we applied a 3-week rolling mean using the rollmean function in R's zoo library, version 1.8-9.
All analyses used R software, version 4.1.2 14; the threshold for statistical significance was considered to be a 2-tailed P < .05,without adjustment for multiple comparisons.
In multivariable linear regression adjusting for baseline manuscript features (article type, study design, and direct submission vs transfer), differences in mean time to return review persisted.Time to return of review was decreased in the pandemic period compared with the prepandemic period (adjusted mean difference of −1.2 days; 95% CI, −0.7 to −1.6) (Figure 1).
A similar pattern of differences to those observed comparing the prepandemic vs pandemic periods was identified in comparisons of manuscripts that did not address COVID-19 (n = 2238) and those that did address COVID-19 (n = 915) ( The mean (SD) number of very good or excellent reviews was greater for COVID-19related manuscripts (1.5 [0.7] vs those not related to COVID-19 (1.4 [0.7]; P < .001).Mean (SD) time to return reviews was also lower: 14.6 (7.0) days for those not COVID related vs 13.7 (6.8) days for those that were COVID related (P = .002).
Figure 2 illustrates changes in peer reviews over time, with the global number of COVID-19 deaths at the bottom for reference. 15In general, the volume of manuscript submissions increased over time, as did the number of reviewers invited, both continuing qualitative trends beginning before the pandemic.Conversely, a decrease in time to return reviews appears to have followed pandemic onset.

Discussion
In this cohort study of 5013 manuscripts with reviews solicited by an open-access general medical journal before and during the COVID-19 pandemic, we did not identify evidence of deterioration in the peer review process.The overall rate of reviewer acceptance of review requests remained stable after pandemic onset.However, the time required for reviewers to complete reviews was modestly shorter during the pandemic, and the mean number of high-quality reviews per manuscript was greater compared with the period before the pandemic.Although the association with time to return reviews persisted after adjustment for potential confounding features, such as study design and manuscript type, we cannot exclude unobserved changes in how the editors invited reviewers-for example, making a greater effort to identify interested reviewers or sending personal notes-that may have coincided with the pandemic.Similar patterns were observed for COVID-19-focused manuscripts.In this case, rates of reviewer acceptance were significantly greater for such manuscripts.Time to return reviews was slightly but statistically significantly shorter, whereas the number of high-quality reviews received per manuscript was slightly greater.
Our results complement those of a recent investigation 7 of peer review during the pandemic.That study found lower rates of review invitation acceptance from health and medical journals among women but not men. 7

Limitations
This study has multiple limitations.First, we did not have access to reviewer-level characteristics, such as gender, age, race and ethnicity, or academic seniority, that would allow us to quantify the differential effect of the pandemic on some reviewers or compare across demographic characteristics, because these measures are not collected by the journal.Second, we cannot necessarily attribute the changes we observed to the pandemic itself.While JAMA Network Open was publishing for 2 years before the pandemic, submissions increased rapidly, particularly in the first year, as journal recognition and reputation increased.Likewise, reviewers' willingness to participate in peer review may have changed over time as they became more aware of the journal.As such, there are likely secular trends that may explain some of the changes we observed; the generalizability of our results to other journals will require further study, and we hope this work will encourage such efforts.
Third, editors' estimates of review quality are entirely subjective, so it is possible that editors took extenuating circumstances into account; for example, editors may have "graded on a curve" during the pandemic, and in reality quality could have remained unchanged or diminished.Finally, the automated flagging of submissions as COVID-19 related may have missed relevant submissions that did not include specific keywords and, likewise, may have flagged submissions that included pandemic-related keywords in the introduction or discussion but were not truly related to COVID-19.
In contextualizing these findings, it is also important to consider other changes in the peer review process occurring during the pandemic.For example, reviewers may have been more or less likely to accept invitations to review manuscripts based on eagerness to review new science about COVID-19, availability due to reduced clinical commitments, unavailability due to intensified clinical commitments, concerns about COVID-related misinformation, or reviewer fatigue associated with increase in volume of manuscripts submitted during the pandemic.Moreover, the editors' behavior may have changed with the increase in number of submissions, such as making more personal requests or opting to proceed with a decision with fewer reviewers for a given manuscript.Despite these limitations, our results may help inform ongoing conversations about quality and burden of peer review during the COVID-19 pandemic.The findings suggest that the pandemic modestly affected the review process in terms of turnaround time and review quality.However, this apparent stability does not address the extent to which reviewer sentiment toward the peer review process may have shifted or the pandemic's effect on the ability of invited reviewers to complete other tasks.Other lines of investigation, including surveys, suggest that the pandemic has negatively affected researchers' quality of life, more so for women than men. 16,17

Conclusions
The findings of this pre-post cohort study suggest that the peer review process at a large, open-access journal has continued to function during the COVID-19 pandemic despite changes in both the volume of submissions and the work and home environments of many peer reviewers.Most encouragingly, during the pandemic, review quality did not appear to have diminished.Still, in light of abundant evidence that COVID-19 has negatively impacted researchers, 16,17 continued efforts to study and improve the peer review process are needed.

Supplementary Material
Refer to Web version on PubMed Central for supplementary material.

Data Sharing Statement:
See the Supplement.

Findings
In this cohort study of 5013 manuscripts reviewed and peer reviews invited by an open-access general medical journal, peer review turnaround time was slightly shorter and editor-reported review quality was modestly increased during the first year of the pandemic.

Meaning
This study found that peer review at a large, open-access general medical journal remained resilient during the first 15 months of the pandemic despite shifts in in the dynamics of the review process.JAMA Netw Open.Author manuscript; available in PMC 2024 February 08.

Figure 1 .
Figure 1.Linear Regression Model of Manuscript Features Associated With Mean Time to Return Reviews Before and During the First Year of the COVID-19 Pandemic Linear regression model of mean time in days to return review and variables associated with mean time in days to return reviews, by manuscript.

Figure 2 .
Figure 2. Change in Characteristics of Peer Review Over Time Before and During the First Year of the COVID-19 Pandemic Changes in aspects of peer reviewer behavior over time, before and during the COVID-19 pandemic, with number of global daily deaths from COVID-19 as an indicator of pandemic intensity.

Table 1 .
Characteristics of Manuscripts Received Before or on March 11, 2020, or Later Turnaround times were not available for 52 manuscripts in the prepandemic period and 64 in the pandemic period.c Reviewer ratings were not available for 55 manuscripts in the prepandemic period and 65 in the pandemic period.
JAMA Netw Open.Author manuscript; available in PMC 2024 February 08.b JAMA Netw Open.Author manuscript; available in PMC 2024 February 08.

Table 2 .
Characteristics of Manuscripts That Did or Did Not Address COVID-19 are from t tests (for means) or χ 2 tests (for categories).b Turnaround times were not available for 54 non-COVID-19-related manuscripts and 10 COVID-19-related manuscripts.c Reviewer ratings were not available for 55 non-COVID-19-related manuscripts and 10 COVID-19-related manuscripts.