Challenges in Identifying the Retracted Status of an Article | Medical Journals and Publishing | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Table 1.  Database Adherence With Following PubMed’s Retraction Procedurea
Database Adherence With Following PubMed’s Retraction Procedurea
Table 2.  Labeling of Retracted Articles on Publisher Websites
Labeling of Retracted Articles on Publisher Websites
1.
Bar-Ilan  J, Halevi  G.  Post retraction citations in context: a case study.   Scientometrics. 2017;113(1):547-565. doi:10.1007/s11192-017-2242-0 PubMedGoogle ScholarCrossref
2.
Budd  JM, Coble  Z, Abritis  A.  An investigation of retracted articles in the biomedical literature.   Proc Assoc Inf Sci Technol. 2016;53(1):1-9. doi:10.1002/pra2.2016.14505301055 Google ScholarCrossref
3.
International Committee of Medical Journal Editors. Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. Published online December 2019. Accessed May 14, 2020. http://www.icmje.org/recommendations/
4.
COPE Council.  COPE Retraction Guidelines. Committee on Publication Ethics; 2019. doi:10.24318/cope.2019.1.4
5.
National Library of Medicine. Errata, retractions, and other linked citations in PubMed. Accessed June 16, 2020. https://www.nlm.nih.gov/bsd/policy/errata.html
6.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Research Letter
    Medical Journals and Publishing
    June 29, 2021

    Challenges in Identifying the Retracted Status of an Article

    Author Affiliations
    • 1Medical College of Wisconsin Libraries, Medical College of Wisconsin, Milwaukee
    • 2Advocate Aurora Library, Advocate Aurora Health, Milwaukee, Wisconsin
    • 3Todd Wehr Library, Carroll University, Waukesha, Wisconsin
    JAMA Netw Open. 2021;4(6):e2115648. doi:10.1001/jamanetworkopen.2021.15648
    Introduction

    Inadvertent or unacknowledged citations to retracted literature are a persistent problem in scholarly publishing.1,2 Inconsistencies in how retraction information is displayed on publisher websites and bibliographic databases hinder determining whether a paper has been retracted, thus contributing to the cause of this problem. This cross-sectional study examines publisher websites and bibliographic databases to check their accordance with industry standards for documenting retracted publications, highlighting challenges readers face in identifying the retracted status of a publication.

    Methods

    A search was performed in PubMed on October 20, 2019, to identify English-language articles published between 2009 and 2019 that were indexed with a publication type “Retracted Publication.” Three articles were selected from the 50 journals that had the most retracted publications in the PubMed search, giving us 150 articles for analysis.

    Articles were examined on publisher websites to document whether the websites followed specific recommendations from the International Committee of Medical Journal Editors (ICMJE) on documenting retractions,3 and we looked for consistency within the websites on how this information was displayed. In accordance with guidelines from Committee on Publishing Ethics (COPE) on retraction,4 the identification of retractions in bibliographic databases were examined. The same articles were reviewed in 6 bibliographic databases that index biomedical literature (PubMed, Ovid MEDLINE, EBSCO CINAHL, ProQuest PsycINFO, Scopus, and Web of Science) to document how the databases displayed retraction information using PubMed’s criteria for updating retracted articles.5 Detailed methods describing the literature search, review, and data abstraction can be found in the eAppendix in the Supplement.

    This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline,6 and a descriptive analysis of the data was performed in Excel (Microsoft Corp). The Medical College of Wisconsin institutional review board office reviewed this project and determined that this study did not qualify as human participants research and was therefore not subject to institutional review board review.

    Results

    Owing to the varying scope of the databases in our study, not all articles were indexed across all databases, and coverage ranged from 100% (150 articles) in PubMed and Ovid MEDLINE to 4% (6 articles) in ProQuest PsycINFO. Performance varied among the 6 bibliographic databases. PubMed and Ovid MEDLINE showed the best performance in adhering to their procedures for updating retracted publications with 87% (650 of 750) compliance with all 5 criteria, whereas ProQuest PsycINFO’s performance was 23% (7 of 30) and EBSCO CINAHL was 7% (8 of 115) (Table 1).

    PubMed, the only free database in our analysis, had the highest performance in documenting retracted publications. ProQuest PsycINFO and EBSCO CINAHL often failed to use labels to indicate an article’s retracted status, such as updating the publication type to retracted or linking to the notice of retraction. Lack of consistency between bibliographic databases results in some users being better informed than others, depending on which database they can access.

    Overall, 112 of 150 articles (75%; 95% CI, 67%-81%) on the publisher websites followed at least 6 of the ICMJE’s 7 recommendations for reporting retractions and demonstrated variability in where publishers displayed the retraction information (Table 2). When we compared the retraction labels within the publisher websites using 3 examples from each of the 50 journals, retraction information was consistently displayed in all 3 of the article abstracts 78% (39 of 50) of the time and 64% (31 of 50) of the time in PDFs. Examples of inconsistencies included: links to the notice of retraction not present on all examples, location of the retraction notice differed, different colors were used to show retraction labels, and some examples did not show retraction information.

    Discussion

    In this study, journal websites and bibliographic databases did not consistently display the retracted status of articles. Guidance on retracting articles is given by the Committee on Publishing Ethics and the ICMJE,3,4 but their recommendations are open to interpretation and not always followed. The ICMJE recommends that retractions should be “prominently labelled,”3 and we propose that an explicit recommendation to add a prefix of “Retracted:” to the title of a retracted publication would fulfill this aim. This change would provide a consistent visual signal to the reader and would change the metadata that can be ingested into citation managers.

    Our study is limited by the number of retracted articles that were reviewed (150 of the entire data set of 7059 articles), and Embase was not included in this study because the authors did not have access to it.

    We hope that by making the retraction information more clearly discoverable and standardized, the number of unintentional and unacknowledged citations of retracted literature will be reduced.

    Back to top
    Article Information

    Accepted for Publication: April 30, 2021.

    Published: June 29, 2021. doi:10.1001/jamanetworkopen.2021.15648

    Correction: This article was corrected on February 28, 2022, to fix errors in Table 1.

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Suelzer EM et al. JAMA Network Open.

    Corresponding Author: Elizabeth M. Suelzer, MLIS, AHIP, Medical College of Wisconsin Libraries, 8701 Watertown Plank Rd, Milwaukee, WI 53226 (esuelzer@mcw.edu).

    Author Contributions: Ms Suelzer had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Suelzer, Ruggeri, Witkowski.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: All authors.

    Critical revision of the manuscript for important intellectual content: Suelzer, Deal, Ruggeri, Witkowski.

    Statistical analysis: Suelzer, Witkowski.

    Administrative, technical, or material support: Deal, Ruggeri, Witkowski.

    Conflict of Interest Disclosures: None reported.

    Additional Information: Preliminary results of this study were posted at the Illinois Digital Environment for Access to Learning and Scholarship repository at https://www.ideals.illinois.edu/handle/2142/108367 and were discussed at the Alfred P. Sloan Foundation–sponsored workshop on Reducing the Inadvertent Spread of Retracted Science in a series of online workshops (October 26 and November 9 and 16, 2020).

    References
    1.
    Bar-Ilan  J, Halevi  G.  Post retraction citations in context: a case study.   Scientometrics. 2017;113(1):547-565. doi:10.1007/s11192-017-2242-0 PubMedGoogle ScholarCrossref
    2.
    Budd  JM, Coble  Z, Abritis  A.  An investigation of retracted articles in the biomedical literature.   Proc Assoc Inf Sci Technol. 2016;53(1):1-9. doi:10.1002/pra2.2016.14505301055 Google ScholarCrossref
    3.
    International Committee of Medical Journal Editors. Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. Published online December 2019. Accessed May 14, 2020. http://www.icmje.org/recommendations/
    4.
    COPE Council.  COPE Retraction Guidelines. Committee on Publication Ethics; 2019. doi:10.24318/cope.2019.1.4
    5.
    National Library of Medicine. Errata, retractions, and other linked citations in PubMed. Accessed June 16, 2020. https://www.nlm.nih.gov/bsd/policy/errata.html
    6.
    von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010 PubMedGoogle ScholarCrossref
    ×