Ongoing Citations of a Retracted Study Involving Cardiovascular Disease, Drug Therapy, and Mortality in COVID-19 | Cardiology | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Figure.  Verified Citations Between May 2020 and May 2021 of the Retracted Study by Mehra et al3
Verified Citations Between May 2020 and May 2021 of the Retracted Study by Mehra et al

Published May 1, 2020, and retracted June 4, 2020.

Table.  Characteristics of the Citations of the Retracted Study by Mehra et al3
Characteristics of the Citations of the Retracted Study by Mehra et al
1.
Mehra  MR, Desai  SS, Ruschitzka  F, Patel  AN.  RETRACTED: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.   Lancet. 2020;S0140-6736(20)31180-6. doi:10.1016/s0140-6736(20)31180-6PubMedGoogle Scholar
2.
Mehra  MR, Ruschitzka  F, Patel  AN.  Retraction—hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.   Lancet. 2020;395(10240):1820. doi:10.1016/S0140-6736(20)31324-6PubMedGoogle ScholarCrossref
3.
Mehra  MR, Desai  SS, Kuy  S, Henry  TD, Patel  AN.  RETRACTION: cardiovascular disease, drug therapy, and mortality in Covid-19.   N Engl J Med. 2020;382(25):e102. doi:10.1056/NEJMoa2007621PubMedGoogle Scholar
4.
Davey  M, Kirchgaessner  S. Surgisphere: mass audit of papers linked to firm behind hydroxychloroquine Lancet study scandal. Accessed April 23, 2021. https://www.theguardian.com/world/2020/jun/10/surgisphere-sapan-desai-lancet-study-hydroxychloroquine-mass-audit-scientific-papers
5.
Piller  C. Who’s to blame? these three scientists are at the heart of the Surgisphere COVID-19 scandal. Accessed April 23, 2021. https://www.sciencemag.org/news/2020/06/whos-blame-these-three-scientists-are-heart-surgisphere-covid-19-scandal
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    EXPAND ALL
    Ongoing citations of a retracted study
    Peppino Peppino, MD, Prof of Pediatric Cardiology | Postgraduate Medical School of Cardiology, Univerisity of Palermo Italy
    Still many  do not know the fraudulent figures coming from a little-known company, Surgisphere. The study was quickly retracted because of concerns about the origin of the data. Personally, I have been involved in calling some authors to exclude the retracted reference.  A national cardiac society published an another expression of concern, calling attention to  this very embarrassing story (1). I join the conclusion of Lee et al (2) to invite ‘peer reviewers, journal editors, and academic institutions to do a better job of addressing the broader issues of ongoing citations of retracted scientific studies and protecting the integrity of the medical literature’.
    1 Calcaterra G, Bassareo PP, Nodari S, Di Franco A, Barilla’ F, Romeo F, An expression of concern on research misconduct during the corona virus disease-2019 pandemic. Jul 2020, Journal of Cardiovascular Medicine, p 838-9. doi: 10.2459/JCM.0000000000001074.
    2 Lee TC, Senecal J, Hsu JM, McDonald EG. Ongoing Citations of a Retracted Study Involving Cardiovascular Disease, Drug Therapy, and Mortality in COVID-19. JAMA Intern Med. Published online August 02, 2021. doi:10.1001/jamainternmed.2021.4112.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Views 9,299
    Citations 0
    Research Letter
    August 2, 2021

    Ongoing Citations of a Retracted Study Involving Cardiovascular Disease, Drug Therapy, and Mortality in COVID-19

    Author Affiliations
    • 1Division of Infectious Diseases, Department of Medicine, McGill University, Montréal, Quebec, Canada
    • 2Clinical Practice Assessment Unit, Department of Medicine, McGill University Health Centre, Montréal, Quebec, Canada
    • 3Faculty of Medicine and Health Sciences, McGill University, Montréal, Quebec, Canada
    • 4Division of General Internal Medicine, Department of Medicine, McGill University Health Centre, Montréal, Quebec, Canada
    JAMA Intern Med. Published online August 2, 2021. doi:10.1001/jamainternmed.2021.4112

    In Spring 2020, early during the COVID-19 pandemic, 2 prominent studies that used a database from Surgisphere, a little-known company, were quickly retracted because of concerns about fraudulent data. One study1 purported to study the safety and efficacy of hydroxychloroquine for treating hospitalized patients with COVID-19. Published online in the Lancet on May 22, 2020, it was retracted on June 13, 2020.2 As of June 11, 2021, the study had an Altmetric attention score of 23 084. The second study3 claimed to investigate the association between cardiovascular disease, renin-angiotensin–aldosterone system inhibitor therapy, and COVID-19 outcomes. Published online May 1, 2020, in the New England Journal of Medicine (NEJM), it was retracted on June 4, 2020.3 As of June 11, 2021, the study had an Altmetric attention score of 3727. We sought to evaluate the association of the widely publicized retraction4,5 with further citations of the NEJM study.3

    Methods

    On April 2, 2021, and with an update on May 31, 2021, we used Google Scholar to identify works that cited the retracted NEJM article from Mehra et al.3 We verified the citations by reviewing the full text of the articles. For each citing article, we determined the initial date of publication, whether the retraction was cited or otherwise noted, and if a study that was reported in the article used the data in a secondary analysis. We also determined the country of origin, article language, type of article, and the number of secondary citations (ie, citations of the article that cited the retracted study).3 Because our study did not involve human participants, it was deemed exempt from institutional board review.

    Results

    We identified 934 articles that cited the retracted study; of these, 146 (15.6%) were preprints or not from peer reviewed journals, 63 (6.7%) were duplicate citations in Google Scholar, and 2 (0.2%) were false links. We could not locate the full text of 10 articles (1.1%). Of the remaining 713, a citation was verified in 652 (91.4%) (http://individual.utoronto.ca/leet/citingpapers.pdf) (Table). Of the 652 verified citations, 70 citations (10.7%) preceded the month of retraction and 227 (34.8%) of the citations occurred within the first 2 months of the retraction. Most citing articles (355 [54.4%]) were published 3 months or later after the retraction; 181 (27.8%) were published 6 months or later (Table and Figure). In May 2021, 11 months after the article was retracted, it was cited 21 times. Of the 652 verified citations, 115 (17.6%) cited or otherwise noted the retraction. Most citations were used to support a statement in the main text in the article; however, in 17 articles (2.6%) the authors incorporated the data into a new analysis. Two of these articles noted the retraction. Of these 17 articles, 11 (64.7%) were published 3 months or later and 7 (41.2%) were published 6 months or later after the retraction; as of May 31, 2021, the 17 articles had a median of 19 secondary citations (interquartile range, 2-49).

    Discussion

    Although a prominent study about cardiovascular disease, drug therapy, and mortality in COVID-193 was retracted with wide publicity in June 2020,4,5 about a month after it was published, it continues to be widely cited, with 21 new citations in May 2021. We chose to study the article by Mehra et al3 because it was retracted soon after publication, received prominent media attention, and was not subject to controversies, as has been the case for some studies of hydroxychloroquine and COVID-19, the topic of the other prominent article1 using the Surgisphere database that was also retracted2 around the same time.

    The limitations of our analysis were that it was restricted to this single article and the use of a single search engine, Google Scholar, to identify the citing works. Although we removed duplicate entries and articles that were erroneously listed as citing the work, it is possible that we may have missed additional citations that Google Scholar did not identify. Nevertheless, there is no reason for a retracted study to continue to be widely cited in the medical literature months after it was retracted, and, in some instances, for the retracted data to be incorporated into new analyses. Our findings challenge authors, peer reviewers, journal editors, and academic institutions to do a better job of addressing the broader issues of ongoing citations of retracted scientific studies and protecting the integrity of the medical literature.

    Back to top
    Article Information

    Accepted for Publication: June 11, 2021.

    Published Online: August 2, 2021. doi:10.1001/jamainternmed.2021.4112

    Corresponding Author: Todd C. Lee, MD, MPH, Royal Victoria Hospital, 1001 Boulevard Décarie, Room E5.1820, Montréal, QC H4A3J1, Canada (todd.lee@mcgill.ca).

    Author Contributions: Dr Lee had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Lee, McDonald.

    Acquisition, analysis, or interpretation of data: Lee, Senecal, Hsu.

    Drafting of the manuscript: Lee, Hsu, McDonald.

    Critical revision of the manuscript for important intellectual content: Lee, Senecal, McDonald.

    Statistical analysis: Lee.

    Administrative, technical, or material support: McDonald.

    Supervision: Lee, McDonald.

    Conflict of Interest Disclosures: Drs McDonald and Lee were signatories on a public letter asking for retraction of the Surgisphere papers. Dr Lee reported research support from Fonds de Recherche du Quebec–Sante during the conduct of the study. No other disclosures were reported.

    References
    1.
    Mehra  MR, Desai  SS, Ruschitzka  F, Patel  AN.  RETRACTED: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.   Lancet. 2020;S0140-6736(20)31180-6. doi:10.1016/s0140-6736(20)31180-6PubMedGoogle Scholar
    2.
    Mehra  MR, Ruschitzka  F, Patel  AN.  Retraction—hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.   Lancet. 2020;395(10240):1820. doi:10.1016/S0140-6736(20)31324-6PubMedGoogle ScholarCrossref
    3.
    Mehra  MR, Desai  SS, Kuy  S, Henry  TD, Patel  AN.  RETRACTION: cardiovascular disease, drug therapy, and mortality in Covid-19.   N Engl J Med. 2020;382(25):e102. doi:10.1056/NEJMoa2007621PubMedGoogle Scholar
    4.
    Davey  M, Kirchgaessner  S. Surgisphere: mass audit of papers linked to firm behind hydroxychloroquine Lancet study scandal. Accessed April 23, 2021. https://www.theguardian.com/world/2020/jun/10/surgisphere-sapan-desai-lancet-study-hydroxychloroquine-mass-audit-scientific-papers
    5.
    Piller  C. Who’s to blame? these three scientists are at the heart of the Surgisphere COVID-19 scandal. Accessed April 23, 2021. https://www.sciencemag.org/news/2020/06/whos-blame-these-three-scientists-are-heart-surgisphere-covid-19-scandal
    ×