Published May 1, 2020, and retracted June 4, 2020.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Lee TC, Senecal J, Hsu JM, McDonald EG. Ongoing Citations of a Retracted Study Involving Cardiovascular Disease, Drug Therapy, and Mortality in COVID-19. JAMA Intern Med. Published online August 02, 2021. doi:10.1001/jamainternmed.2021.4112
In Spring 2020, early during the COVID-19 pandemic, 2 prominent studies that used a database from Surgisphere, a little-known company, were quickly retracted because of concerns about fraudulent data. One study1 purported to study the safety and efficacy of hydroxychloroquine for treating hospitalized patients with COVID-19. Published online in the Lancet on May 22, 2020, it was retracted on June 13, 2020.2 As of June 11, 2021, the study had an Altmetric attention score of 23 084. The second study3 claimed to investigate the association between cardiovascular disease, renin-angiotensin–aldosterone system inhibitor therapy, and COVID-19 outcomes. Published online May 1, 2020, in the New England Journal of Medicine (NEJM), it was retracted on June 4, 2020.3 As of June 11, 2021, the study had an Altmetric attention score of 3727. We sought to evaluate the association of the widely publicized retraction4,5 with further citations of the NEJM study.3
On April 2, 2021, and with an update on May 31, 2021, we used Google Scholar to identify works that cited the retracted NEJM article from Mehra et al.3 We verified the citations by reviewing the full text of the articles. For each citing article, we determined the initial date of publication, whether the retraction was cited or otherwise noted, and if a study that was reported in the article used the data in a secondary analysis. We also determined the country of origin, article language, type of article, and the number of secondary citations (ie, citations of the article that cited the retracted study).3 Because our study did not involve human participants, it was deemed exempt from institutional board review.
We identified 934 articles that cited the retracted study; of these, 146 (15.6%) were preprints or not from peer reviewed journals, 63 (6.7%) were duplicate citations in Google Scholar, and 2 (0.2%) were false links. We could not locate the full text of 10 articles (1.1%). Of the remaining 713, a citation was verified in 652 (91.4%) (http://individual.utoronto.ca/leet/citingpapers.pdf) (Table). Of the 652 verified citations, 70 citations (10.7%) preceded the month of retraction and 227 (34.8%) of the citations occurred within the first 2 months of the retraction. Most citing articles (355 [54.4%]) were published 3 months or later after the retraction; 181 (27.8%) were published 6 months or later (Table and Figure). In May 2021, 11 months after the article was retracted, it was cited 21 times. Of the 652 verified citations, 115 (17.6%) cited or otherwise noted the retraction. Most citations were used to support a statement in the main text in the article; however, in 17 articles (2.6%) the authors incorporated the data into a new analysis. Two of these articles noted the retraction. Of these 17 articles, 11 (64.7%) were published 3 months or later and 7 (41.2%) were published 6 months or later after the retraction; as of May 31, 2021, the 17 articles had a median of 19 secondary citations (interquartile range, 2-49).
Although a prominent study about cardiovascular disease, drug therapy, and mortality in COVID-193 was retracted with wide publicity in June 2020,4,5 about a month after it was published, it continues to be widely cited, with 21 new citations in May 2021. We chose to study the article by Mehra et al3 because it was retracted soon after publication, received prominent media attention, and was not subject to controversies, as has been the case for some studies of hydroxychloroquine and COVID-19, the topic of the other prominent article1 using the Surgisphere database that was also retracted2 around the same time.
The limitations of our analysis were that it was restricted to this single article and the use of a single search engine, Google Scholar, to identify the citing works. Although we removed duplicate entries and articles that were erroneously listed as citing the work, it is possible that we may have missed additional citations that Google Scholar did not identify. Nevertheless, there is no reason for a retracted study to continue to be widely cited in the medical literature months after it was retracted, and, in some instances, for the retracted data to be incorporated into new analyses. Our findings challenge authors, peer reviewers, journal editors, and academic institutions to do a better job of addressing the broader issues of ongoing citations of retracted scientific studies and protecting the integrity of the medical literature.
Accepted for Publication: June 11, 2021.
Published Online: August 2, 2021. doi:10.1001/jamainternmed.2021.4112
Corresponding Author: Todd C. Lee, MD, MPH, Royal Victoria Hospital, 1001 Boulevard Décarie, Room E5.1820, Montréal, QC H4A3J1, Canada (firstname.lastname@example.org).
Author Contributions: Dr Lee had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Lee, McDonald.
Acquisition, analysis, or interpretation of data: Lee, Senecal, Hsu.
Drafting of the manuscript: Lee, Hsu, McDonald.
Critical revision of the manuscript for important intellectual content: Lee, Senecal, McDonald.
Statistical analysis: Lee.
Administrative, technical, or material support: McDonald.
Supervision: Lee, McDonald.
Conflict of Interest Disclosures: Drs McDonald and Lee were signatories on a public letter asking for retraction of the Surgisphere papers. Dr Lee reported research support from Fonds de Recherche du Quebec–Sante during the conduct of the study. No other disclosures were reported.
Create a personal account or sign in to: