Assessment of Citations of the Retracted Article by Wakefield et al With Fraudulent Claims of an Association Between Vaccination and Autism

IMPORTANCE The number of citations can be used to show the influence of an article or to measure the validity of a research study. The article by Wakefield et al that fraudulently reported an association between vaccination and autism continues to accumulate citations even after it was retracted. OBJECTIVES To examine the characteristics of citations from scholarly literature that reference the 1998 article by Wakefield et al and to investigate whether authors are accurately citing retracted references. DESIGN, SETTING, AND PARTICIPANTS In this cross-sectional bibliographic analysis of the scholarly publications that cited a 1998 article by Wakefield et al, cited references were collected from a Web of Science Core Collection search performed on March 11, 2019. A total of 1211 articles were identified, with 58 citing works excluded because they were non–English-language publications or the citation to the study by Wakefield et al could not be located by reviewers. Citing works consisted of books, research articles, letters, editorials, news items, and other scholarly literature. Citations to the article by Wakefield et al were identified and analyzed by 2 reviewers in a blinded screening. Reviewers assigned a characteristic to each citation and indicated whether the retraction was documented.


Introduction
In an era of information overload, it can be challenging to find influential articles in any given field, and one method of identifying such influential articles is to look at citation counts.p 290) The raw number does not show the intent of an author in deciding to cite another's work, and there can be a variety of motivations behind the intent. 5,6For instance, citing a work in a negative manner can be used by researchers to self-correct flawed research and aid in scientific debate. 7Value would be added to citation count numbers if more information was provided concerning the context of the citations. 5,6However, obtaining information about citation characteristics is laborious, and using natural language processing is a relatively new and emerging field. 7,8tracted articles continue to be cited frequently without listing the retraction, 9,10 perhaps because there are no clear guidelines on whom is responsible for ensuring that retracted articles are properly cited as being retracted. 11The Committee on Publication Ethics (COPE) states, "Retracted articles should be clearly identified as such in all electronic sources," 12(p 202) but the committee does not offer guidelines on how and when authors should document that an article has been retracted.
The International Committee of Medical Journal Editors (ICMJE) notes that not all journals check the accuracy of references in submitted publications. 13The omission of a work's retracted status in the citation can lead to the perception that the cited work is valid, 14 requiring scholars to spend time, effort, and money to correct the misperceptions of the problematic work. 15e purpose of this cross-sectional study was to examine how authors of scholarly literature cited the 1998 article by Wakefield et al 16 that purported to show an association between the measles-mumps-rubella (MMR) vaccine and autism.The article was retracted in 2 stages: in 2004, 10 of the original 12 authors issued a partial retraction of the interpretation that there is a causal link between the MMR vaccine and autism (1 of the original authors was unable to be contacted and Wakefield was not an author included in this partial retraction), 17 and in 2010, editors of The Lancet published a notice of retraction of the article because of false claims made in the original article. 18spite the partial retraction and notice of retraction, the 1998 article by Wakefield et al 16 continues to accumulate a significant amount of citations.In 2014, of the 58 million references in Web of Science, only 14 499 works (0.026%) have been cited more than 1000 times. 19The 1998 article by Wakefield et al 16 had accumulated more than 1211 citations by March 2019, and the number continues to increase.According to the Web of Science Core Collection, in April 2019, the article by Wakefield et al 16 was the ninth most-cited reference indexed with the topic of autism of more than 57 600 references and the second most-cited reference indexed with the topic of measles vaccines of more than 900 references.The present study was not the first citation analysis of the article by Wakefield et al. 16 For instance, Chen and Leydesdorff 8,20 used citation data from Web of Science to look for patterns in how journals and research fields cited the retracted article by Wakefield et al. 16 However, these patterns do not examine the context of the citations.The present study is novel because it is the first time, to our knowledge, that each citation to the article by Wakefield et al 16 was analyzed to see how the author cited the article, specifically assessing whether the author affirmed or negated the study by Wakefield et al. 16 Reviewers examined the citing works to evaluate whether the retracted status of the article by Wakefield et al 16  interchangeably; thus, a glossary of terms is given in Table 1.

Methods
In this cross-sectional study, we (E.M.S., J.D., K.L.H., B.R., R.S., and E.W.) conducted a cited reference search on March 11, 2019, in the Web of Science Core Collection to identify scholarly literature that cited the 1998 article by Wakefield et al. 16 The reviewers' Web of Science Core Collection subscription includes access to Science Citation Index Expanded, Social Science Citation Index, Arts & Humanities Citation Index, and Book Citation Index, allowing the reviewers to find cited references that span all scholarly disciplines, including clinical medicine, social sciences, immunology, and neurology and behavior.Web of Science indexes research articles, letters, editorials, news items, proceedings literature, books, and other scholarly literature from sources that demonstrate high levels of editorial rigor and meet a well-defined set of criteria, 22 thus allowing a good representation of the scholarly community's reaction to the 1998 article by Wakefield et al. 16 The Cited Reference Search feature in Web of Science allows for a comprehensive search of bibliographies and reference lists of all items that are indexed in Web of Science.This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. 23The Medical College of Wisconsin Institutional Review Board Office reviewed this project and determined that this study did not qualify as human subjects research and was therefore not subject to institutional review board review.
We identified 1211 citations to the 1998 article by Wakefield et al 16 as of March 11, 2019.Fiftyeight articles were excluded from the citation analysis because the works were not written in English or the citation to the article by Wakefield et al 16 could not be located by the reviewers.A total of 1153 citing works were included in the citation analysis.Bibliographic information and the full-text copy of each citing work was uploaded into Covidence systematic review software (Veritas Health Innovation).Covidence is a web-based software platform that is used to manage the screening process and extract data for systematic reviews; it allows blinded screening and custom tagging of records.
Citing works were reviewed to determine the characteristic of the citation using an established taxonomy 5,21 (Table 2).Each citing work underwent a blinded screening by 2 of us (E.M.S., J.D., K.L.H., B.R., R.S., and/or E.W.) who located the citations within the text of the work and independently assigned them a characteristic.Disagreement about the characteristics were brought before the group (E.M.S., J.D., K.L.H., B.R., R.S., and E.W.) for consensus.If citations fit into more than 1 category or the article by Wakefield et al 16 was cited more than once, a stepwise approach was used to assign the category. 21In the stepwise approach, citations were first screened to assess whether they fit in the categories of negative, affirmative, or contrastive; if not, they were screened for the category of persuasive; if not, citations were screened for the categories of assumptive, perfunctory, methodologic, or conceptual.If a citation could not be located, the citing work was excluded from the review.For multiple citation occurrences, the citation was categorized as did not meet any of these criteria, the citing works were brought before the group for consensus.
Every citing work in the sample was assigned one characteristic.
Not only were the citing works screened to determine the characteristic of each citation, they also were examined to determine whether the retracted status of the article by Wakefield et al 16 was identified in the citation or in the reference list.A citing work was labeled as retraction referenced if the author specifically used the word retracted, retract, or retraction.

Statistical Analysis
Bibliographic information, characteristics, and retraction information were exported from Covidence as a CSV file, and Excel (Microsoft Corp) was used to analyze the data.A descriptive analysis of the data was performed with a focus on frequencies and percentages.The sample consisted of all the included citing works retrieved from the Web of Science search apart from the 58 excluded articles.

Characteristics
Of the 1153 citing works, 838 (72.7%) were negative, 106 (9.2%) perfunctory, 94 (8.2%) affirmative, and 60 (5.2%) assumptive.The other characteristics (conceptual, contrastive, methodologic, and persuasive) had a combined total of 55 (4.8%).Since the article by Wakefield et al 16 was initially published, authors have mostly cited the article in a negative manner (Figure 1).Authors who affirmed the article by Wakefield et al 16 in their citations comprised 94 of the 1153 total citing works (8.2%).Of the affirmative citations, 49 of 94 (52.1%) were published between 1998 and 2003, before the partial retraction of the article by Wakefield et al. 16 Wakefield was an author of 15 of 94 articles with affirmative citations.Of these 15 affirmative self-citing works, 10 include at least 1 coauthor of the original article.The only other self-citing work by the original authors of the article by Wakefield et al 16 is the 2004 partial retraction by Murch et al, 17 which was characterized as negative.

Retractions
From the date of the partial retraction on March 4, 2004, to March 11, 2019, a total of 881 works were published that cited the article by Wakefield et al. 16 These citing works were screened to determine whether they documented the partial and/or notice of retraction to the article by Wakefield et al, 16 and in this period, 493 of 881 (56.0%) of the citing works documented either retraction.Of 57 citing works published in 2004, a total of 10 (17.5%) documented the partial retraction in the citations or the reference lists.However, the reviewers began the retraction analysis (Figure 2) in 2005 to We found that authors documented the retraction in different ways.Of the 493 citing works that documented either retraction, 211 (42.8%) documented the retraction in both the citation and  Retractions to the article by Wakefield et al 16 were published in 2004 and 2010. 17,18 was identified in the citation or reference list.This study replicated the methods used by Leung et al 21 in their article, 1980 Letter on the Risk of Opioid Addiction.The conclusion of Leung et al 21(pp 2194-2195) highlights "the potential consequences of inaccurate citation and underscores the need for diligence when citing previously published studies."Reviewers planned to use the methods of Leung et al 21 to examine another influential article of questionable quality.Terms such as reference and citation are sometimes used

Figure 1 .
Figure 1.Characteristics of References to the Article by Wakefield et al by Year of Publication

Figure 2 .
Figure 2. Number of Articles That Referenced the 1998 Article by Wakefield et al by Year After the Partial Retraction and Notice of Retraction Were Published

Table 1 .
Glossary of Terms Used in This Article Self-citationAct of an author citing his or her previous work as a reference in subsequent worksJAMA Network

Open | Medical Journals and Publishing Assessment
of Citations of the Retracted Article by Wakefield et al contrastive if any of the citations had this characteristic.Barring this, a citation was categorized as overall negative or affirmative if any of the occurrences had these characteristics.Finally, if citations JAMA Network Open.2019;2(11):e1915552. doi:10.1001/jamanetworkopen.2019.15552(Reprinted) November 15, 2019 3/10 Downloaded From: https://jamanetwork.com/ on 10/08/2023

Table 2 .
16tation Characteristics a compensate for works that were submitted for publication before the partial retraction was published and to ensure that the retraction information was indexed in bibliographic databases, making it more likely to be discovered.Between 2005 and 2010, a total of 123 of 322 (38.2%) citing works documented the partial retraction.In 2010, editors of The Lancet issued a notice of retraction18to the article by Wakefield et al.16Of the 71 citing works published in 2010, a total of 21 (29.6%)documented the 2010 notice of retraction or retracted status of the article.The number of