[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure 1.
Characteristics of References to the Article by Wakefield et al by Year of Publication
Characteristics of References to the Article by Wakefield et al by Year of Publication

Other characteristic categories include conceptual, contrastive, methodologic, and persuasive and include a total of 55 publications. A partial retraction to the article by Wakefield et al16 was published in 2004,17 and another retraction was published in 2010.18

Figure 2.
Number of Articles That Referenced the 1998 Article by Wakefield et al by Year After the Partial Retraction and Notice of Retraction Were Published
Number of Articles That Referenced the 1998 Article by Wakefield et al by Year After the Partial Retraction and Notice of Retraction Were Published

Retractions to the article by Wakefield et al16 were published in 2004 and 2010.17,18

Table 1.  
Glossary of Terms Used in This Article
Glossary of Terms Used in This Article
Table 2.  
Citation Characteristicsa
Citation Characteristicsa
1.
Van Raan  AF.  The use of bibliometric analysis in research performance assessment and monitoring of interdisciplinary scientific developments.  Technol Assess Theory Prac. 2003;1(12):20-29.Google Scholar
2.
Vaughan  L, Shaw  D.  Web citation data for impact assessment: a comparison of four science disciplines.  J Am Soc Inf Sci Technol. 2005;56(10):1075-1087. doi:10.1002/asi.20199Google ScholarCrossref
3.
Cole  JR. A short history of the use of citations as a measure of the impact of scientific and scholarly work. In: Cronin  B, Atkins  HB, eds.  The Web of Knowledge: A Festschrift in Honor of Eugene Garfield. Medford, NJ: Information Today; 2000:281-300.
4.
Garfield  E.  Citation indexes in sociological and historical research.  Am Doc. 1963;14(4):289-291. doi:10.1002/asi.5090140405Google ScholarCrossref
5.
Bornmann  L, Daniel  HD.  What do citation counts measure? a review of studies on citing behavior.  J Doc. 2008;64(1):45-80. doi:10.1108/00220410810844150Google ScholarCrossref
6.
Garfield  E. Can citation indexing be automated? In: Symposium Proceedings, Statistical Association Methods for Mechanized Documentation. Washington, DC: National Bureau of Standards; 1965:189-192. National Bureau of Standards Miscellaneous Publication 269.
7.
Catalini  C, Lacetera  N, Oettl  A.  The incidence and role of negative citations in science.  Proc Natl Acad Sci U S A. 2015;112(45):13823-13826. doi:10.1073/pnas.1502280112PubMedGoogle ScholarCrossref
8.
Chen  CM, Hu  ZG, Milbank  J, Schultz  T.  A visual analytic study of retracted articles in scientific literature.  J Am Soc Inf Sci Technol. 2013;64(2):234-253. doi:10.1002/asi.22755Google ScholarCrossref
9.
Bucci  EM.  On zombie papers.  Cell Death Dis. 2019;10(3):189. doi:10.1038/s41419-019-1450-3PubMedGoogle ScholarCrossref
10.
Pfeifer  MP, Snodgrass  GL.  The continued use of retracted, invalid scientific literature.  JAMA. 1990;263(10):1420-1423. doi:10.1001/jama.1990.03440100140020PubMedGoogle ScholarCrossref
11.
Bar-Ilan  J, Halevi  G.  Post retraction citations in context: a case study.  Scientometrics. 2017;113(1):547-565. doi:10.1007/s11192-017-2242-0PubMedGoogle ScholarCrossref
12.
Wager  E, Barbour  V, Yentis  S, Kleinert  S, Council  C; COPE Council.  Retractions: guidance from the Committee on Publication Ethics (COPE).  Maturitas. 2009;64(4):201-203. doi:10.1016/j.maturitas.2009.09.018PubMedGoogle ScholarCrossref
13.
International Committee of Medical Journal Editors. Manuscript preparation: preparing for submission. http://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html. Published 2018. Accessed June 11, 2019, 2019.
14.
Madlock-Brown  CR, Eichmann  D.  The (lack of) impact of retraction on citation networks.  Sci Eng Ethics. 2015;21(1):127-137. doi:10.1007/s11948-014-9532-1PubMedGoogle ScholarCrossref
15.
Budd  JM, Sievert  M, Schultz  TR.  Phenomena of retraction: reasons for retraction and citations to the publications.  JAMA. 1998;280(3):296-297. doi:10.1001/jama.280.3.296PubMedGoogle ScholarCrossref
16.
Wakefield  AJ, Murch  SH, Anthony  A,  et al.  Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children  [retracted in: Lancet. 2010;375(9713):445].  Lancet. 1998;351(9103):637-641. doi:10.1016/S0140-6736(97)11096-0PubMedGoogle ScholarCrossref
17.
Murch  SH, Anthony  A, Casson  DH,  et al.  Retraction of an interpretation  [retraction of: Wakefield AJ, Murch SH, Anthony A, et al. In: Lancet 1998;351(9103):637-641].  Lancet. 2004;363(9411):750. doi:10.1016/S0140-6736(04)15715-2PubMedGoogle ScholarCrossref
18.
The Editors of The Lancet.  Retraction–Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children  [retraction of: Wakefield AJ, Murch SH, Anthony A, et al. In: Lancet 1998;351(9103):637-641].  Lancet. 2010;375(9713):445. doi:10.1016/S0140-6736(10)60175-4PubMedGoogle ScholarCrossref
19.
Van Noorden  R, Maher  B, Nuzzo  R.  The top 100 papers.  Nature. 2014;514(7524):550-553. doi:10.1038/514550aPubMedGoogle ScholarCrossref
20.
Chen  CM, Leydesdorff  L.  Patterns of connections and movements in dual-map overlays: a new method of publication portfolio analysis.  J Am Soc Inf Sci Technol. 2014;65(2):334-351. doi:10.1002/asi.22968Google ScholarCrossref
21.
Leung  PTM, Macdonald  EM, Stanbrook  MB, Dhalla  IA, Juurlink  DNA.  1980 Letter on the risk of opioid addiction.  N Engl J Med. 2017;376(22):2194-2195. doi:10.1056/NEJMc1700150PubMedGoogle ScholarCrossref
22.
Testa  J. The selection process for the Book Citation Index in Web of Science. Clarivate Analytics. https://clarivate.com/webofsciencegroup/essays/selection-process-book-citation-index-web-science/. Published 2017. Accessed September 22, 2019.
23.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010PubMedGoogle ScholarCrossref
24.
Christie  CDC.  Causal link between MMR vaccination and autism: fact or fiction?  West Indian Med J. 2000;49(4):266-267.PubMedGoogle Scholar
25.
Fombonne  E, Cook  EH.  MMR and autistic enterocolitis: consistent epidemiological failure to find an association.  Mol Psychiatry. 2003;8(2):133-134. doi:10.1038/sj.mp.4001266PubMedGoogle ScholarCrossref
26.
Salisbury  DM, Beverley  PCL, Miller  E.  Vaccine programmes and policies.  Br Med Bull. 2002;62:201-211. doi:10.1093/bmb/62.1.201PubMedGoogle ScholarCrossref
27.
O’Brien  SJ, Jones  IG, Christie  P.  Autism, inflammatory bowel disease, and MMR vaccine.  Lancet. 1998;351(9106):906-907. doi:10.1016/S0140-6736(05)70318-4PubMedGoogle ScholarCrossref
28.
Afzal  MA, Pipkin  PA, Minor  PD.  Absence of chicken myelin basic protein residues in commercial formulations of MMR vaccine.  Vaccine. 2000;19(4-5):442-446. doi:10.1016/S0264-410X(00)00202-4PubMedGoogle ScholarCrossref
29.
Nicoll  A, Elliman  D, Ross  E.  MMR vaccination and autism 1998.  BMJ. 1998;316(7133):715-716. doi:10.1136/bmj.316.7133.715PubMedGoogle ScholarCrossref
30.
Huang  DB, Wu  JJ, Tyring  SK.  A review of licensed viral vaccines, some of their safety concerns, and the advances in the development of investigational viral vaccines.  J Infect. 2004;49(3):179-209. doi:10.1016/j.jinf.2004.05.018PubMedGoogle ScholarCrossref
31.
Bushara  KO.  Neurologic presentation of celiac disease.  Gastroenterology. 2005;128(4)(suppl 1):S92-S97. doi:10.1053/j.gastro.2005.02.018PubMedGoogle ScholarCrossref
32.
Peacock  G, Yeargin-Allsopp  M.  Autism spectrum disorders: prevalence and vaccines.  Pediatr Ann. 2009;38(1):22-25. doi:10.3928/00904481-20090101-09PubMedGoogle ScholarCrossref
33.
Tettenborn  MA.  Autism, inflammatory bowel disease, and MMR vaccine.  Lancet. 1998;351(9112):1357. doi:10.1016/S0140-6736(05)79087-5PubMedGoogle ScholarCrossref
34.
Singh  VK, Lin  SX, Yang  VC.  Serological association of measles virus and human herpesvirus-6 with brain autoantibodies in autism.  Clin Immunol Immunopathol. 1998;89(1):105-108. doi:10.1006/clin.1998.4588PubMedGoogle ScholarCrossref
35.
Thrower  D.  Autism, bowel inflammation, and measles.  Lancet. 2002;359(9323):2113. doi:10.1016/S0140-6736(02)08920-1PubMedGoogle ScholarCrossref
36.
Sandler  RH, Finegold  SM, Bolte  ER,  et al.  Short-term benefit from oral vancomycin treatment of regressive-onset autism.  J Child Neurol. 2000;15(7):429-435. doi:10.1177/088307380001500701PubMedGoogle ScholarCrossref
37.
Blaylock  RL, Strunecka  A.  Immune-glutamatergic dysfunction as a central mechanism of the autism spectrum disorders.  Curr Med Chem. 2009;16(2):157-170. doi:10.2174/092986709787002745PubMedGoogle ScholarCrossref
38.
Horvath  K, Papadimitriou  JC, Rabsztyn  A, Drachenberg  C, Tildon  JT.  Gastrointestinal abnormalities in children with autistic disorder.  J Pediatr. 1999;135(5):559-563. doi:10.1016/S0022-3476(99)70052-1PubMedGoogle ScholarCrossref
39.
Finegold  SM, Molitoris  D, Song  Y,  et al.  Gastrointestinal microflora studies in late-onset autism.  Clin Infect Dis. 2002;35(suppl 1):S6-S16. doi:10.1086/341914PubMedGoogle ScholarCrossref
40.
Furlano  RI, Anthony  A, Day  R,  et al.  Colonic CD8 and gamma delta T-cell infiltration with epithelial damage in children with autism.  J Pediatr. 2001;138(3):366-372. doi:10.1067/mpd.2001.111323PubMedGoogle ScholarCrossref
41.
Kirsch  R, Riddell  RH.  Histopathological alterations in irritable bowel syndrome.  Mod Pathol. 2006;19(12):1638-1645. doi:10.1038/modpathol.3800704PubMedGoogle ScholarCrossref
42.
US National Library of Medicine. Errata, retractions, and other linked citations in PubMed. https://www.nlm.nih.gov/bsd/policy/errata.html. Accessed September 22, 2019.
43.
American Psychological Association. Publication Manual of the American Psychological Association. 6th ed. Washington, DC: American Psychological Association; 2010.
44.
Iverson  C, Christiansen  S, Flanagin  A. 3.11.4, References, Retractions. In:  AMA Manual of Style: A Guide for Authors and Editors. New York, NY: Oxford University Press; 2009.
45.
Oransky  I. Want to check for retractions in your personal library—and get alerts—for free? now you can. Retraction Watch Website. https://retractionwatch.com/2019/06/12/want-to-check-for-retractions-in-your-personal-library-and-get-alerts-for-free-now-you-can/. Updated June 12, 2019. Accessed September 22, 2019.
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Medical Journals and Publishing
    November 15, 2019

    Assessment of Citations of the Retracted Article by Wakefield et al With Fraudulent Claims of an Association Between Vaccination and Autism

    Author Affiliations
    • 1Medical College of Wisconsin Libraries, Medical College of Wisconsin, Milwaukee
    • 2Ziebert Medical Library, Advocate Aurora West Allis Medical Center, West Allis, Wisconsin
    • 3Todd Wehr Library, Carroll University, Waukesha, Wisconsin
    JAMA Netw Open. 2019;2(11):e1915552. doi:10.1001/jamanetworkopen.2019.15552
    Key Points español 中文 (chinese)

    Question  What are the characteristics of citations of the retracted 1998 article by Wakefield et al that purported to show an association between the measles-mumps-rubella vaccine and autism?

    Findings  In this cross-sectional bibliographic analysis of 1153 works citing the article by Wakefield et al, citation characteristics were mostly negative, but since the notice of retraction was issued in 2010, many of the citing works published afterward did not indicate that the article was retracted.

    Meaning  The findings suggest that improvements are needed from publishers, bibliographic databases, and citation management software to ensure that retracted articles are accurately documented.

    Abstract

    Importance  The number of citations can be used to show the influence of an article or to measure the validity of a research study. The article by Wakefield et al that fraudulently reported an association between vaccination and autism continues to accumulate citations even after it was retracted.

    Objectives  To examine the characteristics of citations from scholarly literature that reference the 1998 article by Wakefield et al and to investigate whether authors are accurately citing retracted references.

    Design, Setting, and Participants  In this cross-sectional bibliographic analysis of the scholarly publications that cited a 1998 article by Wakefield et al, cited references were collected from a Web of Science Core Collection search performed on March 11, 2019. A total of 1211 articles were identified, with 58 citing works excluded because they were non–English-language publications or the citation to the study by Wakefield et al could not be located by reviewers. Citing works consisted of books, research articles, letters, editorials, news items, and other scholarly literature. Citations to the article by Wakefield et al were identified and analyzed by 2 reviewers in a blinded screening. Reviewers assigned a characteristic to each citation and indicated whether the retraction was documented.

    Main Outcomes and Measures  The characteristics of citations to the article by Wakefield et al, were categorized as negative, affirmative, or contrastive; if not, persuasive; and if not, assumptive, perfunctory, methodologic, or conceptual. Whether the partial retraction or notice of retraction was included in the citing work was also documented.

    Results  Among the 1153 citing works included in this analysis, the most common citation characteristics were negative (838 [72.7%]) followed by perfunctory (106 [9.2%]) and affirmative (94 [8.2%]). A total of 123 of 322 citing works (38.2%) published between 2005 and 2010 documented the partial retraction. After the notice of retraction was published in 2010, the percentage of citing works that documented the partial retraction and/or notice of retraction between 2011 and 2018 increased to 360 of 502 (71.7%).

    Conclusions and Relevance  Since the article by Wakefield et al was initially published, authors have mostly negated the findings of the study. A significant number of authors did not document retractions of the article by Wakefield et al. The findings suggest that improvements are needed from publishers, bibliographic databases, and citation management software to ensure that retracted articles are accurately documented.

    Introduction

    In an era of information overload, it can be challenging to find influential articles in any given field, and one method of identifying such influential articles is to look at citation counts. High citation counts are often equated with articles with high impact factors,1-3 but “impact is not the same as importance or significance.”4(p 290) The raw number does not show the intent of an author in deciding to cite another’s work, and there can be a variety of motivations behind the intent.5,6 For instance, citing a work in a negative manner can be used by researchers to self-correct flawed research and aid in scientific debate.7 Value would be added to citation count numbers if more information was provided concerning the context of the citations.5,6 However, obtaining information about citation characteristics is laborious, and using natural language processing is a relatively new and emerging field.7,8

    Retracted articles continue to be cited frequently without listing the retraction,9,10 perhaps because there are no clear guidelines on whom is responsible for ensuring that retracted articles are properly cited as being retracted.11 The Committee on Publication Ethics (COPE) states, “Retracted articles should be clearly identified as such in all electronic sources,”12(p 202) but the committee does not offer guidelines on how and when authors should document that an article has been retracted. The International Committee of Medical Journal Editors (ICMJE) notes that not all journals check the accuracy of references in submitted publications.13 The omission of a work’s retracted status in the citation can lead to the perception that the cited work is valid,14 requiring scholars to spend time, effort, and money to correct the misperceptions of the problematic work.15

    The purpose of this cross-sectional study was to examine how authors of scholarly literature cited the 1998 article by Wakefield et al16 that purported to show an association between the measles-mumps-rubella (MMR) vaccine and autism. The article was retracted in 2 stages: in 2004, 10 of the original 12 authors issued a partial retraction of the interpretation that there is a causal link between the MMR vaccine and autism (1 of the original authors was unable to be contacted and Wakefield was not an author included in this partial retraction),17 and in 2010, editors of The Lancet published a notice of retraction of the article because of false claims made in the original article.18 Despite the partial retraction and notice of retraction, the 1998 article by Wakefield et al16 continues to accumulate a significant amount of citations. In 2014, of the 58 million references in Web of Science, only 14 499 works (0.026%) have been cited more than 1000 times.19 The 1998 article by Wakefield et al16 had accumulated more than 1211 citations by March 2019, and the number continues to increase. According to the Web of Science Core Collection, in April 2019, the article by Wakefield et al16 was the ninth most-cited reference indexed with the topic of autism of more than 57 600 references and the second most-cited reference indexed with the topic of measles vaccines of more than 900 references. The present study was not the first citation analysis of the article by Wakefield et al.16 For instance, Chen and Leydesdorff8,20 used citation data from Web of Science to look for patterns in how journals and research fields cited the retracted article by Wakefield et al.16 However, these patterns do not examine the context of the citations. The present study is novel because it is the first time, to our knowledge, that each citation to the article by Wakefield et al16 was analyzed to see how the author cited the article, specifically assessing whether the author affirmed or negated the study by Wakefield et al.16 Reviewers examined the citing works to evaluate whether the retracted status of the article by Wakefield et al16 was identified in the citation or reference list.

    This study replicated the methods used by Leung et al21 in their article, 1980 Letter on the Risk of Opioid Addiction. The conclusion of Leung et al21(pp 2194-2195) highlights “the potential consequences of inaccurate citation and underscores the need for diligence when citing previously published studies.” Reviewers planned to use the methods of Leung et al21 to examine another influential article of questionable quality. Terms such as reference and citation are sometimes used interchangeably; thus, a glossary of terms is given in Table 1.

    Methods

    In this cross-sectional study, we (E.M.S., J.D., K.L.H., B.R., R.S., and E.W.) conducted a cited reference search on March 11, 2019, in the Web of Science Core Collection to identify scholarly literature that cited the 1998 article by Wakefield et al.16 The reviewers’ Web of Science Core Collection subscription includes access to Science Citation Index Expanded, Social Science Citation Index, Arts & Humanities Citation Index, and Book Citation Index, allowing the reviewers to find cited references that span all scholarly disciplines, including clinical medicine, social sciences, immunology, and neurology and behavior. Web of Science indexes research articles, letters, editorials, news items, proceedings literature, books, and other scholarly literature from sources that demonstrate high levels of editorial rigor and meet a well-defined set of criteria,22 thus allowing a good representation of the scholarly community’s reaction to the 1998 article by Wakefield et al.16 The Cited Reference Search feature in Web of Science allows for a comprehensive search of bibliographies and reference lists of all items that are indexed in Web of Science. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.23 The Medical College of Wisconsin Institutional Review Board Office reviewed this project and determined that this study did not qualify as human subjects research and was therefore not subject to institutional review board review.

    We identified 1211 citations to the 1998 article by Wakefield et al16 as of March 11, 2019. Fifty-eight articles were excluded from the citation analysis because the works were not written in English or the citation to the article by Wakefield et al16 could not be located by the reviewers. A total of 1153 citing works were included in the citation analysis. Bibliographic information and the full-text copy of each citing work was uploaded into Covidence systematic review software (Veritas Health Innovation). Covidence is a web-based software platform that is used to manage the screening process and extract data for systematic reviews; it allows blinded screening and custom tagging of records.

    Citing works were reviewed to determine the characteristic of the citation using an established taxonomy5,21 (Table 2). Each citing work underwent a blinded screening by 2 of us (E.M.S., J.D., K.L.H., B.R., R.S., and/or E.W.) who located the citations within the text of the work and independently assigned them a characteristic. Disagreement about the characteristics were brought before the group (E.M.S., J.D., K.L.H., B.R., R.S., and E.W.) for consensus. If citations fit into more than 1 category or the article by Wakefield et al16 was cited more than once, a stepwise approach was used to assign the category.21 In the stepwise approach, citations were first screened to assess whether they fit in the categories of negative, affirmative, or contrastive; if not, they were screened for the category of persuasive; if not, citations were screened for the categories of assumptive, perfunctory, methodologic, or conceptual. If a citation could not be located, the citing work was excluded from the review. For multiple citation occurrences, the citation was categorized as contrastive if any of the citations had this characteristic. Barring this, a citation was categorized as overall negative or affirmative if any of the occurrences had these characteristics. Finally, if citations did not meet any of these criteria, the citing works were brought before the group for consensus. Every citing work in the sample was assigned one characteristic.

    Not only were the citing works screened to determine the characteristic of each citation, they also were examined to determine whether the retracted status of the article by Wakefield et al16 was identified in the citation or in the reference list. A citing work was labeled as retraction referenced if the author specifically used the word retracted, retract, or retraction.

    Statistical Analysis

    Bibliographic information, characteristics, and retraction information were exported from Covidence as a CSV file, and Excel (Microsoft Corp) was used to analyze the data. A descriptive analysis of the data was performed with a focus on frequencies and percentages. The sample consisted of all the included citing works retrieved from the Web of Science search apart from the 58 excluded articles.

    Results
    Characteristics

    Of the 1153 citing works, 838 (72.7%) were negative, 106 (9.2%) perfunctory, 94 (8.2%) affirmative, and 60 (5.2%) assumptive. The other characteristics (conceptual, contrastive, methodologic, and persuasive) had a combined total of 55 (4.8%). Since the article by Wakefield et al16 was initially published, authors have mostly cited the article in a negative manner (Figure 1). Authors who affirmed the article by Wakefield et al16 in their citations comprised 94 of the 1153 total citing works (8.2%). Of the affirmative citations, 49 of 94 (52.1%) were published between 1998 and 2003, before the partial retraction of the article by Wakefield et al.16 Wakefield was an author of 15 of 94 articles with affirmative citations. Of these 15 affirmative self-citing works, 10 include at least 1 coauthor of the original article. The only other self-citing work by the original authors of the article by Wakefield et al16 is the 2004 partial retraction by Murch et al,17 which was characterized as negative.

    Retractions

    From the date of the partial retraction on March 4, 2004, to March 11, 2019, a total of 881 works were published that cited the article by Wakefield et al.16 These citing works were screened to determine whether they documented the partial and/or notice of retraction to the article by Wakefield et al,16 and in this period, 493 of 881 (56.0%) of the citing works documented either retraction. Of 57 citing works published in 2004, a total of 10 (17.5%) documented the partial retraction in the citations or the reference lists. However, the reviewers began the retraction analysis (Figure 2) in 2005 to compensate for works that were submitted for publication before the partial retraction was published and to ensure that the retraction information was indexed in bibliographic databases, making it more likely to be discovered. Between 2005 and 2010, a total of 123 of 322 (38.2%) citing works documented the partial retraction. In 2010, editors of The Lancet issued a notice of retraction18 to the article by Wakefield et al.16 Of the 71 citing works published in 2010, a total of 21 (29.6%) documented the 2010 notice of retraction or retracted status of the article. The number of authors who documented the partial retraction or the notice of retraction between 2011 and 2018 increased to 360 of 502 (71.7%). Since 2013, the percentage of authors who have documented either retraction continually improved, and in 2018, a total of 54 of 61 citing works (88.5%) documented either retraction.

    We found that authors documented the retraction in different ways. Of the 493 citing works that documented either retraction, 211 (42.8%) documented the retraction in both the citation and the reference list; 189 (38.3%) documented the retraction in their citations, but the reference to the study by Wakefield et al16 in the reference list did not include a variation of the word retract or a reference of the notice of retraction or of the partial retraction; and 93 (18.9%) documented either retraction in the reference list or included the word retract in the reference to the article by Wakefield et al16 but not in the citation.

    Discussion

    The first aim of this study was to assign a characteristic to each citing work of the article by Wakefield et al16 to determine how authors cited the information. Most authors cited the article by Wakefield et al16 in a negative manner. Even before the article was formally retracted, authors were asserting that there were problems with the study, including the small sample size24-26 and the lack of epidemiologic evidence to support the increased risk of autism after receiving the MMR vaccine.27-29 Later, authors reported that the conclusions of the article by Wakefield et al16 were not reproducible.30-32 Soon after publication of the study, scholars may have recognized that it was flawed. Thus, although this was a highly influential article, it is an example of how high citation count may not equate with a high-quality work.

    Articles that affirmed the article by Wakefield et al16 in their citations comprised only 8% of the total cited references. There were common themes among these citing works. Some authors stated that the article confirmed parents’ observations of their autistic children,33-35 others affirmed the gut-brain connection to autism,36-38 and others cited the article by Wakefield et al16 in their own article’s introduction or discussion section as a reason to support their research studies on gut diseases.39-41

    The second aim of this study was to record whether the retractions were accurately documented. The ICMJE recommendations for manuscript preparations suggest that authors are responsible for ensuring that reference lists are accurate, that authors use PubMed as an authoritative source for information about retractions, and that authors should note the retracted status of the article in the citation when citing a retracted article.13 Despite these recommendations, 142 of the 502 citing works (28.3%) published after 2011 did not document either retraction or note the retracted status in the citation.

    There are challenges in identifying and documenting retracted literature. First, bibliographic databases have differing policies on indexing retractions. According to Web of Science (Product Support, Clarivate Analytics, written communication, September 25, 2019), if a work is specifically identified as having been retracted, the title of the original item is updated to include “Retracted Article” along with the published notice of the retraction’s citation information. The Document Type “Retracted Publication” is added to the record for the original item. The notice of retraction is also indexed using the same title and authors as the original item so that any title or author search that retrieves the original will also retrieve the notice of retraction. The title of the notice of retraction is also updated to include “(Retraction of)” along with the retracted item’s citation information. The Document Type “Retraction” will be added to the record of the notice of retraction.

    Because Web of Science alters the original publication’s record by adding the words “retracted article” in the title of the original publication and by including “retraction of” in the record for the notice of retraction, the work’s retracted status is evident. Even after downloading into citation management software, the title of the retracted work includes “retracted article,” making it hard to overlook the retracted status.

    In contrast to Web of Science, PubMed connects citations for original articles and citations for notices of retraction based on information supplied by the journal publisher within their database and includes a banner that indicates that the publication is retracted but does not change the title of the original publication. “The Publication Type ‘Retraction of Publication’ is assigned to the citation for the retraction notice, and the citation is linked to the citation for the retracted article. The Publication Type ‘Retracted Publication’ is assigned to the retracted article. Citations for retraction notices contain the phrase ‘Retraction of: [article title],’ and citations for retracted articles contain the phrase ‘Retraction in: [article title].’”42

    Although it is clear that an article has been retracted when searching within PubMed, after downloading and/or depositing into citation management software, only the publication type identifier indicates that the work has been retracted. Many information users may overlook this indication.

    Second, journals have differing policies on how they update articles that have been retracted. For example, JAMA adds a watermark with the word retraction in red letters to the article record on their website, and the PDF versions of retracted articles are marked with a header to alert the reader that the article has been retracted, per COPE recommendations.12 However, when the article information is downloaded into citation management software, the retracted status is not evident. The Lancet, on the other hand, updates the item record of the article to include the word retracted at the beginning of the title. When article information is downloaded into citation management software, the retraction information is labeled. Changing the article title makes it easier for bibliographic databases to capture the retraction information as well. We recommend that publishers add a label, such as “retracted,” to the title of articles that have been issued notices of retraction.

    Third, not all citation styles provide guidelines for citing retractions. The Publication Manual of the American Psychological Association, sixth edition43 does not contain instructions for citing a retracted article. The AMA Manual of Style has a reference style for retractions, but citation management software, such as EndNote, RefWorks, and Zotero, do not automatically include the additional information required by the AMA Manual of Style.44 According to the AMA citation style,44 the citation to the 1998 article by Wakefield et al16 should appear as follows:

    1. Wakefield AJ, Murch SH, Anthony A, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children [retracted in: Lancet. 2010;375(9713):445]. Lancet. 1998;351(9103):637-641.

    The text within the brackets was manually added to the reference by the authors. In this example, only the notice of retraction was documented. It is not clear from the AMA Manual of Style whether notice of partial retraction should also be included in the citation. Authors may not be aware that their citation management software is not properly citing a retracted article, and journal editors and article reviewers are likely relying on authors to ensure that their citations are correct.

    Fourth, a retraction can occur after a bibliographic reference has been downloaded into citation management software; thus, an information user may not be aware of a subsequent retraction. Zotero45 now includes an enhancement that will identify retracted articles that have been downloaded into a Zotero library, but not all citation management software includes this feature. The retracted status of the article by Wakefield et al16 is well known in the scholarly community, but authors failed to cite it accurately. To ensure the integrity of scholarly articles and research, better care needs to be taken to ensure that retracted articles are properly cited.

    Limitations

    This study has limitations. The reviewers relied solely on Web of Science to identify citing works of the 1998 article by Wakefield et al.16 In addition, assigning a single characteristic to a citing work was challenging and subjective.

    Conclusions

    Although the 1998 article by Wakefield by al16 continues to accumulate citations, most works that cited it did so in a negative manner. The retracted status of the article by Wakefield article et al16 is well known in the scholarly community, but the findings suggest that authors failed to cite it accurately. To ensure the integrity of scholarly articles and research, we believe that better care needs to be taken to ensure that retracted articles are properly cited and that it is ultimately the authors’ responsibility to ensure that their citations are accurately documented. Improvements can be made to the indexing procedures of bibliographic databases, journal publisher procedures for updating retracted articles, and citation management software products to make it more apparent when articles have been retracted. In addition, stronger guidelines from the ICMJE, COPE, and citation styles on how to cite retracted articles appear to be needed. We also believe that authors should take additional steps to verify their citations by using bibliographic databases, such as PubMed and Retraction Watch, or enlisting the help of librarians and that journal editors should hold authors more accountable for checking their references.

    Back to top
    Article Information

    Accepted for Publication: September 26. 2019.

    Published: November 15, 2019. doi:10.1001/jamanetworkopen.2019.15552

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Suelzer EM et al. JAMA Network Open.

    Corresponding Author: Elizabeth M. Suelzer, MLIS, AHIP, Medical College of Wisconsin Libraries, 8701 Watertown Plank Rd, Milwaukee, WI 53226 (esuelzer@mcw.edu).

    Author Contributions: Ms Suelzer had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Suelzer, Deal, Ruggeri, Witkowski.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Suelzer, Hanus, Ruggeri, Witkowski.

    Critical revision of the manuscript for important intellectual content: All authors.

    Administrative, technical, or material support: Suelzer, Deal, Hanus, Sieracki.

    Supervision: Suelzer.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: This study was supported by grant RE-95-17-0025-17 from the Institute of Museum and Library Services.

    Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: The Medical Library Association (MLA) Research Training Institute (RTI) staff provided the training, support, and encouragement to perform this research; Dan Eastwood, MS, Medical College of Wisconsin Biostatistics Consulting Service, provided advice and assistance with the statistics; and Jodi Philbrick, PhD, University of North Texas, provided mentorship and valuable input. Neither Mr Eastwood nor Dr Philbrick were compensated for their work. The Institute of Museum and Library Sciences partially funded the MLA RTI program pertaining to administrative and instructional costs.

    References
    1.
    Van Raan  AF.  The use of bibliometric analysis in research performance assessment and monitoring of interdisciplinary scientific developments.  Technol Assess Theory Prac. 2003;1(12):20-29.Google Scholar
    2.
    Vaughan  L, Shaw  D.  Web citation data for impact assessment: a comparison of four science disciplines.  J Am Soc Inf Sci Technol. 2005;56(10):1075-1087. doi:10.1002/asi.20199Google ScholarCrossref
    3.
    Cole  JR. A short history of the use of citations as a measure of the impact of scientific and scholarly work. In: Cronin  B, Atkins  HB, eds.  The Web of Knowledge: A Festschrift in Honor of Eugene Garfield. Medford, NJ: Information Today; 2000:281-300.
    4.
    Garfield  E.  Citation indexes in sociological and historical research.  Am Doc. 1963;14(4):289-291. doi:10.1002/asi.5090140405Google ScholarCrossref
    5.
    Bornmann  L, Daniel  HD.  What do citation counts measure? a review of studies on citing behavior.  J Doc. 2008;64(1):45-80. doi:10.1108/00220410810844150Google ScholarCrossref
    6.
    Garfield  E. Can citation indexing be automated? In: Symposium Proceedings, Statistical Association Methods for Mechanized Documentation. Washington, DC: National Bureau of Standards; 1965:189-192. National Bureau of Standards Miscellaneous Publication 269.
    7.
    Catalini  C, Lacetera  N, Oettl  A.  The incidence and role of negative citations in science.  Proc Natl Acad Sci U S A. 2015;112(45):13823-13826. doi:10.1073/pnas.1502280112PubMedGoogle ScholarCrossref
    8.
    Chen  CM, Hu  ZG, Milbank  J, Schultz  T.  A visual analytic study of retracted articles in scientific literature.  J Am Soc Inf Sci Technol. 2013;64(2):234-253. doi:10.1002/asi.22755Google ScholarCrossref
    9.
    Bucci  EM.  On zombie papers.  Cell Death Dis. 2019;10(3):189. doi:10.1038/s41419-019-1450-3PubMedGoogle ScholarCrossref
    10.
    Pfeifer  MP, Snodgrass  GL.  The continued use of retracted, invalid scientific literature.  JAMA. 1990;263(10):1420-1423. doi:10.1001/jama.1990.03440100140020PubMedGoogle ScholarCrossref
    11.
    Bar-Ilan  J, Halevi  G.  Post retraction citations in context: a case study.  Scientometrics. 2017;113(1):547-565. doi:10.1007/s11192-017-2242-0PubMedGoogle ScholarCrossref
    12.
    Wager  E, Barbour  V, Yentis  S, Kleinert  S, Council  C; COPE Council.  Retractions: guidance from the Committee on Publication Ethics (COPE).  Maturitas. 2009;64(4):201-203. doi:10.1016/j.maturitas.2009.09.018PubMedGoogle ScholarCrossref
    13.
    International Committee of Medical Journal Editors. Manuscript preparation: preparing for submission. http://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html. Published 2018. Accessed June 11, 2019, 2019.
    14.
    Madlock-Brown  CR, Eichmann  D.  The (lack of) impact of retraction on citation networks.  Sci Eng Ethics. 2015;21(1):127-137. doi:10.1007/s11948-014-9532-1PubMedGoogle ScholarCrossref
    15.
    Budd  JM, Sievert  M, Schultz  TR.  Phenomena of retraction: reasons for retraction and citations to the publications.  JAMA. 1998;280(3):296-297. doi:10.1001/jama.280.3.296PubMedGoogle ScholarCrossref
    16.
    Wakefield  AJ, Murch  SH, Anthony  A,  et al.  Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children  [retracted in: Lancet. 2010;375(9713):445].  Lancet. 1998;351(9103):637-641. doi:10.1016/S0140-6736(97)11096-0PubMedGoogle ScholarCrossref
    17.
    Murch  SH, Anthony  A, Casson  DH,  et al.  Retraction of an interpretation  [retraction of: Wakefield AJ, Murch SH, Anthony A, et al. In: Lancet 1998;351(9103):637-641].  Lancet. 2004;363(9411):750. doi:10.1016/S0140-6736(04)15715-2PubMedGoogle ScholarCrossref
    18.
    The Editors of The Lancet.  Retraction–Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children  [retraction of: Wakefield AJ, Murch SH, Anthony A, et al. In: Lancet 1998;351(9103):637-641].  Lancet. 2010;375(9713):445. doi:10.1016/S0140-6736(10)60175-4PubMedGoogle ScholarCrossref
    19.
    Van Noorden  R, Maher  B, Nuzzo  R.  The top 100 papers.  Nature. 2014;514(7524):550-553. doi:10.1038/514550aPubMedGoogle ScholarCrossref
    20.
    Chen  CM, Leydesdorff  L.  Patterns of connections and movements in dual-map overlays: a new method of publication portfolio analysis.  J Am Soc Inf Sci Technol. 2014;65(2):334-351. doi:10.1002/asi.22968Google ScholarCrossref
    21.
    Leung  PTM, Macdonald  EM, Stanbrook  MB, Dhalla  IA, Juurlink  DNA.  1980 Letter on the risk of opioid addiction.  N Engl J Med. 2017;376(22):2194-2195. doi:10.1056/NEJMc1700150PubMedGoogle ScholarCrossref
    22.
    Testa  J. The selection process for the Book Citation Index in Web of Science. Clarivate Analytics. https://clarivate.com/webofsciencegroup/essays/selection-process-book-citation-index-web-science/. Published 2017. Accessed September 22, 2019.
    23.
    von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.  Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010PubMedGoogle ScholarCrossref
    24.
    Christie  CDC.  Causal link between MMR vaccination and autism: fact or fiction?  West Indian Med J. 2000;49(4):266-267.PubMedGoogle Scholar
    25.
    Fombonne  E, Cook  EH.  MMR and autistic enterocolitis: consistent epidemiological failure to find an association.  Mol Psychiatry. 2003;8(2):133-134. doi:10.1038/sj.mp.4001266PubMedGoogle ScholarCrossref
    26.
    Salisbury  DM, Beverley  PCL, Miller  E.  Vaccine programmes and policies.  Br Med Bull. 2002;62:201-211. doi:10.1093/bmb/62.1.201PubMedGoogle ScholarCrossref
    27.
    O’Brien  SJ, Jones  IG, Christie  P.  Autism, inflammatory bowel disease, and MMR vaccine.  Lancet. 1998;351(9106):906-907. doi:10.1016/S0140-6736(05)70318-4PubMedGoogle ScholarCrossref
    28.
    Afzal  MA, Pipkin  PA, Minor  PD.  Absence of chicken myelin basic protein residues in commercial formulations of MMR vaccine.  Vaccine. 2000;19(4-5):442-446. doi:10.1016/S0264-410X(00)00202-4PubMedGoogle ScholarCrossref
    29.
    Nicoll  A, Elliman  D, Ross  E.  MMR vaccination and autism 1998.  BMJ. 1998;316(7133):715-716. doi:10.1136/bmj.316.7133.715PubMedGoogle ScholarCrossref
    30.
    Huang  DB, Wu  JJ, Tyring  SK.  A review of licensed viral vaccines, some of their safety concerns, and the advances in the development of investigational viral vaccines.  J Infect. 2004;49(3):179-209. doi:10.1016/j.jinf.2004.05.018PubMedGoogle ScholarCrossref
    31.
    Bushara  KO.  Neurologic presentation of celiac disease.  Gastroenterology. 2005;128(4)(suppl 1):S92-S97. doi:10.1053/j.gastro.2005.02.018PubMedGoogle ScholarCrossref
    32.
    Peacock  G, Yeargin-Allsopp  M.  Autism spectrum disorders: prevalence and vaccines.  Pediatr Ann. 2009;38(1):22-25. doi:10.3928/00904481-20090101-09PubMedGoogle ScholarCrossref
    33.
    Tettenborn  MA.  Autism, inflammatory bowel disease, and MMR vaccine.  Lancet. 1998;351(9112):1357. doi:10.1016/S0140-6736(05)79087-5PubMedGoogle ScholarCrossref
    34.
    Singh  VK, Lin  SX, Yang  VC.  Serological association of measles virus and human herpesvirus-6 with brain autoantibodies in autism.  Clin Immunol Immunopathol. 1998;89(1):105-108. doi:10.1006/clin.1998.4588PubMedGoogle ScholarCrossref
    35.
    Thrower  D.  Autism, bowel inflammation, and measles.  Lancet. 2002;359(9323):2113. doi:10.1016/S0140-6736(02)08920-1PubMedGoogle ScholarCrossref
    36.
    Sandler  RH, Finegold  SM, Bolte  ER,  et al.  Short-term benefit from oral vancomycin treatment of regressive-onset autism.  J Child Neurol. 2000;15(7):429-435. doi:10.1177/088307380001500701PubMedGoogle ScholarCrossref
    37.
    Blaylock  RL, Strunecka  A.  Immune-glutamatergic dysfunction as a central mechanism of the autism spectrum disorders.  Curr Med Chem. 2009;16(2):157-170. doi:10.2174/092986709787002745PubMedGoogle ScholarCrossref
    38.
    Horvath  K, Papadimitriou  JC, Rabsztyn  A, Drachenberg  C, Tildon  JT.  Gastrointestinal abnormalities in children with autistic disorder.  J Pediatr. 1999;135(5):559-563. doi:10.1016/S0022-3476(99)70052-1PubMedGoogle ScholarCrossref
    39.
    Finegold  SM, Molitoris  D, Song  Y,  et al.  Gastrointestinal microflora studies in late-onset autism.  Clin Infect Dis. 2002;35(suppl 1):S6-S16. doi:10.1086/341914PubMedGoogle ScholarCrossref
    40.
    Furlano  RI, Anthony  A, Day  R,  et al.  Colonic CD8 and gamma delta T-cell infiltration with epithelial damage in children with autism.  J Pediatr. 2001;138(3):366-372. doi:10.1067/mpd.2001.111323PubMedGoogle ScholarCrossref
    41.
    Kirsch  R, Riddell  RH.  Histopathological alterations in irritable bowel syndrome.  Mod Pathol. 2006;19(12):1638-1645. doi:10.1038/modpathol.3800704PubMedGoogle ScholarCrossref
    42.
    US National Library of Medicine. Errata, retractions, and other linked citations in PubMed. https://www.nlm.nih.gov/bsd/policy/errata.html. Accessed September 22, 2019.
    43.
    American Psychological Association. Publication Manual of the American Psychological Association. 6th ed. Washington, DC: American Psychological Association; 2010.
    44.
    Iverson  C, Christiansen  S, Flanagin  A. 3.11.4, References, Retractions. In:  AMA Manual of Style: A Guide for Authors and Editors. New York, NY: Oxford University Press; 2009.
    45.
    Oransky  I. Want to check for retractions in your personal library—and get alerts—for free? now you can. Retraction Watch Website. https://retractionwatch.com/2019/06/12/want-to-check-for-retractions-in-your-personal-library-and-get-alerts-for-free-now-you-can/. Updated June 12, 2019. Accessed September 22, 2019.
    ×