From the School of Information Science and Learning Technologies, University of Missouri–Columbia.
Context.— This study examined the impact of retracted articles on biomedical communication.
Objective.— To examine publications identified in the biomedical literature as having
been retracted, to ascertain why and by whom the publications were retracted
and to what extent citations of later-retracted articles continue to be incorporated
in subsequent work.
Design.— A search of MEDLINE from 1966 through August 1997 for articles that
had been retracted.
Main Outcome Measures.— Characteristics of retractions and citations to articles after
Results.— A total of 235 articles had been retracted. Error was acknowledged in
relation to 91 articles; results could not be replicated in 38; misconduct
was evident in 86; and no clear reason was given in 20. Of the 235 articles,
190 were retracted by some or all of the authors; 45 were retracted by a person
or organization other than the author(s). The 235 retracted articles were
cited 2034 times after the retraction notice. Examination of 299 of those
citations reveals that in only 19 instances was the retraction noted; the
remaining 280 citations treated the retracted article either explicitly (n=17)
or implicitly (n=263) as though it were valid research.
Conclusion.— Retracted articles continue to be cited as valid work in the biomedical
literature after publication of the retraction; these citations signal potential
problems for biomedical science.
VARIOUS FACTORS can lead to the retraction of a publication. There have
been some notorious cases of scientific misconduct in recent years. LaFollette1 and Whitley et al2
report on some of the more widely publicized cases, and a collection of essays
edited by Lock and Wells3 addresses generally
the matters of fraud and misconduct. While these concerns are legitimate and
pressing, the biomedical literature is also affected by error that can render
the reported results of research useless at best and dangerous at worst. The
work done by Stewart and Feder4 focuses on
the occurrence of error. Awareness of the retraction and reasons for retraction
might affect the frequency with which such articles are cited subsequent to
Based on the available background research, a set of expectations can
be stated: the authors are the ones doing the retracting; most retractions
occur because of scientific misconduct or unavoidable error; the entire article
should be regarded as an invalid scientific article; and retracted publications
continue to be cited after retraction.
The data for analysis came from a MEDLINE search (1966-August 1997)
that used the publication type "retraction of publication." This strategy
includes 2 limitations: an article must be formally retracted to be assigned
this publication type, and indexers must recognize the retraction and assign
the publication type. Further, this study focuses entirely on retraction,
so corrections and errata are not included. The search yielded 235 retracted
articles. This population is a very small subset of the MEDLINE database,
but it does include all publications formally identified as retractions. The
next task was to classify the articles according to the following characteristics:
who retracted the publication; what content was retracted; why the article
(or portion of the article) was retracted; and how long after publication
the retraction occurred.
To determine citing activity, Science Citation Index was searched to identify all citations to each of the retracted publications.
A 1-year period after publication of the retraction was inserted before a
citation was considered as postretraction to allow for indexing of the retraction
to be in place. Searchers of the MEDLINE database would then have ready access
to the retraction statement. Also, the 1-year period compensates for publication
lag; that is, if a manuscript is in press at the time a retraction statement
is published, then that article could, in good faith, contain citations to
retracted items. This study focuses on postretraction citations appearing
in journals indexed in the Abridged Index Medicus
(AIM), since that source contains the most clinically
relevant journals. The postretraction citations were divided into 3 categories:
the citing article acknowledged the retraction, the citing article explicitly
cited the retracted article as presenting valid research, or the citing article
implicitly cited the retracted article as valid. In addition to noting the
category of citation, the kind of citing publication (letter, review article,
or article) and where in the citing article (introduction, methods, results,
discussion, or conclusions) the citation occurs were recorded.
One element of the phenomenon of retraction addressed was the length
of time between publication of an article and its retraction. The mean time
from publication to retraction was 28 months. This mean includes the retractions
by 1 author of 4 articles 10 years after their publication. Controlling for
this anomalous case yields a mean time from publication to retraction of 25.8
months (range, 2-197 months).
One or more of the authors retracted 190 of the 235 articles; 45 were
retracted by others, including institutional investigating committees or deans,
journal editors, or legal counsels. Retraction of 91 articles occurred because
of some kind of error. The categories of error identified were the following:
error in the methods or analysis in 23, problems with the data in 37, and
problems with the sample (such as contamination) in 31. A total of 86 articles
were retracted because of misconduct or presumed misconduct. Retraction was
classified as being due to misconduct only if the statement of retraction
clearly admits to wrongdoing on the part of one or more of the authors. Presumed
misconduct refers to those instances where one or more of the authors raises
serious questions about the efficacy of the work done by other authors. An
additional 38 articles were retracted because the author(s) could not replicate
the results. These were categorized separately because it is not possible
to determine if the results were not replicable because of unavoidable error
or because of some wrongdoing. The remaining 20 articles were retracted for
other unclassifiable reasons. In 3 instances the retractors were unclear as
to reason, in 3 cases idiosyncratic reasons were given, and in 14 cases no
reason was given. In 200 instances the entire article was retracted; in 35
cases part, but not all, of the article was retracted.
The 235 articles received a total of 2034 postretraction citations.
Of these citations, 299 appeared in journals indexed in AIM. The majority of the citations were found in articles reporting
research or clinical practice (n=277). Only 14 citations appeared in letters,
and 5 of those letters referred to the retraction. Review articles accounted
for 8 citations, and only 1 of those acknowledged the retraction. Of the 299
postretraction citations appearing in AIM journals,
19 acknowledged the retraction in some way. Given that 5 of these acknowledgments
appeared in letters and 1 appeared in a review article, only 13 articles reporting
research made specific mention of the retraction. Of the remaining citations,
17 explicitly treated the retracted article as valid, usually by naming the
authors of the article or mentioning specific elements of their findings or
methods. The remaining 263 citations include implicit approval of the retracted
work, usually in the form of brief mention or bibliographic reference in a
passage that in no way questions the validity of the research.
The second of the stated expectations proved to be somewhat problematic;
the first, third, and fourth were shown to be, with a few exceptions, supported.
The results of this study strongly indicate that retraction of a publication,
even though the retraction may be visible in the journal and is clearly noted
in the MEDLINE database, does not ensure that all subsequent researchers will
be alerted to the retraction and will cease making reference to the retracted
work. It seems to matter little if the cause of retraction is error or misconduct;
citations to any retracted article may well continue.
While 263 of the citing articles in this study embody implicit positive
citation to retracted publications, and while these citations tend to appear
in the introduction (n=117) or discussion (n=153) sections of articles, the
citations still ensure that the retracted articles continue to appear in citation
indexes and that they may be retrieved by readers of the citing article. (It
should be noted that 47 citations appear in the methods section, 6 in the
results, and 2 in the conclusion.) If a researcher comes upon one of the citing
articles and finds the work done there of use, then he or she may turn to
cited works and incorporate them into his or her work as though the work were
valid. Such a researcher, who does not retrieve the information through a
formally structured MEDLINE search, may be unaware of the retraction.
It should be pointed out that biomedical science tends to be self-correcting;
that is, work that is not replicable because of error or misconduct is usually
dismissed in time. However, there may be a great deal of time, effort, and
money spent in discovering that some research is not useful. If erroneous
or fraudulent work lives on in the literature, the amount of time, effort,
and money to correct work may be even greater. It should also be noted that
this study focuses on work that has been acknowledged to be based on error
or misconduct. The question remains as to how much erroneous or fraudulent
work goes undetected or unacknowledged. That is a larger question that should
be addressed by the biomedical community; the evidence provided by this study
suggests that it is a serious question with profound implications both for
research and for clinical practice.
Budd JM, Sievert M, Schultz TR. Phenomena of RetractionReasons for Retraction and Citations to the Publications. JAMA. 1998;280(3):296–297. doi:10.1001/jama.280.3.296