Context Controversy and uncertainty ensue when the results of clinical research
on the effectiveness of interventions are subsequently contradicted. Controversies
are most prominent when high-impact research is involved.
Objectives To understand how frequently highly cited studies are contradicted or
find effects that are stronger than in other similar studies and to discern
whether specific characteristics are associated with such refutation over
time.
Design All original clinical research studies published in 3 major general
clinical journals or high-impact-factor specialty journals in 1990-2003 and
cited more than 1000 times in the literature were examined.
Main Outcome Measure The results of highly cited articles were compared against subsequent
studies of comparable or larger sample size and similar or better controlled
designs. The same analysis was also performed comparatively for matched studies
that were not so highly cited.
Results Of 49 highly cited original clinical research studies, 45 claimed that
the intervention was effective. Of these, 7 (16%) were contradicted by subsequent
studies, 7 others (16%) had found effects that were stronger than those of
subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely
unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted
or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008). Among randomized trials, studies with
contradicted or stronger effects were smaller (P = .009)
than replicated or unchallenged studies although there was no statistically
significant difference in their early or overall citation impact. Matched
control studies did not have a significantly different share of refuted results
than highly cited studies, but they included more studies with “negative”
results.
Conclusions Contradiction and initially stronger effects are not unusual in highly
cited research of clinical interventions and their outcomes. The extent to
which high citations may provoke contradictions and vice versa needs more
study. Controversies are most common with highly cited nonrandomized studies,
but even the most highly cited randomized trials may be challenged and refuted
over time, especially small ones.
Clinical research on important questions about the efficacy of medical
interventions is sometimes followed by subsequent studies that either reach
opposite conclusions or suggest that the original claims were too strong.
Such disagreements may upset clinical practice and acquire publicity in both
scientific circles and in the lay press. Several empirical investigations
have tried to address whether specific types of studies are more likely to
be contradicted and to explain observed controversies. For example, evidence
exists that small studies may sometimes be refuted by larger ones.1,2
Similarly, there is some evidence on disagreements between epidemiological
studies and randomized trials.3-5 Prior
investigations have focused on a variety of studies without any particular
attention to their relative importance and scientific impact. Yet, most research
publications have little impact while a small minority receives most attention
and dominates scientific thinking and clinical practice. Impact is difficult
to measure in all its dimensions. However, the number of citations received
by a publication is a surrogate of the attention it has received in the scientific
literature and its influence on scientific debate and progress. Citations
are readily and objectively counted in established databases.6 High
citation count does not necessarily mean that these studies are accepted;
citations may sometimes be critical of an article. Nevertheless, citation
count is a measure of how much a study has occupied the thinking of other
scientists and has drawn attention—for good or bad.
It is important to evaluate the replication of clinical research studies
that have the highest citation impact. How frequently are such studies eventually
contradicted by other research or are found to have too strong results compared
with subsequent evidence? Is this more common for specific types of studies?
Answering these questions would be useful for interpreting the results of
influential clinical research.
Eligible Original Studies
Eligible original studies for this analysis included all publications
that had received more than 1000 Institute for Scientific Information (ISI)–indexed6 citations; had been published between 1990 and 2003
in the 3 general medical journals with the current highest impact factor (New England Journal of Medicine, JAMA, Lancet) or in medical
specialty journals with impact factor exceeding 7.0 (according to the Journal
Citation Reports 2003) that are likely to publish clinical research (including
in decreasing impact factor, the Journal of the National
Cancer Institute, Gastroenterology, Annals of Internal Medicine, Circulation, Journal of Clinical Oncology, Archives of General Psychiatry, Blood, Hepatology, American Journal of Respiratory and Critical Care
Medicine, Diabetes, Brain, Annals of Neurology, Journal of the American College of Cardiology, Diabetes
Care, Journal of the American Society of Nephrology, Arthritis and Rheumatism, and the American Journal of Psychiatry); addressed the efficacy
of therapeutic or preventive interventions; and pertained to primary data
(excluding reviews and meta-analyses).
Citation counts for articles published between January 1, 1990, and
December 31, 2003, in these journals were downloaded from ISI. Citation counts
are censored on August 20, 2004. All articles with more than 1000 citations
were screened further. Studies with group authorship may be cited in various
ways; therefore, I summed up citations cataloged under different entries for
the same article (using the first author name, group abbreviations, and anonymous
entries).7 The total citation count does not
capture the few citations for which wrong name, journal, volume, or page might
have been cited. Since citations depend on the time interval since publication,
a separate citation count was limited to the first 3 years after the publication
year.
Other Clinical Research on the Same Questions
For each eligible original study, a search was performed to identify
whether there had been any other concurrently or subsequently published clinical
research addressing the same question. Other research was considered eligible,
only if the sample size was close to or larger than that of the highly cited
original study or if it used a theoretically better controlled design. Thus,
for highly cited randomized trials, I perused all randomized trials having
at least 30% of the sample size of the eligible highly cited original study.
Whenever available, quantitative meta-analyses of trials were used as summaries
of trial results. Whenever several pertinent meta-analyses were available,
the one including the largest number of studies was preferred. For highly
cited nonrandomized studies, subsequently published pertinent randomized trials
and meta-analyses thereof were eligible regardless of sample size; nonrandomized
evidence was also considered, if randomized trials were not available.
Concurrently or subsequently published evidence was identified in PubMed
using searches that combined terms pertaining to the tested interventions,
disease and outcome, and terms pertinent to the search of randomized trials
and meta-analyses. Searches followed the Cochrane algorithms for finding meta-analyses
and randomized trials.8
Data Extraction and Classification of Studies
For each eligible original study, I recorded the study name, intervention,
disease and outcomes of interest, study design, sample size, main conclusions,
and citation counts. For the articles presenting or summarizing other relevant
research, I recorded the study design, total sample size, and the findings
as compared with those of the original highly cited study.
Highly cited studies were classified as negative (when they claimed
the tested experimental intervention was ineffective, harmful, or no better
from the control intervention), unchallenged (when no other clinical research
of eligible design and sample size was available to validate the claimed efficacy),
contradicted, initially stronger effects, or replicated effects. The classification
of studies in these categories was based on the final interpretation of the
results by the authors in the “Abstract” and “Discussion”
sections of their original publications. Highly cited articles were classified
according to whether their authors suggested that an intervention was overall
effective or ineffective. When both benefits and harms or caveats were presented,
I focused on the net conclusion of whether the experimental intervention merits
consideration for use in clinical practice. Subsequent research was classified
in the same manner. Contradiction was declared when the original highly cited
study claimed the intervention to be effective, while subsequent research
showed it to be ineffective. When both original and subsequent research claimed
the intervention was effective, studies were compared further regarding the
effect size for the major clinical outcome, the durability of the treatment
effect, and the generalizability and applicability to various settings. Initially
stronger effects were defined when the relative risk reduction for the main
outcome in the subsequent research was half or less compared with what had
been proposed by the original highly cited study (regardless of whether confidence
intervals might overlap or not), or when the subsequent research showed that
the originally proposed benefit was of short duration or its applicability
and generalizability was limited. Classification of the studies independently
by another investigator yielded a highly similar profile (weighted Cohen κ = 0.92).
Correlates of Contradicted or Initially Stronger Effects
Among original highly cited studies with efficacy claims, analyses examined
whether those with contradicted or initially stronger effects differed from
the replicated and unchallenged ones in study design, publication year, sample
size, type of disease (heart disease vs other), journal of publication, citation
count, early citation count, and average citations per year after publication.
Comparisons used the Mann-Whitney U test for continuous
variables and Fisher exact test for binary variables.
Comparison of Highly Cited Articles Against Less Cited Articles
To evaluate whether highly cited studies differ from other studies that
are not so highly cited in their findings and potential for contradiction,
a control group of articles pertaining to the assessment of interventions
was also assembled. Control-group articles were 1:1 matched for journal, year
of publication, and design (randomized vs nonrandomized) against each of the
highly cited articles. Control articles were selected by screening chronologically
the contents of the pertinent journals for each pertinent year starting July
1 (to ensure approximately similar follow-up for citations with the highly
cited articles against which they were matched). Other research was searched
and the control articles were categorized in a similar fashion as described
for the highly cited articles above. Differences between highly cited and
control articles were examined with conditional logistic regression to account
for matching.
Analyses were performed in SPSS version 12.0 (SPSS Inc, Chicago, Ill)
and StatXact (Cytel Corp, Boston, Mass). P values
are 2-tailed, and P<.05 was considered statistically
significant.
One hundred fifteen articles published between 1990 and 2003 had received
more than 1000 citations (major general clinical journals, n = 91;
specialty journals, n = 24). Of those, 66 were excluded (nonsystematic
reviews or editorials, n = 20; meta-analyses, n = 7; case-control
studies of risk factors, n = 12; prevalence or incidence studies,
n = 8; cohort studies of risk factors, n = 3; recommendations,
n = 3; prognostic models, n = 4; time-trend analysis,
n = 1; case series, n = 1; presentations of interviews,
instruments, or assays n = 3, classification criteria n = 4).
The remaining 49 articles were eligible (Table
1)9-57 of
which 47 had appeared in major general medical journals. They included 43
randomized trials, 4 prospective cohorts, and 2 case series. In recent years
(1998 through 2003), the 3 general journals have published an almost equal
number of highly cited articles (New England Journal of
Medicine, n = 4; JAMA, n = 3; Lancet, n = 3). A smaller proportion of highly
cited articles published in specialty journals than those published in general
journals were eligible for the analysis (2/24 vs 47/91, P<.001), because highly cited articles in specialized journals were
mostly nonsystematic reviews or editorials (10/24); classification criteria
(4/24); or descriptions of standardized interviews, instruments, and assays
(3/24). Many diverse disciplines were represented, but the most common topic
was heart disease (n = 27).
Four eligible highly cited studies showed no efficacy for the tested
interventions. They contradicted prior claims for potential efficacy of vitamin
E, beta carotene, and retinol for lung cancer and/or coronary artery disease;
and showed an increased risk of coronary artery disease with hormone therapy
in postmenopausal women (Table 2).
Of the 45 eligible highly cited studies with efficacy claims (Table 2), 7 (16%) were contradicted by subsequent
research, and another 7 (16%) were found to have initially stronger effects.
In all these 14 cases (Box 1),
subsequent studies were either larger or better controlled (randomized vs
a nonrandomized original study). The findings of 20 highly cited articles
(44%) were replicated (also with a larger sample size in subsequent research
compared with the original highly cited study) and 11 (24%) had remained largely
unchallenged.58-78
Box Section Ref IDBox 1. Contradicted and Initially Stronger Effects in Highly Cited
Studies
Contradicted Findings
The Nurses’ Health Study,13 a prospective
cohort, found a 44% relative risk reduction in coronary artery disease events
in women receiving hormone therapy. A small randomized trial42 found
major beneficial effects of this intervention on surrogate markers of coronary
artery disease (lipoprotein and fibrinogen levels) claiming that this should
translate to a major clinical benefit. Although the latter trial was not refuted
at the level of surrogate outcomes, inferences for the anticipated effects
on clinical outcomes were contradicted. The Women’s Health Initiative,46 a large randomized trial, found that estrogen and
progestin significantly increased the relative risk of coronary events by
29% among postmenopausal women, and refuting results were also seen in another
large randomized trial, the Heart and Estrogen/progestin Replacement Study
(HERS).44
Two large prospective cohort studies, the Health Professionals Follow-Up
study20 and the Nurses’ Health Study,21 found that vitamin E was significantly associated
with a decreased risk of coronary artery disease and a trial of 2002 patients
also suggested a 47% relative risk reduction for cardiovascular deaths or
nonfatal myocardial infarction with vitamin E.51 However,
an even larger randomized trial66 subsequently
showed absolutely no beneficial effect for vitamin E on coronary artery disease
(relative risk 1.05 for cardiovascular deaths and 1.02 for myocardial infarction).
A small randomized trial (n = 200) suggested that the human
IgM monoclonal antibody to endotoxin could almost halve mortality due to gram-negative
sepsis.15 A subsequent randomized trial of
more than 10-fold larger sample size62 found
a nonsignificant 11% relative risk increase for mortality.
Finally, a small series of 9 patients22 proposed
that nitric oxide inhalation is very effective in patients with respiratory
distress syndrome by improving oxygenation. However 5 randomized trials involving
535 patients67 failed to show any clinical
benefit.
Initially Stronger Effects
The early results of a trial on zidovudine monotherapy in asymptomatic
patients with human immunodeficiency virus infection9 showed
a significant 60% relative risk reduction against disease progression in the
first year. The short-term benefit was not exaggerated. Yet this effect was
short-lived and the benefit was lost after 18 months both in the same trial
and also as shown in a subsequent meta-analysis.58
A randomized trial of 395 patients18 showed
that immediate angioplasty was superior to thrombolysis with tissue plasminogen
activator in acute myocardial infarction, achieving a 58% relative risk reduction
for death or reinfarction. However, a subsequent meta-analysis with more than
2500 patients65 suggested that the benefit
is probably much smaller (relative risk reduction 30%) and the largest and
most recent trial that involved both specialized and nonspecialized centers
had not shown any sizeable benefit of angioplasty (nonsignificant 20% risk
reduction for death and nonsignificant 33% risk reduction for reinfarction).
Two randomized trials of 410 and 520 patients, respectively,26,27 showed that stents were superior
to balloon angioplasty for management of coronary artery disease with 31%
and 42% relative risk reductions, respectively, in the need for revasularization.
Current evidence, as summarized by a meta-analysis of almost 10 000 patients,
suggests that the benefit is probably much smaller that originally thought
(approximately 10% relative risk reduction), and unblinding may have led to
an increased effect on repeat angioplasty in these trials.69
Another trial suggested a prime role for tissue plasminogen activator
in acute ischemic stroke.29 However, subsequent
evidence has narrowed indications and the intervention is considered effective
mostly when given very early after symptom onset.70
Carotid endarterectomy was initially reported to achieve a 5.9% absolute
risk reduction for stroke or death, projected at 5 years,43 in
patients with asymptomatic stenosis of the carotid artery exceeding 60%. A
meta-analysis of several trials suggested a more modest benefit with 2% absolute
risk reduction at 3.1 years.75
Finally, a cohort study of 805 people found a 68% adjusted relative
risk reduction for coronary artery disease with flavonoids48 while
a meta-analysis of prospective cohorts with total sample size exceeding 100 000
suggests only a 20% relative risk reduction in the top vs bottom third of
flavonoid uptake.76
Comparison of Contradicted or Initially Stronger vs Replicated or Unchallenged
Findings
Five of 6 highly cited nonrandomized studies had been contradicted or
had initially stronger effects while this was seen in only 9 of 39 highly
cited randomized trials (P = .008). Table 3 shows that trials with contradicted or
initially stronger effects had significantly smaller sample sizes and tended
to be older than those with replicated or unchallenged findings. There were
no significant differences on the type of disease. The proportion of contradicted
or initially stronger effects did not differ significantly across journals
(P = .60). There was also no significant
difference in the number of citations received in the first 3 years between
these 2 groups or in the overall number of citations over time although the
citations per year tended to be nonsignificantly fewer in trials with contradicted
or initially stronger effects.
Comparison of Highly Cited Articles Against Less-Cited Control Articles
Of the 49 articles in the control group79-127 (with
median of 157 citations, range 38-815, until 2004), the findings of 2 articles91,119 were contradicted128,129 and
8 studies82,90,92,95,96,109,110,117 had
initially stronger effects130-137 (Box 2); 20 articles79-81,83,86-89,101,103,104,106,108,111,112,118,123,125-127 contained
“positive” findings that were replicated,68,138-155 8
studies93,97,98,102,107,114,115,120 remained
unchallenged, and 11 studies84,85,94,99,100,105,113,114,119,120,122 did
not have any “positive” results; in 7 articles with some “positive”
finding,79,87,91,98,108,112,120 there
were also other interventions evaluated that had “negative” results
although this mixture of “positive” and “negative”
results had not been observed in any of the highly cited articles. The control
articles had a larger number of “negative” findings compared with
the highly cited articles (matched odds ratio [OR], 8; 95% confidence interval
[CI], 1.8-34; P = .006 for any “negative”
finding; and matched OR, 3.3; 95% CI, 0.92-12.0, P = .07
for exclusively “negative” findings). The highly cited articles
did not have a smaller proportion of contradicted or initially stronger effects
than the control articles if anything there was a trend for more contradicted
or initially stronger effects in the highly cited articles (matched OR, 1.6;
95% CI, 0.6-4.0; P = .35; matched OR, 6.0;
95% CI, 0.7-50; P = .10 when limited to
contradicted findings).
Contradicted Findings
In a prospective cohort,91 vitamin A
was inversely related to breast cancer (relative risk in the highest quintile,
0.84; 95% confidence interval [CI], 0.71-0.98) and vitamin A supplementation
was associated with a reduced risk (P = .03)
in women at the lowest quintile group; in a randomized trial128 exploring
further the retinoid-breast cancer hypothesis, fenretinide treatment of women
with breast cancer for 5 years had no effect on the incidence of second breast
malignancies.
A trial (n = 51) showed that cladribine significantly improved
the clinical scores of patients with chronic progressive multiple sclerosis.119 In a larger trial of 159 patients, no significant
treatment effects were found for cladribine in terms of changes in clinical
scores.129
Initially Stronger Effects
A trial (n = 28) of aerosolized ribavirin in infants receiving
mechanical ventilation for severe respiratory syncytial virus infection82 showed significant decreases in mechanical ventilation
(4.9 vs 9.9 days) and hospital stay (13.3 vs 15.0 days). A meta-analysis of
3 trials (n = 104) showed a decrease of only 1.8 days in the duration
of mechanical ventilation and a nonsignificant decrease of 1.9 days in duration
of hospitalization.130
A trial (n = 406) of intermittent diazepam administered during
fever to prevent recurrence of febrile seizures90 showed
a significant 44% relative risk reduction in seizures. The effect was smaller
in other trials and the overall risk reduction was no longer formally significant131; moreover, the safety profile of diazepam was deemed
unfavorable to recommend routine preventive use.
A case-control and cohort study evaluation92 showed
that the increased risk of sudden infant death syndrome among infants who
sleep prone is increased by use of natural-fiber mattresses, swaddling, and
heating in bedrooms. Several observational studies have been done since, and
they have provided inconsistent results on these interventions, in particular,
they disagree on the possible role of overheating.132
A trial of 54 children95 showed that
the steroid budenoside significantly reduced the croup score by 2 points at
4 hours, and significantly decreased readmissions by 86%. A meta-analysis
(n = 3736)133 showed a significant
improvement in the Westley score at 6 hours (1.2 points), and 12 hours (1.9
points), but not at 24 hours. Fewer return visits and/or (re)admissions occurred
in patients treated with glucocorticoids, but the relative risk reduction
was only 50% (95% CI, 24%-64%).
A trial (n = 55) showed that misprostol was as effective as
dinoprostone for termination of second-trimester pregnancy and was associated
with fewer adverse effects than dinoprostone.96 A
subsequent trial134 showed equal efficacy,
but a higher rate of adverse effects with misoprostol (74%) than with dinoprostone
(47%).
A trial (n = 50) comparing botulinum toxin vs glyceryl trinitrate
for chronic anal fissure concluded that both are effective alternatives to
surgery but botulinum toxin is the more effective nonsurgical treatment (1
failure vs 9 failures with nitroglycerin).109 In
a meta-analysis135 of 31 trials, botulinum
toxin compared with placebo showed no significant efficacy (relative risk
of failure, 0.75; 95% CI, 0.32-1.77), and was also no better than glyceryl
trinitrate (relative risk of failure, 0.48; 95% CI, 0.21-1.10); surgery was
more effective than medical therapy in curing fissure (relative risk of failure,
0.12; 95% CI, 0.07-0.22).
A trial of acetylcysteine (n = 83) showed that it was highly
effective in preventing contrast nephropathy (90% relative risk reduction).110 There have been many more trials and many meta-analyses
on this topic. The latest meta-analysis136 shows
a nonsignificant 27% relative risk reduction with acetylcysteine.
A trial of 129 stunted Jamaican children found that both nutritional
supplementation and psychosocial stimulation improved the mental development
of stunted children; children who got both interventions had additive benefits
and achieved scores close to those of nonstunted children.117 With
long-term follow-up, however, it was found that the benefits were small and
the 2 interventions no longer had additive effects.137
Original highly cited articles about medical interventions are published
almost exclusively in 3 general medical journals. Actually, there has been
an approximate equal share of very highly cited articles among these 3 journals
since 1998 as impact factor differences have diminished among these 3 journals.
Articles in specialty journals that reach such high numbers of citations are
usually review articles or articles describing tools useful to specific diseases
rather than original data. Contradicted and potentially exaggerated findings
are not uncommon in the most visible and most influential original clinical
research: 16% of the top-cited clinical research articles on postulated effective
medical interventions that have been published within the last 15 years have
been contradicted by subsequent clinical studies and another 16% have been
found to have initially stronger effects than subsequent research. Contradiction
or initially stronger effects have been encountered in 5 of 6 cases for which
nonrandomized designs were used, but even randomized trials have not escaped
controversy. More than a third of the top-cited randomized trials published
from 1990 through 1995 have already been affected, while for more recent trials,
the time frame is still early and more may be contradicted in the future.
Sample size seems to be important, with smaller sample sizes in trials that
have met controversy vs those that have not.
The classification of studies in this analysis involves many judgments
pertaining to the complexity of studying a given research question with somewhat
different populations, interventions, durations, and outcomes. However, these
studies are widely known for their inferences and this is also proven by the
high interrater agreement. Nevertheless, it should also be acknowledged that
although the classification was performed in duplicate, the searches were
performed by only 1 investigator. It is unavoidable that some other investigators
may feel differently about the categorization of specific studies, especially
for topics that may also have heavy debates surrounding them. However, this
is unlikely to change the aggregate picture about refutation rates.
The examination of contradictions and refutations offers a fascinating
look at the process of science. Four of the highly cited articles examined
herein were refuting investigations with “negative” results. However,
in a sense, even the other highly cited articles with “positive”
results refuted prior knowledge and practice by introducing new concepts and
proposing new interventions. We should acknowledge that there is no proof
that the subsequent studies and meta-analyses were necessarily correct. A
perfect gold standard is not possible in clinical research, so we can only
interpret results of studies relative to other studies. Whenever new research
fails to replicate early claims for efficacy or suggests that efficacy is
more limited than previously thought, it is not necessary that the original
studies were totally wrong and the newer ones are correct simply because they
are larger or better controlled. Alternative explanations for these discrepancies
may include differences in disease spectrum, eligibility criteria, or the
use of concomitant interventions.156 Different
studies on the same question are typically not replicas of each other. In
fact discrepancies may be interesting on their own because they require careful
scrutiny of the data and reappraisal of our beliefs. Thus, it is probably
not surprising that the citation rate of these refuted studies did not seem
to be much affected. Nevertheless, the controversy generates considerable
uncertainty for clinical practice and none of the contradicted interventions
is currently recommended by practice guidelines.
The mere fact that a study is highly cited suggests that there is a
strong active interest in the questions addressed from a clinical or research
perspective. This may increase the chances that other, larger trials may eventually
be conducted. However, for most clinical questions of interest, no large trials
are ever conducted and evidence is based only on small trials or nonrandomized
studies.157 Small trials or meta-analyses thereof
may often be refuted subsequently by large trials1,2 when
such large trials are performed. Small studies using surrogate markers may
also sometimes lead to erroneous clinical inferences.158 There
were only 2 studies with typical surrogate markers among the highly cited
studies examined herein, but both were subsequently contradicted in their
clinical extrapolations about the efficacy of nitric oxide22 and
hormone therapy.42 In the case of initially
stronger effects, the differences in the effect sizes could often be within
the range of what would be expected based on chance variability. This reinforces
the notion that results from clinical studies, especially early ones, should
be interpreted using not only the point estimates but also the uncertainty
surrounding them. However, besides differences in effect sizes, most initially
stronger effects pertained also to issues of durability, generalizability,
or applicability of the proposed effects, as discussed above. Thus, clinicians
should be aware that these important aspects may not be fully settled when
an important treatment breakthrough is announced.
A third of the most-cited clinical research seems to have replication
problems, and this seems to be as large, if not larger, than the vast majority
of other, less-cited clinical research. The current analysis found that matched
studies that were not so highly cited had a greater proportion of “negative”
findings and similar or smaller proportions of contradicted results as the
highly cited ones. Publication bias159,160 and
time-lag bias161,162 favoring
the rapid and prominent publication of “positive” findings may
underlie some of the observed phenomena. Highly cited articles are already
a selected sample with underrepresentation of “negative” findings
compared with the average article on interventions published in major journals.
It is possible that high-profile journals may tend to publish occasionally
very striking findings and that this may lead to some difficulty in replicating
some of these findings.163 Poynard et al164 evaluated the conclusions of hepatology-related
articles published between 1945 and 1999 and found that, overall, 60% of these
conclusions were considered to be true in 2000 and that there was no difference
between randomized and nonrandomized studies or high- vs low-quality studies.
Allowing for somewhat different definitions, the higher rates of refutation
and the generally worse performance of nonrandomized studies in the present
analysis may stem from the fact that I focused on a selected sample of the
most noticed and influential clinical research. For such highly cited studies,
the turnaround of “truth” may be faster; in particular nonrandomized
studies may be more likely to be probed and challenged than nonrandomized
studies published in the general literature.
Finally, a certain proportion of highly cited trials may remain unchallenged.
Sometimes the evidence from the original study may seem so overwhelming that
further similar studies are deemed unethical to perform. The original study
may be widely considered as a milestone for clinical practice and may provide
the gold standard for testing new interventions. However, sometimes other,
validating research may be in the works. Clinical research is time-consuming
and challenging results may take several years to generate and publish. Therefore
evidence from recent trials, no matter how impressive, should be interpreted
with caution, when only one trial is available. It is important to know whether
other similar or larger trials are still ongoing or being planned. Therefore,
transparent and thorough trial registration is of paramount importance165 in order to limit premature claims for efficacy.
Corresponding Author: John P. A. Ioannidis,
MD, Department of Hygiene and Epidemiology, University of Ioannina School
of Medicine, Ioannina 45110, Greece (jioannid@cc.uoi.gr).
Author Contributions: Dr Ioannidis had full
access to all of the data in the study and takes responsibility for the integrity
of the data and the accuracy of the data analysis.
Financial Disclosures: None reported.
Acknowledgment: I thank Dr Tom Trikalinos for
classifying independently the status of the highly cited articles.
1.LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized,
controlled trials.
N Engl J Med. 1997;337:536-5429262498
Google ScholarCrossref 2.Cappelleri JC, Ioannidis JP, Schmid CH.
et al. Large trials vs meta-analysis of smaller trials: how do their results
compare?
JAMA. 1996;276:1332-13388861993
Google ScholarCrossref 3.Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials.
N Engl J Med. 2000;342:1878-188610861324
Google ScholarCrossref 4.Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy
of research designs.
N Engl J Med. 2000;342:1887-189210861325
Google ScholarCrossref 5.Ioannidis JP, Haidich AB, Pappa M.
et al. Comparison of evidence of treatment effects in randomized and nonrandomized
studies.
JAMA. 2001;286:821-83011497536
Google ScholarCrossref 7.Dickersin K, Scherer R, Suci ES, Gil-Montero M. Problems with indexing and citation of articles with group authorship.
JAMA. 2002;287:2772-277412038908
Google ScholarCrossref 9.Volberding PA, Lagakos SW, Koch MA.
et al. the AIDS Clinical Trials Group of the National Institute of Allergy
and Infectious Diseases. Zidovudine in asymptomatic human immunodeficiency virus infection:
a controlled trial in persons with fewer than 500 CD4-positive cells per cubic
millimeter.
N Engl J Med. 1990;322:941-9491969115
Google ScholarCrossref 10.Brown G, Albers JJ, Fisher LD.
et al. Regression of coronary artery disease as a result of intensive lipid-lowering
therapy in men with high levels of apolipoprotein B.
N Engl J Med. 1990;323:1289-12982215615
Google ScholarCrossref 11.Moertel CG, Fleming TR, Macdonald JS.
et al. Levamisole and fluorouracil for adjuvant therapy of resected colon
carcinoma.
N Engl J Med. 1990;322:352-3582300087
Google ScholarCrossref 12.Cohn JN, Johnson G, Ziesche S.
et al. A comparison of enalapril with hydralazine-isosorbide dinitrate in
the treatment of chronic congestive heart failure.
N Engl J Med. 1991;325:303-3102057035
Google ScholarCrossref 13.Stampfer MJ, Colditz GA, Willett WC.
et al. Postmenopausal estrogen therapy and cardiovascular disease: ten-year
follow-up from the Nurses' Health Study.
N Engl J Med. 1991;325:756-7621870648
Google ScholarCrossref 14.North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients
with high-grade carotid stenosis.
N Engl J Med. 1991;325:445-4531852179
Google ScholarCrossref 15.Ziegler EJ, Fisher CJ Jr, Sprung CL.
et al. the HA-1A Sepsis Study Group. Treatment of gram-negative bacteremia and septic shock with HA-1A human
monoclonal antibody against endotoxin: a randomized, double-blind, placebo-controlled
trial.
N Engl J Med. 1991;324:429-4361988827
Google ScholarCrossref 16.The SOLVD Investigators. Effect of enalapril on survival in patients with reduced left ventricular
ejection fractions and congestive heart failure.
N Engl J Med. 1991;325:293-3022057034
Google ScholarCrossref 17.Pfeffer MA, Braunwald E, Moye LA.
et al. the SAVE Investigators. Effect of captopril on mortality and morbidity in patients with left
ventricular dysfunction after myocardial infarction: results of the survival
and ventricular enlargement trial.
N Engl J Med. 1992;327:669-6771386652
Google ScholarCrossref 18.Grines CL, Browne KF, Marco J.
et al. the Primary Angioplasty in Myocardial Infarction Study Group. A comparison of immediate angioplasty with thrombolytic therapy for
acute myocardial infarction.
N Engl J Med. 1993;328:673-6798433725
Google ScholarCrossref 19.Lewis EJ, Hunsicker LG, Bain RP, Rohde RD.the Collaborative Study Group. The effect of angiotensin-converting-enzyme inhibition on diabetic
nephropathy.
N Engl J Med. 1993;329:1456-14628413456
Google ScholarCrossref 20.Rimm EB, Stampfer MJ, Ascherio A, Giovannucci E, Colditz GA, Willett WC. Vitamin E consumption and the risk of coronary heart disease in men.
N Engl J Med. 1993;328:1450-14568479464
Google ScholarCrossref 21.Stampfer MJ, Hennekens CH, Manson JE, Colditz GA, Rosner B, Willett WC. Vitamin E consumption and the risk of coronary disease in women.
N Engl J Med. 1993;328:1444-14498479463
Google ScholarCrossref 22.Rossaint R, Falke KJ, Lopez F, Slama K, Pison U, Zapol WM. Inhaled nitric oxide for the adult respiratory distress syndrome.
N Engl J Med. 1993;328:399-4058357359
Google ScholarCrossref 23.The Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and
progression of long-term complications in insulin-dependent diabetes mellitus.
N Engl J Med. 1993;329:977-9868366922
Google ScholarCrossref 24.The EPIC Investigation. Use of a monoclonal antibody directed against the platelet glycoprotein
IIb/IIIa receptor in high-risk coronary angioplasty.
N Engl J Med. 1994;330:956-9618121459
Google ScholarCrossref 25.Connor EM, Sperling RS, Gelber R.
et al. Pediatric AIDS Clinical Trials Group Protocol 076 Study Group. Reduction of maternal-infant transmission of human immunodeficiency
virus type 1 with zidovudine treatment.
N Engl J Med. 1994;331:1173-11807935654
Google ScholarCrossref 26.Fischman DL, Leon MB, Baim DS.
et al. Stent Restenosis Study Investigators. A randomized comparison of coronary-stent placement and balloon angioplasty
in the treatment of coronary artery disease.
N Engl J Med. 1994;331:496-5018041414
Google ScholarCrossref 27.Serruys PW, de Jaegere P, Kiemeneij F.
et al. BENESTENT Study Group. A comparison of balloon-expandable-stent implantation with balloon
angioplasty in patients with coronary artery disease.
N Engl J Med. 1994;331:489-4958041413
Google ScholarCrossref 28.The Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group. The effect of vitamin E and beta carotene on the incidence of lung
cancer and other cancers in male smokers.
N Engl J Med. 1994;330:1029-10358127329
Google ScholarCrossref 29.The National Institute of Neurological Disorders and Stroke rt-PA Stroke
Study Group. Tissue plasminogen activator for acute ischemic stroke.
N Engl J Med. 1995;333:1581-15877477192
Google ScholarCrossref 30.Shepherd J, Cobbe SM, Ford I.
et al. West of Scotland Coronary Prevention Study Group. Prevention of coronary heart disease with pravastatin in men with hypercholesterolemia.
N Engl J Med. 1995;333:1301-13077566020
Google ScholarCrossref 31.Sacks FM, Pfeffer MA, Moye LA.
et al. Cholesterol and Recurrent Events Trial Investigators. The effect of pravastatin on coronary events after myocardial infarction
in patients with average cholesterol levels.
N Engl J Med. 1996;335:1001-10098801446
Google ScholarCrossref 32.Packer M, Bristow MR, Cohn JN.
et al. US Carvedilol Heart Failure Study Group. The effect of carvedilol on morbidity and mortality in patients with
chronic heart failure.
N Engl J Med. 1996;334:1349-13558614419
Google ScholarCrossref 33.Omenn GS, Goodman GE, Thornquist MD.
et al. Effects of a combination of beta carotene and vitamin A on lung cancer
and cardiovascular disease.
N Engl J Med. 1996;334:1150-11558602180
Google ScholarCrossref 34.Ridker PM, Cushman M, Stampfer MJ, Tracy RP, Hennekens CH. Inflammation, aspirin, and the risk of cardiovascular disease in apparently
healthy men.
N Engl J Med. 1997;336:973-9799077376
Google ScholarCrossref 35.Hammer SM, Squires KE, Hughes MD.
et al. AIDS Clinical Trials Group 320 Study Team. A controlled trial of two nucleoside analogues plus indinavir in persons
with human immunodeficiency virus infection and CD4 cell counts of 200 per
cubic millimeter or less.
N Engl J Med. 1997;337:725-7339287227
Google ScholarCrossref 36.The EPILOG Investigators. Platelet glycoprotein IIb/IIIa receptor blockade and low-dose heparin
during percutaneous coronary revascularization.
N Engl J Med. 1997;336:1689-16969182212
Google ScholarCrossref 37.McHutchison JG, Gordon SC, Schiff ER.
et al. Hepatitis Interventional Therapy Group. Interferon alfa-2b alone or in combination with ribavirin as initial
treatment for chronic hepatitis C.
N Engl J Med. 1998;339:1485-14929819446
Google ScholarCrossref 38.The Long-Term Intervention with Pravastatin in Ischaemic Disease (LIPID)
Study Group. Prevention of cardiovascular events and death with pravastatin in patients
with coronary heart disease and a broad range of initial cholesterol levels.
N Engl J Med. 1998;339:1349-13579841303
Google ScholarCrossref 39.Pitt B, Zannad F, Remme WJ.
et al. Randomized Aldactone Evaluation Study Investigators. The effect of spironolactone on morbidity and mortality in patients
with severe heart failure.
N Engl J Med. 1999;341:709-71710471456
Google ScholarCrossref 40.Yusuf S, Sleight P, Pogue J, Bosch J, Davies R, Dagenais G.the Heart Outcomes Prevention Evaluation Study Investigators. Effects of an angiotensin-converting-enzyme inhibitor, ramipril, on
cardiovascular events in high-risk patients.
N Engl J Med. 2000;342:145-15310639539
Google ScholarCrossref 41.SHEP Cooperative Research Group. Prevention of stroke by antihypertensive drug treatment in older persons
with isolated systolic hypertension: final results of the Systolic Hypertension
in the Elderly Program (SHEP).
JAMA. 1991;265:3255-32642046107
Google ScholarCrossref 42.The Writing Group for the PEPI Trial. Effects of estrogen or estrogen/progestin regimens on heart disease
risk factors in postmenopausal women.
JAMA. 1995;273:199-2087807658
Google ScholarCrossref 43.Executive Committee for the Asymptomatic Carotid Atherosclerosis Study. Endarterectomy for asymptomatic carotid artery stenosis.
JAMA. 1995;273:1421-14287723155
Google ScholarCrossref 44.Hulley S, Grady D, Bush T.
et al. Heart and Estrogen/progestin Replacement Study (HERS) Research Group. Randomized trial of estrogen plus progestin for secondary prevention
of coronary heart disease in postmenopausal women.
JAMA. 1998;280:605-6139718051
Google ScholarCrossref 45.Downs JR, Clearfield M, Weis S.
et al. Primary prevention of acute coronary events with lovastatin in men
and women with average cholesterol levels: results of AFCAPS/TexCAPS.
JAMA. 1998;279:1615-16229613910
Google ScholarCrossref 46.Rossouw JE, Anderson GL, Prentice RL.
et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal
women: principal results from the Women's Health Initiative randomized controlled
trial.
JAMA. 2002;288:321-33312117397
Google ScholarCrossref 47.MRC Vitamin Study Research Group. Prevention of neural tube defects: results of the Medical Research
Council Vitamin Study.
Lancet. 1991;338:131-1371677062
Google ScholarCrossref 48.Hertog MG, Feskens EJ, Hollman PC, Katan MB, Kromhout D. Dietary antioxidant flavonoids and risk of coronary heart disease:
the Zutphen Elderly Study.
Lancet. 1993;342:1007-10118105262
Google ScholarCrossref 49. Randomised trial of cholesterol lowering in 4444 patients with coronary
heart disease: the Scandinavian Simvastatin Survival Study (4S).
Lancet. 1994;344:1383-13897968073
Google Scholar 50.CAPRIE Steering Committee. A randomised, blinded, trial of clopidogrel vs aspirin in patients
at risk of ischaemic events (CAPRIE).
Lancet. 1996;348:1329-13398918275
Google ScholarCrossref 51.Stephens NG, Parsons A, Schofield PM, Kelly F, Cheeseman K, Mitchinson MJ. Randomised controlled trial of vitamin E in patients with coronary
disease: Cambridge Heart Antioxidant Study (CHAOS).
Lancet. 1996;347:781-7868622332
Google ScholarCrossref 52.Hansson L, Zanchetti A, Carruthers SG.
et al. HOT Study Group. Effects of intensive blood-pressure lowering and low-dose aspirin in
patients with hypertension: principal results of the Hypertension Optimal
Treatment (HOT) randomised trial.
Lancet. 1998;351:1755-17629635947
Google ScholarCrossref 53.Poynard T, Marcellin P, Lee SS.
et al. International Hepatitis Interventional Therapy Group (IHIT). Randomised trial of interferon alpha-2b plus ribavirin for 48 weeks
or for 24 weeks vs interferon alpha-2b plus placebo for 48 weeks for treatment
of chronic infection with hepatitis C virus.
Lancet. 1998;352:1426-14329807989
Google ScholarCrossref 54.UK Prospective Diabetes Study (UKPDS) Group. Effect of intensive blood-glucose control with metformin on complications
in overweight patients with type 2 diabetes (UKPDS 34).
Lancet. 1998;352:854-8659742977
Google ScholarCrossref 55. The Cardiac Insufficiency Bisoprolol Study II (CIBIS-II): a randomised
trial.
Lancet. 1999;353:9-1310023943
Google ScholarCrossref 56.Castaigne S, Cjomienne C, Daniel MT.
et al. All-trans-retinoic acid as a differentiation therapy for acute promyelocytic
leukemia, 1: clinical results.
Blood. 1990;76:1704-17092224119
Google Scholar 57.Fisher B, Costantino JP, Wickerham DL.
et al. Tamoxifen for prevention of breast cancer: report of the National Surgical
Adjuvant Breast and Bowel Project P-1 Study.
J Natl Cancer Inst. 1998;90:1371-13889747868
Google ScholarCrossref 58.Ioannidis JP, Cappelleri JC, Lau J.
et al. Early or deferred zidovudine therapy in HIV-infected patients without
an AIDS-defining illness.
Ann Intern Med. 1995;122:856-8667741372
Google ScholarCrossref 59.Law MR, Wald NJ, Rudnicka AR. Quantifying effect of statins on low density lipoprotein cholesterol,
ischaemic heart disease, and stroke: systematic review and meta-analysis.
BMJ. 2003;326:142312829554
Google ScholarCrossref 60.Gill S, Loprinzi CL, Sargent DJ.
et al. Pooled analysis of fluorouracil-based adjuvant therapy for stage II
and III colon cancer: who benefits and by how much?
J Clin Oncol. 2004;22:1797-180615067028
Google ScholarCrossref 61.Rothwell PM, Eliasziw M, Gutnikov SA.
et al. Analysis of pooled data from the randomised controlled trials of endarterectomy
for symptomatic carotid stenosis.
Lancet. 2003;361:107-11612531577
Google ScholarCrossref 62.McCloskey RV, Straube RC, Sanders C, Smith SM, Smith CR.CHESS Trial Study Group. Treatment of septic shock with human monoclonal antibody HA-1A: a randomized,
double-blind, placebo-controlled trial.
Ann Intern Med. 1994;121:1-58198341
Google ScholarCrossref 63.Garg R, Yusuf S.Collaborative Group on ACE Inhibitor Trials. Overview of randomized trials of angiotensin-converting enzyme inhibitors
on mortality and morbidity in patients with heart failure.
JAMA. 1995;273:1450-14567654275
Google ScholarCrossref 64.Rodrigues EJ, Eisenberg MJ, Pilote L. Effects of early and late administration of angiotensin-converting
enzyme inhibitors on mortality after myocardial infarction.
Am J Med. 2003;115:473-47914563504
Google ScholarCrossref 65.Cucherat M, Bonnefoy E, Tremeau G. Primary angioplasty vs intravenous thrombolysis for acute myocardial
infarction.
Cochrane Database Syst Rev. 2003;3:CD00156012917910
Google Scholar 66.Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P.The Heart Outcomes Prevention Evaluation Study Investigators. Vitamin E supplementation and cardiovascular events in high-risk patients.
N Engl J Med. 2000;342:154-16010639540
Google ScholarCrossref 67.Sokol J, Jacobs SE, Bohn D. Inhaled nitric oxide for acute hypoxemic respiratory failure in children
and adults.
Cochrane Database Syst Rev. 2003;1:CD00278712535438
Google Scholar 68.Karvouni E, Katritsis DG, Ioannidis JP. Intravenous glycoprotein IIb/IIIa receptor antagonists reduce mortality
after percutaneous coronary interventions.
J Am Coll Cardiol. 2003;41:26-3212570940
Google ScholarCrossref 69.Brophy JM, Belisle P, Joseph L. Evidence for use of coronary stents: a hierarchical bayesian meta-analysis.
Ann Intern Med. 2003;138:777-78612755549
Google ScholarCrossref 70.Hacke W, Donnan G, Fieschi C.
et al. Association of outcome with early stroke treatment: pooled analysis
of ATLANTIS, ECASS, and NINDS rt-PA stroke trials.
Lancet. 2004;363:768-77415016487
Google ScholarCrossref 71.Brophy JM, Joseph L, Rouleau JL. Beta-blockers in congestive heart failure: a Bayesian meta-analysis.
Ann Intern Med. 2001;134:550-56011281737
Google ScholarCrossref 72.Yazdanpanah Y, Sissoko D, Egger M, Mouton Y, Zwahlen M, Chene G. Clinical efficacy of antiretroviral combination therapy based on protease
inhibitors or non-nucleoside analogue reverse transcriptase inhibitors: indirect
comparison of controlled trials.
BMJ. 2004;328:24914742351
Google ScholarCrossref 73.Kjaergard LL, Krogsgaard K, Gluud C. Interferon alfa with or without ribavirin for chronic hepatitis C:
systematic review of randomised trials.
BMJ. 2001;323:1151-115511711405
Google ScholarCrossref 74.Staessen JA, Gasowski J, Wang JG.
et al. Risks of untreated and treated isolated systolic hypertension in the
elderly: meta-analysis of outcome trials.
Lancet. 2000;355:865-87210752701
Google ScholarCrossref 75.Benavente O, Moher D, Pham B. Carotid endarterectomy for asymptomatic carotid stenosis: a meta-analysis.
BMJ. 1998;317:1477-14809831572
Google ScholarCrossref 76.Huxley RR, Neil HA. The relation between dietary flavonol intake and coronary heart disease
mortality: a meta-analysis of prospective cohort studies.
Eur J Clin Nutr. 2003;57:904-90812879084
Google ScholarCrossref 77.Tallman MS, Andersen JW, Schiffer CA.
et al. All-trans-retinoic acid in acute promyelocytic leukemia.
N Engl J Med. 1997;337:1021-10289321529
Google ScholarCrossref 78.Cuzick J, Powles T, Veronesi U.
et al. Overview of the main outcomes in breast-cancer prevention trials.
Lancet. 2003;361:296-30012559863
Google ScholarCrossref 79.Watts NB, Harris ST, Genant HK.
et al. Intermittent cyclical etidronate treatment of postmenopausal osteoporosis.
N Engl J Med. 1990;323:73-792113611
Google ScholarCrossref 80.Smith P, Arnesen H, Holme I. The effect of warfarin on mortality and reinfarction after myocardial
infarction.
N Engl J Med. 1990;323:147-1522194126
Google ScholarCrossref 81.Hussey GD, Klein M. A randomized, controlled trial of vitamin A in children with severe
measles.
N Engl J Med. 1990;323:160-1642194128
Google ScholarCrossref 82.Smith DW, Frankel LR, Mathers LH, Tang AT, Ariagno RL, Prober CG. A controlled trial of aerosolized ribavirin in infants receiving mechanical
ventilation for severe respiratory syncytial virus infection.
N Engl J Med. 1991;325:24-291904551
Google ScholarCrossref 83.The National Institute of Child Health and Human Developments Intravenous
Immunoglobulin Study Group. Intravenous immune globulin for the prevention of bacterial infections
in children with symptomatic human immunodeficiency virus infection.
N Engl J Med. 1991;325:73-801675763
Google ScholarCrossref 84.Green DM, Zevon MA, Lowrie G, Seigelstein N, Hall B. Congenital anomalies in children of patients who received chemotherapy
for cancer in childhood and adolescence.
N Engl J Med. 1991;325:141-1462052058
Google ScholarCrossref 85.Bradley JD, Brandt KD, Katz BP, Kalasinski LA, Ryan SI. Comparison of an antiinflammatory dose of ibuprofen, an analgesic dose
of ibuprofen, and acetaminophen in the treatment of patients with osteoarthritis
of the knee.
N Engl J Med. 1991;325:87-912052056
Google ScholarCrossref 86.Crawford J, Ozer H, Stoller R.
et al. Reduction by granulocyte colony-stimulating factor of fever and neutropenia
induced by chemotherapy in patients with small-cell lung cancer.
N Engl J Med. 1991;325:164-1701711156
Google ScholarCrossref 87.Maggioni AP, Franzosi MG, Santoro E, White H, Van de Werf F, Tognoni G.Gruppo Italiano per lo Studio della Sopravvivenza nell'Infarto Miocardico
II (GISSI-2) and The International Study Group. The risk of stroke in patients with acute myocardial infarction after
thrombolytic and antithrombotic treatment.
N Engl J Med. 1992;327:1-61598096
Google ScholarCrossref 88.Packer M, Gheorghiade M, Young JB.
et al. RADIANCE Study. Withdrawal of digoxin from patients with chronic heart failure treated
with angiotensin-converting enzyme inhibitors.
N Engl J Med. 1993;329:1-78505940
Google ScholarCrossref 89.Jenkins DJ, Wolever TM, Rao AV.
et al. Effect on blood lipids of very high intakes of fiber in diets low in
saturated fat and cholesterol.
N Engl J Med. 1993;329:21-268389421
Google ScholarCrossref 90.Rosman NP, Colton T, Labazzo J.
et al. A controlled trial of diazepam administered during febrile illnesses
to prevent recurrence of febrile seizures.
N Engl J Med. 1993;329:79-848510706
Google ScholarCrossref 91.Hunter DJ, Manson JE, Colditz GA.
et al. A prospective study of the intake of vitamins C, E, and A and the risk
of breast cancer.
N Engl J Med. 1993;329:234-2408292129
Google ScholarCrossref 92.Ponsonby AL, Dwyer T, Gibbons LE, Cochrane JA, Wang YG. Factors potentiating the risk of sudden infant death syndrome associated
with the prone position.
N Engl J Med. 1993;329:377-3828326970
Google ScholarCrossref 93.Winterkorn JM, Kupersmith MJ, Wirtschafter JD, Forman S. Treatment of vasospastic amaurosis fugax with calcium-channel blockers.
N Engl J Med. 1993;329:396-3988326973
Google ScholarCrossref 94.Greenberg ER, Baron JA, Tosteson TD.
et al. Polyp Prevention Study Group. A clinical trial of antioxidant vitamins to prevent colorectal adenoma.
N Engl J Med. 1994;331:141-1478008027
Google ScholarCrossref 95.Klassen TP, Feldman ME, Watters LK, Sutcliffe T, Rowe PC. Nebulized budesonide for children with mild-to-moderate croup.
N Engl J Med. 1994;331:285-2898022437
Google ScholarCrossref 96.Jain JK, Mishell DR Jr. A comparison of intravaginal misoprostol with prostaglandin E2 for
termination of second-trimester pregnancy.
N Engl J Med. 1994;331:290-2938022438
Google ScholarCrossref 97.Loprinzi CL, Michalak JC, Quella SK.
et al. Megestrol acetate for the prevention of hot flashes.
N Engl J Med. 1994;331:347-3528028614
Google ScholarCrossref 98.O'Connell MJ, Martenson JA, Wieand HS.
et al. Improving adjuvant therapy for rectal cancer by combining protracted-infusion
fluorouracil with radiation therapy after curative surgery.
N Engl J Med. 1994;331:502-5078041415
Google ScholarCrossref 99.Suarez FL, Savaiano DA, Levitt MD. A comparison of symptoms after the consumption of milk or lactose-hydrolyzed
milk by people with self-reported severe lactose intolerance.
N Engl J Med. 1995;333:1-47776987
Google ScholarCrossref 100.Singh SN, Fletcher RD, Fisher SG.
et al. Survival Trial of Antiarrhythmic Therapy in Congestive Heart Failure. Amiodarone in patients with congestive heart failure and asymptomatic
ventricular arrhythmia.
N Engl J Med. 1995;333:77-827539890
Google ScholarCrossref 101.Bhasin S, Storer TW, Berman N.
et al. The effects of supraphysiologic doses of testosterone on muscle size
and strength in normal men.
N Engl J Med. 1996;335:1-78637535
Google ScholarCrossref 102.Stegeman CA, Tervaert JW, de Jong PE, Kallenberg CG.Dutch Co-Trimoxazole Wegener Study Group. Trimethoprim-sulfamethoxazole (co-trimoxazole) for the prevention of
relapses of Wegener's granulomatosis.
N Engl J Med. 1996;335:16-208637536
Google ScholarCrossref 103.van Hensbroek MB, Onyiorah E, Jaffar S.
et al. A trial of artemether or quinine in children with cerebral malaria.
N Engl J Med. 1996;335:69-758649492
Google ScholarCrossref 104.van der Horst CM, Saag MS, Cloud GA.
et al. National Institute of Allergy and Infectious Diseases Mycoses Study
Group and AIDS Clinical Trials Group. Treatment of cryptococcal meningitis associated with the acquired immunodeficiency
syndrome.
N Engl J Med. 1997;337:15-219203426
Google ScholarCrossref 105.Levine RJ, Hauth JC, Curet LB.
et al. Trial of calcium to prevent preeclampsia.
N Engl J Med. 1997;337:69-769211675
Google ScholarCrossref 106.Musch DC, Martin DF, Gordon JF, Davis MD, Kuppermann BD.the Ganciclovir Implant Study Group. Treatment of cytomegalovirus retinitis with a sustained-release ganciclovir
implant.
N Engl J Med. 1997;337:83-909211677
Google ScholarCrossref 107.Adams RJ, McKie VC, Hsu L.
et al. Prevention of a first stroke by transfusions in children with sickle
cell anemia and abnormal results on transcranial Doppler ultrasonography.
N Engl J Med. 1998;339:5-119647873
Google ScholarCrossref 108.Stefanick ML, Mackey S, Sheehan M, Ellsworth N, Haskell WL, Wood PD. Effects of diet and exercise in men and postmenopausal women with low
levels of HDL cholesterol and high levels of LDL cholesterol.
N Engl J Med. 1998;339:12-209647874
Google ScholarCrossref 109.Brisinda G, Maria G, Bentivoglio AR, Cassetta E, Gui D, Albanese A. A comparison of injections of botulinum toxin and topical nitroglycerin
ointment for the treatment of chronic anal fissure.
N Engl J Med. 1999;341:65-6910395629
Google ScholarCrossref 110.Tepel M, van der Giet M, Schwarzfeld C, Laufer U, Liermann D, Zidek W. Prevention of radiographic-contrast-agent-induced reductions in renal
function by acetylcysteine.
N Engl J Med. 2000;343:180-18410900277
Google ScholarCrossref 111.Levine AM, Wernz JC, Kaplan L.
et al. Low-dose chemotherapy with central nervous system prophylaxis and zidovudine
maintenance in AIDS-related lymphoma: a prospective multi-institutional trial.
JAMA. 1991;266:84-881710673
Google ScholarCrossref 112.Freeman EW, Rickels K, Sondheimer SJ, Polansky M. A double-blind trial of oral progesterone, alprazolam, and placebo
in treatment of severe premenstrual syndrome.
JAMA. 1995;274:51-577791258
Google ScholarCrossref 113.Williams-Russo P, Sharrock NE, Mattis S, Szatrowski TP, Charlson ME. Cognitive effects after epidural vs general anesthesia in older adults:
a randomized trial.
JAMA. 1995;274:44-507791257
Google ScholarCrossref 114.Meduri GU, Headley AS, Golden E.
et al. Effect of prolonged methylprednisolone therapy in unresolving acute
respiratory distress syndrome: a randomized controlled trial.
JAMA. 1998;280:159-1659669790
Google ScholarCrossref 115.Gulick RM, Mellors JW, Havlir D.
et al. Simultaneous vs sequential initiation of therapy with indinavir, zidovudine,
and lamivudine for HIV-1 infection: 100-week follow-up.
JAMA. 1998;280:35-419660361
Google ScholarCrossref 116.Grady D, Herrington D, Bittner V.
et al. Cardiovascular disease outcomes during 6.8 years of hormone therapy:
Heart and Estrogen/progestin Replacement Study follow-up (HERS II).
JAMA. 2002;288:49-5712090862
Google ScholarCrossref 117.Grantham-McGregor SM, Powell CA, Walker SP, Himes JH. Nutritional supplementation, psychosocial stimulation, and mental development
of stunted children: the Jamaican Study.
Lancet. 1991;338:1-51676083
Google ScholarCrossref 118.Ghana VAST Study Team. Vitamin A supplementation in northern Ghana: effects on clinic attendances,
hospital admissions, and child mortality.
Lancet. 1993;342:7-128100345
Google ScholarCrossref 119.Sipe JC, Romine JS, Koziol JA, McMillan R, Zyroff J, Beutler E. Cladribine in treatment of chronic progressive multiple sclerosis.
Lancet. 1994;344:9-137912347
Google ScholarCrossref 120.Karonga Prevention Trial Group. Randomised controlled trial of single BCG, repeated BCG, or combined
BCG and killed
Mycobacteriumleprae vaccine for prevention of leprosy and tuberculosis in Malawi.
Lancet. 1996;348:17-248691924
Google ScholarCrossref 121.Waldo AL, Camm AJ, deRuyter H.
et al. the SWORD Investigators. Effect of d-sotalol on mortality in patients with left ventricular
dysfunction after recent and remote myocardial infarction.
Lancet. 1996;348:7-128691967
Google ScholarCrossref 122.Veronesi U, Maisonneuve P, Costa A.
et al. Italian Tamoxifen Prevention Study. Prevention of breast cancer with tamoxifen: preliminary findings from
the Italian randomised trial among hysterectomised women.
Lancet. 1998;352:93-979672273
Google Scholar 123.The EPISTENT Investigators. Randomised placebo-controlled and balloon-angioplasty-controlled trial
to assess safety of coronary stenting with use of platelet glycoprotein-IIb/IIIa
blockade.
Lancet. 1998;352:87-929672272
Google Scholar 124.CLIP Group (Cancer of the Liver Italian Programme). Tamoxifen in treatment of hepatocellular carcinoma: a randomised controlled
trial.
Lancet. 1998;352:17-209800740
Google ScholarCrossref 125.Torp-Pedersen C, Kober L.TRACE Study Group. Effect of ACE inhibitor trandolapril on life expectancy of patients
with reduced left-ventricular function after acute myocardial infarction.
Lancet. 1999;354:9-1210406358
Google ScholarCrossref 126.Negrin RS, Haeuber DH, Nagler A.
et al. Maintenance treatment of patients with myelodysplastic syndromes using
recombinant human granulocyte colony-stimulating factor.
Blood. 1990;76:36-431694702
Google Scholar 127.Budman DR, Berry DA, Cirrincione CT.
et al. the Cancer and Leukemia Group B. Dose and dose intensity as determinants of outcome in the adjuvant
treatment of breast cancer.
J Natl Cancer Inst. 1998;90:1205-12119719081
Google ScholarCrossref 128.Veronesi U, De Palo G, Marubini E.
et al. Randomized trial of fenretinide to prevent second breast malignancy
in women with early breast cancer.
J Natl Cancer Inst. 1999;91:1847-185610547391
Google ScholarCrossref 129.Rice GP, Filippi M, Comi G.Cladribine MRI Study Group. Cladribine and progressive MS: clinical and MRI outcomes of a multicenter
controlled trial.
Neurology. 2000;54:1145-115510720289
Google ScholarCrossref 130.Ventre K, Randolph A. Ribavirin for respiratory syncytial virus infection of the lower respiratory
tract in infants and young children.
Cochrane Database Syst Rev. 2004;4:CD00018115494991
Google Scholar 131.Rantala H, Tarkka R, Uhari M. A meta-analytic review of the preventive treatment of recurrences of
febrile seizures.
J Pediatr. 1997;131:922-9259427902
Google ScholarCrossref 132.Creery D, Mikrogianakis A. Sudden infant death syndrome.
Clin Evid. 2003;10:457-46715555102
Google Scholar 133.Russell K, Wiebe N, Saenz A.
et al. Glucocorticoids for croup.
Cochrane Database Syst Rev. 2004;1:CD00195514973975
Google Scholar 134.Makhlouf AM, Al-Hussaini TK, Habib DM, Makarem MH. Second-trimester pregnancy termination: comparison of three different
methods.
J Obstet Gynaecol. 2003;23:407-41112881083
Google ScholarCrossref 135.Nelson R. A systematic review of medical therapy for anal fissure.
Dis Colon Rectum. 2004;47:422-43114994109
Google ScholarCrossref 136.Nallamothu BK, Shojania KG, Saint S.
et al. Is acetylcysteine effective in preventing contrast-related nephropathy?
a meta-analysis.
Am J Med. 2004;117:938-94715629733
Google ScholarCrossref 137.Grantham-McGregor SM, Walker SP, Chang SM, Powell CA. Effects of early childhood supplementation with and without stimulation
on later development in stunted Jamaican children.
Am J Clin Nutr. 1997;66:247-2539250101
Google Scholar 138.Adachi JD, Roux C, Pitt PI.
et al. A pooled data analysis on the use of intermittent cyclical etidronate
therapy for the prevention and treatment of corticosteroid induced bone loss.
J Rheumatol. 2000;27:2424-243111036840
Google Scholar 139.Anand SS, Yusuf S. Oral anticoagulant therapy in patients with coronary artery disease:
a meta-analysis.
JAMA. 1999;282:2058-206710591389
Google ScholarCrossref 140.Fawzi WW, Chalmers TC, Herrera MG, Mosteller F. Vitamin A supplementation and child mortality: a meta-analysis.
JAMA. 1993;269:898-9038426449
Google ScholarCrossref 141.Spector SA, Gelber RD, McGrath N.
et al. Pediatric AIDS Clinical Trials Group. A controlled trial of intravenous immune globulin for the prevention
of serious bacterial infections in children receiving zidovudine for advanced
human immunodeficiency virus infection.
N Engl J Med. 1994;331:1181-11877935655
Google ScholarCrossref 142.Lyman GH, Kuderer NM, Djulbegovic B. Prophylactic granulocyte colony-stimulating factor in patients receiving
dose-intensive cancer chemotherapy: a meta-analysis.
Am J Med. 2002;112:406-41111904116
Google ScholarCrossref 143.Dundar Y, Hill R, Dickson R, Walley T. Comparative efficacy of thrombolytics in acute myocardial infarction:
a systematic review.
QJM. 2003;96:103-11312589008
Google ScholarCrossref 144.Hood WB Jr, Dans AL, Guyatt GH, Jaeschke R, McMurray JJ. Digitalis for treatment of congestive heart failure in patients in
sinus rhythm.
Cochrane Database Syst Rev. 2004;2:CD00290115106182
Google Scholar 145.Anderson JW, Randles KM, Kendall CW, Jenkins DJ. Carbohydrate and fiber recommendations for individuals with diabetes:
a quantitative assessment and meta-analysis of the evidence.
J Am Coll Nutr. 2004;23:5-1714963049
Google ScholarCrossref 146.Giorgi A, Weatherby RP, Murphy PW. Muscular strength, body composition and health responses to the use
of testosterone enanthate: a double blind study.
J Sci Med Sport. 1999;2:341-35510710012
Google ScholarCrossref 147.Artemether-Quinine Meta-analysis Study Group. A meta-analysis using individual patient data of trials comparing artemether
with quinine in the treatment of severe falciparum malaria.
Trans R Soc Trop Med Hyg. 2001;95:637-65011816438
Google ScholarCrossref 148.Saag MS, Cloud GA, Graybill JR.
et al. National Institute of Allergy and Infectious Diseases Mycoses Study
Group. A comparison of itraconazole vs fluconazole as maintenance therapy
for AIDS-associated cryptococcal meningitis.
Clin Infect Dis. 1999;28:291-29610064246
Google ScholarCrossref 149.Martin DF, Kuppermann BD, Wolitz RA, Palestine AG, Li H, Robinson CA.Roche Ganciclovir Study Group. Oral ganciclovir for patients with cytomegalovirus retinitis treated
with a ganciclovir implant.
N Engl J Med. 1999;340:1063-107010194235
Google ScholarCrossref 150.Nieman DC, Brock DW, Butterworth D, Utter AC, Nieman CC. Reducing diet and/or exercise training decreases the lipid and lipoprotein
risk factors of moderately obese women.
J Am Coll Nutr. 2002;21:344-35012166532
Google ScholarCrossref 151.Kaplan LD, Straus DJ, Testa MA.
et al. National Institute of Allergy and Infectious Diseases AIDS Clinical
Trials Group. Low-dose compared with standard-dose m-BACOD chemotherapy for non-Hodgkin's
lymphoma associated with human immunodeficiency virus infection.
N Engl J Med. 1997;336:1641-16489171066
Google ScholarCrossref 152.Diegoli MS, da Fonseca AM, Diegoli CA, Pinotti JA. A double-blind trial of four medications to treat severe premenstrual
syndrome.
Int J Gynaecol Obstet. 1998;62:63-679722128
Google ScholarCrossref 153.Flather MD, Yusuf S, Kober L.
et al. ACE-Inhibitor Myocardial Infarction Collaborative Group. Long-term ACE-inhibitor therapy in patients with heart failure or left-ventricular
dysfunction: a systematic overview of data from individual patients.
Lancet. 2000;355:1575-158110821360
Google ScholarCrossref 154.Ossenkoppele GJ, van der Holt B, Verhoef GE.
et al. Dutch-Belgian Hemato-Oncology Cooperative Group. A randomized study of granulocyte colony-stimulating factor applied
during and after chemotherapy in patients with poor risk myelodysplastic syndromes:
a report from the HOVON Cooperative Group.
Leukemia. 1999;13:1207-121310450748
Google ScholarCrossref 155.Citron ML, Berry DA, Cirrincione C.
et al. Randomized trial of dose-dense vs conventionally scheduled and sequential
vs concurrent combination chemotherapy as postoperative adjuvant treatment
of node-positive primary breast cancer: first report of Intergroup Trial C9741/Cancer
and Leukemia Group B Trial 9741.
J Clin Oncol. 2003;21:1431-143912668651
Google ScholarCrossref 156.Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is not enough.
Lancet. 1998;351:123-1279439507
Google ScholarCrossref 157.Ioannidis JP, Cappelleri JC, Lau J. Issues in comparisons between meta-analyses and large trials.
JAMA. 1998;279:1089-10939546568
Google ScholarCrossref 158.Fleming TR, DeMets DL. Surrogate end points in clinical trials: are we being misled?
Ann Intern Med. 1996;125:605-6138815760
Google ScholarCrossref 159.Dickersin K, Min YI. Publication bias: the problem that won't go away.
Ann N Y Acad Sci. 1993;703:135-1468192291
Google ScholarCrossref 160.Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.
Lancet. 1991;337:867-8721672966
Google ScholarCrossref 161.Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study
of clinical research projects.
BMJ. 1997;315:640-6459310565
Google ScholarCrossref 162.Ioannidis JP. Effect of the statistical significance of results on the time to completion
and publication of randomized efficacy trials.
JAMA. 1998;279:281-2869450711
Google ScholarCrossref 163.Wheatley K, Clayton D. Be skeptical about unexpected large treatment effects: the case of
an MRC AML12 randomization.
Control Clin Trials. 2003;24:66-7012559643
Google ScholarCrossref 164.Poynard T, Munteanu M, Ratziu V.
et al. Truth survival in clinical research: an evidence-based requiem?
Ann Intern Med. 2002;136:888-89512069563
Google ScholarCrossref 165.DeAngelis C, Drazen JM, Frizelle FA.
et al. Clinical trial registration: a statement from the International Committee
of Medical Journal Editors.
N Engl J Med. 2004;351:1250-125115356289
Google ScholarCrossref