Context Controversy and uncertainty ensue when the results of clinical research
on the effectiveness of interventions are subsequently contradicted. Controversies
are most prominent when high-impact research is involved.
Objectives To understand how frequently highly cited studies are contradicted or
find effects that are stronger than in other similar studies and to discern
whether specific characteristics are associated with such refutation over
time.
Design All original clinical research studies published in 3 major general
clinical journals or high-impact-factor specialty journals in 1990-2003 and
cited more than 1000 times in the literature were examined.
Main Outcome Measure The results of highly cited articles were compared against subsequent
studies of comparable or larger sample size and similar or better controlled
designs. The same analysis was also performed comparatively for matched studies
that were not so highly cited.
Results Of 49 highly cited original clinical research studies, 45 claimed that
the intervention was effective. Of these, 7 (16%) were contradicted by subsequent
studies, 7 others (16%) had found effects that were stronger than those of
subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely
unchallenged. Five of 6 highly-cited nonrandomized studies had been contradicted
or had found stronger effects vs 9 of 39 randomized controlled trials (P = .008). Among randomized trials, studies with
contradicted or stronger effects were smaller (P = .009)
than replicated or unchallenged studies although there was no statistically
significant difference in their early or overall citation impact. Matched
control studies did not have a significantly different share of refuted results
than highly cited studies, but they included more studies with “negative”
results.
Conclusions Contradiction and initially stronger effects are not unusual in highly
cited research of clinical interventions and their outcomes. The extent to
which high citations may provoke contradictions and vice versa needs more
study. Controversies are most common with highly cited nonrandomized studies,
but even the most highly cited randomized trials may be challenged and refuted
over time, especially small ones.