Persistence of Contradicted Claims in the Literature

Context Some research findings based on observational epidemiology are contradicted by randomized trials, but may nevertheless still be supported in some scientific circles.

Objectives To evaluate the change over time in the content of citations for 2 highly cited epidemiological studies that proposed major cardiovascular benefits associated with vitamin E in 1993; and to understand how these benefits continued being defended in the literature, despite strong contradicting evidence from large randomized clinical trials (RCTs). To examine the generalizability of these findings, we also examined the extent of persistence of supporting citations for the highly cited and contradicted protective effects of beta-carotene on cancer and of estrogen on Alzheimer disease.

Data Sources For vitamin E, we sampled articles published in 1997, 2001, and 2005 (before, early, and late after publication of refuting evidence) that referenced the highly cited epidemiological studies and separately sampled articles published in 2005 and referencing the major contradicting RCT (HOPE trial). We also sampled articles published in 2006 that referenced highly cited articles proposing benefits associated with beta-carotene for cancer (published in 1981 and contradicted long ago by RCTs in 1994-1996) and estrogen for Alzheimer disease (published in 1996 and contradicted recently by RCTs in 2004).

Data Extraction The stance of the citing articles was rated as favorable, equivocal, and unfavorable to the intervention. We also recorded the range of counterarguments raised to defend effectiveness against contradicting evidence.

Results For the 2 vitamin E epidemiological studies, even in 2005, 50% of citing articles remained favorable. A favorable stance was independently less likely in more recent articles, specifically in articles that also cited the HOPE trial (odds ratio for 2001, 0.05 [95% confidence interval, 0.01-0.19; P < .001] and the odds ratio for 2005, 0.06 [95% confidence interval, 0.02-0.24; P < .001], as compared with 1997), and in general/internal medicine vs specialty journals. Among articles citing the HOPE trial in 2005, 41.4% were unfavorable. In 2006, 62.5% of articles referencing the highly cited article that had proposed beta-carotene and 61.7% of those referencing the highly cited article on estrogen effectiveness were still favorable; 100% and 96%, respectively, of the citations appeared in specialty journals; and citations were significantly less favorable (P = .001 and P = .009, respectively) when the major contradicting trials were also mentioned. Counterarguments defending vitamin E or estrogen included diverse selection and information biases and genuine differences across studies in participants, interventions, cointerventions, and outcomes. Favorable citations to beta-carotene, long after evidence contradicted its effectiveness, did not consider the contradicting evidence.

Conclusion Claims from highly cited observational studies persist and continue to be supported in the medical literature despite strong contradictory evidence from randomized trials.

Some research findings that have received wide attention in the scientific community, as proven by the high citation counts of the respective articles, are eventually contradicted by subsequent evidence.¹ A number of such high-profile contradictions pertain to differences between nonrandomized and randomized studies. For example, the effect of vitamin E on cardiovascular disease prevention has been in the center of a major debate in clinical research over the last 2 decades. Vitamin E is known to have antioxidant activity, and a long list of citations in the preclinical literature on antioxidants^2-4 suggested that these agents may be beneficial for cancer and cardiovascular disease. Two highly cited publications suggested in the 1990s that vitamin E could decrease cardiovascular disease risk by almost half in men and in women.^5,6 However, subsequent randomized trials showed no benefit or even suggested increased harm.^7,8 Several other highly prominent contradictions have also been recorded pertaining to the effects of other dietary components and hormones.^9-15 The prominent refutation of the epidemiological studies has spurred considerable controversy for observational epidemiology in general.^16-21

Such debate offers opportunities to study what happens to the scientific literature, when a highly prominent claim is refuted. How quickly are such beliefs abandoned? Is there still literature citing the contradicted studies despite their refutation? What counterarguments are used by the citing articles to defend the original claims? To answer these questions, we performed citation content analysis for the 2 most highly cited articles that proposed vitamin E benefits. We evaluated the change in favorable vs unfavorable citations over time and recorded the counterarguments that were used to continue supporting the belief in vitamin E effectiveness. To assess the generalizability of our findings, we also examined the extent to which 2 other major contradicted claims, the preventive effectiveness of beta-carotene for cancer and estrogens for Alzheimer dementia, continue to be supported in the current literature.

We focused on 2 highly cited articles published in 1993. These articles presented data from 2 observational cohorts^5,6 and showed consistently that vitamin E was associated with major decreases in the relative risk (RR) of cardiovascular events (0.63, 95% confidence interval [CI], 0.47-0.84 in men and RR, 0.59; 95% CI, 0.38-0.91 in women for those receiving vitamin E for 2 years vs none). These 2 articles are the most-cited papers on benefits from vitamin E supplementation and they have received 1395 and 1234 citations, respectively, until the end of 2006. Based on these articles, vitamin E was considered cardioprotective for many years. Several smaller studies suggested direct or indirect evidence supporting this claim.

A randomized trial of 2002 patients (CHAOS) published in 1996 also found a 47% relative risk reduction for cardiovascular events.⁷ However, many randomized trials subsequently found no cardiovascular benefit. The most-cited contradicting trial (HOPE) was published in January 2000 and found an RR of 1.05 (95% CI, 0.95-1.16) for cardiovascular events,⁸ an effect entirely incompatible with estimates of the epidemiological studies. A meta-analysis published in late 2004 concluded that at high doses, vitamin E significantly increased the risk of death (RR, 1.04; 95% CI, 1.01-1.07).²² Publication of the CHAOS and HOPE trials have also accumulated a large number of citations (1172 and 704 citations by the end of 2006, respectively) and the meta-analysis is also highly cited, despite the short time since its publication (226 citations by the end of 2006, the most-cited article published in in the field of clinical medicine in 2004 according to Essential Science Indicators). A recent meta-analysis²³ even concluded that among high-quality trials, vitamin E increases mortality regardless of dose (RR, 1.04; 95% CI, 1.01-1.07 in low-bias trials). Vitamin E supplementation is not currently recommended by practice guidelines.^24,25

Citation Curves. We downloaded annual citation counts from Thomson Scientific ISI Web of Knowledge for each of the 2 highly cited epidemiological studies between 1993 and 2006 and also assessed the number articles citing at least 1 of the 2 studies. For reference standard, we examined the total annual citation curves for all the articles published in the same year (1993) and in the same journal as the 2 highly cited epidemiological studies.

Selection of the Citing Articles. We sampled citations to the 2 highly cited epidemiological studies at 3 different and equidistant years: 1997, 2001, and 2005. The first selected year (1997) represents the peak of annual citations and may be perceived to be the time when the evidence was the strongest in favor of vitamin E (shortly after the additional support offered by the CHAOS trial published in 1996).⁷ The second selected year (2001) corresponds with an early period after major refutation (1-2 years after the HOPE results).⁸ The third selected year (2005) corresponds with a late period after major refutation; meta-analysis had even shown increased harm with vitamin E. To allow for a fairly similar number of citations analyzed at each selected year, we sampled every third citation in 1997, every second citation in 2001, and all citations in 2005 among citations made to either or both highly cited epidemiological articles.

Our purpose was not to study the overall literature on a research topic in which contradiction of the original studies has arisen. The boundaries of such a literature review are very difficult, if not impossible, to define. On the contrary, we aimed to examine the citing behavior of the scientific literature toward the original studies that have been contradicted. The content analysis of this set of citations is likely to yield a set of references that is enriched in positions that allude to or even try to defend the original claims. This can give insights on how extensively, and with what arguments, these claims are defended despite the ensued contradiction.

Characteristics of the Citing Articles. For each eligible citation, we retrieved the full text of the citing article. We recorded the first author, journal, and country(ies) of investigators. We classified each article depending on whether it had primary data or not (reviews, meta-analyses, editorials, letters, other), and articles with primary data were further categorized depending on whether they were derived from a randomized trial or from nonrandomized studies. Additionally, we retrieved the 2005 impact factor for each journal that published an eligible article²⁶ and recorded whether (per Web of Knowledge classification) the journal was classified in the general/internal medicine category vs some specialty (including both clinical and basic sciences). We also recorded which of the 2001 and 2005 articles had also cited the HOPE trial,⁸ the most highly cited contradicting publication to date on this topic.

Citation Content Analysis. We assessed how many times each of the 2 highly cited epidemiological studies was cited with a reference in each citing article. For each time that each article was cited, we recorded the exact phrase or sentence in which the reference(s) appeared and any preceding or following sentences that elaborated on the same argument(s). When these articles were cited multiple times in the same citing article, we captured the text on all of these appearances.

We first excluded citations that were erroneous (irrelevant, apparently an error of the authors), and those that were not pertinent to cardiovascular disease prevention and vitamin E, but instead to some other aspect of the 2 highly cited articles (eg, association of vitamin C with chronic diseases that was also commented in the original highly cited articles) or some other generic issue (eg, referring to similar methods or questionnaires being used as in the vitamin E studies). When the context of the citation was pertinent to the association between vitamin E and cardiovascular disease prevention, we categorized the overall stance of the citing article as favorable, equivocal, or unfavorable.

The categorization depended on whether the arguments were suggesting that vitamin E had beneficial effects (favorable), both favorable and unfavorable arguments existed without any clear preference given to either (equivocal), or vitamin E was claimed to be ineffective or harmful (unfavorable). When both favorable and unfavorable arguments were presented but the authors eventually took sides in one direction, the article was accordingly categorized as either favorable or unfavorable. For categorization, we cumulatively considered all the expounded arguments in each citing article.

Data extraction was performed by 2 independent investigators; discrepancies were resolved by consensus and arbitration by a third investigator.

The primary outcome was the proportion of articles citing the highly cited epidemiological studies that were favorable, equivocal, and unfavorable about vitamin E effectiveness for cardiovascular disease prevention.

The main hypothesis was that these proportions should markedly change between 1997, 2001, and 2005, unless beliefs in vitamin E effectiveness remained unchanged. Secondary hypotheses evaluated whether any additional characteristics of the citing article besides year of publication (country of origin, article type, impact factor, journal field, article also cited the contradicting HOPE trial) were related to its stance.

The primary hypothesis was evaluated with the Jonckheere-Terpstra test for multiple-ordered categories. The secondary hypotheses were evaluated with the Kruskal-Wallis analysis of variance for single-ordered variables and the Jonckheere-Terpstra test for multiple-ordered variables.

We also performed analyses to examine the independent association of different characteristics on the overall stance of a citing article. Unfavorable and equivocal citations were merged because they occurred fewer times than favorable ones. We used univariate logistic regressions to examine the association between each of the characteristics mentioned previously with a favorable stance. Variables with a P value of less than .10 in univariate analyses were considered also in a multivariate analysis. Categorical independent variables were treated with multiple dummy variables. The regression used step-wise backward elimination of variables that had a P value of greater than .05. Forward selection of variables yielded similar models. For the multivariate analyses, we first constructed a variable that considered both the publication year and whether citation to the HOPE trial was made (categories: 1997, 2001 and citing HOPE, 2001 not citing HOPE, 2005 and citing HOPE, and 2005 not citing HOPE), since year and citation to the HOPE trial are by default strongly correlated (citing HOPE not applicable for 1997 articles; HOPE was published in 2000). Quantitative analyses were performed using SPSS version 13.0 (SPSS Inc, Chicago, Illinois) and StatXact (Cytel Corp, Boston, Massachusetts). P values were 2-tailed, and a P value of less than .05 was considered statistically significant.

Content Analysis for Citations to the HOPE Trial. Articles selected because they cite the 2 highly cited epidemiological studies may be more likely to be favorable to vitamin E use compared with articles that would be selected because they cite contradicting studies. Therefore, we also created a separate group of articles in which we sampled every third article that cited in 2005 the HOPE trial,⁸ regardless of whether it cited the 2 highly cited epidemiological studies or not. Through the same process, we identified the proportion of favorable, equivocal, and unfavorable citations to vitamin E use.

Qualitative Counterarguments. We created a qualitative list of the different types of arguments that have been made to counter the accumulating evidence that vitamin E is harmful or not effective. We categorized counterarguments according to allusion to biases and genuine differences in study participants, interventions, cointerventions, and outcomes using the PICO structure.²⁷

To examine the generalizability of our main findings on vitamin E, we also investigated 2 other examples for which observational claims have been subsequently contradicted by large randomized trials. We used as highly cited articles the most-cited articles that had proposed these claims. We selected a claim that had been made a long time ago and had also been contradicted long ago (beta-carotene for cancer prevention), and a claim that had been contradicted very recently (estrogen for dementia prevention). We then examined the current stance (favorable, equivocal, or unfavorable) of citing articles. We chose for citation content analysis the calendar year 2006, ie, a decade and 2 years, respectively, after the major contradicting studies were published.

Beta-carotene was initially supported by many epidemiological studies and laboratory investigations as a potent chemoprevention against cancer. The most-cited article in this literature is an influential review of the epidemiological and other nonrandomized studies that was published in 1981.⁹ This review received 1119 citations by the end of 2006. Randomized trials, nevertheless, found no benefit or harm with beta-carotene use. The 3 most-cited trials on this topic are the Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group¹⁰ (primary outcome, lung cancer; published in 1994; 1640 citations through 2006), the Beta-Carotene and Retinol Efficacy Trial¹¹ (primary outcome, lung cancer using carotene combined with retinol; published in 1996; 1296 citations through 2006), and the Physicians's Health Study¹² (primary outcome, all cancers; published in 1996; 1087 citations through 2006). These trials found relative risks of 1.18 (95% CI, 1.03-1.36), 1.28 (95% CI, 1.04-1.57) and 0.98 (95% CI, 0.91-1.06), respectively. Beta-carotene supplementation is not recommended by any guidelines currently.

Estrogens were also supported by many epidemiological studies and laboratory investigations as strong neuroprotective agents that could diminish the risk of dementia. The most-cited article is an observational study¹³ published in 1996 that found a 60% (95% CI, 15-78) RR reduction in postmenopausal women taking estrogens. This study has received 915 citations through 2006. Early randomized trials could not replicate these benefits and in mid-2004, the Women's Health Initiative Memory Study RCT published its results showing a trend for increased risk of dementia with estrogens in postmenopausal women (RR, 1.49; 95% CI, 0.83-2.66)¹⁴ and worsening of cognition.¹⁵ Estrogens are also not recommended as preventive intervention for dementia currently.

For each of these 2 topics, we constructed citation curves for the highly cited epidemiological articles, retrieved the articles citing the articles in 2006, evaluated the stance (favorable, equivocal, or unfavorable) of the citing articles, and captured counterarguments raised to defend the effectiveness of these interventions using the same methods as for vitamin E. For 3 articles, the 2 independent reviewers disagreed on the stance of the citation and the third investigator arbitrated on the discrepancy.

Citation Curves Over Time. The citation curve for the 2 vitamin E epidemiological articles largely paralleled the citation curve for all the articles published in the same journal in 1993: early rapid increase, peak in 1997 or 1998, and slow decline until 2001 (Figure 1). However, in 2002 and beyond, the relative decrease in citations was much steeper for the 2 vitamin E articles than for the total citations to all articles published in the same journal. The citation rate in 2006 for all articles published in 1993 continued to be more than half (55%) of the citation rate in 1997, while the 2 vitamin E articles had decreased to 20% of their peak annual citation rate by that time.

Characteristics of Eligible Citing Articles. We selected for citation analysis 176 citing articles, of which 56 articles were published in 1997, 61 in 2001, and 59 in 2005 (Table 1). We could not retrieve 2 articles from 2001 and 2 from 2005; thus, we finally analyzed 172 publications (Table 1).

Seventy-six articles (44.2%) included at least 1 author from an institution located in the United States. Ninety-seven (56.4%) articles included primary data, 23 (23.7%) of which pertained to data from randomized trials. The citing articles were published in journals with a median impact factor 2.310 (interquartile range, 1.52-4.04). Twenty-six (16.0%) appeared in general or internal medicine journals. Seventy-two (62.1%) of the articles in 2001 and 2005 cited also the contradicting HOPE trial (Table 1).

Of the 172 articles, one had entirely erroneously cited 1 of the 2 articles⁵ and the citations in another 6 articles were not pertinent to vitamin E and cardiovascular disease prevention. Thus, 165 articles were eligible for categorizing a stance on vitamin E in cardiovascular prevention.

Overall Stance and Evolution Over Time. Overall, 101 citing articles (61.2%) were favorable, 36 (21.8%) were equivocal, and 28 (17.0%) were unfavorable (Table 1). Categorization by 2 independent investigators was concordant (weighted κ 0.91, 95% CI, 0.87-0.94).

Citing articles showed significant difference in their stance over time (P = .0002). The proportion of unfavorable articles increased from 1.9% in 1997 to 14.3% in 2001 and to 33.9% in 2005. Despite a decrease in the proportion of favorable articles, these still represented 50% of the total in 2005. The stance of the articles overall was also significantly more favorable, less unfavorable, or both, when articles were not originating from the United States (P = .046), when articles included nonrandomized primary data (P < .001), when specialty journals were involved (P < .001), and when the HOPE trial was not cited (P < .001) (Table 2).

Independent Associations of Citing Article Characteristics With Favorable Stance. In multivariate analyses, the odds of a citing article having a favorable stance were approximately 20 times lower in 2001 and 2005, as compared with 1997, when the HOPE trial was also cited (odds ratios were 0.05 and 0.06, respectively), but not necessarily when the HOPE trial was not cited (Table 3). Moreover, the odds of a favorable stance were 12 times lower in articles published in general and internal medicine journals than in articles published in other journals (Table 3).

Overall Stance of Articles Citing the HOPE Trial in 2005. In a sample of 29 articles published in 2005 that had cited the HOPE study, 6 (20.7%) were still favorable to vitamin E, 11 (37.9%) were equivocal, and 12 (41.4%) were unfavorable. Eight of these articles had also cited one or both of the 1993 highly cited epidemiological studies. Excluding these 8 articles, there were 6 (28.6%) favorable citations to vitamin E, 9 (42.9%) equivocal citations, and 6 (28.6%) unfavorable citations.

Qualitative List of Counterarguments. Typical examples of counterarguments are shown in Box 1.^28-34 Alluded biases included study selection bias in meta-analyses or information bias due to incomplete recording of outcome events. Genuine diversity between studies in favor of vitamin E and trials with negative or harmful effects focused on baseline patient characteristics (ie, genetic background, dietary habits, stage of atherosclerotic disease, oxidative stress status, and lifestyle of study participants); vitamin E intervention—type, dosage, and bioavailability (ie, use of synthetic vs natural form of tocopherol, use of small vs higher doses of tocopherol, ingestion of vitamin E with vs without lipid-rich meals, use of balanced intake vs single antioxidant supplementation, or discrepancies in antioxidant levels in blood or tissues before and after supplementation); concomitant interventions (ie, patients supplemented with a harmful cointervention or lacking an additional useful antioxidant cointervention); and duration of follow-up (short-term vs long-term follow up studies). Diverse biological mechanisms were invoked in support.

Biases

Selection bias: meta-analysis did not put its results in perspective by reviewing the context of research on vitamin E including the many positive observational and interventional studies²⁸

Information bias: mortality estimates from CHAOS^a came from a research letter, not a peer-reviewed study, and included data after the study was officially ended, and thus subject to information bias²⁸

Genuine Diversity

Participant Characteristics

Genetic characteristics: genetic background of study subjects might have contributed to the differential results²⁹

Dietary habits: discrepancies may be explained by differences in the antioxidant content of the basal diet of the sample population under investigation³⁰

Stage of disease: some antioxidants, eg, vitamin E, might be more effective in the early phase of atherosclerosis, but much less so in the advanced clinically overt stage present in the majority of patients evaluated in clinical trials³¹

Oxidative stress status: studies that have included healthy subjects with decreased oxidative stress while vitamin E reduced oxidative stress in smokers (a condition of increased oxidative stress)³¹

Lifestyle characteristics: lifestyle of study subjects might have contributed to the differential results²⁹

Intervention: Vitamin E Form, Dose, Bioavailability

Vitamin E form: some trials utilized synthetic tocopherol, whose efficacy is not equivalent to the natural form³¹

Vitamin E dose: an adequate intake in the lowest intake category or a low interindividual variation intake may explain some of the negative findings³²

Vitamin E bioavailability: no control on how antioxidant vitamins were ingested: the bioavailability of vitamin E is higher when it is taken with lipid-rich meals. Antioxidant levels were not consistently measured in blood or tissues before and after supplementation: the same intake may produce different levels in distinct individuals³¹

Co-interventions

Beta-carotene (harmful co-intervention): most of the evidence for an elevated mortality risk came from two trials that administered vitamin E together with beta-carotene³³

Lack of appropriate cointervention: . . . single antioxidant supplementation might not be a good strategy, since antioxidant defenses normally behave as a network: therefore balanced intake is likely important³¹

Outcomes

Duration of follow-up: the possibility that antioxidants need to be taken more than five years to have a significant effect on atherosclerotic plaque formation cannot be dismissed³⁴

^aRefers to the mortality data of CHAOS, which, contrary to the main publication of the trial on cardiovascular events, had shown no benefit from vitamin E.

Citations to the highly cited article proposing the possibility of beta-carotene effectiveness for cancer prevention did not fall more steeply than those of the average article published in the same journal in the same year. Conversely, the decline in citation rate took a decade longer to start than for the average paper. The decline was heralded by the publication of the most prominent contradicting trials (Figure 2). Citations to the highly cited epidemiological study on estrogen use for dementia prevention largely followed the pattern of the citations in the same journal for articles published in the same year. The contradiction is recent and there was only a modestly steeper decline in 2006 (Figure 2).

The highly cited studies received 17citations for beta-carotene and 48 citations for estrogen in 2006 (Table 4) and 1 citing article could not be retrieved for each. For beta-carotene, 10 citing articles (62.5%) were favorable, 3 (18.8%) equivocal, and 3 (18.8%) unfavorable. For estrogen, 29 citing articles (61.7%) were favorable, 14 (29.8%) equivocal, and 4 (8.5%) unfavorable. All beta-carotene citations and all but 2 estrogen citations appeared in specialty journals. The overall stance of the citing articles was significantly more favorable when the contradicting Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group and Women's Health Initiative Memory Study were not cited (P = .001 and P = .009, respectively).

Of the 10 favorable and the 3 equivocal beta-carotene citations, only 1 raised counterarguments against the contradicting evidence, claiming³⁵ that “the effectiveness of these carotenoids as antioxidants depends upon a number of factors (eg, concentration, cell type, cell status, timing of insult exposure, location in the cell, interaction with other antioxidants, etc)”. The other 9 favorable citing articles (2 reviews and 7 experimental articles on human tissue and animals) simply did not cite any trials that had contradicted beta-carotene effectiveness.

For Alzheimer disease, counterarguments (Box 2)^36-43 to support estrogen pertained to various selection biases; issues related to the participants, including differences in age, menopausal symptoms, prior hormonal treatment, and stage of disease (before vs after the onset of dementia process); differences in the intervention scheme, including estrogen form, route of administration and regimen, and cointerventions; and choice of outcome definitions (Box 2).

Biases

Selection bias—general: the dichotomy between the observational and recent prospective studies may be due to selection bias³⁶

Different baseline risk despite randomization: there were baseline differences on the global cognitive test—low scorers (more frequent in the treatment than placebo arm) were at much greater risk for developing dementia³⁷

Different risk factors at baseline despite randomization: significantly more women with a history of hypertension were randomized to active treatment than to placebo (41% vs 38%), but more women with a history of stroke were allocated to the placebo group (2.0% vs 1.3%)³⁷

Genuine Diversity

Participant Characteristics

Age: women were an average age of 68 when entering the study. These issues may limit the generalizability³⁸

Background disease: the majority of the dementia diagnoses in the WHI study seemed to be related to vascular disease³⁸

Prior treatment: many variables may contribute to the discrepancy; these include prior hormone replacement history³⁹

Concomitant symptoms: women with vasomotor symptoms were excluded from the Women's Health Initiative clinical trial if it was anticipated that symptoms would affect treatment compliance³⁷

Stage of disease: estrogen replacement therapy may be applied to delay the progression of AD pathogenesis but not to recover the lost functions⁴⁰

Intervention: Estrogen Form, Route of Administration, Dosage

Estrogen source and mode of delivery: other factors have also been identified for consideration in interpretation of the WHI study including the source of hormone (equine estrogens as compared to synthetic human forms of these hormones) and mode of delivery (cyclic vs continuous)⁴¹

Estrogen preparation, route and mode of delivery: although informative, the interpretation of the WHI studies is limited by the hormone preparations used, their route of administration, the regimen of hormone administration (ie, continuous daily therapy vs cyclic affect concentrations and localization of antiapoptotic proteins, which appear to exert their antiapoptotic effects through maintenance of mitochondrial membrane potential in the face of cellular stresses⁴²

Estrogen dose: in contrast, in vitro exposure of neurons to estrogen if the dose is high enough, can exacerbate degeneration⁴³

Co-interventions

Progestin (harmful co-intervention): progestin included in the estrogen replacement therapy could compromise estrogen's effect^40a

Outcomes

Type of endpoints: WHI did not consider Alzheimer disease as a specific endpoint, whereas most observational studies looked specifically at Alzheimer disease risk³⁷

^aWHI generated randomized data both for estrogen and for estrogen plus progestin regimens

Citations to the 2 highly cited observational studies proposing an association of vitamin E with reduced cardiovascular events became less favorable over time, as contradicting data from randomized trials accumulated. Nevertheless, despite the eventual accumulation of strongly refuting evidence, even in 2005, half of the articles citing these epidemiological studies were still favorable to the vitamin E claim. Even among articles that cited the contradicting HOPE trial rather than the positive epidemiological studies, the majority in 2005 still could not conclude that vitamin E was ineffective. Many counterarguments were raised to defend vitamin E in the face of contradictory evidence from RCTs. In a similar fashion, in 2006 more than half of the articles citing the highly cited epidemiologic articles on beta-carotene for cancer prevention and estrogen for dementia prevention remained favorable for these interventions. For beta-carotene, after a decade had passed from the contradiction of its effectiveness, counterarguments were uncommon: citing articles simply did not mention the contradicting trials. Conversely, for estrogen, a claim for which contradiction has been more recent, many counterarguments (of similar breadth as for vitamin E) were raised to defend its effectiveness.

We observed an apparent split of stance in the scientific literature. The persistent favorable stance toward the contradicted interventions was particularly prominent in articles published in specialty journals of both clinical and basic science disciplines. Specialist articles apparently continued to use references to the highly cited observational studies to support their own lines of research. The presence of refuting data were not mentioned in many articles. Other articles did report data with contrary results, but they raised also a wide array of counterarguments to support the observational claim. Most nonrandomized studies published in specialty journals show positive results.^18,44,45 Apparently, there is also a citation bias selecting positive citations.^46,47 Conversely, for journals with a more general medical audience, apparently the contradicting randomized data carried more weight than the observational data. For beta-carotene and estrogen, almost all analyzed citations appeared in specialty journals.

Our citation content analysis also highlights another aspect of the existing antithesis between randomized and observational research.^17,48-51 Apparently, the same data are used and interpreted entirely differently by different investigators depending on whether they supported findings from the randomized trials or observational studies. However, when randomized and observational studies disagree, it is incorrect to assume that nonrandomized studies are always wrong. Disagreements and contradictions appear also between randomized trials, even large ones, and also in many other research fields in which other designs are used.

In the evaluation of counterarguments, we encountered almost any source of bias, genuine diversity, and biological reasoning invoked to defend the original observations. While some or even many of these counterarguments may be valid, this is also consistent with a belief that is defended at all cost. The defense of the observational associations was persistent, despite the availability of very strong contradicting randomized evidence on the same topic. Thus, one wonders whether any contradicted associations may ever be entirely abandoned, if such strong randomized evidence is not considered as much stronger evidence on the topic. For most associations and questions of medical interest, either no randomized data exist, or the randomized evidence is minimal or of poor quality.^52,53

Our data also suggest that contradiction through randomized trials may lead eventually to a decrease in the absolute frequency of citations to the epidemiological studies. However, this may occur with considerable delay and a considerable segment of the literature continues to cite the contradicted articles long after the contradiction. The articles that cited these observational studies continued to be predominantly favorable. Moreover, even when we considered articles that referenced the most prominent contradicting trial against vitamin E, clearly unfavorable citations for vitamin E were still the minority. Beta-carotene, in particular, offers the opportunity to examine what happens when many years have passed after the contradiction: a citation rate of decreasing (but still substantial) volume continues to support the contradicted claims without even mentioning the contradicting evidence or raising counterarguments.

Sometimes investigator beliefs in scientific circles may have similar psychological characteristics as the nonscientific beliefs observed in other areas of society. The wish bias of individuals, irrespective of topic, can be large and may also influence the interpretation of scientific results. Such bias has been discussed and demonstrated in the past for several other societal and scientific efforts.^54-59 Wish bias does not necessarily mean that the defended beliefs are wrong. Moreover, it can be difficult to discern whether perpetuated beliefs are based on careful consideration of all evidence and differential interpretation, inappropriate entrenchment of old information, lack of dissemination of newer data, or purposeful silencing of their existence. Regardless of the reasons, better communication between research specialists and evidence-based clinical science⁶⁰ may improve this situation and may lead to more rational and concerted translational efforts in basic, preclinical, and clinical research.

Corresponding Author: John P. A. Ioannidis, MD, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, University Campus, Ioannina, 45110 Greece (jioannid@cc.uoi.gr).

Author Contributions: Dr Ioannidis had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Tatsioni, Bonitsis, Ioannidis.

Acquisition of data: Tatsioni, Bonitsis, Ioannidis.

Analysis and interpretation of data: Tatsioni, Bonitsis, Ioannidis.

Drafting of the manuscript: Tatsioni, Ioannidis.

Critical revision of the manuscript for important intellectual content: Bonitsis.

Statistical analysis: Tatsioni, Ioannidis.

Study supervision: Ioannidis.

Funding/Support: None.