Relative Citation Impact of Various Study Designs in the Health Sciences

Context The relative merits of various study designs and their placement in hierarchies of evidence are often discussed. However, there is limited knowledge about the relative citation impact of articles using various study designs.

Objective To determine whether the type of study design affects the rate of citation in subsequent articles.

Design and Setting We measured the citation impact of articles using various study designs—including meta-analyses, randomized controlled trials, cohort studies, case-control studies, case reports, nonsystematic reviews, and decision analysis or cost-effectiveness analysis—published in 1991 and in 2001 for a sample of 2646 articles.

Main Outcome Measure The citation count through the end of the second year after the year of publication and the total received citations.

Results Meta-analyses received more citations than any other study design both in 1991 (P<.05 for all comparisons) and in 2001 (P<.001 for all comparisons) and both in the first 2 years and in the longer term. More than 10 citations in the first 2 years were received by 32.4% of meta-analyses published in 1991 and 43.6% of meta-analyses published in 2001. Randomized controlled trials did not differ significantly from epidemiological studies and nonsystematic review articles in 1991 but clearly became the second-cited study design in 2001. Epidemiological studies, nonsystematic review articles, and decision and cost-effectiveness analyses had relatively similar impact; case reports received negligible citations. Meta-analyses were cited significantly more often than all other designs after adjusting for year of publication, high journal impact factor, and country of origin. When limited to studies addressing treatment effects, meta-analyses received more citations than randomized trials.

Conclusion Overall, the citation impact of various study designs is commensurate with most proposed hierarchies of evidence.

Several authors and organizations have proposed hierarchies of evidence, based on the relative reliability of various types of study designs.^1-4 Although many people recognize that expert opinions and nonsystematic reviews provide the least reliable level of information,^5,6 such articles continue to have a massive influential presence.⁷ Controlled studies assume higher places in hierarchies of evidence than uncontrolled studies, and randomized trials are considered the gold standard for clinical research.^1-4 However, randomized trials cannot be conducted for all questions of interest⁸ and there is debate on whether they give different results than nonrandomized studies.^9-14 Finally, meta-analyses are becoming increasingly frequent in the literature. Meta-analyses are often placed at the highest level of evidence,^1-4 despite their critics.^15,16 No hierarchy of evidence is unanimously accepted.

An important issue is whether the impact of various studies is different and changing over time. Impact on clinical practice and decision making is difficult to measure comprehensively. However, one important measure of impact is the use of citations in the published literature. Citations have limitations,¹⁷ but they provide an objective measurement of how often scientists use a specific published work. One may ask: What is the relative citation impact of published articles using various types of designs? Is this impact commensurate with the proposed hierarchies of evidence? Has it changed over time? We aimed to answer these questions using citation analysis.

We compared the citation impact across various study designs and between studies published in 1991 and 2001. We searched the Institute for Scientific Information (ISI) Science Citation Index at the Web of Science Database (www.isinet.com) for meta-analyses, randomized controlled trials (RCTs), cohort studies, case-control studies, case reports, nonsystematic reviews, and decision analysis or cost-effectiveness analysis records published in 1991 and 2001. These types of publications cover the major, readily identifiable designs used in collecting and synthesizing medical information. Secondarily, meta-analyses were also classified as meta-analyses including RCTs vs others. Both meta-analyses and RCTs were also classified according to their subject or purpose (treatment effect [therapy or prevention], prognosis, diagnosis, and etiology or association for meta-analyses; treatment effect and diagnosis for RCTs).

It is impractical to identify and analyze all the tens of thousands of publications fitting in these study designs. Often it is impossible to accurately classify the study design unless the whole article is carefully scrutinized. Sometimes even this may not suffice. Thus we used a strategy that aimed to yield an adequate number of relevant publications for each design with high specificity in characterizing design. The search strategies for each type of publication sought the appearance of the relevant study design terms in the article title (TI). Meta-analysis was searched with TI = meta-analy* or metaanaly*, randomized controlled trial with TI=random* AND TI=trial, decision analysis or cost-effectiveness analysis with TI=decision analy* OR TI=cost effectiveness analy* OR TI=cost-effectiveness analy*, nonsystematic review with TI=review NOT TI=systemat* NOT TI=meta-analy* NOT TI=overview NOT TI=case report*, case-control study with TI=case control study, cohort study with TI=cohort study, and case report with TI=case report NOT TI=review NOT TI=overview . When the search algorithm yielded an excessive number of articles, we screened systematically 1:5 or 1:10 batches of records, for study designs with 1200 to 3000 records and more than 3000 records retrieved in a year, respectively.

Two investigators (N.A.P. and A.A.A) independently screened both the title and abstract of identified articles. Articles were eligible if they represented applications of the type of study design under which they were identified. We excluded ISI records without abstract; letters; editorials; news and meeting abstracts; methodology-and-theory articles; and articles not on human subjects or material, not on health, or both. Discrepancies were discussed between investigators; a third investigator (J.P.A.I.) resolved disagreements.

For each article eligible for citation analysis, we recorded total citations until December 10, 2004; citations received up to the end of the second year after publication (1991-1993 and 2001-2003, respectively); country of authors; and journal.

The main analyses addressed citation counts for 1991-1993 and 2001-2003 (early citations). Most articles are rarely cited, if at all, during the same year in which they were published, but the citation count of the 2 subsequent years is representative (it forms the basis of estimating journal impact factors). Secondary analyses counted total citations until December 10, 2004 (long-term impact); this time frame unavoidably differed between the 1991 and 2001 publication cohorts.

Citation counts per publication type and year were summarized with medians and interquartile ranges (citation distributions are left-skewed).¹⁸ Mann-Whitney U tests and Kruskal-Wallis analysis of variance compared 2 or several groups, respectively.

We also identified articles that received more than 10 citations in the first 2 years (approximately the top 10% most-cited ISI-indexed articles in Clinical Medicine).¹⁹ Logistic regressions addressed the year and type of publication (dummy variables) as predictors of more than 10 citations in 2 years, adjusting also for country of authors and high journal impact factor.

Analyses were conducted using SPSS statistical software version 12.0 (SPSS Inc, Chicago, Ill). P values are 2-tailed. Statistical significance was considered at the .05 level.

We identified 17 813 articles (6052 from 1991, 11 761 from 2001) and screened 5769 of those for eligibility (1936 and 3833, respectively); 2646 articles (904 and 1742, respectively) were eligible for citation counting (Table 1).

Both in 1991 and in 2001, there was a significant difference in citation counts between various study designs (P<.001). Citation counts were statistically significantly higher in the 2001 publications compared with the 1991 publications for all designs (P<.05), except for cohort studies and decision and cost-effectiveness analyses (Table 1).

Both in 1991 and 2001, meta-analyses received the highest number of citations and RCTs were second (Table 1). For 1991, comparisons against other designs were always formally statistically significant, with the exception of decision and cost-effectiveness analysis publications (P = .11), and the difference against RCTs was also modest (P = .04); for 2001, P values were <.001 for meta-analyses compared with any other design. Differences in citation counts for other designs were more subtle, except for the case reports that always had negligible citation impact.

Twenty-three meta-analyses (32.4%) published in 1991 received more than 10 citations within 2 years. This rose to 43.6% (116/266) for meta-analyses published in 2001. Randomized controlled trials had the next highest impact (23.2% [76/328] and 29.5% [78/264], respectively). Other designs had percentages in the range of 10% to 25%, except for case reports for which less than 1% received more than 10 citations (Figure 1). In multivariable logistic regression, articles published in 2001 (odds ratio [OR], 1.56; 95% confidence interval [CI], 1.23-1.99), having US authors (OR, 1.69; 95% CI, 1.37-2.08), and published in journals with impact factors greater than 10 (OR, 12.8; 95% CI, 8.4-19.5) were more likely to be cited more than 10 times than articles published in 1991, without US authors, or in other journals; and meta-analyses were significantly more likely to be cited more than 10 times compared with all other designs (for RCTs, OR, 0.49; 95% CI, 0.36-0.68; for cohort studies, OR, 0.46; 95% CI, 0.34-0.63; for case-control studies, OR, 0.37; 95% CI, 0.27-0.52; for case reports, OR, 0.01; 95% CI, 0.00-0.04; for nonsystematic reviews, OR, 0.47; 95% CI, 0.31-0.73; and for decision or cost-effectiveness analysis articles, OR, 0.29; 95% CI, 0.16-0.51 vs meta-analyses).

Both in 1991 and in 2001, there was a statistically significant difference in citation count between various designs (P<.001, Figure 2). For 1991, meta-analyses were statistically significantly cited more times than all other designs (P<.05 for all comparisons). Conversely, RCTs had significantly more citations only from case reports (P<.001) and possibly decision or cost-effectiveness analysis articles (P = .05) but did not differ significantly in citation impact vs other designs. Case reports were statistically significantly less cited than anything else (P<.001 for all comparisons). For 2001, meta-analyses had greater impact than all other designs (P<.001 for all comparisons) and RCTs were cited significantly more times than all the remaining designs (P<.05 for all comparisons). Case reports had once again a very low impact (P<.001 for all comparisons). All other comparisons of designs were not statistically significant.

Citations of subgroups of meta-analyses and RCTs are shown in Table 2. There were no statistically significant differences in the citations received by meta-analyses including or not including RCTs, both in 1991 and 2001 and both for the 2 years and for the long term (P>.19 for all analyses). Similarly, citations did not differ significantly for meta-analyses of different purpose or subject (P>.58 for all analyses). Meta-analyses addressing treatment effects tended to receive more citations than RCTs of treatment effects in 1991 (P = .08 for 2-year citations, P = .10 for long-term citations), and the difference became more clear in 2001 (P = .001 and P = .001, respectively).

The citation impact of various study designs follows the order proposed by most current theoretical hierarchies of evidence.^1-3 On average meta-analyses currently receive more citations than any other type of study design. Meta-analyses have clearly surpassed in citation impact both decision or cost-effectiveness analysis articles and RCTs, against which they had mostly modest differences, if any, in the early 1990s. Although RCTs have become the second most cited study design, decision or cost-effectiveness analysis has not followed this growth. Epidemiological studies are now lagging behind randomized research; however, this was not as evident for articles published in 1991. Nonsystematic reviews continue to have a citation impact similar to that of epidemiological studies. Finally, case reports have negligible impact.

The superiority in citation impact of meta-analyses and secondarily RCTs is consistent with the prominent role given to these designs by evidence-based medicine,^1-4 despite the criticisms leveled against both designs.^15,20 The further dissemination of hierarchies of evidence may further increase the citations for meta-analyses and RCTs. If the proposal that each study should start and end when a meta-analysis is adopted,²¹ meta-analyses may become even more highly cited. Interestingly, high citations for meta-analyses extend to meta-analyses of nonrandomized research. Of course, we acknowledge that primary studies are required for quantitative syntheses ever to be performed.

The relative impact of epidemiological research has lost ground recently.Perhaps there is increasing uncertainty due to the refutation of several key cohort studies on important questions such as vitamins or hormone therapy.¹¹ Decision or cost-effectiveness analysis has also not managed to keep a high impact. Nevertheless, many important questions simply cannot be answered with randomized research.

Also many nonsystematic reviews continue to be published. In our study, we excluded nonquantitative reviews that seemed to use some systematic approaches. Empirical evaluations of orthopedic and general medical journals have shown that systematic reviews received double the number of citations compared with nonsystematic ones.^22,23 Efforts to enhance the accuracy and usefulness of all reviews are important because even nonsystematic expert reviews are still extensively read by practitioners.²⁴

Some caveats should be discussed. First, higher citation rates in articles published in 2001 than in those published in 1991 probably reflect simply the increase of journals worldwide (especially journal articles listed by ISI). Second, we excluded several types of reports such as nonhuman studies and hybrid designs (eg, reports describing both cohort and case-control studies). However, we wanted to focus sharply on the key study designs. Third, we did not exclude self-citations. Fourth, we used very strict screening criteria to ensure high specificity for characterizing study designs. Most studies probably still do not mention their design in their title. It is unknown whether among studies having the same design, those that state it in the title would get more citations or less. Nevertheless, even if such differences exist, they probably would not affect selectively some study designs over others.

Finally, a citation does not guarantee the respect of the citing investigators. Occasionally a study may be cited only to be criticized or dismissed. Nevertheless, citation still means that the study is active in the scientific debate. Moreover, we should acknowledge that citation impact does not necessarily translate into clinical or scientific impact, but this is extremely difficult to measure and could vary on a case-by-case basis. Allowing for these caveats, our evaluation provides empirical evidence on the relative impact of various study designs in the literature.

Corresponding Author: John P. A. Ioannidis, MD, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece (jioannid@cc.uoi.gr).

Author Contributions: Dr Ioannidis had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Ioannidis, Patsopoulos.

Acquisition of data: Ioannidis, Patsopoulos, Analatos.

Analysis and interpretation of data: Ioannidis, Patsopoulos, Analatos.

Drafting of the manuscript: Ioannidis, Patsopoulos.

Critical revision of the manuscript for important intellectual content: Analatos.

Statistical analysis: Ioannidis, Patsopoulos, Analatos.

Study supervision: Ioannidis.

Acknowledgment: We thank Tom Trikalinos, MD, for help in retrieving articles and Anastasios E. Germenis, MD, for encouragement.