Author Affiliations: Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina (Drs Ioannidis and Contopoulos-Ioannidis, and Ms Haidich), Department of Hygiene and Epidemiology, University of Athens School of Medicine (Ms Pappa and Mr Pantazis) and Laikon General Hospital (Drs Kokori and Tektonidou), Athens, Greece; Department of Pediatrics, George Washington University School of Medicine, Washington, DC (Dr Contopoulos-Ioannidis); and Division of Clinical Care Research, Department of Medicine, Tufts University School of Medicine, Boston, Mass (Drs Ioannidis and Lau).
Context There is substantial debate about whether the results of nonrandomized
studies are consistent with the results of randomized controlled trials on
the same topic.
Objectives To compare results of randomized and nonrandomized studies that evaluated
medical interventions and to examine characteristics that may explain discrepancies
between randomized and nonrandomized studies.
Data Sources MEDLINE (1966–March 2000), the Cochrane Library (Issue 3, 2000),
and major journals were searched.
Study Selection Forty-five diverse topics were identified for which both randomized
trials (n = 240) and nonrandomized studies (n = 168) had been performed and
had been considered in meta-analyses of binary outcomes.
Data Extraction Data on events per patient in each study arm and design and characteristics
of each study considered in each meta-analysis were extracted and synthesized
separately for randomized and nonrandomized studies.
Data Synthesis Very good correlation was observed between the summary odds ratios of
randomized and nonrandomized studies (r = 0.75; P<.001); however, nonrandomized studies tended to show
larger treatment effects (28 vs 11; P = .009). Between-study
heterogeneity was frequent among randomized trials alone (23%) and very frequent
among nonrandomized studies alone (41%). The summary results of the 2 types
of designs differed beyond chance in 7 cases (16%). Discrepancies beyond chance
were less common when only prospective studies were considered (8%). Occasional
differences in sample size and timing of publication were also noted between
discrepant randomized and nonrandomized studies. In 28 cases (62%), the natural
logarithm of the odds ratio differed by at least 50%, and in 15 cases (33%),
the odds ratio varied at least 2-fold between nonrandomized studies and randomized
Conclusions Despite good correlation between randomized trials and nonrandomized
studies—in particular, prospective studies—discrepancies beyond
chance do occur and differences in estimated magnitude of treatment effect
are very common.
Ioannidis JPA, Haidich A, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, Contopoulos-Ioannidis DG, Lau J. Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies. JAMA. 2001;286(7):821-830. doi:10.1001/jama.286.7.821