[Skip to Content]
[Skip to Content Landing]
February 1, 1995

Empirical Evidence of Bias: Dimensions of Methodological Quality Associated With Estimates of Treatment Effects in Controlled Trials

Author Affiliations

From the Division of STD/HIV Prevention, National Center for Prevention Services, Centers for Disease Control and Prevention, Atlanta, Ga (Dr Schulz); the UK Cochrane Centre, Oxford, England (Drs Schulz and Chalmers); Department of Epidemiology and Population Sciences, London (England) School of Hygiene and Tropical Medicine (Dr Schulz and Mr Hayes); and Medical Statistics Laboratory, Imperial Cancer Research Fund, London (Mr Altman).

JAMA. 1995;273(5):408-412. doi:10.1001/jama.1995.03520290060030

Objective.  —To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects.

Design.  —An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects.

Data Sources.  —Meta-analyses from the Cochrane Pregnancy and Childbirth Database.

Main Outcome Measures.  —The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding.

Results.  —Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P<.001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P=.01), with odds ratios being exaggerated by 17%.

Conclusions.  —This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials.(JAMA. 1995;273:408-412)

Chalmers TC, Matta RJ, Smith H Jr, Kunzler AM.  Evidence favoring the use of anticoagulants in the hospital phase of acute myocardial infarction.  N Engl J Med . 1977;297:1091-1096.Crossref
Pocock SJ.  Allocation of patients to treatment in clinical trials.  Biometrics . 1979;35:183-197.Crossref
Sacks H, Chalmers TC, Smith H.  Randomized versus historical controls for clinical trials.  Am J Med . 1982;72:233-240.Crossref
Chalmers TC, Celano P, Sacks HS, Smith H Jr.  Bias in treatment assignment in controlled clinical trials.  N Engl J Med . 1983;309:1358-1361.Crossref
Miller JN, Colditz GA, Mosteller F.  How study design affects outcomes in comparisons of therapy, II: surgical.  Stat Med . 1989;8:455-466.Crossref
Colditz GA, Miller JN, Mosteller F.  How study design affects outcomes in comparisons of therapy, I: medical.  Stat Med . 1989;8:441-454.Crossref
Devine EC, Cook TD.  A meta-analytic analysis of effects of psychoeducational interventions on length of post-surgical hospital stay.  Nurs Res . 1983; 32:267-274.Crossref
Wortman PM, Yeaton WH.  Synthesis of results in controlled trials of coronary artery bypass graft surgery.  In: Light RJ, ed.  Evaluation Studies Review Annual . Beverly Hills, Calif: Sage; 1983;8:536-551.
Gillman MW, Runyan DK.  Bias in treatment assignment in controlled clinical trials .  N Engl J Med . 1984;310:1610-1611.
Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC.  An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials.  Control Clin Trials . 1990;11:339-352.Crossref
Chalmers TC, Smith H Jr, Blackburn B, et al.  A method for assessing the quality of a randomized control trial.  Control Clin Trials . 1981;2:31-49.Crossref
Chalmers I, ed.  Oxford Database of Perinatal Trials: Version 1.2, Disk Issue 8 . New York, NY: Oxford University Press Inc; 1992.
Greenland S.  A critical look at some popular meta-analytic methods.  Am J Epidemiol . 1994;140: 290-296.
Chalmers I, Enkin M, Keirse MJNC, eds.  Effective Care in Pregnancy and Childbirth . New York, NY: Oxford University Press Inc; 1989.
Enkin MW, Keirse MJNC, Renfrew MJ, Neilson JP.  Cochrane Database of Systematic Reviews . Oxford, England: Update Software; 1993.
Chalmers I, Hetherington J, Newdick M, et al.  The Oxford database of perinatal trials: developing a register of published reports of controlled trials.  Control Clin Trials . 1986;7:306-325.Crossref
Chalmers I, Hetherington J, Elbourne D, Keirse MJNC, Enkin M.  Materials and methods used in synthesizing evidence to evaluate the effects of care during pregnancy and childbirth.  In: Chalmers I, Enkin M, Keirse MJN, eds.  Effective Care in Pregnancy and Childbirth, Volume 1: Pregnancy . New York, NY: Oxford University Press Inc; 1989:39-65.
Schulz KF, Chalmers I, Grimes DA, Altman DG.  Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals.  JAMA . 1994;272:125-128.Crossref
Liberati A, Himel HN, Chalmers TC.  A quality assessment of randomized control trials of primary treatment of breast cancer.  J Clin Oncol . 1986;4: 942-951.
Dean AG, Dean JA, Burton AH, Dicker RC.  Epi Info, Version 5: A Word Processing, Database, and Statistics Program for Epidemiology on Microcomputers . Atlanta, Ga: Centers for Disease Control and Prevention; 1990.
Francis B, Green M, Payne C, et al.  GLIM 4: The Statistical System for Generalized Linear Interactive Modelling . New York, NY: Oxford University Press Inc; 1993.
Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbé KA.  Incorporating variations in the quality of individual randomized trials into meta-analysis.  J Clin Epidemiol . 1992;45:255-265.Crossref
Schulz KF.  Methodological Quality and Bias in Randomised Controlled Trials . London, England: University of London; 1994. Thesis.
Aitkin M, Anderson D, Francis B, Hinde J.  Statistical Modeling in GLIM . New York, NY: Oxford University Press Inc; 1989.
Mosteller F, Gilbert JP, McPeek B.  Reporting standards and research strategies for controlled trials.  Control Clin Trials . 1980;1:37-58.Crossref
DerSimonian R, Charette LJ, McPeek B, Mosteller F.  Reporting on methods in clinical trials.  N Engl J Med . 1982;306:1332-1337.Crossref
Altman DG, Doré CJ.  Randomisation and baseline comparisons in clinical trials.  Lancet . 1990;335: 149-153.Crossref
Peto R.  Why do we need systematic overviews of randomized trials?  Stat Med . 1987;6:233-244.Crossref
Altman DG.  Randomisation: essential for reducing bias.  BMJ . 1991;302:1481-1482.Crossref
Gøtzsche PC.  Multiple publication of reports of drug trials.  Eur J Clin Pharmacol . 1989;36:429-432.Crossref
The Standards of Reporting Trials Group.  A proposal for structured reporting of randomized controlled trials.  JAMA . 1994;272:1926-1931.Crossref