[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Original Contribution
June 12, 2002

Correlation of Quality Measures With Estimates of Treatment Effect in Meta-analyses of Randomized Controlled Trials

Author Affiliations

Author Affiliations: Evidence-based Practice Center, Division of Clinical Care Research, Tufts University School of Medicine, New England Medical Center, Boston, Mass (Drs Balk, Bonis, Moskowitz, Schmid, Wang, and Lau); and the Biomedical Research Institute, Foundation for Research and Technology Hellas, Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece (Dr Ioannidis). Dr Moskowitz is now with the Division of General Pediatrics, Mount Sinai Hospital, Mount Sinai School of Medicine, New York, NY.

JAMA. 2002;287(22):2973-2982. doi:10.1001/jama.287.22.2973

Context Specific features of trial quality may be associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas.

Objective To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs).

Design Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale.

Main Outcome Measures Relative ORs comparing overall treatment effect (summary OR) of high vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies.

Results Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent.

Conclusions Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses.