Context.— The extent of concordance between meta-analyses and large trials on
the same topic has been investigated with different protocols. Inconsistent
conclusions created confusion regarding the validity of these major tools
of clinical evidence.
Objective.— To evaluate protocols comparing meta-analyses and large trials in order
to understand if and why they disagree on the concordance of these 2 clinical
research methods.
Design.— Systematic comparison of protocol designs, study selection, definitions
of agreement, analysis methods, and reported discrepancies between large trials
and meta-analyses.
Results.— More discrepancies were claimed when large trials were selected from
influential journals (which may prefer trials disagreeing with prior evidence)
than from already performed meta-analyses (which may target homogeneous trials)
and when both primary and secondary (rather than only primary) end points
were considered. Depending on how agreement was defined, kappa coefficients
varied from 0.22 (low agreement) to 0.72 (excellent agreement). The correlation
of treatment effects between large trials and meta-analyses varied from −0.12
to 0.76, but was more similar (0.50-0.76) when only primary end points were
considered. When both the magnitude and uncertainty of treatment effects were
considered, large trials disagreed with meta-analyses 10% to 23% of the time.
Discrepancies were attributed to different disease risks, variable protocols,
quality, and publication bias.
Conclusions.— Comparisons of large trials with meta-analyses may reach different conclusions
depending on how trials and meta-analyses are selected and how end points
and agreement are defined. Scrutiny of these 2 major research methods can
enhance our appreciation of both for guiding medical practice.