[Skip to Content]
[Skip to Content Landing]
JAMA Guide to Statistics and Methods
October 10, 2017

Gatekeeping Strategies for Avoiding False-Positive Results in Clinical Trials With Many Comparisons

Author Affiliations
  • 1Department of Emergency Medicine, Harbor-UCLA Medical Center, Torrance, California
  • 2Los Angeles Biomedical Research Institute, Torrance, California
  • 3Berry Consultants, LLC, Austin, Texas
JAMA. 2017;318(14):1385-1386. doi:10.1001/jama.2017.13276

Clinical trials characterizing the effects of an experimental therapy rarely have only a single outcome of interest. In a previous report in JAMA,1 the CLEAN-TAVI investigators evaluated the benefits of a cerebral embolic protection device for stroke prevention during transcatheter aortic valve implantation. The primary end point was the reduction in the number of ischemic lesions observed 2 days after the procedure. The investigators were also interested in 16 secondary end points involving measurement of the number, volume, and timing of cerebral lesions in various brain regions. Statistically comparing a large number of outcomes using the usual significance threshold of .05 is likely to be misleading because there is a high risk of falsely concluding that a significant effect is present when none exists.2 If 17 comparisons are made when there is no true treatment effect, each comparison has a 5% chance of falsely concluding that an observed difference exists, leading to a 58% chance of falsely concluding at least 1 difference exists. The formula 1 −[1 −α]N can be used to calculate the chance of obtaining at least 1 falsely significant result, when there is no true underlying difference between the groups (in this case α is .05 and N is 17 for the number of tests).