Clinical trials characterizing the effects of an experimental therapy rarely have only a single outcome of interest. In a previous report in JAMA,1 the CLEAN-TAVI investigators evaluated the benefits of a cerebral embolic protection device for stroke prevention during transcatheter aortic valve implantation. The primary end point was the reduction in the number of ischemic lesions observed 2 days after the procedure. The investigators were also interested in 16 secondary end points involving measurement of the number, volume, and timing of cerebral lesions in various brain regions. Statistically comparing a large number of outcomes using the usual significance threshold of .05 is likely to be misleading because there is a high risk of falsely concluding that a significant effect is present when none exists.2 If 17 comparisons are made when there is no true treatment effect, each comparison has a 5% chance of falsely concluding that an observed difference exists, leading to a 58% chance of falsely concluding at least 1 difference exists. The formula 1 −[1 −α]N can be used to calculate the chance of obtaining at least 1 falsely significant result, when there is no true underlying difference between the groups (in this case α is .05 and N is 17 for the number of tests).
Yadav K, Lewis RJ. Gatekeeping Strategies for Avoiding False-Positive Results in Clinical Trials With Many Comparisons. JAMA. 2017;318(14):1385–1386. doi:10.1001/jama.2017.13276