Prevalence of Multiplicity and Appropriate Adjustments Among Cardiovascular Randomized Clinical Trials Published in Major Medical Journals | Cardiology | JAMA Network Open | JAMA Network
[Skip to Navigation]
Table 1.  Characteristics of 511 Included Randomized Clinical Trials
Characteristics of 511 Included Randomized Clinical Trials
Table 2.  Multiplicity Adjustment
Multiplicity Adjustment
Table 3.  Methods Used to Adjust for Multiplicity
Methods Used to Adjust for Multiplicity
Table 4.  Comparisons of Reporting for Intervention Type, Funding Source, Trial Size, and Trial Type
Comparisons of Reporting for Intervention Type, Funding Source, Trial Size, and Trial Type
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Medical Journals and Publishing
    April 17, 2020

    Prevalence of Multiplicity and Appropriate Adjustments Among Cardiovascular Randomized Clinical Trials Published in Major Medical Journals

    Author Affiliations
    • 1Department of Medicine, John H. Stroger, Jr. Hospital of Cook County, Chicago, Illinois
    • 2Department of Medicine, Dow University of Health Sciences, Karachi, Pakistan
    • 3Department of Medicine, Guthrie Robert Packer Hospital, Sayre, Pennsylvania
    • 4Department of Medicine, Mayo Clinic, Rochester, Minnesota
    • 5Division of Cardiovascular Medicine, Department of Internal Medicine, University of Oklahoma Health Sciences Center, Oklahoma City
    • 6Division of Cardiovascular Medicine, Baptist Health Louisville, Louisville, Kentucky
    • 7Medical Research Council Biostatistics Unit, Department of Biostatistics, University of Cambridge, Cambridge, United Kingdom
    • 8Population Health Sciences Institute, Department of Biostatistics, Newcastle University, Newcastle upon Tyne, United Kingdom
    • 9Cardiovascular Division, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
    • 10Cardiology Section, Department of Medicine, Veterans Administration Boston Healthcare System, Boston, Massachusetts
    • 11Cardiovascular Research Foundation, Department of Cardiovascular Medicine, Columbia University Medical Center, New York, New York
    • 12Heart & Vascular Center, Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts
    • 13Heart, Vascular & Thoracic Institute, Tomsich Family Department of Cardiovascular Medicine, Cleveland Clinic, Cleveland, Ohio
    JAMA Netw Open. 2020;3(4):e203082. doi:10.1001/jamanetworkopen.2020.3082
    Key Points español 中文 (chinese)

    Questions  What is the prevalence of multiplicity among cardiovascular randomized clinical trials published in 6 medical journals with a high impact factor, and how frequently are multiplicity adjustments made in these trials?

    Findings  In this cross-sectional study, data were collected from the past issues of 6 journals with high impact factors published between August 1, 2015, and July 31, 2018. Of 511 cardiovascular randomized clinical trials included in this analysis, 300 had some form of multiplicity; of these 300, only 85 adjusted for multiplicity.

    Meaning  Among contemporary cardiovascular randomized clinical trials, it appears that multiplicity adjustments are infrequently reported.

    Abstract

    Importance  Multiple analyses in a clinical trial can increase the probability of inaccurately concluding that there is a statistically significant treatment effect. However, to date, it is unknown how many randomized clinical trials (RCTs) perform adjustments for multiple comparisons, the lack of which could lead to erroneous findings.

    Objectives  To assess the prevalence of multiplicity and whether appropriate multiplicity adjustments were performed among cardiovascular RCTs published in 6 medical journals with a high impact factor.

    Design, Setting, and Participants  In this cross-sectional study, cardiovascular RCTs were selected from all over the world, characterized as North America, Western Europe, multiregional, and rest of the world. Data were collected from past issues of 3 cardiovascular journals (Circulation, European Heart Journal, and Journal of the American College of Cardiology) and 3 general medicine journals (JAMA, The Lancet, and The New England Journal of Medicine) with high impact factors published between August 1, 2015, and July 31, 2018. Supplements and trial protocols of each of the included RCTs were also searched for multiplicity. Data were analyzed December 20 to 27, 2018.

    Exposures  Data from the selected RCTs were extracted and verified independently by 2 researchers using a structured data instrument. In case of disagreement, a third reviewer helped to achieve consensus. An RCT was considered to have multiple treatment groups if it had more than 2 arms; multiple outcomes were defined as having more than 1 primary outcome, and multiple analyses were defined as analysis of the same outcome variable in multiple ways. Multiplicity was examined only for the analysis of the primary end point.

    Main Outcomes and Measures  Outcomes of interest were percentages of primary analyses that performed multiplicity adjustment of primary end points.

    Results  Of 511 cardiovascular RCTs included in this analysis, 300 (58.7%) had some form of multiplicity; of these 300, only 85 (28.3%) adjusted for multiplicity. Intervention type and funding source had no statistically significant association with the reporting of multiplicity risk adjustment. Trials that assessed mortality vs nonmortality outcomes were more likely to contain a multiplicity risk in their primary analysis (66.3% [177 of 267] vs 50.4% [123 of 244]; P < .001), and larger trials vs smaller trials were less likely to make any adjustments for multiplicity (35.6% [52 of 146] vs 21.4% [33 of 154]; P = .001).

    Conclusions and Relevance  Findings from this study suggest that cardiovascular RCTs published in medical journals with high impact factors demonstrate infrequent adjustments to correct for multiple comparisons in the primary end point. These parameters may be improved by more standardized reporting.

    Introduction

    Previous studies1-4 have raised concerns about selective reporting of outcomes in randomized clinical trials (RCTs). However, few reports have focused on multiplicity, which (along with incomplete reporting) is a major factor contributing to nonreproducibility of published claims.5 Multiplicity refers to the “potential inflation of type I error rate as a result of multiple testing, for example because of multiple subgroup comparisons, comparisons across multiple treatment arms, analysis of multiple outcomes, and multiple analyses of the same outcome at different times.”6

    Negative consequences associated with multiplicity could be prevented by complete and accurate reporting of analyses outlined in the registered trial protocols. Multiplicity could also be mitigated by statistical adjustment when multiple analyses are specified a priori. Several statistical methods, such as defining coprimary outcome variables, performing various stepwise procedures,7-10 applying methods for multiple-group comparisons11,12 and including gatekeeping or hierarchical testing, have been proposed for multiplicity adjustment.13,14

    To our knowledge, no study has reported on the prevalence of multiplicity among cardiovascular RCTs and, when applicable, whether appropriate multiplicity adjustments were implemented. To fill this knowledge gap, we conducted a cross-sectional study of cardiovascular RCTs published in medical journals with high impact factors to assess the reporting quality of statistical analyses, including the frequency with which multiplicity adjustments were reported.

    Methods

    This cross-sectional study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.15 It also followed methods from the American Heart Association on standards for cardiac prevention and treatment studies.16

    Data Sources and Search Strategy

    Three cardiovascular journals (Circulation, European Heart Journal, and Journal of the American College of Cardiology) and 3 general medicine journals (JAMA, The Lancet, and The New England Journal of Medicine) published between August 1, 2015, and July 31, 2018, were searched for general trial characteristics, multiplicity error, and multiplicity correction to assess the pool of recent and contemporary cardiovascular clinical trials. Data were analyzed December 20 to 27, 2018. These journals were chosen based on their high impact factor, broad readership, and reputation of publishing important clinical trials used in the development of guidelines. Supplements and trial protocols of each of the included RCTs were also searched for general trial characteristics, multiplicity error, and multiplicity correction.

    Study Selection

    Articles were selected if they reported results of cardiovascular RCTs and compared at least 2 treatment groups. Excluded were brief communications, research letters, and animal studies. Data from the selected RCTs were extracted and verified by 2 of us (M.S.K. and Z.N.A.) independently using a structured data instrument and then cross-checked by another of us (T.J.S).

    Data Extraction

    Data were extracted from both the primary and secondary articles. Primary articles were defined as reports on an empirical research study conducted by the authors analyzing data collected for the first time, while secondary articles were studies derived from data collected and analyzed from primary articles. We analyzed and extracted data only from the analysis of the primary end point of each RCT because multiplicity in a secondary analysis is generally exploratory or hypothesis generating. A multiplicity coding manual was developed to investigate the reporting of primary statistical analyses, multiple analyses, and adjustments for multiplicity issues (eAppendix in the Supplement). The multiplicity coding manual was pretested and modified by coding 15 articles initially. Two of us (M.S.K. and Z.N.A.) coded each article separately and discussed any inconsistencies in the data and modified the multiplicity coding manual accordingly. The rest of the articles were then coded according to this multiplicity coding manual. The complete published articles were searched for general trial characteristics, multiplicity error, and multiplicity correction by the coders, along with additional supplementary material (eg, trial protocols and appendixes if they were referred to in the article). The order of the articles was randomized for each coder.

    To measure the extent of agreement between the 2 independent coders, the κ statistic was used and calculated according to the methods by Landis and Koch.17 The frequency of discrepancies between the coders was computed using the Kappa Calculator (Statistics Solutions),18 and the κ statistics were assessed for several outcomes. There was substantial agreement in reproducibility for the presence of multiplicity, with κ = 0.76 (95% CI, 0.51-0.90) in the main text and κ = 0.78 (95% CI, 0.55-0.89) after adjusting for these multiplicity errors. Overall interobserver agreement in extracting data was good, and any discrepancies were resolved after discussion. When consensus could not be reached, another of us (T.J.S.) arbitrated. Finally, a post hoc search was done to assess if the authors of articles stated that their trial was exploratory or hypothesis generating in any section of the article.

    General RCT Characteristics

    The following information was extracted from the RCTs: (1) the number of randomized participants; (2) region of the world where the trial was conducted (North America, Western Europe, multiregional, or rest of the world [multiregional was defined as any trial that had multiple sites across the world, and rest of the world was defined as any trial having sites in a region that was not located in either North America or Western Europe]); (3) intervention type (drugs, procedures [eg, a different approach or method of implementing treatment], medical devices, surgery, testing or imaging, or other [eg, diet]); and (4) funding source. Trial size was extracted (as a proxy for trial phase considering its inconsistent definition) and categorized as small (≤500 participants per group) or large (>500 participants per group). Trial type (prespecification of a primary end point) was categorized and extracted as either a mortality trial (defined as any trial where the primary outcome was mortality during treatment) or a nonmortality trial.

    Outcome Data

    Data were also extracted for whether the article had risk of multiplicity (ie, contained multiple analyses, a term that encompasses any of the following: multiple treatment groups, multiple outcome variables, and multiple analyses of the same outcome variable) and whether the authors defined the methods used for multiplicity correction. An RCT was considered to have multiple treatment groups if it had more than 2 arms, multiple outcome variables were defined as having more than 1 primary outcome, and multiple analyses were defined as analysis of the same outcome variable in multiple ways. All 3 of these scenarios were weighted equally. We considered multiplicity adjustment sufficient when an article outlined that it attempted to adjust for multiple comparisons.

    Statistical Analysis

    Descriptive statistics were used to assess the proportion of RCTs with (1) multiple primary analyses and (2) a multiplicity adjustment for the analysis of the primary end point. Also recorded were the class of multiplicity in the primary analysis (multiple treatment groups, multiple outcome variables, or multiple analyses of the same outcome variable) and frequencies of each of the methods used to adjust for multiplicity in the primary analysis. Multiplicity was examined only for the analysis of the primary end point. It was deemed unnecessary for secondary analyses, which are generally exploratory or hypothesis generating. Outcomes of interest were percentages of primary analyses that performed multiplicity adjustment of primary end points. Two-sided χ2 tests were used to examine the association between (1) intervention type, (2) funding source, (3) trial size, and (4) trial type; it was noted whether risk for a familywise error because of multiple comparisons was present and whether the RCT adjusted for multiple comparisons. The method described by Holm8 adjusts for multiple comparisons between type of intervention type, funding source, trial size, and trial type. According to this method, the smallest P value from all planned comparisons is compared with a significance level of .05 divided by K, where K represents the number of comparisons to be made. If the null hypothesis is rejected, the next smallest P value is compared with a significance level of P = .05 divided by K minus 1, and so on until the null hypothesis can no longer be rejected. In this scenario, a total of 8 comparisons were made; therefore, the significance level was set to α = .006 (according to .05 ÷ by K − 1, where K = 8 in this case) in the initial step and to α = .05 in the last step (α being the Holm-corrected significance level). A statistical software package (SPSS, version 23; IBM) was used for all analyses.

    Results
    Literature Search

    The initial search identified 2166 trials, which were transferred to a reference management software program (EndNote, Clarivate Analytics). The titles and abstracts of the identified studies were then screened to exclude irrelevant studies. Full-text studies were subsequently obtained and evaluated for the remaining 1273 reports. After assessing for relevance, 511 articles were included in the final analysis.

    Study Characteristics

    Of 511 cardiovascular RCTs included in this analysis, 123 (24.1%) were published in Journal of the American College of Cardiology, 112 (21.9%) in Circulation, 107 (20.9%) in European Heart Journal, 71 (13.9%) in The New England Journal of Medicine, 55 (10.8%) in The Lancet, and 43 (8.4%) in JAMA (Table 1). Approximately half (248 [48.5%]) of the trials were industry funded, and approximately half (243 [47.6%]) were large trials. Approximately half (251 [49.1%]) of the trials used a drug intervention. A total of 229 trials (44.8%) made use of composite outcomes as their primary outcome variable.

    Risk of Multiplicity

    Of 511 cardiovascular RCTs included in this analysis, 300 (58.7%) had some form of multiplicity (282 of 511 [55.2%] did not mention whether they did or did not adjust for multiplicity). Of these 300 trials, 81 (27.0%) had multiple treatment groups, 45 (15.0%) identified multiple outcome variables as primary, 170 (56.7%) had multiple analyses of the same outcome variable, 3 (1.0%) had multiple treatment groups and multiple outcome variables, and 1 (0.3%) had multiple treatment groups and multiple analyses (Table 2).

    Multiplicity Adjustment

    Among 300 RCTs, only 85 (28.3%) adjusted for multiplicity for all primary analyses (Table 2). Of 511 trials, 289 (56.6%) did not mention whether they did or did not attempt to adjust for multiple comparisons. Forty-one trials (48.2%) had multiple analyses of the same outcome variable that adjusted for multiplicity, 22 (25.9%) had multiple treatment groups that adjusted for multiplicity, and 19 (22.4%) had multiple outcome variables that adjusted for multiplicity. The individual multiplicity correction tests are also listed in Table 3.

    Limitation Specified of Performing Multiplicity Adjustment

    Of 300 trials with multiplicity error risk, 19 (6.3%) were exploratory or hypothesis generating. Twelve of these trials mentioned this exploratory nature in the Discussion section of the article, 5 mentioned it in the Methods section, and 2 mentioned it in more than 1 section of the article. Of the 85 trials that adjusted for multiplicity, 68 (80.0%) mentioned that they adjusted for multiplicity in the main text of the article, and 17 (20.0%) only mentioned it in the supplement or trial protocol.

    Determinants of Performing Multiplicity

    Intervention type and funding source had no statistically significant association with the reporting of multiplicity risk adjustment (Table 4). Trials that assessed mortality vs nonmortality outcomes were more likely to contain a multiplicity risk in their primary analysis (66.3% [177 of 267] vs 50.4% [123 of 244]; P < .001). Although larger trials had no association with specifying an analysis of the primary end point or containing a multiplicity error risk within their analysis, they were less likely than smaller trials to make any adjustments to correct for multiplicity issues (35.6% [52 of 146] vs 21.4% [33 of 154]; P = .001). All of these results were statistically significant after application of the Holm test.

    Discussion

    Our report demonstrates that 58.7% of 511 cardiovascular RCTs included in this analysis contained multiple analyses within their methods and that 55.2% of the total RCTs did not report whether they adjusted for multiple comparisons. Trials that assessed mortality were more likely than nonmortality trials to have some form of multiplicity, which is not unexpected because mortality is usually not the sole end point. However, because of the exigent nature of the mortality component and because some researchers consider mortality a safety end point as well, authors might be inclined to claim an association even if the overall end point fails to show effectiveness. These results have important implications for the performance and interpretation of cardiovascular RCTs.

    Articles mentioning that they did not adjust for multiplicity often provided some justification, such as stating that their study was exploratory or hypothesis generating. Some justifications were unique to the trial; for example, 1 article mentioned that a chance finding could not be ruled out because of multiple testing and the sample size of the subgroups.19 It is possible that results might change before and after multiplicity adjustment; for example, in a trial where P = .046 for the primary outcome, the adjusted P value may have been different after multiplicity correction.20

    Among 85 of 511 included articles that adjusted for multiplicity for all primary analyses (Table 3), half of the trials used a composite outcome as their primary outcome. Composite outcomes allow increased statistical precision and efficiency with fewer participants to detect a statistically significant difference among comparators,21 especially in the case of total mortality, which is a rare event requiring more power and an extended follow-up to show a difference between interventions.22 Although the use of composite end points is acceptable and in some instances beneficial, it may also increase the risk of introducing a multiplicity error if the observed treatment effect was associated with a softer clinical end point.21,23 For example, a trial where all-cause mortality, myocardial infarction, and recurrent angina are components of a composite end point, recurrent angina might be considered the softest of the 3 components. The present study did not examine the details of each composite end point of every trial or consider composite end points to be a source of multiplicity; however it suggests the need for future investigators and researchers to apply methods to avoid the possibility of a multiplicity error, as described by Sankoh et al.21

    Uncertainty in interpreting research results is common and may be attributable to a lack of statistical power or the use of questionable research practices, or it may reflect decisions a researcher makes to conduct a trial.24 These uncertainties might explain gaps in the reporting of multiplicity and adjustments made. Conversely, one could also argue that such gaps are less a reflection of multiplicity issues but rather reflect the unavailability of the trial protocol and statistical analysis plan. We suggest that all RCTs in medical journals should describe the trial protocol–specific analytic plan, including the methods used to adjust for multiple comparisons or acknowledgment of the lack of correction for multiplicity. We believe that this inclusion is especially relevant because most clinicians do not have statistical expertise.

    Because the criteria used to classify trial phase (ie, phase 1, 2, or 3) were inconsistent among the RCTs in this study, trial size was used as a proxy for trial phase. For drug interventions, smaller trials (≤500 participants per group) may more likely reflect early to middle stages of development, and larger trials (>500 participants per group) may more likely reflect confirmatory stages.25,26 Among 161 RCTs, Gewandter et al27 found no association between trial size and funding source, with multiplicity adjustment most likely because of the limited power of a study to perform such an analysis. Our analysis included a larger sample of both industry-sponsored RCTs (n = 248) and large RCTs with multiplicity issues (n = 154) (Table 4). We found that smaller trials were more likely to be adjusted for multiplicity. Funding source had no association with adjusting for multiplicity. This observation suggests that RCTs of drugs in early to middle stages of development may be more likely to adjust for multiplicity.

    The appropriateness of testing procedures is guided by information on statistical features of a study design or analytic strategy and differs depending on whether there is a single source of multiplicity or several sources and whether there are multiple treatment groups, multiple outcome variables, or multiple analyses of the same outcome variable. Dmitrienko and D’Agostino23 provide some guidance on how to choose the most appropriate test for multiplicity corrections; they state that nonparametric tests, such as the Holm test, can be applied to most multiplicity problems involving a single source of multiplicity. In cases where the association between statistical tests is known, such as in clinical trials with several dose-control comparisons and patient populations, more specific parametric tests, such as the Dunnett test, may be applied. In an effort to better explain types of multiple analyses and multiple outcome variables, detailed examples are listed in eTable 1 in the Supplement.

    Limitations

    This study has several limitations. First, we assessed the reporting quality of the methods used for multiplicity adjustments but not necessarily the quality of statistical practices used. Because the methods may have been prespecified but not stated in the articles,28,29 this report is subject to reporting bias. Second, studies were recorded as having adjusted for multiplicity in the primary analysis only if the authors adjusted for all instances of multiplicity. However, this approach does not consider the trials that tried to adjust for some but not all sources of multiple comparisons. Also, whether a study had multiple treatment groups, multiple outcome variables, or multiple analyses of the same outcome variable, all had the same weight in terms of adjusting for multiplicity and thus were considered equally. A third limitation is that we only evaluated the primary outcomes. It is important to note that the secondary end point in a sequence may often influence a conclusion. For instance, if a trial finds a statistically significant difference in major adverse cardiovascular events (myocardial infarction, stroke, and admission for heart failure) and the next end point in the sequence is the myocardial infarction rate, which is not statistically significant, it would be incorrect to conclude a nominally statistically significant stroke association. In addition, we were unable to differentiate our analysis by trial type. Although the objective of this study was to evaluate the overall prevalence of multiplicity among cardiovascular RCTs, it must be remembered that phase 3 trials hold the most importance from a public health perspective, and multiplicity is of lesser concern in phase 2 trials.

    Conclusions

    This cross-sectional study found frequent inconsistencies associated with multiplicity in primary analysis reporting among cardiovascular RCTs published in medical journals with a high impact factor. These findings adversely reflect on the robustness of data published in journals that carry global reach and generate evidence that can transform clinical guidelines and practice. Our findings suggest that investigators should be encouraged to adjust for multiplicity when warranted. Practical guidelines for multiplicity adjustment in clinical trials (eg, recommendations by Proschan and Waclawiw30) can be consulted. We think that this information should ideally be prespecified in the Methods section of clinical trials before unblinding of the study data (eTable 2 in the Supplement). We believe that it should be the collective responsibility of journal editors, peer reviewers, and readers to pay close attention to the Methods and Statistical Analysis sections of articles reporting clinical trial results to ensure that multiplicity issues have been addressed.

    Back to top
    Article Information

    Accepted for Publication: February 17, 2020.

    Published: April 17, 2020. doi:10.1001/jamanetworkopen.2020.3082

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Khan MS et al. JAMA Network Open.

    Corresponding Author: Ankur Kalra, MD, Vascular and Thoracic Institute, Tomsich Family Department of Cardiovascular Medicine, Cleveland Clinic, Cleveland Clinic Akron General, 224 W Exchange St, Ste 225, Akron, OH 44302 (kalraa@ccf.org).

    Author Contributions: Drs Maaz Shah Khan and Ansari had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Muhammad Shahzeb Khan, Siddiqi, Riaz, Stone, Kalra.

    Acquisition, analysis, or interpretation of data: Maaz Shah Khan, Ansari, Siddiqi, S.U. Khan, Asad, Mandrola, Wason, Warraich, Bhatt, Kapadia, Kalra.

    Drafting of the manuscript: Muhammad Shahzeb Khan, Maaz Shah Khan, Ansari, Siddiqi, Asad, Mandrola, Kalra.

    Critical revision of the manuscript for important intellectual content: Muhammad Shahzeb Khan, Ansari, Siddiqi, S.U. Khan, Riaz, Asad, Wason, Warraich, Stone, Bhatt, Kapadia, Kalra.

    Statistical analysis: Muhammad Shahzeb Khan, Maaz Shah Khan, Siddiqi, Wason.

    Obtained funding: Kalra.

    Administrative, technical, or material support: Maaz Shah Khan, Ansari, Asad, Mandrola, Kalra.

    Supervision: Muhammad Shahzeb Khan, Asad, Warraich, Stone, Kapadia, Kalra.

    Conflict of Interest Disclosures: Dr Stone reported receiving personal fees or other support from Abiomed, Ablative Solutions, Ancora, Applied Therapeutics, ARIA, BioStar, Cagent Vascular, Cardiac Success, Cook, Gore, HeartFlow, MAIA Pharmaceuticals, Matrizyme, MedFocus, Miracor, Neovasc, Orchestra BioMed, Qool Therapeutics, REVA, Robocath, Shockwave, SpectraWAVE, Teruma, TherOx, Valfix, Vascular Dynamics, Vectorious, and V-Wave. Dr Bhatt reported receiving grants, travel support, or personal fees from or having other relationships (board or committee membership, chair, editor, or trustee) with Abbott, Afimmune, Amarin, American College of Cardiology, American Heart Association, Amgen, AstraZeneca, Baim Institute for Clinical Research, Bayer, Belvoir Publications, Biotronik, Boehringer Ingelheim, Boston Scientific, Boston VA Research Institute, Bristol-Myers Squibb, Cardax, Cereno Scientific, Chiesi, Cleveland Clinic, Clinical Cardiology, CSI, CSL Behring, Duke Clinical Research Institute, Eisai, Elsevier, Ethicon, Ferring Pharmaceuticals, FlowCo, Forest Laboratories/AstraZeneca, Fractyl, Harvard Clinical Research Institute (now Baim Institute for Clinical Research), HMP Global, Idorsia, Ironwood, Ischemix, Journal of the American College of Cardiology, Lexicon, Lilly, Mayo Clinic, Medscape Cardiology, Medtelligence/ReachMD, Medtronic, Merck, Mount Sinai School of Medicine, Novo Nordisk, Pfizer, PhaseBio, PLx Pharma, Population Health Research Institute, Regado Biosciences, Regeneron, Roche, Sanofi-Aventis, Slack Publications, Society of Cardiovascular Patient Care, St Jude Medical (now Abbott), Svelte, Synaptic, Takeda, The Medicines Company, TobeSoft, US Department of Veterans Affairs, and WebMD. No other disclosures were reported.

    References
    1.
    Gandhi  R, Jan  M, Smith  HN, Mahomed  NN, Bhandari  M.  Comparison of published orthopaedic trauma trials following registration in ClinicalTrials.gov.   BMC Musculoskelet Disord. 2011;12(1):278. doi:10.1186/1471-2474-12-278 PubMedGoogle ScholarCrossref
    2.
    Huić  M, Marušić  M, Marušić  A.  Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy.   PLoS One. 2011;6(9):e25258. doi:10.1371/journal.pone.0025258 PubMedGoogle Scholar
    3.
    Mathieu  S, Boutron  I, Moher  D, Altman  DG, Ravaud  P.  Comparison of registered and published primary outcomes in randomized controlled trials.   JAMA. 2009;302(9):977-984. doi:10.1001/jama.2009.1242 PubMedGoogle ScholarCrossref
    4.
    Milette  K, Roseman  M, Thombs  BD.  Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review.   J Psychosom Res. 2011;70(3):205-217. doi:10.1016/j.jpsychores.2010.09.015 PubMedGoogle ScholarCrossref
    5.
    Goodman  SN, Fanelli  D, Ioannidis  JP.  What does research reproducibility mean?   Sci Transl Med. 2016;8(341):341ps12. doi:10.1126/scitranslmed.aaf5027PubMedGoogle Scholar
    6.
    Li  G, Taljaard  M, Van den Heuvel  ER,  et al.  An introduction to multiplicity issues in clinical trials: the what, why, when and how.   Int J Epidemiol. 2017;46(2):746-755.PubMedGoogle Scholar
    7.
    Dmitrienko  A, Tamhane  AC.  Gatekeeping procedures with clinical trial applications.   Pharm Stat. 2007;6(3):171-180. doi:10.1002/pst.291 PubMedGoogle ScholarCrossref
    8.
    Holm  S.  A simple sequentially rejective multiple test procedure.   Scand J Stat. 1979;6(2):65-70.Google Scholar
    9.
    Hochberg  Y.  A sharper Bonferroni procedure for multiple tests of significance.   Biometrika. 1988;75(4):800-802. doi:10.1093/biomet/75.4.800 Google ScholarCrossref
    10.
    Hommel  G.  A stagewise rejective multiple test procedure based on a modified Bonferroni test.   Biometrika. 1988;75(2):383-386. doi:10.1093/biomet/75.2.383 Google ScholarCrossref
    11.
    Tukey  JW.  Exploratory Data Analysis. Addison-Wesley; 1977.
    12.
    Dunnett  CW.  A multiple comparison procedure for comparing several treatments with a control.   J Am Stat Assoc. 1955;50(272):1096-1121. doi:10.1080/01621459.1955.10501294 Google ScholarCrossref
    13.
    Dmitrienko  A, Wiens  BL, Tamhane  AC, Wang  X.  Tree-structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives [published correction appears in Stat Med. 2008;27(17):3452].   Stat Med. 2007;26(12):2465-2478. doi:10.1002/sim.2716 PubMedGoogle ScholarCrossref
    14.
    Westfall  PH, Krishen  A.  Optimally weighted, fixed sequence and gatekeeper multiple testing procedures.   J Stat Plan Inference. 2001;99(1):25-40. doi:10.1016/S0378-3758(01)00077-5 Google ScholarCrossref
    15.
    von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010 PubMedGoogle ScholarCrossref
    16.
    Rao  G, Lopez-Jimenez  F, Boyd  J,  et al; American Heart Association Council on Lifestyle and Cardiometabolic Health; Council on Cardiovascular and Stroke Nursing; Council on Cardiovascular Surgery and Anesthesia; Council on Clinical Cardiology; Council on Functional Genomics and Translational Biology; and Stroke Council.  Methodological standards for meta-analyses and qualitative systematic reviews of cardiac prevention and treatment studies: a scientific statement from the American Heart Association.   Circulation. 2017;136(10):e172-e194. doi:10.1161/CIR.0000000000000523 PubMedGoogle ScholarCrossref
    17.
    Landis  JR, Koch  GG.  The measurement of observer agreement for categorical data.   Biometrics. 1977;33(1):159-174. doi:10.2307/2529310 PubMedGoogle ScholarCrossref
    18.
    Statistics Solutions. Kappa Calculator. Accessed August 16, 2019. https://www.statisticssolutions.com/kappa-calculator/
    19.
    Steffel  J, Giugliano  RP, Braunwald  E,  et al.  Edoxaban versus warfarin in atrial fibrillation patients at risk of falling: ENGAGE AF–TIMI 48 analysis [published correction appears in J Am Coll Cardiol. 2017;70(4):512-513].   J Am Coll Cardiol. 2016;68(11):1169-1178. doi:10.1016/j.jacc.2016.06.034 PubMedGoogle ScholarCrossref
    20.
    Bobrow  K, Farmer  AJ, Springer  D,  et al.  Mobile phone text messages to support treatment adherence in adults with high blood pressure (SMS-Text Adherence Support [StAR]): a single-blind, randomized trial.   Circulation. 2016;133(6):592-600.PubMedGoogle Scholar
    21.
    Sankoh  AJ, Li  H, D’Agostino  RB  Sr.  Use of composite endpoints in clinical trials.   Stat Med. 2014;33(27):4709-4714. doi:10.1002/sim.6205 PubMedGoogle ScholarCrossref
    22.
    Khan  SU, Rahman  H, Talluri  S, Kaluski  E.  The clinical benefits and mortality reduction associated with catheter ablation in subjects with atrial fibrillation: a systematic review and meta-analysis.   JACC Clin Electrophysiol. 2018;4(5):626-635. doi:10.1016/j.jacep.2018.03.003 PubMedGoogle ScholarCrossref
    23.
    Dmitrienko  A, D’Agostino  RB  Sr.  Multiplicity considerations in clinical trials.   N Engl J Med. 2018;378(22):2115-2122. doi:10.1056/NEJMra1709701 PubMedGoogle ScholarCrossref
    24.
    Silberzahn  R, Uhlmann  EL, Martin  DP,  et al.  Many analysts, one data set: making transparent how variations in analytic choices affect results.   Adv Methods Pract Psychol Sci. 2018;1(3):337-356. doi:10.1177/2515245917747646 Google ScholarCrossref
    25.
    Moore  RA.  What works for whom? determining the efficacy and harm of treatments for pain.   Pain. 2013;154(suppl 1):S77-S86. doi:10.1016/j.pain.2013.03.024 PubMedGoogle ScholarCrossref
    26.
    Moore  A, Eccleston  C, Derry  S,  et al; ACTINPAIN Writing Group of the IASP Special Interest Group on Systematic Reviews in Pain Relief; Cochrane Pain, Palliative and Supportive Care Systematic Review Group Editors.  “Evidence” in chronic pain: establishing best practice in the reporting of systematic reviews.   Pain. 2010;150(3):386-389. doi:10.1016/j.pain.2010.05.011 PubMedGoogle ScholarCrossref
    27.
    Gewandter  JS, Smith  SM, McKeown  A,  et al.  Reporting of primary analyses and multiplicity adjustment in recent analgesic clinical trials: ACTTION systematic review and recommendations.   Pain. 2014;155(3):461-466. doi:10.1016/j.pain.2013.11.009 PubMedGoogle ScholarCrossref
    28.
    Hill  CL, LaValley  MP, Felson  DT.  Discrepancy between published report and actual conduct of randomized clinical trials.   J Clin Epidemiol. 2002;55(8):783-786. doi:10.1016/S0895-4356(02)00440-7 PubMedGoogle ScholarCrossref
    29.
    Soares  HP, Daniels  S, Kumar  A,  et al; Radiation Therapy Oncology Group.  Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group.   BMJ. 2004;328(7430):22-24. doi:10.1136/bmj.328.7430.22 PubMedGoogle ScholarCrossref
    30.
    Proschan  MA, Waclawiw  MA.  Practical guidelines for multiplicity adjustment in clinical trials.   Control Clin Trials. 2000;21(6):527-539. doi:10.1016/S0197-2456(00)00106-9 PubMedGoogle ScholarCrossref
    ×