Noninferiority clinical trials are designed to determine whether an intervention is not worse than a comparator by a prespecified difference, known as the noninferiority margin (ΔNI). These trials are useful when comparing standard therapies with novel treatments that may be easier to use, are less costly, or have fewer adverse effects.1 The number of noninferiority trials is increasing; however, particularly compared with superiority trials, the interpretation of noninferiority trials is not straightforward. Superiority trials can be classified as positive or negative based on whether the null hypothesis can be rejected (ie, P < .05). In contrast, noninferiority trials can be interpreted as noninferior, inconclusive, or not noninferior according to the location of the 95% CIs in relation to the prespecified margin (Figure). Inconclusive and not noninferior results are often both presented as “negative” results (eg, failure to meet noninferiority criteria). This can lead readers to erroneously interpret an inconclusive trial as showing evidence that the novel treatment is worse, while in reality the findings are indeterminate and further research is required to determine noninferiority.
To better understand how often inconclusive noninferiority trials are misinterpreted, we studied the interpretation of 2 recent noninferiority trials (American College of Surgeons Oncology Group [ACOSOG2] Z6051 and Australasian Laparoscopic Cancer of the Rectum [ALaCaRT3]) evaluating the surgical treatment of rectal cancer that had inconclusive results regarding the noninferiority of laparoscopy in terms of quality of surgical resection (Figure). Both trials used correct yet ambiguous wording (ie, failed to meet the criteria for noninferiority) to report their findings.2,3 Publication of these 2 randomized clinical trials (RCTs) reignited concerns regarding the oncologic safety of the laparoscopic approach for rectal cancer.2,3 To evaluate the frequency of misinterpretation of these inconclusive trials, we examined publications citing the ACOSOG Z6051 and ALaCaRT trials and determined whether interpretations were concordant with the findings of the primary studies.
We used Scopus to identify articles citing the ACOSOG Z6051 and ALaCaRT trials2,3 from their publication (October 6, 2015) to February 15, 2018. We excluded articles not published in English, book chapters, and news pieces. Full texts were reviewed to identify articles that discussed the main findings of the trials. Sentences discussing the RCT results were extracted and independently reviewed by 2 of us (S.A.A. and F.D.). We recorded whether articles complied with the Consolidated Standards of Reporting Trials (CONSORT) recommendations4 for the interpretation of noninferiority trials (Figure) and whether the authors acknowledged the inconclusive nature of the results. No institutional review board approval of the study was required.
We identified 150 unique publications citing the ACOSOG Z6051 and ALaCaRT trials,2,3 with 130 articles having available full text in English. Seventy-one of 130 articles (54.6%) discussed the main findings of at least 1 of the trials. A substantial proportion of articles (20 of 71 [28.2%]) incorrectly interpreted the RCT findings as suggestive of inferiority of laparoscopic surgery (Table). Although the remainder of the articles included interpretations compatible with the CONSORT recommendations, most (38 of 51 [74.5%]) used ambiguous wording akin to that used in the trials (ie, failed to meet the criteria for noninferiority). Less than one-quarter of the publications (12 of 51 [23.5%]) acknowledged that the results were inconclusive.
This cross-sectional analysis of 71 articles discussing the ACOSOG Z6051 and ALaCaRT trials2,3 demonstrates that noninferiority trials with inconclusive results are often misinterpreted or described using correct yet ambiguous language that may obscure the true trial results. Most previous analyses of noninferiority trials have focused on quality and completeness of reporting of design variables and results.
This study has some limitations. These include the use of Scopus, which may not have accurately identified all articles citing the 2 trials studied. Moreover, our findings might not be generalizable to other noninferiority trials.
Our analysis is the first that we know of to explore the interpretation of inconclusive noninferiority trials among knowledge users. Appropriate interpretation of these studies is important because decisions about the use of new therapies increasingly rely on data from noninferiority trials. Explicit statements in publications of noninferiority trials when the findings are inconclusive would aid interpretation and avoid erroneous conclusions that novel treatments are inferior.
Accepted for Publication: June 10, 2018.
Corresponding Author: Nancy N. Baxter, MD, PhD, Division of General Surgery, Department of Surgery, University of Toronto, St Michael’s Hospital, 040-16 Cardinal Carter Wing, 30 Bond St, Toronto, ON M5B 1W8, Canada (baxtern@smh.ca).
Published Online: September 26, 2018. doi:10.1001/jamasurg.2018.3222
Author Contributions: Drs Acuna and Dossa had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: All authors.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Acuna.
Critical revision of the manuscript for important intellectual content: All authors.
Supervision: Baxter.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was funded by a Canadian Institutes of Health Research (CIHR) Operating Grant (reference 115164) and a CIHR Foundation Grant (reference 148470).
Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
2.Fleshman
J, Branda
M, Sargent
DJ,
et al. Effect of laparoscopic-assisted resection vs open resection of stage II or III rectal cancer on pathologic outcomes: the ACOSOG Z6051 randomized clinical trial.
JAMA. 2015;314(13):1346-1355. doi:
10.1001/jama.2015.10529PubMedGoogle ScholarCrossref 3.Stevenson
AR, Solomon
MJ, Lumley
JW,
et al; ALaCaRT Investigators. Effect of laparoscopic-assisted resection vs open resection on pathological outcomes in rectal cancer: the ALaCaRT randomized clinical trial.
JAMA. 2015;314(13):1356-1363. doi:
10.1001/jama.2015.12009PubMedGoogle ScholarCrossref 6.Carmichael
JC, Keller
DS, Baldini
G,
et al. Clinical practice guidelines for enhanced recovery after colon and rectal surgery from the American Society of Colon and Rectal Surgeons and Society of American Gastrointestinal and Endoscopic Surgeons.
Dis Colon Rectum. 2017;60(8):761-784. doi:
10.1097/DCR.0000000000000883PubMedGoogle ScholarCrossref