[Skip to Navigation]
Sign In
Figure.  Consolidated Standards of Reporting Trials (CONSORT)–Recommended Interpretation of Noninferiority Findings and Results of the ACOSOG Z6051 and ALaCaRT Trials2,3
Consolidated Standards of Reporting Trials (CONSORT)–Recommended Interpretation of Noninferiority Findings and Results of the ACOSOG Z6051 and ALaCaRT Trials

Error bars indicate 95% CIs. A, Shown are examples of noninferiority (NI) trial results and their interpretation according to the CONSORT recommendations. In example 1, if the 95% CI lies wholly to the left of zero, the new intervention is superior. In examples 2 and 3, if the 95% CI lies to the left of ΔNI and includes zero, the new treatment is noninferior but not shown to be superior. In example 4, if the 95% CI lies wholly to the left of ΔNI and wholly to the right of zero, the new treatment is noninferior in the sense already defined, but it is also inferior in the sense that a null treatment difference is excluded. In examples 5 and 6, if the 95% CI includes ΔNI and zero, the difference is not significant, but the result regarding noninferiority is inconclusive. In example 7, if the 95% CI includes ΔNI and is wholly to the right of zero, the difference is statistically significant, but the result is inconclusive regarding possible inferiority. Last, in example 8, if the 95% CI is wholly above ΔNI, the new treatment is not noninferior (looking-glass opposite of noninferior as inferior is also used to refer to the opposite to superior). The vertical dotted lines indicate the noninferiority margin (∆NI). B, Results of the ACOSOG Z6051 and ALaCaRT trials are presented as risk differences and 95% CIs with their respective ΔNI. The ACOSOG Z6051 noninferiority margin is risk difference 6.0 (Δ2); the ALaCaRT noninferiority margin is risk difference 8.0 (Δ1). As seen, the results of the ACOSOG Z6051 trial correspond to example 6, and the results of the ALaCaRT trial correspond to example 7. The vertical dotted lines indicate the noninferiority margin from the ACOSOG Z6051 trial (∆1) and the ALaCaRT trial (∆2).

Table.  Categorization of the Interpretation of the ACOSOG Z6051 and ALaCaRT Trials2,3 and Examples
Categorization of the Interpretation of the ACOSOG Z6051 and ALaCaRT Trials and Examples
1.
Kaul  S, Diamond  GA.  Good enough: a primer on the analysis and interpretation of noninferiority trials.  Ann Intern Med. 2006;145(1):62-69. doi:10.7326/0003-4819-145-1-200607040-00011PubMedGoogle ScholarCrossref
2.
Fleshman  J, Branda  M, Sargent  DJ,  et al.  Effect of laparoscopic-assisted resection vs open resection of stage II or III rectal cancer on pathologic outcomes: the ACOSOG Z6051 randomized clinical trial.  JAMA. 2015;314(13):1346-1355. doi:10.1001/jama.2015.10529PubMedGoogle ScholarCrossref
3.
Stevenson  AR, Solomon  MJ, Lumley  JW,  et al; ALaCaRT Investigators.  Effect of laparoscopic-assisted resection vs open resection on pathological outcomes in rectal cancer: the ALaCaRT randomized clinical trial.  JAMA. 2015;314(13):1356-1363. doi:10.1001/jama.2015.12009PubMedGoogle ScholarCrossref
4.
Piaggio  G, Elbourne  DR, Pocock  SJ, Evans  SJ, Altman  DG; CONSORT Group.  Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement.  JAMA. 2012;308(24):2594-2604. doi:10.1001/jama.2012.87802PubMedGoogle ScholarCrossref
5.
Burstein  HJ, Krilov  L, Aragon-Ching  JB,  et al.  Clinical cancer advances 2017: annual report on progress against cancer from the American Society of Clinical Oncology.  J Clin Oncol. 2017;35(12):1341-1367. doi:10.1200/JCO.2016.71.5292PubMedGoogle ScholarCrossref
6.
Carmichael  JC, Keller  DS, Baldini  G,  et al.  Clinical practice guidelines for enhanced recovery after colon and rectal surgery from the American Society of Colon and Rectal Surgeons and Society of American Gastrointestinal and Endoscopic Surgeons.  Dis Colon Rectum. 2017;60(8):761-784. doi:10.1097/DCR.0000000000000883PubMedGoogle ScholarCrossref
7.
Stevenson  AR.  The future for laparoscopic rectal cancer surgery.  Br J Surg. 2017;104(6):643-645. doi:10.1002/bjs.10503PubMedGoogle ScholarCrossref
Research Letter
January 2019

Frequency of Misinterpretation of Inconclusive Noninferiority Trials: The Case of the Laparoscopic vs Open Resection for Rectal Cancer Trials

Author Affiliations
  • 1Division of General Surgery, Department of Surgery, University of Toronto, Toronto, Ontario, Canada
JAMA Surg. 2019;154(1):90-92. doi:10.1001/jamasurg.2018.3222

Noninferiority clinical trials are designed to determine whether an intervention is not worse than a comparator by a prespecified difference, known as the noninferiority margin (ΔNI). These trials are useful when comparing standard therapies with novel treatments that may be easier to use, are less costly, or have fewer adverse effects.1 The number of noninferiority trials is increasing; however, particularly compared with superiority trials, the interpretation of noninferiority trials is not straightforward. Superiority trials can be classified as positive or negative based on whether the null hypothesis can be rejected (ie, P < .05). In contrast, noninferiority trials can be interpreted as noninferior, inconclusive, or not noninferior according to the location of the 95% CIs in relation to the prespecified margin (Figure). Inconclusive and not noninferior results are often both presented as “negative” results (eg, failure to meet noninferiority criteria). This can lead readers to erroneously interpret an inconclusive trial as showing evidence that the novel treatment is worse, while in reality the findings are indeterminate and further research is required to determine noninferiority.

To better understand how often inconclusive noninferiority trials are misinterpreted, we studied the interpretation of 2 recent noninferiority trials (American College of Surgeons Oncology Group [ACOSOG2] Z6051 and Australasian Laparoscopic Cancer of the Rectum [ALaCaRT3]) evaluating the surgical treatment of rectal cancer that had inconclusive results regarding the noninferiority of laparoscopy in terms of quality of surgical resection (Figure). Both trials used correct yet ambiguous wording (ie, failed to meet the criteria for noninferiority) to report their findings.2,3 Publication of these 2 randomized clinical trials (RCTs) reignited concerns regarding the oncologic safety of the laparoscopic approach for rectal cancer.2,3 To evaluate the frequency of misinterpretation of these inconclusive trials, we examined publications citing the ACOSOG Z6051 and ALaCaRT trials and determined whether interpretations were concordant with the findings of the primary studies.

Methods

We used Scopus to identify articles citing the ACOSOG Z6051 and ALaCaRT trials2,3 from their publication (October 6, 2015) to February 15, 2018. We excluded articles not published in English, book chapters, and news pieces. Full texts were reviewed to identify articles that discussed the main findings of the trials. Sentences discussing the RCT results were extracted and independently reviewed by 2 of us (S.A.A. and F.D.). We recorded whether articles complied with the Consolidated Standards of Reporting Trials (CONSORT) recommendations4 for the interpretation of noninferiority trials (Figure) and whether the authors acknowledged the inconclusive nature of the results. No institutional review board approval of the study was required.

Results

We identified 150 unique publications citing the ACOSOG Z6051 and ALaCaRT trials,2,3 with 130 articles having available full text in English. Seventy-one of 130 articles (54.6%) discussed the main findings of at least 1 of the trials. A substantial proportion of articles (20 of 71 [28.2%]) incorrectly interpreted the RCT findings as suggestive of inferiority of laparoscopic surgery (Table). Although the remainder of the articles included interpretations compatible with the CONSORT recommendations, most (38 of 51 [74.5%]) used ambiguous wording akin to that used in the trials (ie, failed to meet the criteria for noninferiority). Less than one-quarter of the publications (12 of 51 [23.5%]) acknowledged that the results were inconclusive.

Discussion

This cross-sectional analysis of 71 articles discussing the ACOSOG Z6051 and ALaCaRT trials2,3 demonstrates that noninferiority trials with inconclusive results are often misinterpreted or described using correct yet ambiguous language that may obscure the true trial results. Most previous analyses of noninferiority trials have focused on quality and completeness of reporting of design variables and results.

Limitations

This study has some limitations. These include the use of Scopus, which may not have accurately identified all articles citing the 2 trials studied. Moreover, our findings might not be generalizable to other noninferiority trials.

Conclusions

Our analysis is the first that we know of to explore the interpretation of inconclusive noninferiority trials among knowledge users. Appropriate interpretation of these studies is important because decisions about the use of new therapies increasingly rely on data from noninferiority trials. Explicit statements in publications of noninferiority trials when the findings are inconclusive would aid interpretation and avoid erroneous conclusions that novel treatments are inferior.

Back to top
Article Information

Accepted for Publication: June 10, 2018.

Corresponding Author: Nancy N. Baxter, MD, PhD, Division of General Surgery, Department of Surgery, University of Toronto, St Michael’s Hospital, 040-16 Cardinal Carter Wing, 30 Bond St, Toronto, ON M5B 1W8, Canada (baxtern@smh.ca).

Published Online: September 26, 2018. doi:10.1001/jamasurg.2018.3222

Author Contributions: Drs Acuna and Dossa had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Acuna.

Critical revision of the manuscript for important intellectual content: All authors.

Supervision: Baxter.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was funded by a Canadian Institutes of Health Research (CIHR) Operating Grant (reference 115164) and a CIHR Foundation Grant (reference 148470).

Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Kaul  S, Diamond  GA.  Good enough: a primer on the analysis and interpretation of noninferiority trials.  Ann Intern Med. 2006;145(1):62-69. doi:10.7326/0003-4819-145-1-200607040-00011PubMedGoogle ScholarCrossref
2.
Fleshman  J, Branda  M, Sargent  DJ,  et al.  Effect of laparoscopic-assisted resection vs open resection of stage II or III rectal cancer on pathologic outcomes: the ACOSOG Z6051 randomized clinical trial.  JAMA. 2015;314(13):1346-1355. doi:10.1001/jama.2015.10529PubMedGoogle ScholarCrossref
3.
Stevenson  AR, Solomon  MJ, Lumley  JW,  et al; ALaCaRT Investigators.  Effect of laparoscopic-assisted resection vs open resection on pathological outcomes in rectal cancer: the ALaCaRT randomized clinical trial.  JAMA. 2015;314(13):1356-1363. doi:10.1001/jama.2015.12009PubMedGoogle ScholarCrossref
4.
Piaggio  G, Elbourne  DR, Pocock  SJ, Evans  SJ, Altman  DG; CONSORT Group.  Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement.  JAMA. 2012;308(24):2594-2604. doi:10.1001/jama.2012.87802PubMedGoogle ScholarCrossref
5.
Burstein  HJ, Krilov  L, Aragon-Ching  JB,  et al.  Clinical cancer advances 2017: annual report on progress against cancer from the American Society of Clinical Oncology.  J Clin Oncol. 2017;35(12):1341-1367. doi:10.1200/JCO.2016.71.5292PubMedGoogle ScholarCrossref
6.
Carmichael  JC, Keller  DS, Baldini  G,  et al.  Clinical practice guidelines for enhanced recovery after colon and rectal surgery from the American Society of Colon and Rectal Surgeons and Society of American Gastrointestinal and Endoscopic Surgeons.  Dis Colon Rectum. 2017;60(8):761-784. doi:10.1097/DCR.0000000000000883PubMedGoogle ScholarCrossref
7.
Stevenson  AR.  The future for laparoscopic rectal cancer surgery.  Br J Surg. 2017;104(6):643-645. doi:10.1002/bjs.10503PubMedGoogle ScholarCrossref
×