Controlled clinical trials, which are used to guide the decisions made by patients, clinicians, and policy makers, often only report measures of relative effect.1,2 However, absolute measures, such as the absolute risk reduction (ARR), the number needed to treat (NNT), and the number needed to harm (NNH), which measure the difference in the observed risk of an event between 2 interventions and the number of patients who need to be treated to achieve 1 additional favorable or adverse outcome, respectively, can be easier to interpret, more clinically meaningful, and less likely to exaggerate differences when outcome risk is low.3 In part because only 5% of trials published in highly cited journals before 1998 reported NNT and/or ARR,4 the Consolidated Standards of Reporting Trials (CONSORT) statement recommended that trials with binary outcomes report both relative and absolute measures.5 We assessed the recent trends and characteristics of absolute measure reporting in highly cited medical journals to determine if there have been improvements over time.
We identified the 6 most-cited medical journals according to InCites Journal Citation Reports (Clarivate Analytics 2019) (Table 1). For each journal, we reviewed all issues published in 2001, 2007, 2013, and 2019 to identify all controlled clinical trials that reported analyses testing superiority of the intervention to control and abstract-level binary outcomes, including hazard ratios. For eligible trials, we identified key study characteristics and recorded whether at least 1 abstract-level positive (P < .05) binary efficacy and/or safety outcome was reported. Next, we determined whether any NNT, NNH, and/or ARR was reported in the abstract and/or full text. For each NNT/NNH, we recorded if reporting was for primary or secondary end points and whether 95% CIs, P values, and corresponding effect estimates were provided. Fisher exact and Mann-Whitney U tests were conducted in R, version 3.4.0 (R Foundation for Statistical Computing) (2-sided P < .05). Because publicly available data were used, this study did not require ethics approval or patient consent.
We identified 875 controlled trials meeting the aforementioned criteria , of which 76 (8.7%) reported at least 1 NNT, 8 (0.9%) reported at least 1 NNH, and 249 (28.5%) reported at least 1 ARR (Table 1). In total, 292 trials (33.4%) reported at least 1 NNT, NNH, and/or ARR. A total of 80 (9.1%) reported at least 1 NNT and/or NNH, which remained relatively constant between 2001 and 2019; ARR reporting increased from 26 of 140 (18.6%) to 105 of 282 (37.2%; P < .001).
Trials in the therapeutic area of oncology had the lowest rates of reporting NNT, NNH, and/or ARR, but there were no differences by intervention tested, patient follow-up, enrollment, or funding sources (Table 2). Trials with at least 1 statistically significant end point were more likely to report an NNT/NNH than those without (75 of 624 [12.0%] vs 5 of 251 [2.0%]; P < .001).
Among all 197 NNT/NNH reports, 95 (48.2%) were for primary end points and 76 (38.6%) had a 95% CI and/or P value. There were 114 NNT/NNH reports with a corresponding effect estimate reported anywhere in the text, of which 88 (77.2%) were statistically significant; 55 (48.2%) only had corresponding relative measures.
Among 875 controlled trials with binary outcomes and/or hazard ratios published in highly cited general medical journals, fewer than one-tenth reported at least 1 NNT or NNH, but more than one-quarter reported at least 1 ARR. The majority of NNT/NNH reports were presented for statistically significant end points but without 95% CIs or P values. These findings raise concerns about persistent incomplete and selective reporting of absolute measures in controlled trials published in widely cited general medical journals.
This study may not be generalizable to all journals and fields, and we did not review supplemental materials. Although we observed a 4-fold increase in reporting NNT and a 6-fold increase in reporting ARR, compared with a prior study examining trials through 1998,4 the present findings continue to support concerns about absolute measure reporting across journals and study designs.6 Despite the limitations of the ARR, which depends on the baseline risk of the population to be treated, and NNT/NNH, which may be misinterpreted, additional explicit statements from journals on absolute measure reporting requirements and greater oversight of the CONSORT checklist can help improve reporting practices and trial interpretability.
Accepted for Publication: July 25, 2020.
Published Online: November 23, 2020. doi:10.1001/jamainternmed.2020.4799
Corresponding Author: Joshua D. Wallach, PhD, MS, Department of Environmental Health Sciences, Yale School of Public Health, 60 College St, 4th Floor, Room 411, New Haven, CT 06510 (joshua.wallach@yale.edu).
Author Contributions: Dr Wallach had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Elliott, Dhruva, Wallach.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Elliott, Wallach.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Elliott, Skydel, Wallach.
Supervision: Wallach.
Conflict of Interest Disclosures: Dr Dhruva reported travel reimbursement from the US Food and Drug Administration, Center for Drug Research and Evaluation/Office of Medical Policy, and National Evaluation System for Health Technology Coordinating Center, as well as research grant funding from the National Heart, Lung, and Blood Institute of the National Institutes of Health, the Greenwall Foundation, and the National Evaluation System for Health Technology Coordinating Center. Dr Ross reported grants from the US Food and Drug Administration, Johnson and Johnson, Medical Devices Innovation Consortium, the Agency for Healthcare Research and Quality of the US Department of Health and Human Services, the National Heart, Lung, and Blood Institute of the National Institutes of Health, Laura and John Arnold Foundation, the US Centers for Medicare & Medicaid Services, Medtronic Inc, and the British Medical Association, as well as personal fees from the American Medical Association, outside the submitted work, and is a former associate editor of JAMA Internal Medicine and a current research editor at the British Medical Journal. Dr Wallach has received research support through Yale University from the Laura and John Arnold Foundation and the US Food and Drug Administration outside the submitted work. No other disclosures were reported.
Additional Information: Requests for the dataset can be made to the corresponding author at joshua.wallach@yale.edu.
2.King
NB, Harper
S, Young
ME. Use of relative and absolute effect measures in reporting health inequalities: structured review.
BMJ. 2012;345:e5774. doi:
10.1136/bmj.e5774PubMedGoogle Scholar 3.Faraone
SV. Interpreting estimates of treatment effects: implications for managed care.
P T. 2008;33(12):700-711.
PubMedGoogle Scholar