The NNT is calculated assuming a 30% reduction in relative risk over 10 years.
eTable 1. Percentage of patients who developed ESRD within 10 years of cohort entry and corresponding ARR and NNT assuming a 30% reduction in relative risk
eTable 2. Percentage of patients with diabetes mellitus who developed ESRD within 3 years of cohort entry and corresponding ARR and NNT assuming a 30% reduction in relative risk
eTable 3. Percentage of patients not prescribed an angiotensin-converting enzyme (ACE) inhibitor or angiotensin II receptor blocker (ARB) at baseline who developed ESRD within 3 years of cohort entry and corresponding ARR and NNT assuming a 30% reduction in relative risk
eTable 4. Percentage of patients who developed ESRD or died with an eGFR <15 mL/min/1.73 m2 within 3 years of cohort entry and corresponding ARR and NNT assuming a 30% reduction in relative risk
Customize your JAMA Network experience by selecting one or more topics from the list below.
O’Hare AM, Hotchkiss JR, Kurella Tamura M, et al. Interpreting Treatment Effects From Clinical Trials in the Context of Real-World Risk InformationEnd-Stage Renal Disease Prevention in Older Adults. JAMA Intern Med. 2014;174(3):391–397. doi:10.1001/jamainternmed.2013.13328
Copyright 2014 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.
Older adults are often excluded from clinical trials. The benefit of preventive interventions tested in younger trial populations may be reduced when applied to older adults in the clinical setting if they are less likely to survive long enough to experience those outcomes targeted by the intervention.
To extrapolate a treatment effect similar to those reported in major randomized clinical trials of angiotensin-converting enzyme inhibitors and angiotensin II receptor blockers for prevention of end-stage renal disease (ESRD) to a real-world population of older patients with chronic kidney disease.
Design, Setting, and Participants
Simulation study in a retrospective cohort conducted in Department of Veterans Affairs medical centers. We included 371 470 patients 70 years or older with chronic kidney disease.
Level of estimated glomerular filtration rate (eGFR) and proteinuria.
Main Outcomes and Measures
Among members of this cohort, we evaluated the expected effect of a 30% reduction in relative risk on the number needed to treat (NNT) to prevent 1 case of ESRD over a 3-year period. These limits were selected to mimic the treatment effect achieved in major trials of angiotensin-converting enzyme inhibitors and angiotensin II receptor blockers for prevention of ESRD. These trials have reported relative risk reductions of 23% to 56% during observation periods of 2.6 to 3.4 years, yielding NNTs to prevent 1 case of ESRD of 9 to 25.
The NNT to prevent 1 case of ESRD among members of this cohort ranged from 16 in patients with the highest baseline risk (eGFR of 15-29 mL/min/1.73 m2 with a dipstick proteinuria measurement of ≥2+) to 2500 for those with the lowest baseline risk (eGFR of 45-59 mL/min/1.73 m2 with negative or trace proteinuria and eGFR of ≥60 mL/min/1.73 m2 with dipstick proteinuria measurement of 1+). Most patients belonged to groups with an NNT of more than 100, even when the exposure time was extended over 10 years and in all sensitivity analyses.
Conclusions and Relevance
Differences in baseline risk and life expectancy between trial subjects and real-world populations of older adults with CKD may reduce the marginal benefit to individual patients of interventions to prevent ESRD.
Elderly patients are often underrepresented in randomized clinical trials.1-3 Extrapolating the results of trials conducted in younger adults to older patients in the clinical setting offers several challenges. Differences in underlying disease processes, in the presence and severity of coexisting comorbid conditions, and in the clinical context in which interventions are deployed may modify the efficacy, tolerability, and relevance of interventions tested in clinical trials when applied to older adults in real-world clinical settings.4-10
Many interventions recommended in older adults are intended to prevent or to delay the onset of nonfatal health outcomes. For these interventions, differences in life expectancy and baseline risk between trial populations and real-world populations of older adults may further modify the expected benefit.11-13 Patients whose life expectancy is more limited or whose baseline risk for the outcome of interest is lower than that of the trial population may have less opportunity to benefit from preventive interventions with known efficacy. Thus, in considering treatments intended to lower the risk of nonfatal health outcomes, older patients and their health care providers must weigh available information on efficacy in the context of their likelihood of experiencing the relevant outcome during their remaining lifetime and their own treatment priorities.11-14
To illustrate the importance of interpreting treatment effects from clinical trials in the context of real-world risk information, we conducted a simulation study in which we applied a relative risk reduction similar to that achieved in major randomized clinical trials of angiotensin-converting enzyme (ACE) inhibitors and angiotensin II receptor blockers (ARBs) for the prevention of end-stage renal disease (ESRD) to a real-world cohort of elderly patients with chronic kidney disease (CKD). These agents have been shown to slow progression to ESRD in clinical trials that were conducted in younger populations at high risk for ESRD.15-17 However, the benefit of these agents in reducing the risk of ESRD in an individual patient is conditional on that patient’s likelihood of surviving long enough to develop ESRD, a quantity that may vary as a function of life expectancy and baseline risk for ESRD. We therefore hypothesized that the marginal benefit to individual patients of a treatment effect similar to that achieved in major trials of ACE inhibitors and ARBs would differ when applied to a real-world cohort of older adults with CKD.
The treatment effect and exposure time were selected after reviewing the design of and relative risk reductions achieved in trials included in 2 recent systematic reviews of ESRD prevention.15,16 We considered the subset of 4 trials that enrolled at least 350 participants and found a lower incidence of ESRD (defined as dialysis or transplant) among participants treated with an ACE inhibitor or an ARB compared with participants in the control arm18-21 (Table 1). None of these trials enrolled adults older than 70 years, 2 of the 4 required that participants have diabetes mellitus,18,19 3 required that participants have proteinuria,18-20 and all enrolled participants with renal insufficiency. Mortality rates among members of the control groups ranged from 0 to 20.3% across trials. Based on the duration (2.6 to 3.4 years) and relative risk reductions (23% to 56%) achieved in these trials, we selected a 30% reduction in the relative risk for ESRD and an observation period of 3 years.
We assumed that the same reduction in relative risk would apply to a real-world population of older patients with CKD. We used information on observed survival and incident ESRD among members of this cohort to determine the probability that a patient would develop ESRD within 3 years of cohort entry. We then estimated the absolute reduction in risk for ESRD that would occur if patients experienced a 30% reduction in the relative risk for ESRD during this time frame. The estimated number needed to treat (NNT) to prevent 1 case of ESRD was then calculated as the reciprocal of the expected absolute risk reduction. To account for the effect of death during follow-up on the NNT and to parallel the method for calculation of the NNT in trials, we calculated rates of ESRD during the 3-year observation period among the denominator of patients who entered the cohort, regardless of whether they died during follow-up. These estimates are presented for groups defined by level of estimated glomerular filtration rate (eGFR) and proteinuria because both are strongly associated with survival and the risk for ESRD and because trials selected for patients with relatively high levels of proteinuria and low levels of renal function who were at higher risk for ESRD.
We used the same approach to evaluate the effect of a longer exposure to the treatment effect over 10 years. For members of this cohort who died within this time frame, the 10-year NNT reported here would represent an optimistic estimate of the NNT had the treatment been continued throughout their remaining lifetime. Our analyses were thus designed to account for 2 reasons a patient’s absolute risk for ESRD might differ from that observed in the trials. First, they may have a lower baseline risk of developing ESRD. Second, they may be less likely to live long enough for their disease to progress to ESRD.
We estimated the NNT to prevent 1 case of ESRD within 3 years of cohort entry in the following sensitivity analyses. First, because some trials were conducted only among participants with diabetes mellitus, we restricted the analysis to patients with a diagnostic code for diabetes during the year before cohort entry. Second, because some cohort members were already receiving ACE inhibitors or ARBs at cohort entry, we repeated our analyses after excluding these patients. Third, to account for possible age differences in the uptake of renal replacement therapy, we expanded the definition of ESRD to include patients who died during follow-up and whose most recent eGFR measurement before death was less than 15 mL/min/1.73 m2. Finally, we repeated the primary analysis using relative risk reductions of 23% and 56%, respectively, to reflect the full range of treatment effects reported in individual trials.
We identified 790 342 patients 70 years or older who were not receiving long-term dialysis, had not received a kidney transplant, and had at least 1 outpatient measurement of serum creatinine obtained at a Department of Veterans Affairs (VA) medical center from October 1, 2000, through September 30, 2001. The date of the first measurement of serum creatinine during this period was taken as the date of cohort entry. For patients who had a dipstick assay for measurement of proteinuria level, we used the most recently available assay result before the measurement of serum creatinine. All analyses were conducted among the subset of patients with stages I to IV CKD, defined as an eGFR of 15 to 60 mL/min/1.73 m2 or a proteinuria finding of at least 1+ on a dipstick (n = 371 470). Measurements of serum creatinine levels were obtained from the VA Decision Support System Laboratory Results file (which records the results of serum creatinine tests obtained at VA medical centers). Proteinuria dipstick results were obtained from the VA Corporate Data Warehouse. Information on age was ascertained from the VA Vital Status file. Information on race was ascertained from Medicare and VA sources, with preference for Medicare race data when available. Patients with diabetes mellitus were identified using inpatient and outpatient diagnostic code searches of VA and Medicare sources. Prescription of ACE inhibitors and ARBs at the time of cohort entry was ascertained using the VA Decision Support System Pharmacy files. Information on the date of death was obtained from the VA Vital Status file (a comprehensive source of death data for veterans), which was available through July 23, 2013.22 End-stage renal disease, defined as initiation of long-term dialysis or receipt of a kidney transplant, was ascertained by linkage to the US Renal Data System, a national registry for ESRD. Follow-up for ESRD was available through September 30, 2011. The study was approved by the institutional review board at the VA Puget Sound Healthcare System. Informed consent was waived for this study.
The mean (SD) age of cohort patients was 77.8 (4.6) years, 7525 (2.0%) were women, 34 154 (9.2%) were African American, 174 879 (47.1%) had a diagnostic code for diabetes mellitus, and 140 647 (37.9%) had an active prescription for an ACE inhibitor or an ARB at the time of cohort entry. Mean (SD) eGFR was 48.0 (11.7) mL/min/1.73 m2. Most patients had moderate reductions in eGFR in the range of 30 to 59 mL/min/1.73 m2 and negative, trace, or unmeasured proteinuria (Table 2).
Overall, 1.1% of cohort members reached ESRD within 3 years of cohort entry, ranging from 0.13% of those in the lowest risk groups (those with an eGFR ≥60 mL/min/1.73 m2 and proteinuria dipstick findings of 1+ and those with an eGFR of 45-59 mL/min/1.73 m2 with negative or trace proteinuria) to 21.17% for those in the highest risk group (eGFR of 15-29 mL/min/1.73 m2 and proteinuria dipstick findings of ≥2+) (Table 2). Numbers needed to treat for a 30% reduction in the relative risk of ESRD during the 3-year exposure period ranged from 16 for the highest-risk group to 2500 for the lowest-risk groups. Overall, 91% of cohort members belonged to a group for which the NNT exceeded 100.
Median survival for members of the study cohort was 6.7 years (interquartile range [25th-75th percentiles], 3.2-11.4 years), ranging from 8.1 years for patients with an eGFR of 45 to 59 mL/min/1.73 m2 and negative or trace proteinuria to 3.1 years among those with an eGFR of 15 to 29 mL/min/1.73 m2 and proteinuria dipstick findings of 2+ or greater (Table 3). Overall, 23.3% of the cohort died within 3 years and 68.6% died within 10 years of cohort entry. Ten-year mortality rates ranged from 60.6% of patients with an eGFR 45 to 59 mL/min/1.73 m2 and negative or trace proteinuria to 95.0% of those with an eGFR of 15 to 29 mL/min/1.73 m2 and proteinuria dipstick findings of 2+ or greater. Overall, 3.3% of patients reached ESRD within 10 years of cohort entry, ranging from 0.8% of those with an eGFR of 45 to 59 mL/min/1.73 m2 and negative or trace proteinuria to 35.1% of those with an eGFR of 15 to 29 mL/min/1.73 m2 and proteinuria dipstick findings of 2+ or greater (Figure and Supplement [eTable 1]). Corresponding NNTs ranged from 435 to 10, and 73% of cohort members belonged to a group for whom the NNT exceeded 100 over the 10-year follow-up period.
In the sensitivity analyses, the 3-year NNT ranged from 14 to 2500 among patients with diabetes mellitus, and 89% of cohort members with diabetes belonged to a group with an NNT greater than 100 (Supplement [eTable 2]). Among patients not receiving an ACE inhibitor or an ARB at cohort entry (n = 230 823), the 3-year NNT ranged from 16 to 3333, and 93% belonged to a group with an NNT greater than 100 (Supplement [eTable 3]). When the definition of ESRD was expanded to include death with an eGFR of less than 15 mL/min/1.73 m2, the 3-year NNT ranged from 12 to 588 and 91% of cohort members belonged to a group with an NNT greater than 100 (Supplement [eTable 4]). When we applied a 23% instead of a 30% relative risk reduction for ESRD to the overall cohort, the 3-year NNT ranged from 21 to 3333 and 93% of patients belonged to a group with an NNT greater than 100. A 56% relative risk reduction conferred 3-year NNTs ranging from 8 to 1667, and 91% of patients belonged to a group with an NNT greater than 100.
Major ESRD prevention trials have achieved NNTs of 9 to 25 over trial durations of 2.6 to 3.4 years, with relative risk reductions from 23% to 56% in populations with a baseline risk for ESRD ranging from 14.8% to 25.5% during follow-up.18-21 When extrapolated to this real-world cohort of adults 70 years or older with CKD, a treatment effect within this range would be expected to yield NNTs ranging from 16 for those at highest risk for ESRD to 2500 for those at lowest risk, with most patients belonging to groups with an NNT greater than 100.
Older adults with limited life expectancy and complex comorbidity must often choose between a large number of recommended interventions intended to restore or maintain health.6,13 End-stage renal disease is but one example of the many disease outcomes for which preventive interventions are available and may be recommended in an older adult. Information on the NNT to achieve a given clinical outcome can be useful in this context because it conveys information on the effort needed to achieve a treatment effect in a way that can be readily understood by patients and providers.23-25 The most effective treatments have NNTs close to 1, indicating that only a few patients must be treated to achieve the desired outcome in a single patient. As the NNT increases, the marginal benefit to the individual patient decreases. Although fixed NNT thresholds are often used to define effectiveness,26 the NNT and its counterpart—the number needed to harm—are appealing precisely because they allow for flexibility in how individual patients weigh the benefits and burdens of different treatments.
Inhibition of the renin-angiotensin system constitutes the ESRD-preventive strategy that is most strongly supported by the results of randomized clinical trials.16 Trials supporting the effectiveness of these agents for prevention of ESRD enrolled mainly younger participants who were at relatively high risk for this outcome, especially those with proteinuria.15-17 In a real-world cohort of older adults with CKD, we observed much greater heterogeneity in baseline risk of ESRD compared with trial populations, leading to a much larger variation in the NNT for an equivalent degree of relative risk reduction. In general, only among the small subset of patients with the lowest levels of eGFR and highest levels of proteinuria did a 30% reduction in relative risk yield NNTs comparable to those reported in the aforementioned trials. The NNTs were far higher for the large majority of patients with earlier-stage CKD, particularly those with lower levels of proteinuria. If anything, these results probably overestimate the expected benefit of available interventions to prevent ESRD among members of this cohort, given the paucity of evidence to support the efficacy of ACE inhibitors and ARBs for ESRD prevention in older adults with CKD and in those without proteinuria.15,27
Clinical trials of interventions targeted at outcomes such as ESRD that may take many years to develop must often balance the need to optimize power with pragmatic constraints on trial recruitment and duration. Selection of participants at high risk for the outcome of interest can help to support these opposing goals. However, extrapolation of treatment effects in high-risk trial populations to real-world populations at lower risk rests on the assumption that similar benefits will accrue over longer periods. This assumption may not be justified in patients at much lower risk for the outcome and/or with more limited life expectancy. To evaluate the potential effect of longer-term exposure on the benefit of the intervention, we applied the same 30% relative risk reduction over a 10-year time frame. Ten-year NNTs represent an optimistic estimate of the NNT, had the exposure been extended over the remaining lifetime of the 68.6% of cohort members who died during this time frame. Even with this longer follow-up period, most cohort members belonged to groups for which the NNT far exceeded those achieved in trial populations over much shorter periods.
The simulation described here is intended to illustrate the potential importance of interpreting treatment effects from randomized clinical trials in the context of risk information from real-world clinical settings. Our study is not intended to provide all the information that would be needed to support treatment decisions about ESRD prevention among older adults in the clinical setting. First, interventions to prevent ESRD may also favorably affect a range of other desired treatment targets (eg, doubling of serum creatinine levels, movement to a more advanced stage of kidney disease, and reduction in cardiovascular risk and mortality). The NNTs for interventions targeted at outcomes that are more common than ESRD (eg, loss of eGFR) or that simultaneously reduce the risk of ESRD and death would be expected to be lower than those reported here for ESRD. Second, we assumed equal efficacy in all patients (ie, the same reduction in relative risk for all groups). In reality, the efficacy of interventions to prevent ESRD may also vary depending on patient characteristics, such as level of renal function and proteinuria.16,17,28 Because older adults and those without proteinuria have generally been excluded from major randomized clinical trials of interventions targeted at ESRD, the efficacy of these interventions in older patients and in patients without proteinuria is uncertain.15,28 Third, we assumed that patients with similar levels of eGFR and proteinuria had a similar baseline risk of ESRD. However, as for survival time, the risk of ESRD probably varies even within these strata.29 Finally, the overall benefit of an intervention depends on the relationship between effectiveness and harm,12,30 and the harms of interventions to prevent ESRD probably differ between real-world populations of older adults with CKD and trial populations.
Our study has several limitations. First, our results for this predominantly male cohort may not be generalizable to older women with CKD. Because rates of ESRD are higher in men than in women,31 the results presented here probably overestimate the benefit of an equivalent reduction in relative risk among populations with a more balanced sex distribution. Second, information on the level of proteinuria was missing for a substantial number of cohort patients. Nevertheless, the percentage of cohort members missing information on proteinuria is similar to that reported for other real-world cohorts defined by the presence of serum creatinine measures.32-34 We chose to retain these patients in the cohort to assess the NNT for all groups within the population with an eGFR of less than 60 mL/min/1.73 m2. Finally, 10-year NNTs may be overly optimistic given that the percentage of relative risk reduction for any intervention is likely to diminish as the size of the unaffected population decreases.
Differences in baseline risk and life expectancy may substantially modify the benefit of ESRD-preventive interventions when applied to real-world populations of older adults compared with trial populations. This study highlights the importance of interpreting treatment effects from randomized clinical trials in the context of risk information from real-world clinical settings. This consideration may be particularly relevant in older adults because they are often underrepresented in clinical trials and their risk for experiencing the outcome of interest during their remaining lifetime may be very different from that for younger trial populations.
Accepted for Publication: October 29, 2013.
Corresponding Author: Ann M. O’Hare, MA, MD, Department of Veterans Affairs Puget Sound Healthcare System, 1160 S Columbian Way, Seattle, WA 98108 (email@example.com).
Published Online: January 13, 2014. doi:10.1001/jamainternmed.2013.13328.
Author Contributions: Dr O’Hare had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: O’Hare, Hotchkiss, Covinsky.
Acquisition of data: O’Hare, Batten.
Analysis and interpretation of data: O’Hare, Kurella Tamura, Larson, Hemmelgarn, Batten, Do, Covinsky.
Drafting of the manuscript: O’Hare, Batten.
Critical revision of the manuscript for important intellectual content: O’Hare, Hotchkiss, Kurella Tamura, Larson, Hemmelgarn, Do, Covinsky.
Statistical analysis: O’Hare, Batten, Do, Covinsky.
Obtained funding: O’Hare.
Administrative, technical, or material support: O’Hare, Kurella Tamura, Larson, Hemmelgarn.
Study supervision: O’Hare, Larson.
Conflict of Interest Disclosures: Dr Kurella Tamura reports having served on a scientific advisory board for Amgen. No other disclosures were reported.
Funding/Support: This study was supported by grant K24AG029812 (Dr Covinsky) and research funding (Drs O’Hare and Larson) from the National Institute on Aging, by VA Health Services Research and Development (Dr O’Hare), and by the Roy and Vi Baay Chair in Kidney Research (Dr Hemmelgarn).
Role of the Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: Whitney Showalter, BA, Department of Veterans Affairs Puget Sound Healthcare System, assisted with securing and maintaining institutional review board approvals and data use agreements for this study. Jeff Todd-Stenberg, BS, Department of Veterans Affairs Puget Sound Healthcare System, provided guidance on data acquisition and data use agreements for this project. Charles Maynard, PhD, assisted with acquisition of mortality data for this study. These contributors are employees of the Department of Veterans Affairs and did not receive any non-salary compensation for their work on this project.