Expected per-patient net OS benefit is calculated using the described quantitative framework. The sample sizes displayed were calculated by summing trial populations across all studies that met each PFS threshold.
Incremental social value is calculated using the described quantitative framework. These calculations use medium patient benefit parameters of $22 000 for incremental lifetime treatment cost and 22 months of delay between PFS and OS evidence publications.
For parameter values below and to the right of each line, it is better to grant access based on PFS data alone, given the particular PFS threshold illustrated. For values above and to the left of each line, it is better to wait for overall survival (OS) data before providing access. Dashed horizontal lines correspond to “high treatment cost,” which is our upper bound cost estimate ($62 000); “medium treatment cost,” which is our mean estimate ($22 000); and “low treatment cost,” which is our lower bound ($2000).
eAppendix. Supplementary Materials—Data, Parameters, and Calculations
eTable 1. Net PFS and OS Means by Trial
eTable 2. Expected Values by PFS Net Benefit Threshold
eTable 3. Incremental Treatment Costs
eTable 4. Incremental Social Value by PFS Net Benefit Threshold
eTable 5. Indifference Points for Payers When Choosing Between OS and PFS Policies
eTable 6. Expected Net Benefit for Median PFS and OS
eTable 7. Expected and Incremental Social Value for Median PFS and OS Analysis
eTable 8. Indifference Points for Median PFS and OS Analysis
eTable 9. Incremental Social Value by PFS Benefit Threshold and Value of a Statistical Life Year
eTable 10. Incremental Social Value by PFS Benefit Threshold and Incremental Lifetime Treatment Cost
eTable 11. NSCLC Trials With Separate PFS and OS Publications, 2009-2014
eTable 12. Incremental Social Value by PFS Benefit Threshold and Publication Delay
eTable 13. Parameter Values Used for Sensitivity Analyses
eTable 14. Sensitivity Analysis Scenarios Results
eTable 15. Net PFS Benefits for Clinical Uncertainty Sensitivity Analysis
eTable 16. Clinical Uncertainty and Publication Bias Sensitivity Analysis Results
Customize your JAMA Network experience by selecting one or more topics from the list below.
Lakdawalla DN, Chou JW, Linthicum MT, MacEwan JP, Zhang J, Goldman DP. Evaluating Expected Costs and Benefits of Granting Access to New Treatments on the Basis of Progression-Free Survival in Non–Small-Cell Lung Cancer. JAMA Oncol. 2015;1(2):196–202. doi:10.1001/jamaoncol.2015.0203
Copyright 2015 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.
Surrogate end points may be used as proxy for more robust clinical end points. One prominent example is the use of progression-free survival (PFS) as a surrogate for overall survival (OS) in trials for oncologic treatments. Decisions based on surrogate end points may expedite regulatory approval but may not accurately reflect drug efficacy. Payers and clinicians must balance the potential benefits of earlier treatment access based on surrogate end points against the risks of clinical uncertainty.
To present a framework for evaluating the expected net benefit or cost of providing early access to new treatments on the basis of evidence of PFS benefits before OS results are available, using non–small-cell lung cancer (NSCLC) as an example.
Design, Setting, and Participants
A probabilistic decision model was used to estimate expected incremental social value of the decision to grant access to a new treatment on the basis of PFS evidence. The model analyzed a hypothetical population of patients with NSCLC who could be treated during the period between PFS and OS evidence publication. Estimates for delay in publication of OS evidence following publication of PFS evidence, expected OS benefit given PFS benefit, incremental cost of new treatment, and other parameters were drawn from the literature on treatment of NSCLC.
Main Outcomes and Measures
Incremental social value of early access for each additional patient per month (in 2014 US dollars).
For “medium-value” model parameters, early reimbursement of drugs with any PFS benefit yields an incremental social cost of more than $170 000 per newly treated patient per month. In contrast, granting early access on the basis of PFS benefit between 1 and 3.5 months produces more than $73 000 in incremental social value. Across the full range of model parameter values, granting access for drugs with PFS benefit between 3 and 3.5 months is robustly beneficial, generating incremental social value ranging from $38 000 to more than $1 million per newly treated patient per month, whereas access for all drugs with any PFS benefit is usually not beneficial.
Conclusions and Relevance
The value of providing access to new treatments on the basis of surrogate end points, and PFS in particular, likely varies considerably. Payers and clinicians should carefully consider how to use PFS data in balancing potential benefits against costs in each particular disease.
In drug approval and coverage decisions, waiting for the most complete data on long-term effectiveness can delay access by years. Since 1992, the US Food and Drug Administration (FDA) has provided accelerated approval of promising treatments for serious conditions on the basis of “surrogate” end points to expedite patient access to potentially effective innovations.1 While granting earlier access to new treatments, this strategy also risks introducing treatments that later prove ineffective. Payers and clinicians must weigh the benefits and risks of granting access without complete efficacy information.
In the case of cancer drug approval, overall survival (OS) is generally considered the “gold-standard” outcome. Overall survival measures the time from treatment randomization until death.2 The value of using drugs with demonstrated OS benefit is clearly defined, widely accepted, and well understood.2,3 However, collecting OS data often takes years.3,4 The social benefits generated by new treatments might be even higher if they could be accessed before OS data collection were completed.
One solution is to base access decisions on surrogate end points. Perhaps the most salient example is progression-free survival (PFS), which measures time elapsed between treatment randomization and objective tumor progression (or death, if that occurs first).2(pp8-10) Compared with OS, PFS can be measured in smaller samples of patients or over shorter intervals because disease progression is typically observed earlier and more frequently than mortality. Conversely, PFS is not perfectly correlated with OS.5,6 In some cases, drugs with PFS benefits fail to show OS benefits. Policymakers, payers, and physicians must decide how to balance this uncertainty of clinical benefit against the potential loss of life from delayed access.7-11
In this article, we present a stylized, quantitative framework for weighing the benefits of OS and PFS, using non–small cell lung cancer (NSCLC) treatment trials. Non–small cell lung cancer will affect an estimated 183 600 new US patients in 2015.12(p16) Because it is relatively insensitive to chemotherapy and radiotherapy, NSCLC is difficult to treat, and for most patients, no cure exists.13 The rapid introduction of new targeted therapies for NSCLC makes it a salient case study for the costs and benefits of early access based on PFS.14
Using surrogate end points—eg, progression-free survival (PFS) as a proxy for overall survival (OS)—to make decisions that accelerate access to new drugs requires balancing clinical risk against the potential for great benefit.
Decision model estimated social benefit (life years) and cost (US$) for non–small-cell lung cancer.
A positive PFS correctly predicted a positive OS 71% of the time (15 of 21 randomized clinical trials).
Granting access for drugs that provide greater than 3 months PFS is robustly beneficial.
Granting access for all drugs with any increase in PFS is not beneficial.
This research did not involve human subjects and was therefore exempt from institutional review board review.
Our primary outcome was the incremental social value of providing early access to a new NSCLC treatment on the basis of PFS evidence alone, per newly treated patient per month. When this value exceeds zero, early access is more efficient from a societal perspective, and vice versa.
The net social benefit of access based on PFS data equals the expected value of earlier drug availability, less the expected cost of errors due to reliance on PFS alone, across all patients treated. Because the total social benefit scales proportionally with the monthly number of patients treated with a new drug and because this monthly number of patients is uncertain, we divide total benefit by the monthly number of patients initiating treatment and thus remove this parameter from consideration. The resulting “incremental social value of early access per newly treated patient per month,” hereafter simply “incremental social value,” captures the value of treating 1 additional patient with a clinically appropriate new therapy. It is the product of 3 components:(months of delay between PFS and OS publications) × (net change in survival months per patient treated) × (value of each survival month).The months of delay between PFS and OS publications is the mean duration, in months, of delay in access caused by waiting for OS. The net change in survival months per patient is the difference between the clinical value of successes per patient and the clinical cost of failures per patient. The clinical value of successes is the product of 2 terms:
The probability of success, or the probability that a drug with a particular PFS benefit will produce positive OS benefit; and
The value of success, or the expected OS benefit net of incremental financial cost (compared with existing treatments), for each patient using a drug with positive OS benefit.
The probability of failure, or the probability that a drug with a particular PFS benefit will produce no positive OS benefit; and
The cost of failure, or the expected reduction in OS benefit net of incremental financial cost, for each patient using a drug with no OS benefit.
The methodology for these value calculations is described in detail in Stevens et al15 and the eAppendix (in the Supplement).
Conventionally, gains in life-years are multiplied by the “value of a statistical life-year” (VSLY) to assign monetary value.16,17 The VSLY is the amount that society would pay to save 1 life-year, when the identity of the affected individual is unknown. Estimates of VSLY vary widely in the literature. The cost-effectiveness literature often cites the value of a quality-adjusted life-year as $50 000 to $100 000 per life-year.18,19 The labor economics, product safety, and workplace safety literatures, on the other hand, publish estimates that center around $300 000 (in 2000 dollars).20 Similar values are reported in studies that use out-of-pocket payment decisions by patients and prescribing decisions by physicians.21,22
We identified a broad range of values for a statistical life-year, from $40 000 to $400 000.19 We consider $100 000, $200 000, and $300 000 as low, medium, and high patient benefit scenarios. The medium scenario corresponds to a widely cited literature review conducted in 200019 reporting a preferred value of $150 000, or approximately $200 000 in 2014 dollars.
To estimate incremental social value as described, we needed 2 pieces of information: (1) the correlation between randomized clinical trial estimates of PFS and OS, which yields probabilities and costs of success and failure; and (2) the incremental cost of new drugs.
To obtain studies containing randomized clinical trial estimates of PFS and OS for a range of drugs, we searched PubMed for phase 3 clinical trials with an NSCLC indication published from 2009 to 2014. We limited our search to phase 3 trials to ensure more uniform trial quality—for example, phase 4 studies are often less rigorous than phase 3 pivotal trials. The years 2009 to 2014 were selected to ensure focus on the most recent innovations in NSCLC therapies. We then applied 3 inclusion criteria: (1) inclusion of PFS or OS outcomes, (2) comparison of 2 or more treatments, and (3) inclusion of Kaplan-Meier tables for the number of patients at risk in each time interval. After identifying an initial set of studies, we conducted a secondary targeted search to identify paired articles for trials with an early PFS publication and a subsequent OS publication.
The analytical framework requires the correlation between mean OS and mean PFS across trials. Clinical trials do not generally contain mean survival or progression benefits because they rarely follow all patients to progression or death. Instead, they typically report median PFS and OS. We addressed this issue by using 2 different sets of estimates and performing a sensitivity analysis. First, we calculated mean PFS and OS estimates using the published data on survival or progression times in the trial, assuming that patients who did not die or progress by the end of the trial did so shortly thereafter. Second, we used the median PFS and OS estimates for sensitivity analyses. The true mean benefits of drugs will typically be higher than either of these approximations, particularly for more effective treatments. Therefore, we understated the value of policies that launch new drugs faster.
In total, we included 27 published clinical trials with 28 comparisons (see eAppendix in the Supplement). We computed the correlation between mean OS and PFS across the trials. We also found 1 paired trial with a delay of 22 months between publication of PFS and OS data. To test sensitivity to this parameter, we relaxed 2 of our 3 inclusion criteria and expanded the trial database to include results of trials at any phase, with or without Kaplan-Meier tables. This resulted in 4 additional paired PFS and OS publications, with publication delays ranging from 7 to 46 months.
To estimate the incremental lifetime cost per treated patient of a novel drug, we conducted a targeted literature review for cost-effectiveness analyses of NSCLC treatments in US dollars. We found 4 published studies with incremental treatment costs (cost of the intervention treatment relative to the comparator) ranging from $2000 to $62 000.23-26 The mean incremental cost across these studies was approximately $22 000 per patient, which served as our medium-value parameter, along with the minimum and maximum values of $2000 and $62 000.
We first calculated the expected OS benefit conditional on PFS greater than 0, 1, 2, and 3 months, respectively. The maximum PFS benefit observed in the sample is 3.52 months, so we treat 3.5 months as an upper bound on all stated benefit ranges; for example, the PFS greater than 1 benefit range includes trials with a PFS benefit of at least 1 month up to 3.5 months. Next, we calculated the incremental social value of using PFS data for (1) the low, medium, and high values of VSLY; and (2) the low, medium, and high values of delay between PFS and OS publications.
Subsequently, we present “indifference curves” along which society is indifferent between granting early access at a given PFS threshold and waiting for OS data. Points on the curves represent an incremental drug cost and VSLY at which the expected net benefit of access based on PFS equals zero.
Finally, we conducted a sensitivity analysis of the effect of “publication bias” on our results.27 Unsuccessful OS trials are less likely to be published. To test the potential effect of this bias on our results, we identified all drug comparisons that had PFS studies without a matching OS publication and assumed that their OS benefits were zero or negative. This assumption is conservative because it presumes the worst-case scenario for publication bias and abstracts from crossover contamination, which would bias downward the PFS-OS correlation in published literature. We reran our analyses using this assumption and present the results.
For the 27 published trials in our example, a positive PFS outcome correctly predicted a positive OS outcome 71% of the time (in 15 of 21 trials with positive PFS outcome). Among the 71% of “successful” cases, the mean OS benefit across trials equaled 1.25 months. Among the 29% of “failed” cases, the mean OS reduction was 0.14 months.
Figure 1 illustrates how expected survival benefits vary with alternative PFS thresholds (defined as PFS greater than or equal to a given number of months). For example, mean OS benefit for trials with positive PFS is approximately 0.85 months. This consists of trials that cover almost 13 000 patients. Narrowing our focus to trials with at least 3-month PFS benefits raises expected OS benefit to 2.85 months based on trials that cover more than 1100 patients.
Figure 2 illustrates the incremental social value of early access, based on different PFS thresholds and values for VSLY (Figure 2A) and publication lag (Figure 2B). It is never socially valuable to grant access to all drugs with positive PFS, regardless of the VSLY or publication lag value. It is always valuable to approve drugs with 3 or more months of PFS benefit. Early access for drugs with PFS benefits greater than 1 or 2 months is valuable if VSLY equals $200 000 or more but not if VSLY is $100 000 or less. For example, the incremental social value is $73 000 for VSLY of $200 000. Lower values of life result in more stringent PFS evidence thresholds. In contrast, changes in the delay between OS and PFS data affect the size of the benefit from early access but never the evidence threshold required.
The 3 lines in Figure 3 illustrate the sensitivity of access decisions to incremental treatment costs and VSLY. The figure shows indifference curves along which society is indifferent between early access and waiting for data. Each point in the figure is a combination of VSLY and incremental cost value. If the point lies below and to the right of an indifference curve for a particular PFS threshold, it is valuable to grant early access at that threshold. To illustrate, granting early access to any drug with positive PFS benefits is worse than waiting for OS, under our medium-value parameters of $200 000 for value of a life-year and $22 000 for incremental treatment cost. However, at these parameters, it is beneficial to grant early access for therapies with PFS greater than 1 month.
Moreover, Figure 3 illustrates that increases in incremental cost must be offset by increases in VSLY to keep society indifferent between early access and waiting for OS data. The 3 dashed horizontal lines contain the low, medium, and high values for incremental treatment cost, of $2000, $22 000, and $62 000. At the high cost value, early access for PFS greater than 3 is beneficial only if VSLY exceeds $260 000. In contrast, at the medium value, it is beneficial for VSLY as low as $180 000.
The sensitivity analysis for publication bias lowers the probability of positive OS, conditional on positive PFS, from 71% to 63%. At $300 000 VSLY, these updated probabilities did not change the PFS benefit threshold at which early access is equivalent to waiting (Table). They did shift benefit thresholds at VSLY of $100 000 (from 2.5 months to greater than 3.5 months) and VSLY of $200 000 (from 1 month to 3 months). We could not calculate the exact threshold for VSLY of $100 000 because we had insufficient data for PFS benefits of greater than 3.5 months.
Data limitations, such as right-censoring in clinical trial data, likely caused our calculations to underestimate the true mean effects of new drugs on OS and PFS. To assess sensitivity, we repeated our calculations using reported median OS and PFS values, which are often preferred by clinicians. Using medians instead of means reduces the PFS thresholds at which providing early access is net beneficial.
Using surrogate end points such as PFS in access decisions may allow patients to benefit from new treatments earlier, but the potential benefits must be weighed against the risks associated with introducing less effective treatments. Our NSCLC example illustrates how the balance of benefits and costs depends crucially on incremental cost and the VSLY. Decision makers can offset higher incremental costs or lower perceived value of life by requiring more stringent PFS benefit thresholds. For example, in our medium-value scenario analyses, a blanket policy launching any drug with positive PFS would always be worse than waiting for OS data. However, increasing the threshold to at least 1 month of PFS benefit would produce benefit in most cases, whereas increasing it to 3 months would produce benefit in all cases.
Apart from economic considerations, the expected benefit of early access also depends on the strength of PFS as a predictor of OS outcomes. For example, when we weakened the correlation between PFS and OS by accounting for potential publication bias, the necessary PFS threshold increased by as many as 2 additional months. More generally, in tumor types or treatment contexts with weaker PFS-OS correlations, decision makers should require greater evidence of PFS benefit before granting early access, and vice versa.
Our calculations reflect the societal perspective. In contrast, the payer perspective likely internalizes less than the full value of lives saved, whereas patients internalize less than the full cost of treatment. As a result, the payer perspective will tilt away from early access based on PFS, whereas the patient perspective will tilt toward early access.
Several limitations of our analysis should be noted. First, the published literature is limited by the history of innovation in a disease area. For NSCLC, the data became sparse at PFS benefit of 3.5 or greater. If newer agents are more effective than older ones, this would require extrapolation beyond what is contained in the historical data.
Second, trials that follow patients over a long period are subject to “crossover contamination,” in which patients switch or “cross over” from the control arm to the intervention arm. When high-risk patients cross over to the intervention because of disease progression or demonstrated efficacy, this results in an underestimate of OS benefits in long-running trials. As such, published estimates of OS might be unreliably low compared with actual OS benefits experienced by patients, and the correlation between published PFS and OS estimates might similarly understate the true correlation.
Third, the expected (mean) value is the relevant metric from a decision-analytic perspective, but trials typically report median benefits rather than expected benefits. Our analysis recovered mean benefits by extrapolating expected values from reported survival curves. This is imperfect because the ultimate survival outcome is unknown for patients who did not die or progress by the end of the trial. Thus, expected survival is always understated.
Fourth, our analysis presents incremental social value of early access and abstracts from the likely possibility of heterogeneous outcomes across patients. If PFS is a better or worse surrogate for some subpopulations, PFS benefit thresholds may need to vary across those subpopulations.
A host of important policies interact with our analysis. For example, there is growing misalignment between the end points that are typically used for drug approval and those that are considered acceptable for reimbursement. The FDA is typically receptive to PFS as a way to expedite drug development and approval.28 Unlike the FDA, payers are not insulated from growth in the cost of therapy. Thus, payer and FDA incentives differ. Payers must weigh the value of early access against the cost of paying for new, and often expensive, therapies that might not be working as well as the PFS data suggest. Our framework helps illustrate how these 2 offsetting forces can be systematically quantified and compared.
Granting early access to novel therapies on the basis of PFS data can provide value to patients in need of life-extending therapy, but at the risk of reimbursing novel therapies that later prove ineffective or harmful. Decision makers must carefully balance the risks and rewards of early access based on surrogate end points such as PFS. We showed how the optimal decision varies substantially with incremental treatment cost, the value of life, and the correlation between PFS and OS. When faced with cheaper therapies, higher values of life, or stronger PFS-OS correlations, decision makers ought to lower thresholds for PFS benefit, and vice versa.
In the example of NSCLC, policies granting early access for all drugs with positive PFS are not preferred except in cases in which incremental treatment cost is far toward the low end of the range. Conversely, granting access for drugs with PFS benefits greater than 1 month was preferred to waiting for OS data for medium patient benefit parameter values, whereas access for those with PFS benefits greater than 2 months was preferred for a relatively robust range of parameter values. Finally, policies granting early access for PFS benefits greater than 3 months were always preferred.
More generally, our framework demonstrates how to make socially efficient decisions for a range of incremental cost, value of life, and PFS benefit assumptions. For example, if future NSCLC breakthroughs bring higher costs, benefit thresholds can be adjusted accordingly using this analytic framework. Similarly, if future breakthroughs demonstrate greater PFS benefit than historical ones, a stronger case can be made for early access and vice versa. The use of PFS in early-access decisions holds both risks and rewards that must be carefully and systematically balanced by society.
Accepted for Publication: January 19, 2015.
Corresponding Author: Darius N. Lakdawalla, PhD, Leonard D. Schaeffer Center for Health Policy and Economics, University of Southern California, 635 Downey Way, Verna and Peter Dauterive Hall (VPD), Second Floor, Los Angeles, CA 90089-3333 (email@example.com).
Published Online: March 19, 2015. doi:10.1001/jamaoncol.2015.0203.
Author Contributions: Dr Lakdawalla and Ms Chou had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: All authors.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Lakdawalla, Chou, Linthicum, MacEwan.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Lakdawalla, Chou, MacEwan.
Obtained funding: Lakdawalla, Zhang, Goldman.
Administrative, technical, or material support: Lakdawalla, Chou, Linthicum, Zhang.
Study supervision: Lakdawalla, MacEwan, Goldman.
Conflict of Interest Disclosures: Drs Lakdawalla and Goldman hold the positions of Partner at Precision Health Economics, a health care consultancy. Ms Chou, Mr Linthicum, and Dr MacEwan are employees of Precision Health Economics. Precision Health Economics maintained all rights to publication subject to a time-limited period for review and comment by Novartis. No other disclosures are reported.
Funding/Support: Drs Lakdawalla and Goldman received grant support for this work from the National Institutes on Aging (grant P01 AG033559). Additional support was provided by Novartis Pharmaceuticals under contract with Precision Health Economics.
Role of the Funder/Sponsor: The National Institutes on Aging had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The coauthor of this study who is also an employee of Novartis (Dr Zhang) was fully involved with all aspects of this research, including the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.
Create a personal account or sign in to: