Of the 9 resubmissions, 7 were previously submitted and received a negative reimbursement recommendation and 2 were withdrawn from consideration before the final reimbursement recommendation was delivered.
The dashed blue line represents the overall median gain in PFS, excluding values of 0 months (ie, no statistically significant difference between the intervention and comparator). The orange vertical lines represent the first and third quartile of PFS. Median (interquartile range) PFS gains were 4.0 (2.4-7.1) months.
The dashed blue line represents the overall median gain in OS, excluding values of 0 months (ie, no statistically significant difference between the intervention and comparator). The orange vertical lines represent the first and third quartile of OS. Median (interquartile range) OS gains were 3.6 (2.2-6.2) months.
eTable 1. Examples of Trial Selection in the Present Study
eTable 2. Characteristics of the pCODR Submissions for Solid-Tumor Indications Between 2011-2020
eTable 3. List of All 104 Oncology Drugs With Final Recommendations From pCODR Between 2011-2020
eTable 4. Select Clinical Outcomes of All 104 Oncology Drugs With Final Recommendations From pCODR Between 2011-2020
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
Meyers DE, Jenei K, Chisamore TM, Gyawali B. Evaluation of the Clinical Benefit of Cancer Drugs Submitted for Reimbursement Recommendation Decisions in Canada. JAMA Intern Med. 2021;181(4):499–508. doi:10.1001/jamainternmed.2020.8588
What are the differences in the clinical evidence and benefit between cancer drugs that received a positive vs those that received a negative reimbursement recommendation from the pan-Canadian Oncology Drug Review (pCODR)?
In this cohort study of 104 reimbursement recommendation decisions, 78 received a positive recommendation from the pCODR. Of these, 72 were based on phase 3 randomized clinical trials and 39 had proven overall survival benefit with median gains of 3.7 months.
Results of this study found that only half of the pCODR-recommended cancer drugs showed improved survival, and the survival gains were usually modest; these findings suggest that, despite the pCODR framework, cancer drugs without meaningful patient benefit continue to be reimbursed in the Canadian market.
Cancer drugs approved by the US Food and Drug Administration have come under scrutiny for marginal clinical benefits; however, the clinical benefits of cancer drugs recommended for reimbursement in Canada have not been adequately studied.
To assess the differences in the clinical evidence and benefit of cancer drugs that received a positive vs a negative recommendation for provincial reimbursement in Canada.
Design, Setting, and Participants
This cohort study obtained publicly available regulatory documents from the pan-Canadian Oncology Drug Review (pCODR) and corresponding clinical trial documentation. All cancer drugs with a solid tumor indication that were submitted from the inception of the pCODR (July 2011) to February 2020 were evaluated. To be included, submissions had to have a final reimbursement recommendation; submissions that were incomplete, were withdrawn, or had a pending decision were excluded.
A completed reimbursement recommendation decision from the pCODR.
Main Outcomes and Measures
Final reimbursement recommendation (positive vs negative); trial characteristics; and relevant clinical outcomes (ie, overall survival [OS] and progression-free survival [PFS]), including the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS) scores available at the time of pCODR assessment.
Between 2011 and 2020, the pCODR issued 104 reimbursement recommendation decisions for cancer drugs with a solid tumor indication. Among these drug submissions, 78 (75.0%) received a positive recommendation, of which 72 (92.3%) were conditional. Drugs that received a positive recommendation compared with those with a negative recommendation were more likely to have phase 3 randomized clinical trial design (92.3% [72 of 78] vs 53.8% [14 of 26]; P < .001) and have substantial benefit according to the ESMO-MCBS scores (61.5% [48 of 78] vs 19.2% [5 of 26]; P < .001). The most common primary end points associated with the successful submissions were PFS (53.9%) and OS (32.1%). Overall, 39 of 78 submissions (50.0%) that received a positive recommendation had shown OS benefit, with median (interquartile range) OS gains of 3.7 (2.7-6.5) months.
Conclusions and Relevance
This cohort study found that, although the pCODR takes into account the magnitude of clinical benefit, only half of the cancer drugs that received a positive recommendation had evidence of improved OS and the survival gains were usually modest. These results suggest that, although the pCODR helps filter out some cancer drugs with low quality of evidence and low magnitude of benefit, cancer drugs without meaningful patient benefit continue to enter the Canadian market; these findings are important for making reimbursement policy decisions globally.
The clinical benefit and cost of cancer drugs have come under scrutiny in recent years.1-4 Although the cost of cancer drugs continues to escalate,1,2 the clinical benefits of most modern cancer drugs seem to be either modest or uncertain.3,4 Cancer drugs are prescribed to patients to improve either the duration or the quality of life, but ideally both. However, analyses of cancer drugs approved by the US Food and Drug Administration (FDA)3,5 or the European Medicines Agency6 revealed concerns that most of these drugs lacked evidence that they improved the overall survival (OS) or quality of life (QOL) of patients with cancer or that these clinically meaningful end points were not evaluated in relevant clinical trials. Instead, the use of surrogate measures, such as progression-free survival (PFS) or response rates for drug approval decisions, has increased. These surrogate measures are known to poorly correlate with improvements in OS or QOL for most tumor types.7-10
For drugs that are associated with improved OS, the gains in OS are usually minimal. An analysis of cancer drugs approved in the US between 2002 and 2014 showed that less than half of these approvals met the threshold for meaningful benefit,3 which was set by the American Society of Clinical Oncology standard.11 Notwithstanding this lack of substantial clinical benefit, new cancer drugs routinely cost upward of US $10 000 per month.1 This disconnect between benefit and cost of cancer drugs has partly been ascribed to the FDA not taking the magnitude of benefit, or the cost-effectiveness of drugs, into account in its approval process.12
In Canada, the pan-Canadian Oncology Drug Review (pCODR), a subsidiary of the Canadian Agency for Drugs and Technologies in Health, provides a reimbursement recommendation for cancer drugs that takes clinical benefit, cost-effectiveness, patient-based values, and adoption feasibility into account.13 In this cohort study, we systematically reviewed all cancer drugs submitted to the pCODR for reimbursement decision and assessed the availability and quality of clinical evidence at the time of recommendation. In addition, we assessed the differences in those metrics between drugs that received a positive and those that received a negative recommendation for provincial reimbursement.
According to the Common Rule, institutional review board approval was not required for this study because all of the data used in this study were obtained from public records and did not include individual-level, patient-identifying information. This study of published reports did not involve patient-identifying data and was not submitted for institutional review board approval. Data were collected between March 1, 2020, and May 1, 2020.
For a cancer drug to be marketed in Canada, it must first be approved for sale by Health Canada, appraised for a reimbursement recommendation by the pCODR, and then funded for reimbursement at the provincial level. Although the pCODR has no mandate to assess all drugs that receive approval from Health Canada, manufacturers that wish to have their drugs in the Canadian market generally submit to the pCODR at the same time as, or after, Health Canada approval. After the pCODR assesses the submitted data for clinical and economic benefit, the pCODR Expert Review Committee provides a nonbinding recommendation (yes, conditionally yes, or no) for provincial reimbursement. The pCODR makes these reports publicly available online.14
Conditional recommendations are usually conditional on cost-effectiveness and are considered to be positive recommendations (ie, recommended for funding at the provincial level). The conditional recommendations process in Canada is not similar to the accelerated approval pathway in the US, which signifies uncertainty in the evidence and mandates a confirmatory trial for conversion to a regular approval. No postmarketing confirmatory trials are mandated for the pCODR conditional recommendations. For example, in 2019, pembrolizumab was conditionally recommended for the adjuvant treatment of advanced melanoma based on limitations in cost-effectiveness. As of September 2020, the drug has been funded in 7 provinces with various eligibility criteria, whereas 2 provinces remain in negotiations. Despite being funded, pembrolizumab remains conditionally recommended for this indication. Drugs that have received approval from Health Canada but negative recommendations from the pCODR will not be funded but can still be used in Canada through compassionate access programs, private insurance plans, or out-of-pocket costs.
We studied all completed submissions for cancer drugs with a solid tumor indication in adult patients from the inception of the pCODR (July 2011) to February 2020. To be included, submissions had to have received a final reimbursement recommendation; submissions that were incomplete, were withdrawn, or had a pending decision were excluded. Submissions made for biosimilar drugs, pediatric indications, or hematologic malignant neoplasms were excluded. These exclusions were made to ensure the homogeneity of the data set and to allow for assessment of clinical benefits using the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS), a tool developed for solid tumor indications.
For drugs that met the inclusion criteria, the following variables were extracted: final reimbursement recommendation (yes, conditionally yes, or no), tumor indication, treatment setting, line of therapy, end points of the primary clinical trial informing the submission, and relevant clinical outcomes such as the medians and hazard ratios (HRs) of OS or PFS. We dichotomized reimbursement decisions as positive (including conditional) or negative. Outcome data from clinical trials must have been available at the time of the pCODR assessment. All data were extracted from the pCODR documents and cross-referenced with ClinicalTrials.gov to ensure accuracy using the trial name and national clinical trial registration number. When multiple studies were cited as contributing to the body of evidence for a particular submission, we chose the most relevant trial (on the basis of phase, recency, and trial location) for review (eTable 1 in the Supplement). Two of us (D.E.M. and K.J.) extracted the pCODR data separately, and any discrepancies were resolved by consensus with input from an oncologist (B.G.).
We designated a drug as showing improved PFS or OS if a positive difference was found between the intervention and comparator groups and if the CI of the HR did not cross 1.0. If the experimental group did not reach the median PFS and OS at the time of assessment by the pCODR, then we estimated the median PFS and OS by multiplying the median PFS and OS of the control group by the point estimate of the HR, a practice that is in keeping with the calculation of the ESMO-MCBS.15 If both the experimental and control groups had not reached the median despite statistically significant HR and CI, the drug was designated as having a clinical benefit of unknown magnitude and was excluded from the calculation of median PFS and OS gains.
We calculated the median of the median gains in PFS and OS across all the submissions and separately for submissions that received positive and negative reimbursement recommendations, only if the gains were statistically significant. For a conservative estimate, we also included the submissions for which the difference in median PFS and OS was not statistically significant by inputting a value of 0 for median PFS and OS gains for these submissions.
The clinical benefit of cancer drugs within this sample was also categorized using the ESMO-MCBS, a validated framework16 widely used in the assessment of the clinical benefit of cancer drugs. These ESMO-MCBS scores were either extracted from the ESMO website, where available, or calculated using the publicly available score cards.17 The ESMO-MCBS assigns a score of 1 (low) to 5 (high) for drugs used in the advanced or metastatic setting and a score of C (low) to A (high) for drugs that are potentially curative or used in the adjuvant setting. Scores of 4 to 5 and A to B are designated as having substantial or major clinical benefit. These scores take into account the magnitude of not only efficacy but also adverse events and quality of life.
We compared the characteristics of the trials and the clinical benefits of submissions with positive vs negative recommendations. The differences in medians of continuous variables (ie, PFS and OS) and the dependence of categorical variables were tested using permutation testing.18 We chose this approach because we generally did not have sufficient data to rely on asymptotic testing (ie, χ2 test for categorical data). P values were not adjusted for multiple hypothesis testing, and 2-sided P < .05 was considered to be statistically significant. Analyses were performed with R, version 3.6.2 (R Foundation for Statistical Computing). Permutation testing was completed with the infer package (Creative Commons).
A total of 104 cancer drug submissions met the eligibility criteria and were included in this study (Figure 1). Among these submissions, 54 (51.9%) were for new indications of drugs previously submitted to the pCODR, 41 (39.4%) were for novel drugs with no previous submissions, and 9 (8.7%) were resubmissions that either were not completed (n = 2) or received negative decisions (n = 7).
Lung cancer was the most common primary indication (25 [24.0%]), and small molecules were the most common drug type (48 [46.2%]). Most submissions were for the advanced or metastatic setting (97 [93.3%]), with 37 (35.6%) for treatment in the first line (eTables 2 and 3 in the Supplement).
Of the 104 submissions, 79 (76.0%) had OS data and 82 (78.8%) had PFS data available at the time of review. Seventy-eight submissions (75.0%) received a positive recommendation for provincial reimbursement (6 outright recommendations and 72 conditional recommendations), and 26 submissions (25.0%) received a negative recommendation. The most common reason cited for a conditional recommendation was cost-effectiveness (70 of 72 [97.2%]). During the same period, the FDA granted 163 drug approvals for adult solid tumors, of which 121 were regular (74.2%) and 42 were accelerated (25.7%) approvals. None of the 26 submissions that received a negative recommendation from the pCODR had been rejected by the FDA.
No statistically significant difference was observed in the distributions of tumor type, drug class, treatment setting, or line of therapy between submissions that received positive and negative recommendations; however, there were significant differences with regard to certain trial characteristics (Table 1).
Of the drugs that received a positive recommendation for reimbursement, 92.3% (72 of 78) had evidence from a phase 3 randomized clinical trial (RCT) compared with 53.8% (14 of 26) of the drugs that received a negative recommendation (P < .001). The most common primary end points associated with the successful submissions were PFS (53.8%) and OS (32.1%). The most common primary end point among the submissions that received a positive recommendation was PFS (42 [53.8%]) vs response rates among those that received a negative recommendation (10 [38.5%]). In addition, drugs that received a positive recommendation were statistically significantly more likely to have data available for PFS and OS at the time of pCODR assessment (Table 1 and eTable 4 in the Supplement). No significant differences were found in the distribution of submissions with evidence of OS or PFS benefit on the basis of reimbursement recommendations.
Forty-seven submissions had documented OS improvements, of which 39 (83.0%) received a positive recommendation. Twenty-six submissions without evidence of OS benefit and 13 with no OS data available at the time of pCODR assessment also received a positive recommendation. Thus, only 50.0% (39 of 78) of cancer drugs recommended for reimbursement in Canada had shown an improvement in OS.
The differences between submissions that received positive and those that received negative reimbursement recommendations did not reach statistical significance for median (interquartile range) gains in PFS (4.7 [2.8-8.2] vs 2.9 [0.8-4.0]; P = .06) or in OS (3.7 [2.7-6.5] vs 1.9 [1.4-4.5]; P = .06) (Table 1, Figure 2, and Figure 3). When submissions with OS or PFS gains of 0 months were included, the median gains in PFS for submissions with positive vs negative recommendations were 3.9 vs 2.7 (P = .29) and the median gains in OS were 1.9 vs 1.4 (P = .92). These calculations included only those submissions with OS and PFS data available at the time of the recommendation. Submissions that received a positive reimbursement recommendation were more likely to have demonstrated substantial clinical benefit according to the ESMO-MCBS scores compared with those that received a negative recommendation (48 [61.5%] vs 5 [19.2%]; P < .001) (Table 1).
Although the pCODR recommendations are nonbinding, we found 100% concordance between pCODR recommendations and provincial funding decisions for submissions that had received “recommend for funding” and “do not recommend for funding” decisions (based on available data). However, of the 72 conditional recommendations, 6 (8.3%) were generally not funded by the provinces that had completed their deliberations.
We evaluated the characteristics of trials and the magnitude of clinical benefit of cancer drugs with a solid tumor indication submitted to the pCODR for Canadian provincial reimbursement. We found that three-quarters of all submissions received a positive recommendation, of which half demonstrated improved survival with median gains in OS of 3.7 months. Significant differences in trial characteristics were observed, such as randomization, phase, and primary end points, between submissions that received a positive recommendation and submissions that received a negative recommendation. These findings have multiple regulatory and reimbursement policy lessons globally.
These current findings must be placed into context by examining the results of similar studies that evaluated the magnitude of clinical benefit and the quality of evidence for cancer drugs approved by the FDA at different time points (Table 219,20) to facilitate a qualitative comparison. We found that 61.5% of cancer drugs (48 of 78) that received a positive recommendation from the pCODR had demonstrated substantial clinical benefit according to the ESMO-MCBS score, in contrast to the data previously reported for the FDA: 43.8% of drug approvals (46 of 105) based on RCTs21 and 33.8% of drug approvals (45 of 133) based on single-arm trials22 demonstrated substantial clinical benefit using the same scale (ESMO-MCBS). Similarly, although only 6.4% of cancer drugs (5 of 78) that received a positive recommendation by the pCODR were based on response rates, the corresponding percentage was 34.9% of drugs (22 of 63) approved by the FDA.20 For OS, 50.0% of drugs (39 of 78) recommended by the pCODR and 30.2% of drugs (19 of 63) approved by the FDA showed improved OS.20 With regard to the quality of evidence, 92.3% (72 of 78) of positive recommendations by the pCODR but only 75.8% (116 of 153) of FDA approvals were based on RCTs.
One explanation for these differences in results is that the pCODR uses a multicriteria deliberative framework that takes into account not only the presence or absence of clinical benefit but also the magnitude of benefit, quality of evidence, economic evaluation, and feedback from patient and clinician stakeholders. This approach is in contrast to that of the FDA, which considers neither the magnitude of benefit nor the cost. We found that all 26 cancer drugs that were not recommended for reimbursement by the pCODR had previously been approved by the FDA, which is similar to the findings of a previous study.23 Furthermore, although the FDA granted 163 approvals during the same period, the pCODR received only 104 submissions, of which only 78 received a positive reimbursement recommendation. These data support the notion that the pCODR framework might filter out drugs with minimal clinical utility. Integrating the ESMO-MCBS score as a preliminary tool may help the pCODR to further filter out minimally effective cancer drugs at the outset, an approach that is being used in certain countries in Europe and by the World Health Organization in formulating the essential medicine list in cancer.24
Results from the present study identified areas of improvement for the Canadian regulatory system. First, the quality of evidence for most of the submissions that received a positive recommendation was low. Less than one-third of submissions that received a positive recommendation from the pCODR had OS as the primary end point. In the US, the proportion of cancer drugs that are approved on the basis of surrogates has increased over the past decade, peaking at 93% in 2017.19 Although PFS was the most frequently used primary end point in the current study, data suggest that PFS is not a valid surrogate for clinical outcomes such as OS8,10,25 or QOL.9,26 Previous work has shown that more than half of submissions to the pCODR between 2015 and 2018 did not have data on QOL.27 None of the pCODR submissions that we evaluated had QOL as a primary outcome measure. Thus, moving forward, clinical trials should prioritize meaningful clinical end points, such as OS and QOL, to ensure that regulatory decisions are guided by high-quality evidence.
Second, cancer drugs that were recommended for reimbursement in Canadian provinces were not shown to have substantially improved clinical outcomes for patients. Across all submissions with a positive recommendation, the median OS gains were only 3.7 months and the median PFS gains were 4.7 months, which are not substantially different from the survival gains in submissions that received a negative recommendation. Furthermore, nearly 40% of cancer drugs that received a positive recommendation did not reach the threshold for substantial clinical benefit per the ESMO-MCBS. This finding demonstrates room for improvement in the Canadian system.
Previous studies have also examined the pCODR decision process from different angles and have highlighted other areas of improvement. A study by Niraula and Nugent23 found that in a cohort of 91 cancer drugs approved by the FDA, 18 (20%) received a negative recommendation by the pCODR, with higher incremental cost-effectiveness ratio being a factor associated with a negative recommendation. Other studies have found that the economic models submitted to the pCODR by drug manufacturers were often flawed as identified by the economic reviewers28 of the pCODR and cost-effectiveness appeared to be associated with a full vs conditional reimbursement decision rather than a positive vs negative decision.29 In addition, although patient values were considered in stakeholder feedback, both patients30 and clinicians31 involved in the process were shown to have relevant financial conflicts of interest.
This study has some limitations. The study is unique in that it comprehensively summarized and quantified the clinical evidence as well as the magnitude of clinical benefit for cancer drugs that were recommended or not recommended for reimbursement in Canada. It did not, however, assess the role of cost or cost-effectiveness in this decision-making process. The pCODR bases its recommendation decision not only on quality of evidence and magnitude of benefit but also on economic evaluation, stakeholder views, and adoption feasibility; as such, this analysis may not be fully representative of all the aspects that the pCODR includes in its deliberation. We assessed only the quality of evidence on trial phase and randomization evaluated by the pCODR, but we did not assess other characteristics of the trials. A complete assessment is important given recent data that suggested that trials used to inform regulatory decisions for cancer drugs by both the European Medicines Agency32 and the FDA33 were at high risk of bias and had critical design flaws. In addition, we limited our analyses to data that were available to the pCODR at the time of deliberation and did not consider evidence that may have been subsequently generated.
In this cohort study, we found that 75% of submissions to the pCODR for provincial reimbursement of cancer drugs in Canada received a positive reimbursement recommendation; these submissions were mostly based on phase 3 RCTs, with half of them showing improved OS. Cancer drugs recommended for reimbursement in Canada also have issues, such as the use of surrogate end points and a modest margin of clinical benefit. Even in a regulatory system like the pCODR with a central evidence appraisal process, including assessment of evidence quality, magnitude of clinical benefit, and cost-effectiveness, cancer drugs without meaningful patient benefit are entering the market. These findings have implications for regulatory and reimbursement policy decisions in Canada and globally.
Accepted for Publication: December 11, 2020.
Published Online: February 22, 2021. doi:10.1001/jamainternmed.2020.8588
Corresponding Author: Bishal Gyawali, MD, PhD, Division of Cancer Care and Epidemiology, Queen’s University Cancer Research Institute, 10 Stuart St, Kingston, ON K7L 3N6, Canada (firstname.lastname@example.org).
Author Contributions: Dr Meyers and Ms Jenei had full access to all the data in the study and take responsibility for the integrity of the data collection. Mr Chisamore takes responsibility for the accuracy of the data analyses.
Concept and design: Meyers, Jenei, Gyawali.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Meyers, Jenei, Gyawali.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Chisamore.
Administrative, technical, or material support: Meyers, Jenei, Gyawali.
Other - data entry, revision: Jenei.
Conflict of Interest Disclosures: None reported.