Evaluation of the Clinical Benefit of Cancer Drugs Submitted for Reimbursement Recommendation Decisions in Canada | Oncology | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Identification of Completed Pan-Canadian Oncology Drug Review (pCODR) Submissions of Drugs for Solid Tumor Indications Between July 2011 and February 2020
Identification of Completed Pan-Canadian Oncology Drug Review (pCODR) Submissions of Drugs for Solid Tumor Indications Between July 2011 and February 2020

Of the 9 resubmissions, 7 were previously submitted and received a negative reimbursement recommendation and 2 were withdrawn from consideration before the final reimbursement recommendation was delivered.

Figure 2.  Gains in Progression-Free Survival (PFS) of 82 Pan-Canadian Oncology Drug Review (pCODR) Submissions With Data Available at the Time of Assessment
Gains in Progression-Free Survival (PFS) of 82 Pan-Canadian Oncology Drug Review (pCODR) Submissions With Data Available at the Time of Assessment

The dashed blue line represents the overall median gain in PFS, excluding values of 0 months (ie, no statistically significant difference between the intervention and comparator). The orange vertical lines represent the first and third quartile of PFS. Median (interquartile range) PFS gains were 4.0 (2.4-7.1) months.

Figure 3.  Gains in Overall Survival (OS) of 79 Pan-Canadian Oncology Drug Review (pCODR) Submissions With Data Available at the Time of Assessment
Gains in Overall Survival (OS) of 79 Pan-Canadian Oncology Drug Review (pCODR) Submissions With Data Available at the Time of Assessment

The dashed blue line represents the overall median gain in OS, excluding values of 0 months (ie, no statistically significant difference between the intervention and comparator). The orange vertical lines represent the first and third quartile of OS. Median (interquartile range) OS gains were 3.6 (2.2-6.2) months.

Table 1.  Comparison of Characteristics of Cancer Drug Submissions Recommended vs Not Recommended for Reimbursement by Pan-Canadian Oncology Drug Review Between 2011 and 2020
Comparison of Characteristics of Cancer Drug Submissions Recommended vs Not Recommended for Reimbursement by Pan-Canadian Oncology Drug Review Between 2011 and 2020
Table 2.  Trial Characteristics and Clinical Benefits of Cancer Drugs Recommended for Reimbursement by Pan-Canadian Oncology Drug Review vs Cancer Drugs Approved by the US Food and Drug Administrationa
Trial Characteristics and Clinical Benefits of Cancer Drugs Recommended for Reimbursement by Pan-Canadian Oncology Drug Review vs Cancer Drugs Approved by the US Food and Drug Administrationa
1.
Saluja  R, Arciero  VS, Cheng  S,  et al.  Examining trends in cost and clinical benefit of novel anticancer drugs over time.   J Oncol Pract. 2018;14(5):e280-e294. doi:10.1200/JOP.17.00058 PubMedGoogle ScholarCrossref
2.
IQVIA Institute for Human Data Science. Global oncology trends 2019: therapeutics, clinical development and health system implications. May 2019. Accessed July 25, 2020. https://www.iqvia.com/insights/the-iqvia-institute/reports/global-oncology-trends-2019
3.
Fojo  T, Mailankody  S, Lo  A.  Unintended consequences of expensive cancer therapeutics—the pursuit of marginal indications and a me-too mentality that stifles innovation and creativity: the John Conley Lecture.   JAMA Otolaryngol Head Neck Surg. 2014;140(12):1225-1236. doi:10.1001/jamaoto.2014.1570 PubMedGoogle ScholarCrossref
4.
Kumar  H, Fojo  T, Mailankody  S.  An appraisal of clinically meaningful outcomes guidelines for oncology clinical trials.   JAMA Oncol. 2016;2(9):1238-1240. doi:10.1001/jamaoncol.2016.0931 PubMedGoogle ScholarCrossref
5.
Kim  C, Prasad  V.  Cancer drugs approved on the basis of a surrogate end point and subsequent overall survival: an analysis of 5 years of US Food and Drug Administration approvals.   JAMA Intern Med. 2015;175(12):1992-1994. doi:10.1001/jamainternmed.2015.5868 PubMedGoogle ScholarCrossref
6.
Davis  C, Naci  H, Gurpinar  E, Poplavska  E, Pinto  A, Aggarwal  A.  Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009-13.   BMJ. 2017;359:j4530. doi:10.1136/bmj.j4530 PubMedGoogle ScholarCrossref
7.
Chen  EY, Haslam  A, Prasad  V.  FDA acceptance of surrogate end points for cancer drug approval: 1992-2019.   JAMA Intern Med. 2020;180(6):912-914. doi:10.1001/jamainternmed.2020.1097 PubMedGoogle ScholarCrossref
8.
Haslam  A, Hey  SP, Gill  J, Prasad  V.  A systematic review of trial-level meta-analyses measuring the strength of association between surrogate end-points and overall survival in oncology.   Eur J Cancer. 2019;106:196-211. doi:10.1016/j.ejca.2018.11.012 PubMedGoogle ScholarCrossref
9.
Hwang  TJ, Gyawali  B.  Association between progression-free survival and patients’ quality of life in cancer clinical trials.   Int J Cancer. 2019;144(7):1746-1751. doi:10.1002/ijc.31957 PubMedGoogle ScholarCrossref
10.
Booth  CM, Eisenhauer  EA.  Progression-free survival: meaningful or simply measurable?   J Clin Oncol. 2012;30(10):1030-1033. doi:10.1200/JCO.2011.38.7571 PubMedGoogle ScholarCrossref
11.
Ellis  LM, Bernstein  DS, Voest  EE,  et al.  American Society of Clinical Oncology perspective: raising the bar for clinical trials by defining clinically meaningful outcomes.   J Clin Oncol. 2014;32(12):1277-1280. doi:10.1200/JCO.2013.53.8009 PubMedGoogle ScholarCrossref
12.
Darrow  JJ, Avorn  J, Kesselheim  AS.  FDA approval and regulation of pharmaceuticals, 1983-2018.   JAMA. 2020;323(2):164-176. doi:10.1001/jama.2019.20288 PubMedGoogle ScholarCrossref
13.
Canadian Agency for Drugs and Technologies in Health. Procedures for the CADTH Pan-Canadian Oncology Drug Review. April 2020. Accessed June 10, 2020. https://www.cadth.ca/sites/default/files/pcodr/pCODR%27s%20Drug%20Review%20Process/pcodr-procedures.pdf
14.
Canadian Agency for Drugs and Technologies in Health. Reimbursement review reports. Accessed July 25, 2020. https://www.cadth.ca/reimbursement-review-reports
15.
European Society for Medical Oncology. ESMO-Magnitude of Clinical Benefit Scale: evaluation forms version 1.1. Accessed May 5, 2020. https://www.esmo.org/guidelines/esmo-mcbs/scale-evaluation-forms-v1.0-v1.1/scale-evaluation-forms-v1.1
16.
Cherny  NI, Dafni  U, Bogaerts  J,  et al.  ESMO-Magnitude of Clinical Benefit Scale version 1.1.   Ann Oncol. 2017;28(10):2340-2366. doi:10.1093/annonc/mdx310 PubMedGoogle ScholarCrossref
17.
European Society for Medical Oncology. ESMO-MCBS scorecards. Accessed September 23, 2020. https://www.esmo.org/guidelines/esmo-mcbs/esmo-magnitude-of-clinical-benefit-scale
18.
Pesarin  F, Salmaso  L.  Permutation Tests for Complex Data: Theory, Applications and Software. John Wiley & Sons Ltd; 2010. doi:10.1002/9780470689516
19.
Gyawali  B, Sharma  S, Booth  CM.  Is the number of cancer drug approvals a surrogate for regulatory success?   J Cancer Policy. 2019;22:100202. doi:10.1016/j.jcpo.2019.100202Google Scholar
20.
Mailankody  S, Prasad  V.  Five years of cancer drug approvals: innovation, efficacy, and costs.   JAMA Oncol. 2015;1(4):539-540. doi:10.1001/jamaoncol.2015.0373 PubMedGoogle ScholarCrossref
21.
Tibau  A, Molto  C, Ocaña  A,  et al.  Magnitude of clinical benefit of cancer drugs approved by the US Food and Drug Administration.   J Natl Cancer Inst. 2018;110(5):486-492. doi:10.1093/jnci/djx232 PubMedGoogle ScholarCrossref
22.
Tibau  A, Molto  C, Borrell  M,  et al.  Magnitude of clinical benefit of cancer drugs approved by the US Food and Drug Administration based on single-arm trials.   JAMA Oncol. 2018;4(11):1610-1611. doi:10.1001/jamaoncol.2018.4300 PubMedGoogle ScholarCrossref
23.
Niraula  S, Nugent  Z.  New cancer drug approvals from the perspective of a universal healthcare system: analyses of the Pan-Canadian Oncology Drug Review recommendations.   J Natl Compr Canc Netw. 2018;16(12):1460-1466. doi:10.6004/jnccn.2018.7084 PubMedGoogle ScholarCrossref
24.
European Society for Medical Oncology. ESMO-MCBS and the World Health Organisation (WHO) model list of essential medicines. Accessed September 10, 2020. https://www.esmo.org/guidelines/esmo-mcbs/esmo-mcbs-and-the-who
25.
Kemp  R, Prasad  V.  Surrogate endpoints in oncology: when are they acceptable for regulatory and clinical decisions, and are they currently overused?   BMC Med. 2017;15(1):134-137. doi:10.1186/s12916-017-0902-9 PubMedGoogle ScholarCrossref
26.
Kovic  B, Jin  X, Kennedy  SA,  et al.  Evaluating progression-free survival as a surrogate outcome for health-related quality of life in oncology: a systematic review and quantitative analysis.   JAMA Intern Med. 2018;178(12):1586-1596. doi:10.1001/jamainternmed.2018.4710 PubMedGoogle ScholarCrossref
27.
Raymakers  AJN, Regier  DA, Peacock  SJ.  Health-related quality of life in oncology drug reimbursement submissions in Canada: a review of submissions to the Pan-Canadian Oncology Drug Review.   Cancer. 2020;126(1):148-155. doi:10.1002/cncr.32455 PubMedGoogle ScholarCrossref
28.
Masucci  L, Beca  J, Sabharwal  M, Hoch  JS.  Methodological issues in economic evaluations submitted to the Pan-Canadian Oncology Drug Review (pCODR).   Pharmacoecon Open. 2017;1(4):255-263. doi:10.1007/s41669-017-0018-3 PubMedGoogle ScholarCrossref
29.
Skedgel  C, Wranik  D, Hu  M.  The relative importance of clinical, economic, patient values and feasibility criteria in cancer drug reimbursement in Canada: a revealed preferences analysis of recommendations of the Pan-Canadian Oncology Drug Review 2011-2017.   Pharmacoeconomics. 2018;36(4):467-475. doi:10.1007/s40273-018-0610-0 PubMedGoogle ScholarCrossref
30.
Lexchin  J.  Association between commercial funding of Canadian patient groups and their views about funding of medicines: an observational study.   PLoS ONE. 2019;14(2):e0212399. doi:10.1371/journal.pone.0212399 PubMedGoogle Scholar
31.
Lexchin  J.  Financial conflicts of interest of clinicians making submissions to the pan-Canadian Oncology Drug Review: a descriptive study.   BMJ Open. 2019;9(7):e030750. doi:10.1136/bmjopen-2019-030750 PubMedGoogle Scholar
32.
Naci  H, Davis  C, Savović  J,  et al.  Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014-16: cross sectional analysis.   BMJ. 2019;366:l5221. doi:10.1136/bmj.l5221 PubMedGoogle ScholarCrossref
33.
Hilal  T, Gonzalez-Velez  M, Prasad  V.  Limitations in clinical trials leading to anticancer drug approvals by the US Food and Drug Administration.   JAMA Intern Med. 2020;180(8):1108-1115. doi:10.1001/jamainternmed.2020.2250 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    February 22, 2021

    Evaluation of the Clinical Benefit of Cancer Drugs Submitted for Reimbursement Recommendation Decisions in Canada

    Author Affiliations
    • 1Department of Medicine, University of Calgary, Calgary, Alberta, Canada
    • 2School of Population and Public Health, University of British Columbia, Vancouver, British Columbia, Canada
    • 3Leeds, Greenville and Lanark District Health Unit, Brockville, Ontario, Canada
    • 4Division of Cancer Care and Epidemiology, Queen’s University Cancer Research Institute, Kingston, Ontario, Canada
    • 5Department of Oncology, Queen’s University, Kingston, Ontario, Canada
    • 6Department of Public Health Sciences, Queen’s University, Kingston, Ontario, Canada
    JAMA Intern Med. 2021;181(4):499-508. doi:10.1001/jamainternmed.2020.8588
    Key Points

    Question  What are the differences in the clinical evidence and benefit between cancer drugs that received a positive vs those that received a negative reimbursement recommendation from the pan-Canadian Oncology Drug Review (pCODR)?

    Findings  In this cohort study of 104 reimbursement recommendation decisions, 78 received a positive recommendation from the pCODR. Of these, 72 were based on phase 3 randomized clinical trials and 39 had proven overall survival benefit with median gains of 3.7 months.

    Meaning  Results of this study found that only half of the pCODR-recommended cancer drugs showed improved survival, and the survival gains were usually modest; these findings suggest that, despite the pCODR framework, cancer drugs without meaningful patient benefit continue to be reimbursed in the Canadian market.

    Abstract

    Importance  Cancer drugs approved by the US Food and Drug Administration have come under scrutiny for marginal clinical benefits; however, the clinical benefits of cancer drugs recommended for reimbursement in Canada have not been adequately studied.

    Objective  To assess the differences in the clinical evidence and benefit of cancer drugs that received a positive vs a negative recommendation for provincial reimbursement in Canada.

    Design, Setting, and Participants  This cohort study obtained publicly available regulatory documents from the pan-Canadian Oncology Drug Review (pCODR) and corresponding clinical trial documentation. All cancer drugs with a solid tumor indication that were submitted from the inception of the pCODR (July 2011) to February 2020 were evaluated. To be included, submissions had to have a final reimbursement recommendation; submissions that were incomplete, were withdrawn, or had a pending decision were excluded.

    Exposures  A completed reimbursement recommendation decision from the pCODR.

    Main Outcomes and Measures  Final reimbursement recommendation (positive vs negative); trial characteristics; and relevant clinical outcomes (ie, overall survival [OS] and progression-free survival [PFS]), including the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS) scores available at the time of pCODR assessment.

    Results  Between 2011 and 2020, the pCODR issued 104 reimbursement recommendation decisions for cancer drugs with a solid tumor indication. Among these drug submissions, 78 (75.0%) received a positive recommendation, of which 72 (92.3%) were conditional. Drugs that received a positive recommendation compared with those with a negative recommendation were more likely to have phase 3 randomized clinical trial design (92.3% [72 of 78] vs 53.8% [14 of 26]; P < .001) and have substantial benefit according to the ESMO-MCBS scores (61.5% [48 of 78] vs 19.2% [5 of 26]; P < .001). The most common primary end points associated with the successful submissions were PFS (53.9%) and OS (32.1%). Overall, 39 of 78 submissions (50.0%) that received a positive recommendation had shown OS benefit, with median (interquartile range) OS gains of 3.7 (2.7-6.5) months.

    Conclusions and Relevance  This cohort study found that, although the pCODR takes into account the magnitude of clinical benefit, only half of the cancer drugs that received a positive recommendation had evidence of improved OS and the survival gains were usually modest. These results suggest that, although the pCODR helps filter out some cancer drugs with low quality of evidence and low magnitude of benefit, cancer drugs without meaningful patient benefit continue to enter the Canadian market; these findings are important for making reimbursement policy decisions globally.

    Introduction

    The clinical benefit and cost of cancer drugs have come under scrutiny in recent years.1-4 Although the cost of cancer drugs continues to escalate,1,2 the clinical benefits of most modern cancer drugs seem to be either modest or uncertain.3,4 Cancer drugs are prescribed to patients to improve either the duration or the quality of life, but ideally both. However, analyses of cancer drugs approved by the US Food and Drug Administration (FDA)3,5 or the European Medicines Agency6 revealed concerns that most of these drugs lacked evidence that they improved the overall survival (OS) or quality of life (QOL) of patients with cancer or that these clinically meaningful end points were not evaluated in relevant clinical trials. Instead, the use of surrogate measures, such as progression-free survival (PFS) or response rates for drug approval decisions, has increased. These surrogate measures are known to poorly correlate with improvements in OS or QOL for most tumor types.7-10

    For drugs that are associated with improved OS, the gains in OS are usually minimal. An analysis of cancer drugs approved in the US between 2002 and 2014 showed that less than half of these approvals met the threshold for meaningful benefit,3 which was set by the American Society of Clinical Oncology standard.11 Notwithstanding this lack of substantial clinical benefit, new cancer drugs routinely cost upward of US $10 000 per month.1 This disconnect between benefit and cost of cancer drugs has partly been ascribed to the FDA not taking the magnitude of benefit, or the cost-effectiveness of drugs, into account in its approval process.12

    In Canada, the pan-Canadian Oncology Drug Review (pCODR), a subsidiary of the Canadian Agency for Drugs and Technologies in Health, provides a reimbursement recommendation for cancer drugs that takes clinical benefit, cost-effectiveness, patient-based values, and adoption feasibility into account.13 In this cohort study, we systematically reviewed all cancer drugs submitted to the pCODR for reimbursement decision and assessed the availability and quality of clinical evidence at the time of recommendation. In addition, we assessed the differences in those metrics between drugs that received a positive and those that received a negative recommendation for provincial reimbursement.

    Methods

    According to the Common Rule, institutional review board approval was not required for this study because all of the data used in this study were obtained from public records and did not include individual-level, patient-identifying information. This study of published reports did not involve patient-identifying data and was not submitted for institutional review board approval. Data were collected between March 1, 2020, and May 1, 2020.

    Cancer Drug Reimbursement Decision Process in Canada

    For a cancer drug to be marketed in Canada, it must first be approved for sale by Health Canada, appraised for a reimbursement recommendation by the pCODR, and then funded for reimbursement at the provincial level. Although the pCODR has no mandate to assess all drugs that receive approval from Health Canada, manufacturers that wish to have their drugs in the Canadian market generally submit to the pCODR at the same time as, or after, Health Canada approval. After the pCODR assesses the submitted data for clinical and economic benefit, the pCODR Expert Review Committee provides a nonbinding recommendation (yes, conditionally yes, or no) for provincial reimbursement. The pCODR makes these reports publicly available online.14

    Conditional recommendations are usually conditional on cost-effectiveness and are considered to be positive recommendations (ie, recommended for funding at the provincial level). The conditional recommendations process in Canada is not similar to the accelerated approval pathway in the US, which signifies uncertainty in the evidence and mandates a confirmatory trial for conversion to a regular approval. No postmarketing confirmatory trials are mandated for the pCODR conditional recommendations. For example, in 2019, pembrolizumab was conditionally recommended for the adjuvant treatment of advanced melanoma based on limitations in cost-effectiveness. As of September 2020, the drug has been funded in 7 provinces with various eligibility criteria, whereas 2 provinces remain in negotiations. Despite being funded, pembrolizumab remains conditionally recommended for this indication. Drugs that have received approval from Health Canada but negative recommendations from the pCODR will not be funded but can still be used in Canada through compassionate access programs, private insurance plans, or out-of-pocket costs.

    Cancer Drugs and Trials Reviewed

    We studied all completed submissions for cancer drugs with a solid tumor indication in adult patients from the inception of the pCODR (July 2011) to February 2020. To be included, submissions had to have received a final reimbursement recommendation; submissions that were incomplete, were withdrawn, or had a pending decision were excluded. Submissions made for biosimilar drugs, pediatric indications, or hematologic malignant neoplasms were excluded. These exclusions were made to ensure the homogeneity of the data set and to allow for assessment of clinical benefits using the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS), a tool developed for solid tumor indications.

    Data Extraction and Data Set Formulation

    For drugs that met the inclusion criteria, the following variables were extracted: final reimbursement recommendation (yes, conditionally yes, or no), tumor indication, treatment setting, line of therapy, end points of the primary clinical trial informing the submission, and relevant clinical outcomes such as the medians and hazard ratios (HRs) of OS or PFS. We dichotomized reimbursement decisions as positive (including conditional) or negative. Outcome data from clinical trials must have been available at the time of the pCODR assessment. All data were extracted from the pCODR documents and cross-referenced with ClinicalTrials.gov to ensure accuracy using the trial name and national clinical trial registration number. When multiple studies were cited as contributing to the body of evidence for a particular submission, we chose the most relevant trial (on the basis of phase, recency, and trial location) for review (eTable 1 in the Supplement). Two of us (D.E.M. and K.J.) extracted the pCODR data separately, and any discrepancies were resolved by consensus with input from an oncologist (B.G.).

    Calculation of Magnitude of Clinical Benefit

    We designated a drug as showing improved PFS or OS if a positive difference was found between the intervention and comparator groups and if the CI of the HR did not cross 1.0. If the experimental group did not reach the median PFS and OS at the time of assessment by the pCODR, then we estimated the median PFS and OS by multiplying the median PFS and OS of the control group by the point estimate of the HR, a practice that is in keeping with the calculation of the ESMO-MCBS.15 If both the experimental and control groups had not reached the median despite statistically significant HR and CI, the drug was designated as having a clinical benefit of unknown magnitude and was excluded from the calculation of median PFS and OS gains.

    We calculated the median of the median gains in PFS and OS across all the submissions and separately for submissions that received positive and negative reimbursement recommendations, only if the gains were statistically significant. For a conservative estimate, we also included the submissions for which the difference in median PFS and OS was not statistically significant by inputting a value of 0 for median PFS and OS gains for these submissions.

    The clinical benefit of cancer drugs within this sample was also categorized using the ESMO-MCBS, a validated framework16 widely used in the assessment of the clinical benefit of cancer drugs. These ESMO-MCBS scores were either extracted from the ESMO website, where available, or calculated using the publicly available score cards.17 The ESMO-MCBS assigns a score of 1 (low) to 5 (high) for drugs used in the advanced or metastatic setting and a score of C (low) to A (high) for drugs that are potentially curative or used in the adjuvant setting. Scores of 4 to 5 and A to B are designated as having substantial or major clinical benefit. These scores take into account the magnitude of not only efficacy but also adverse events and quality of life.

    Statistical Analysis

    We compared the characteristics of the trials and the clinical benefits of submissions with positive vs negative recommendations. The differences in medians of continuous variables (ie, PFS and OS) and the dependence of categorical variables were tested using permutation testing.18 We chose this approach because we generally did not have sufficient data to rely on asymptotic testing (ie, χ2 test for categorical data). P values were not adjusted for multiple hypothesis testing, and 2-sided P < .05 was considered to be statistically significant. Analyses were performed with R, version 3.6.2 (R Foundation for Statistical Computing). Permutation testing was completed with the infer package (Creative Commons).

    Results

    A total of 104 cancer drug submissions met the eligibility criteria and were included in this study (Figure 1). Among these submissions, 54 (51.9%) were for new indications of drugs previously submitted to the pCODR, 41 (39.4%) were for novel drugs with no previous submissions, and 9 (8.7%) were resubmissions that either were not completed (n = 2) or received negative decisions (n = 7).

    Lung cancer was the most common primary indication (25 [24.0%]), and small molecules were the most common drug type (48 [46.2%]). Most submissions were for the advanced or metastatic setting (97 [93.3%]), with 37 (35.6%) for treatment in the first line (eTables 2 and 3 in the Supplement).

    Differences in Submissions With Positive vs Negative Recommendations

    Of the 104 submissions, 79 (76.0%) had OS data and 82 (78.8%) had PFS data available at the time of review. Seventy-eight submissions (75.0%) received a positive recommendation for provincial reimbursement (6 outright recommendations and 72 conditional recommendations), and 26 submissions (25.0%) received a negative recommendation. The most common reason cited for a conditional recommendation was cost-effectiveness (70 of 72 [97.2%]). During the same period, the FDA granted 163 drug approvals for adult solid tumors, of which 121 were regular (74.2%) and 42 were accelerated (25.7%) approvals. None of the 26 submissions that received a negative recommendation from the pCODR had been rejected by the FDA.

    No statistically significant difference was observed in the distributions of tumor type, drug class, treatment setting, or line of therapy between submissions that received positive and negative recommendations; however, there were significant differences with regard to certain trial characteristics (Table 1).

    Of the drugs that received a positive recommendation for reimbursement, 92.3% (72 of 78) had evidence from a phase 3 randomized clinical trial (RCT) compared with 53.8% (14 of 26) of the drugs that received a negative recommendation (P < .001). The most common primary end points associated with the successful submissions were PFS (53.8%) and OS (32.1%). The most common primary end point among the submissions that received a positive recommendation was PFS (42 [53.8%]) vs response rates among those that received a negative recommendation (10 [38.5%]). In addition, drugs that received a positive recommendation were statistically significantly more likely to have data available for PFS and OS at the time of pCODR assessment (Table 1 and eTable 4 in the Supplement). No significant differences were found in the distribution of submissions with evidence of OS or PFS benefit on the basis of reimbursement recommendations.

    Forty-seven submissions had documented OS improvements, of which 39 (83.0%) received a positive recommendation. Twenty-six submissions without evidence of OS benefit and 13 with no OS data available at the time of pCODR assessment also received a positive recommendation. Thus, only 50.0% (39 of 78) of cancer drugs recommended for reimbursement in Canada had shown an improvement in OS.

    Magnitude of Clinical Benefit for Submissions With Positive vs Negative Recommendations

    The differences between submissions that received positive and those that received negative reimbursement recommendations did not reach statistical significance for median (interquartile range) gains in PFS (4.7 [2.8-8.2] vs 2.9 [0.8-4.0]; P = .06) or in OS (3.7 [2.7-6.5] vs 1.9 [1.4-4.5]; P = .06) (Table 1, Figure 2, and Figure 3). When submissions with OS or PFS gains of 0 months were included, the median gains in PFS for submissions with positive vs negative recommendations were 3.9 vs 2.7 (P = .29) and the median gains in OS were 1.9 vs 1.4 (P = .92). These calculations included only those submissions with OS and PFS data available at the time of the recommendation. Submissions that received a positive reimbursement recommendation were more likely to have demonstrated substantial clinical benefit according to the ESMO-MCBS scores compared with those that received a negative recommendation (48 [61.5%] vs 5 [19.2%]; P < .001) (Table 1).

    Provincial Fate of Cancer Drugs Reviewed by the pCODR

    Although the pCODR recommendations are nonbinding, we found 100% concordance between pCODR recommendations and provincial funding decisions for submissions that had received “recommend for funding” and “do not recommend for funding” decisions (based on available data). However, of the 72 conditional recommendations, 6 (8.3%) were generally not funded by the provinces that had completed their deliberations.

    Discussion

    We evaluated the characteristics of trials and the magnitude of clinical benefit of cancer drugs with a solid tumor indication submitted to the pCODR for Canadian provincial reimbursement. We found that three-quarters of all submissions received a positive recommendation, of which half demonstrated improved survival with median gains in OS of 3.7 months. Significant differences in trial characteristics were observed, such as randomization, phase, and primary end points, between submissions that received a positive recommendation and submissions that received a negative recommendation. These findings have multiple regulatory and reimbursement policy lessons globally.

    These current findings must be placed into context by examining the results of similar studies that evaluated the magnitude of clinical benefit and the quality of evidence for cancer drugs approved by the FDA at different time points (Table 219,20) to facilitate a qualitative comparison. We found that 61.5% of cancer drugs (48 of 78) that received a positive recommendation from the pCODR had demonstrated substantial clinical benefit according to the ESMO-MCBS score, in contrast to the data previously reported for the FDA: 43.8% of drug approvals (46 of 105) based on RCTs21 and 33.8% of drug approvals (45 of 133) based on single-arm trials22 demonstrated substantial clinical benefit using the same scale (ESMO-MCBS). Similarly, although only 6.4% of cancer drugs (5 of 78) that received a positive recommendation by the pCODR were based on response rates, the corresponding percentage was 34.9% of drugs (22 of 63) approved by the FDA.20 For OS, 50.0% of drugs (39 of 78) recommended by the pCODR and 30.2% of drugs (19 of 63) approved by the FDA showed improved OS.20 With regard to the quality of evidence, 92.3% (72 of 78) of positive recommendations by the pCODR but only 75.8% (116 of 153) of FDA approvals were based on RCTs.

    One explanation for these differences in results is that the pCODR uses a multicriteria deliberative framework that takes into account not only the presence or absence of clinical benefit but also the magnitude of benefit, quality of evidence, economic evaluation, and feedback from patient and clinician stakeholders. This approach is in contrast to that of the FDA, which considers neither the magnitude of benefit nor the cost. We found that all 26 cancer drugs that were not recommended for reimbursement by the pCODR had previously been approved by the FDA, which is similar to the findings of a previous study.23 Furthermore, although the FDA granted 163 approvals during the same period, the pCODR received only 104 submissions, of which only 78 received a positive reimbursement recommendation. These data support the notion that the pCODR framework might filter out drugs with minimal clinical utility. Integrating the ESMO-MCBS score as a preliminary tool may help the pCODR to further filter out minimally effective cancer drugs at the outset, an approach that is being used in certain countries in Europe and by the World Health Organization in formulating the essential medicine list in cancer.24

    Results from the present study identified areas of improvement for the Canadian regulatory system. First, the quality of evidence for most of the submissions that received a positive recommendation was low. Less than one-third of submissions that received a positive recommendation from the pCODR had OS as the primary end point. In the US, the proportion of cancer drugs that are approved on the basis of surrogates has increased over the past decade, peaking at 93% in 2017.19 Although PFS was the most frequently used primary end point in the current study, data suggest that PFS is not a valid surrogate for clinical outcomes such as OS8,10,25 or QOL.9,26 Previous work has shown that more than half of submissions to the pCODR between 2015 and 2018 did not have data on QOL.27 None of the pCODR submissions that we evaluated had QOL as a primary outcome measure. Thus, moving forward, clinical trials should prioritize meaningful clinical end points, such as OS and QOL, to ensure that regulatory decisions are guided by high-quality evidence.

    Second, cancer drugs that were recommended for reimbursement in Canadian provinces were not shown to have substantially improved clinical outcomes for patients. Across all submissions with a positive recommendation, the median OS gains were only 3.7 months and the median PFS gains were 4.7 months, which are not substantially different from the survival gains in submissions that received a negative recommendation. Furthermore, nearly 40% of cancer drugs that received a positive recommendation did not reach the threshold for substantial clinical benefit per the ESMO-MCBS. This finding demonstrates room for improvement in the Canadian system.

    Previous studies have also examined the pCODR decision process from different angles and have highlighted other areas of improvement. A study by Niraula and Nugent23 found that in a cohort of 91 cancer drugs approved by the FDA, 18 (20%) received a negative recommendation by the pCODR, with higher incremental cost-effectiveness ratio being a factor associated with a negative recommendation. Other studies have found that the economic models submitted to the pCODR by drug manufacturers were often flawed as identified by the economic reviewers28 of the pCODR and cost-effectiveness appeared to be associated with a full vs conditional reimbursement decision rather than a positive vs negative decision.29 In addition, although patient values were considered in stakeholder feedback, both patients30 and clinicians31 involved in the process were shown to have relevant financial conflicts of interest.

    Limitations

    This study has some limitations. The study is unique in that it comprehensively summarized and quantified the clinical evidence as well as the magnitude of clinical benefit for cancer drugs that were recommended or not recommended for reimbursement in Canada. It did not, however, assess the role of cost or cost-effectiveness in this decision-making process. The pCODR bases its recommendation decision not only on quality of evidence and magnitude of benefit but also on economic evaluation, stakeholder views, and adoption feasibility; as such, this analysis may not be fully representative of all the aspects that the pCODR includes in its deliberation. We assessed only the quality of evidence on trial phase and randomization evaluated by the pCODR, but we did not assess other characteristics of the trials. A complete assessment is important given recent data that suggested that trials used to inform regulatory decisions for cancer drugs by both the European Medicines Agency32 and the FDA33 were at high risk of bias and had critical design flaws. In addition, we limited our analyses to data that were available to the pCODR at the time of deliberation and did not consider evidence that may have been subsequently generated.

    Conclusions

    In this cohort study, we found that 75% of submissions to the pCODR for provincial reimbursement of cancer drugs in Canada received a positive reimbursement recommendation; these submissions were mostly based on phase 3 RCTs, with half of them showing improved OS. Cancer drugs recommended for reimbursement in Canada also have issues, such as the use of surrogate end points and a modest margin of clinical benefit. Even in a regulatory system like the pCODR with a central evidence appraisal process, including assessment of evidence quality, magnitude of clinical benefit, and cost-effectiveness, cancer drugs without meaningful patient benefit are entering the market. These findings have implications for regulatory and reimbursement policy decisions in Canada and globally.

    Back to top
    Article Information

    Accepted for Publication: December 11, 2020.

    Published Online: February 22, 2021. doi:10.1001/jamainternmed.2020.8588

    Corresponding Author: Bishal Gyawali, MD, PhD, Division of Cancer Care and Epidemiology, Queen’s University Cancer Research Institute, 10 Stuart St, Kingston, ON K7L 3N6, Canada (gyawali.bishal@queensu.ca).

    Author Contributions: Dr Meyers and Ms Jenei had full access to all the data in the study and take responsibility for the integrity of the data collection. Mr Chisamore takes responsibility for the accuracy of the data analyses.

    Concept and design: Meyers, Jenei, Gyawali.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Meyers, Jenei, Gyawali.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Chisamore.

    Administrative, technical, or material support: Meyers, Jenei, Gyawali.

    Supervision: Gyawali.

    Other - data entry, revision: Jenei.

    Conflict of Interest Disclosures: None reported.

    References
    1.
    Saluja  R, Arciero  VS, Cheng  S,  et al.  Examining trends in cost and clinical benefit of novel anticancer drugs over time.   J Oncol Pract. 2018;14(5):e280-e294. doi:10.1200/JOP.17.00058 PubMedGoogle ScholarCrossref
    2.
    IQVIA Institute for Human Data Science. Global oncology trends 2019: therapeutics, clinical development and health system implications. May 2019. Accessed July 25, 2020. https://www.iqvia.com/insights/the-iqvia-institute/reports/global-oncology-trends-2019
    3.
    Fojo  T, Mailankody  S, Lo  A.  Unintended consequences of expensive cancer therapeutics—the pursuit of marginal indications and a me-too mentality that stifles innovation and creativity: the John Conley Lecture.   JAMA Otolaryngol Head Neck Surg. 2014;140(12):1225-1236. doi:10.1001/jamaoto.2014.1570 PubMedGoogle ScholarCrossref
    4.
    Kumar  H, Fojo  T, Mailankody  S.  An appraisal of clinically meaningful outcomes guidelines for oncology clinical trials.   JAMA Oncol. 2016;2(9):1238-1240. doi:10.1001/jamaoncol.2016.0931 PubMedGoogle ScholarCrossref
    5.
    Kim  C, Prasad  V.  Cancer drugs approved on the basis of a surrogate end point and subsequent overall survival: an analysis of 5 years of US Food and Drug Administration approvals.   JAMA Intern Med. 2015;175(12):1992-1994. doi:10.1001/jamainternmed.2015.5868 PubMedGoogle ScholarCrossref
    6.
    Davis  C, Naci  H, Gurpinar  E, Poplavska  E, Pinto  A, Aggarwal  A.  Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009-13.   BMJ. 2017;359:j4530. doi:10.1136/bmj.j4530 PubMedGoogle ScholarCrossref
    7.
    Chen  EY, Haslam  A, Prasad  V.  FDA acceptance of surrogate end points for cancer drug approval: 1992-2019.   JAMA Intern Med. 2020;180(6):912-914. doi:10.1001/jamainternmed.2020.1097 PubMedGoogle ScholarCrossref
    8.
    Haslam  A, Hey  SP, Gill  J, Prasad  V.  A systematic review of trial-level meta-analyses measuring the strength of association between surrogate end-points and overall survival in oncology.   Eur J Cancer. 2019;106:196-211. doi:10.1016/j.ejca.2018.11.012 PubMedGoogle ScholarCrossref
    9.
    Hwang  TJ, Gyawali  B.  Association between progression-free survival and patients’ quality of life in cancer clinical trials.   Int J Cancer. 2019;144(7):1746-1751. doi:10.1002/ijc.31957 PubMedGoogle ScholarCrossref
    10.
    Booth  CM, Eisenhauer  EA.  Progression-free survival: meaningful or simply measurable?   J Clin Oncol. 2012;30(10):1030-1033. doi:10.1200/JCO.2011.38.7571 PubMedGoogle ScholarCrossref
    11.
    Ellis  LM, Bernstein  DS, Voest  EE,  et al.  American Society of Clinical Oncology perspective: raising the bar for clinical trials by defining clinically meaningful outcomes.   J Clin Oncol. 2014;32(12):1277-1280. doi:10.1200/JCO.2013.53.8009 PubMedGoogle ScholarCrossref
    12.
    Darrow  JJ, Avorn  J, Kesselheim  AS.  FDA approval and regulation of pharmaceuticals, 1983-2018.   JAMA. 2020;323(2):164-176. doi:10.1001/jama.2019.20288 PubMedGoogle ScholarCrossref
    13.
    Canadian Agency for Drugs and Technologies in Health. Procedures for the CADTH Pan-Canadian Oncology Drug Review. April 2020. Accessed June 10, 2020. https://www.cadth.ca/sites/default/files/pcodr/pCODR%27s%20Drug%20Review%20Process/pcodr-procedures.pdf
    14.
    Canadian Agency for Drugs and Technologies in Health. Reimbursement review reports. Accessed July 25, 2020. https://www.cadth.ca/reimbursement-review-reports
    15.
    European Society for Medical Oncology. ESMO-Magnitude of Clinical Benefit Scale: evaluation forms version 1.1. Accessed May 5, 2020. https://www.esmo.org/guidelines/esmo-mcbs/scale-evaluation-forms-v1.0-v1.1/scale-evaluation-forms-v1.1
    16.
    Cherny  NI, Dafni  U, Bogaerts  J,  et al.  ESMO-Magnitude of Clinical Benefit Scale version 1.1.   Ann Oncol. 2017;28(10):2340-2366. doi:10.1093/annonc/mdx310 PubMedGoogle ScholarCrossref
    17.
    European Society for Medical Oncology. ESMO-MCBS scorecards. Accessed September 23, 2020. https://www.esmo.org/guidelines/esmo-mcbs/esmo-magnitude-of-clinical-benefit-scale
    18.
    Pesarin  F, Salmaso  L.  Permutation Tests for Complex Data: Theory, Applications and Software. John Wiley & Sons Ltd; 2010. doi:10.1002/9780470689516
    19.
    Gyawali  B, Sharma  S, Booth  CM.  Is the number of cancer drug approvals a surrogate for regulatory success?   J Cancer Policy. 2019;22:100202. doi:10.1016/j.jcpo.2019.100202Google Scholar
    20.
    Mailankody  S, Prasad  V.  Five years of cancer drug approvals: innovation, efficacy, and costs.   JAMA Oncol. 2015;1(4):539-540. doi:10.1001/jamaoncol.2015.0373 PubMedGoogle ScholarCrossref
    21.
    Tibau  A, Molto  C, Ocaña  A,  et al.  Magnitude of clinical benefit of cancer drugs approved by the US Food and Drug Administration.   J Natl Cancer Inst. 2018;110(5):486-492. doi:10.1093/jnci/djx232 PubMedGoogle ScholarCrossref
    22.
    Tibau  A, Molto  C, Borrell  M,  et al.  Magnitude of clinical benefit of cancer drugs approved by the US Food and Drug Administration based on single-arm trials.   JAMA Oncol. 2018;4(11):1610-1611. doi:10.1001/jamaoncol.2018.4300 PubMedGoogle ScholarCrossref
    23.
    Niraula  S, Nugent  Z.  New cancer drug approvals from the perspective of a universal healthcare system: analyses of the Pan-Canadian Oncology Drug Review recommendations.   J Natl Compr Canc Netw. 2018;16(12):1460-1466. doi:10.6004/jnccn.2018.7084 PubMedGoogle ScholarCrossref
    24.
    European Society for Medical Oncology. ESMO-MCBS and the World Health Organisation (WHO) model list of essential medicines. Accessed September 10, 2020. https://www.esmo.org/guidelines/esmo-mcbs/esmo-mcbs-and-the-who
    25.
    Kemp  R, Prasad  V.  Surrogate endpoints in oncology: when are they acceptable for regulatory and clinical decisions, and are they currently overused?   BMC Med. 2017;15(1):134-137. doi:10.1186/s12916-017-0902-9 PubMedGoogle ScholarCrossref
    26.
    Kovic  B, Jin  X, Kennedy  SA,  et al.  Evaluating progression-free survival as a surrogate outcome for health-related quality of life in oncology: a systematic review and quantitative analysis.   JAMA Intern Med. 2018;178(12):1586-1596. doi:10.1001/jamainternmed.2018.4710 PubMedGoogle ScholarCrossref
    27.
    Raymakers  AJN, Regier  DA, Peacock  SJ.  Health-related quality of life in oncology drug reimbursement submissions in Canada: a review of submissions to the Pan-Canadian Oncology Drug Review.   Cancer. 2020;126(1):148-155. doi:10.1002/cncr.32455 PubMedGoogle ScholarCrossref
    28.
    Masucci  L, Beca  J, Sabharwal  M, Hoch  JS.  Methodological issues in economic evaluations submitted to the Pan-Canadian Oncology Drug Review (pCODR).   Pharmacoecon Open. 2017;1(4):255-263. doi:10.1007/s41669-017-0018-3 PubMedGoogle ScholarCrossref
    29.
    Skedgel  C, Wranik  D, Hu  M.  The relative importance of clinical, economic, patient values and feasibility criteria in cancer drug reimbursement in Canada: a revealed preferences analysis of recommendations of the Pan-Canadian Oncology Drug Review 2011-2017.   Pharmacoeconomics. 2018;36(4):467-475. doi:10.1007/s40273-018-0610-0 PubMedGoogle ScholarCrossref
    30.
    Lexchin  J.  Association between commercial funding of Canadian patient groups and their views about funding of medicines: an observational study.   PLoS ONE. 2019;14(2):e0212399. doi:10.1371/journal.pone.0212399 PubMedGoogle Scholar
    31.
    Lexchin  J.  Financial conflicts of interest of clinicians making submissions to the pan-Canadian Oncology Drug Review: a descriptive study.   BMJ Open. 2019;9(7):e030750. doi:10.1136/bmjopen-2019-030750 PubMedGoogle Scholar
    32.
    Naci  H, Davis  C, Savović  J,  et al.  Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014-16: cross sectional analysis.   BMJ. 2019;366:l5221. doi:10.1136/bmj.l5221 PubMedGoogle ScholarCrossref
    33.
    Hilal  T, Gonzalez-Velez  M, Prasad  V.  Limitations in clinical trials leading to anticancer drug approvals by the US Food and Drug Administration.   JAMA Intern Med. 2020;180(8):1108-1115. doi:10.1001/jamainternmed.2020.2250 PubMedGoogle ScholarCrossref
    ×