[Skip to Navigation]
Sign In
Figure 1.  Adjusted Kaplan-Meier Curves for Overall Survival
Adjusted Kaplan-Meier Curves for Overall Survival

Curves show survival for alectinib trial data vs ceritinib real-world data (RWD) (A) and alectinib RWD vs ceritinib RWD (B). Numbers at risk are reweighted sample sizes. Log-rank P values are shown.

Figure 2.  Tipping Point Analysis for Missing Eastern Cooperative Oncology Group (ECOG) Performance Status (PS)
Tipping Point Analysis for Missing Eastern Cooperative Oncology Group (ECOG) Performance Status (PS)

Tipping point analysis for varying δ shifts applied to missing baseline ECOG PS in the ceritinib RWD showing adjusted hazard ratios (HRs) comparing alectinib and ceritinib groups. Negative values of δ imply exponentially increasing odds of patients having poorer ECOG PS than expected under missing at random given their covariates. Sufficient balance for the trial data vs RWD comparison was not achieved beyond δ = −2.

Table 1.  Unadjusted Patient Characteristics by Treatment Group
Unadjusted Patient Characteristics by Treatment Group
Table 2.  Propensity Score–Weighted Patient Characteristics by Treatment Group
Propensity Score–Weighted Patient Characteristics by Treatment Group
Table 3.  Adjusted HRs From Complete-Case (Missing Completely at Random) and Multiple Imputation (Missing at Random) Analyses
Adjusted HRs From Complete-Case (Missing Completely at Random) and Multiple Imputation (Missing at Random) Analyses
1.
Hall  PS.  Real-world data for efficient health technology assessment.   Eur J Cancer. 2017;79:235-237. doi:10.1016/j.ejca.2017.04.003PubMedGoogle ScholarCrossref
2.
Oortwijn  W, Sampietro-Colom  L, Trowman  R.  How to deal with the inevitable: generating real-world data and using real-world evidence for HTA purposes—from theory to action.   Int J Technol Assess Health Care. 2019;35(4):346-350. doi:10.1017/S0266462319000400PubMedGoogle ScholarCrossref
3.
Hernán  MA, Hernández-Díaz  S, Robins  JM.  A structural approach to selection bias.   Epidemiology. 2004;15(5):615-625. doi:10.1097/01.ede.0000135174.63482.43PubMedGoogle ScholarCrossref
4.
Roberts  MH, Ferguson  GT.  Real-world evidence: bridging gaps in evidence to guide payer decisions.   Pharmacoecon Open. 2021;5(1):3-11. doi:10.1007/s41669-020-00221-yPubMedGoogle ScholarCrossref
5.
Malone  DC, Brown  M, Hurwitz  JT, Peters  L, Graff  JS.  Real-world evidence: useful in the real world of US payer decision making? how? when? and what studies?   Value Health. 2018;21(3):326-333. doi:10.1016/j.jval.2017.08.3013PubMedGoogle ScholarCrossref
6.
O’Donnell  JC, Le  TK, Dobrin  R,  et al.  Evolving use of real-world evidence in the regulatory process: a focus on immuno-oncology treatment and outcomes.   Future Oncol. 2021;17(3):333-347. doi:10.2217/fon-2020-0591PubMedGoogle ScholarCrossref
7.
Lash  TL, Fox  MP, Cooney  D, Lu  Y, Forshee  RA.  Quantitative bias analysis in regulatory settings.   Am J Public Health. 2016;106(7):1227-1230. doi:10.2105/AJPH.2016.303199PubMedGoogle ScholarCrossref
8.
Lash  TL, Fox  MP, Fink  AK.  Applying Quantitative Bias Analysis to Epidemiologic Data. Springer Science & Business Media; 2011.
9.
Davies  J, Martinec  M, Delmar  P,  et al.  Comparative effectiveness from a single-arm trial and real-world data: alectinib versus ceritinib.   J Comp Eff Res. 2018;7(9):855-865. doi:10.2217/cer-2018-0032PubMedGoogle ScholarCrossref
10.
IQWiG. IQWiG reports: commission No. A17-19—Alectinib (non-small cell lung cancer). May 2, 2017. Accessed September 3, 2021. https://www.iqwig.de/download/a17-19_alectinib_extract-of-dossier-assessment_v1-0.pdf?rev=185028
11.
CADTH. Alecensaro for locally advanced or metastatic non-small lung cancer (second line): details. CADTH project number PC0114-000. August 25, 2017. Accessed September 3, 2021. https://www.cadth.ca/alecensaro-locally-advanced-or-metastatic-non-small-cell-lung-cancer-second-line-details
12.
Ou  S-HI, Ahn  JS, De Petris  L,  et al.  Alectinib in crizotinib-refractory ALK-rearranged non–small-cell lung cancer: a phase II global study.   J Clin Oncol. 2016;34(7):661-668. doi:10.1200/JCO.2015.63.9443PubMedGoogle ScholarCrossref
13.
Shaw  AT, Gandhi  L, Gadgeel  S,  et al.  Alectinib in ALK-positive, crizotinib-resistant, non-small-cell lung cancer: a single-group, multicentre, phase 2 trial.   Lancet Oncol. 2016;17(2):234-242. doi:10.1016/S1470-2045(15)00488-XPubMedGoogle ScholarCrossref
14.
Allan  V, Ramagopalan  SV, Mardekian  J,  et al.  Propensity score matching and inverse probability of treatment weighting to address confounding by indication in comparative effectiveness research of oral anticoagulants.   J Comp Eff Res. 2020;9(9):603-614. doi:10.2217/cer-2020-0013PubMedGoogle ScholarCrossref
15.
Vandenbroucke  JP, von Elm  E, Altman  DG,  et al; STROBE Initiative.  Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration.   PLoS Med. 2007;4(10):e297. doi:10.1371/journal.pmed.0040297PubMedGoogle Scholar
16.
Schabath  MB, Cress  WD, Muñoz-Antonia  T.  Racial and ethnic differences in the epidemiology and genomics of lung cancer.   Cancer Control. 2016;23(4):338-346. doi:10.1177/107327481602300405PubMedGoogle ScholarCrossref
17.
Sammon  CJ, Leahy  TP, Gsteiger  S, Ramagopalan  S.  Real-world evidence and nonrandomized data in health technology assessment: using existing methods to address unmeasured confounding?   J Comp Eff Res. 2020;9(14):969-972. doi:10.2217/cer-2020-0112PubMedGoogle ScholarCrossref
18.
Gemine  RE, Ghosal  R, Collier  G,  et al; LungCast Investigators.  Longitudinal study to assess impact of smoking at diagnosis and quitting on 1-year survival for people with non-small cell lung cancer.   Lung Cancer. 2019;129:1-7. doi:10.1016/j.lungcan.2018.12.028PubMedGoogle ScholarCrossref
19.
Leacy  FP, Floyd  S, Yates  TA, White  IR.  Analyses of sensitivity to the missing-at-random assumption using multiple imputation with delta adjustment: application to a tuberculosis/HIV prevalence survey with incomplete HIV-status data.   Am J Epidemiol. 2017;185(4):304-315. doi:10.1093/aje/kww107PubMedGoogle Scholar
20.
Rezvan  PH, Lee  KJ, Simpson  JA.  Sensitivity analysis within multiple imputation framework using delta-adjustment: application to Longitudinal Study of Australian Children.   Longitudinal Life Course Studies. 2018;9(3):259-278. doi:10.14301/llcs.v9i3.503Google ScholarCrossref
21.
Thorlund  K, Dron  L, Park  JJH, Mills  EJ.  Synthetic and external controls in clinical trials: a primer for researchers.   Clin Epidemiol. 2020;12:457-467. doi:10.2147/CLEP.S242097PubMedGoogle ScholarCrossref
22.
Webster-Clark  M, Jonsson Funk  M, Stürmer  T.  Single-arm trials with external comparators and confounder misclassification: how adjustment can fail.   Med Care. 2020;58(12):1116-1121. doi:10.1097/MLR.0000000000001400PubMedGoogle ScholarCrossref
23.
Lobo  FS, Wagner  S, Gross  CR, Schommer  JC.  Addressing the issue of channeling bias in observational studies with propensity scores analysis.   Res Social Adm Pharm. 2006;2(1):143-151. doi:10.1016/j.sapharm.2005.12.001PubMedGoogle ScholarCrossref
24.
Kuo  CS, Tung  PH, Huang  AC,  et al.  A retrospective study of alectinib versus ceritinib in patients with advanced non-small-cell lung cancer of anaplastic lymphoma kinase fusion in whom crizotinib treatment failed.   BMC Cancer. 2021;21(1):309. doi:10.1186/s12885-021-08005-1PubMedGoogle ScholarCrossref
Original Investigation
Oncology
October 7, 2021

Assessment of Alectinib vs Ceritinib in ALK-Positive Non–Small Cell Lung Cancer in Phase 2 Trials and in Real-world Data

Author Affiliations
  • 1Personalized Healthcare Data Science, Roche Products Limited, Welwyn Garden City, United Kingdom
  • 2Cytel, Inc, Waltham, Massachusetts
  • 3Health Economics, Reimbursement and Outcomes, Roche Products Limited, Welwyn Garden City, United Kingdom
  • 4Global Access, F. Hoffmann–La Roche Ltd, Basel, Switzerland
  • 5Department of Investigational Cancer Therapeutics, Division of Cancer Medicine, The University of Texas MD Anderson Cancer Center, Houston
  • 6Division of Pediatrics, University of Texas MD Anderson Cancer Center, Houston
JAMA Netw Open. 2021;4(10):e2126306. doi:10.1001/jamanetworkopen.2021.26306
Key Points

Question  How robust are conclusions about the comparative effectiveness of the ALK inhibitor alectinib vs ceritinib in crizotinib-refractory, ALK-positive non–small cell lung cancer from indirect comparisons using real-world data (RWD)?

Findings  This comparative effectiveness study including 355 patients found that alectinib exposure was associated with improved survival compared with ceritinib in both single-group trials and multicenter US RWD. Results were robust to a range of plausible assumptions about unmeasured confounding and missing Eastern Cooperative Oncology Group Performance Status and underrecorded comorbidities in RWD.

Meaning  These findings suggest that alectinib is preferable to ceritinib for crizotinib-refractory, ALK-positive non–small cell lung cancer, providing validity to single-group trial data vs RWD comparisons.

Abstract

Importance  Quantitative assessment of bias from unmeasured confounding and missing data can help evaluate uncertainty in findings from indirect comparisons using real-world data (RWD).

Objective  To compare the effectiveness of alectinib vs ceritinib in terms of overall survival (OS) in patients with ALK-positive, crizotinib-refractory, non–small cell lung cancer (NSCLC) and to assess the sensitivity of these findings to unmeasured confounding and missing data assumptions.

Design, Setting, and Participants  This comparative effectiveness research study compared patients from 2 phase 2 alectinib trials and real-world patients. Patients were monitored from June 2013 to March 2020. Comparisons of interest were between alectinib trial data vs ceritinib RWD and alectinib RWD vs ceritinib RWD. RWD treatment groups were selected from nationally representative cancer data from US cancer clinics, the majority from community centers. Participants were ALK-positive patients aged 18 years or older with advanced NSCLC, prior exposure to crizotinib, and Eastern Cooperative Oncology Group Performance Status (PS) of 0 to 2. Data analysis was performed from October 2020 to March 2021.

Exposures  Initiation of alectinib or ceritinib therapy.

Main Outcomes and Measures  The main outcome was OS.

Results  In total, there were 355 patients: 183 (85 men [46.4%]) in the alectinib trial, 91 (43 men [47.3%]) in the ceritinib RWD group, and 81 (38 men [46.9%]) in the alectinib RWD group. Patients in the alectinib trial were younger (mean [SD] age, 52.53 [11.18] vs 57.97 [11.71] years), more heavily pretreated (mean [SD] number of prior therapy lines, 1.95 [0.72] vs 1.47 [0.81]), and had more favorable baseline ECOG PS (ECOG PS of 0 or 1, 165 patients [90.2%] vs 37 patients [77.1%]) than those in the ceritinib RWD group. The alectinib RWD group (mean [SD] age, 58.69 [11.26] years) had more patients with favorable ECOG PS (ECOG PS of 0 or 1, 49 patients [92.4%] vs 37 patients [77.1%]) and more White patients (56 patients [72.7%] vs 53 patients [62.4%]) compared with the ceritinib group. Compared with ceritinib RWD, alectinib-exposed patients had significantly longer OS in alectinib trials (adjusted hazard ratio [HR], 0.59; 95% CI, 0.44-0.75; P < .001) and alectinib RWD (HR, 0.46; 95% CI, 0.29-0.63; P < .001) after adjustment for baseline confounders. For the worst-case HR estimate of 0.59, residual confounding by a hypothetical confounder associated with mortality and treatment by a risk ratio greater than 2.24 was required to reverse the findings. Conclusions were robust to plausible deviations from random missingness for missing ECOG PS and underrecorded comorbidities and central nervous system metastases in RWD.

Conclusions and Relevance  Alectinib exposure was associated with longer OS compared with ceritinib in patients with ALK-positive NSCLC, and only substantial levels of bias examined reversed the findings. These findings suggest that quantitative bias analysis can be a useful tool to address uncertainty of findings for decision-makers considering RWD.

Introduction

For rare indications, for which large sample sizes or randomized comparators are infeasible, real-world data (RWD) can at times be provided for consideration by decision-makers, such as health technology assessment (HTA) agencies, as an external comparator group for reimbursement decisions.1,2 However, the risk of bias3 in nonrandomized studies can pose a threat to their validity even when statistical methods to reduce confounding due to lack of treatment randomization are used, which has made HTA agencies reluctant to use them in their decision-making.4,5 For example, RWD often contain significant amounts of missing observations for important prognostic variables that are not uniformly recorded in routine clinical practice, such as Eastern Cooperative Oncology Group (ECOG) Performance Status (PS) in cancer studies.6 Findings from such studies can be prone to bias if assumptions about the mechanism of missingness are incorrectly specified or if unmeasured confounders are not accounted for.

One way to assess the sensitivity of adjusted measures of association of exposures and outcomes to systematic bias is to compute them over a range of assumptions about missing observations in RWD and unmeasured confounders. Then the robustness of real-world evidence can be evaluated according to how drastic or implausible these assumptions have to be for the observed association to be nullified. Quantitative bias analysis (QBA) methods can be used to identify the impact of missing data and residual confounding on results from nonrandomized studies using RWD.7,8 In this comparative effectiveness study, we build on a previously reported indirect comparison of overall survival (OS) in patients with crizotinib-refractory, ALK-positive, non–small cell lung cancer (NSCLC) using data from single-group alectinib trials and ceritinib-exposed patients from RWD derived from a US database,9 which was submitted to HTA agencies for reimbursement of alectinib.10 In these assessments, concerns were raised about possible bias due to a large amount of missing ECOG PS in RWD and lack of sensitivity analyses to address the effect of unmeasured confounding.11

There is no ongoing or currently planned randomized trial comparing alectinib and ceritinib for the crizotinib-refractory, ALK-positive, NSCLC indication, and alectinib was approved for this indication in the US on the basis of results from 2 phase 2 clinical trials, NP28673 (ClinicalTrials.gov identifier NCT01801111)12 and NP28671 (ClinicalTrials.gov identifier NCT01871805).13 Thus, here we sought to (1) compare OS between patients from single-group alectinib trials and ceritinib-exposed patients in updated RWD (ceritinib RWD group) using propensity score weighting,14 (2) compare OS between patients in RWD exposed to alectinib (alectinib RWD group) or ceritinib to measure real-world effectiveness of alectinib as an additional line of evidence to the trial data vs RWD comparison from the first objective, and (3) quantify the degree of residual confounding and assumptions about missing ECOG PS and underrecorded comorbidities and central nervous system (CNS) metastases that could nullify our findings from the first or second objectives using QBA.

Methods

This comparative effectiveness research study adheres to the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) reporting guideline and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cohort studies.15 Approval for this study was granted by the Copernicus Group institutional review board. Informed consent was waived because the data were deidentified, in accordance with 45 CFR §46.

Study Population

At the time of initiation of alectinib or ceritinib (time 0 for start of follow-up), patients were aged 18 years or older, had a diagnosis of locally advanced or metastatic NSCLC, had a documented ALK rearrangement, and had previously received crizotinib. Patients were monitored from June 2013 to March 2020. RWD treatment groups were selected from nationally representative cancer data from 158 US cancer clinics, the majority from community centers. Patients with ECOG PS greater than 2 at baseline (measured <30 days before or <7 days after time 0) and American Joint Committee on Cancer stage below IIIB were excluded from all analyses. Race was self-reported by patients and was captured in the electronic health record. Race was assessed because it is a factor independently associated with lung cancer incidence and mortality, in part, as a result of differences in genetic susceptibility.16 A flowchart of patient selection is shown in eFigure 1 in the Supplement. Additional detailed methods for this article can be found in the eAppendix in the Supplement.

Statistical Analysis

Along with hazard ratios (HRs), we plotted OS as a function of time using propensity score–weighted Kaplan-Meier curves, median survival times, and P values from log-rank tests. Two-sided P < .05 was considered statistically significant. All analyses were performed in R statistical software version 4.0.5 (R Project for Statistical Computing), and figures were generated in R and Python software version 3.6.9 (Python). Data analysis was performed from October 2020 to March 2021.

Results
Demographic and Clinical Characteristics

In total, there were 355 patients: 183 (85 men [46.4%]) in the alectinib trial, 91 (43 men [47.3%]) in the ceritinib RWD group, and 81 (38 men [46.9%]) in the alectinib RWD group. Compared with the ceritinib RWD group, patients in the alectinib trial group were younger (mean [SD] age, 57.97 [11.71] vs 52.53 [11.18] years), were more heavily pretreated (mean [SD] number of prior therapy lines, 1.47 [0.81] vs 1.95 [0.72] lines; prior chemotherapy, 39 patients [42.9%] vs 136 patients [74.3%]), and had more favorable ECOG PS (ECOG PS of 0 or 1, 37 patients [77.1%] vs 165 patients [90.2%]) (Table 1). A substantially larger number of patients in the alectinib trials than in the ceritinib RWD group had documented CNS metastases (112 patients [61.2%] vs 19 patients [20.9%]), which is possibly due, in part, to differences in recording between clinical trials and real-world clinical practice. ECOG PS was missing for 43 patients (47.3%) in the ceritinib RWD group; no patients in the trial had a missing baseline ECOG PS. Following reweighting, sufficient balance (standardized mean difference, <0.1) was achieved for all covariates (Table 2), and propensity weight distributions showed acceptable overlap with maximum values between weights of 1 and 2 (eFigure 2 in the Supplement). Fewer imbalances in baseline characteristics were identified between alectinib RWD and ceritinib RWD groups. Patients in the alectinib RWD group (mean [SD] age, 58.69 [11.26] years) were more likely to have favorable ECOG PS (ECOG PS of 0 or 1, 49 patients [92.4%] vs 37 patients [77.1%]), to be White (56 patients [72.7%] vs 53 patients [62.4%]), and to be smokers (36 patients [44.4%] vs 30 patients [32.9%]) compared with the ceritinib RWD group (Table 1). ECOG PS was missing for 28 patients (34.6%) in the alectinib RWD group. The only other variables with missing values in these 2 RWD groups were race (6 of 91 patients [6.6%] in the ceritinib group and 4 of 81 patients [4.9%] in the alectinib group) and cancer stage at diagnosis (1 of 91 patients [1.1%] in the ceritinib group and 0 of 81 patients in the alectinib group). Sufficient balance was achieved for all covariates after weighting and distributions of propensity weights almost completely overlapped (Table 2 and eFigure 2 in the Supplement).

Treatment Effectiveness

Among patients with complete baseline confounder data (complete cases), alectinib was associated with significantly lower risk of all-cause mortality in both trial data vs RWD and RWD vs RWD comparisons. The adjusted HR for mortality comparing alectinib trial (180 patients) and ceritinib RWD (45 patients) was 0.55 (95% CI, 0.31-0.95; P = .03) (Table 3). Consistent with these results for trial data vs RWD comparisons, for the comparison of OS in alectinib RWD (52 patients) vs ceritinib RWD, the HR was 0.47 (95% CI, 0.23-0.96; P = .04) (Table 3). Unadjusted HRs were 0.64 (95% CI, 0.43-0.97) and 0.53 (95% CI, 0.30-0.92), respectively. Coefficients from the multivariable analyses are provided in eTable 1 and eTable 2 in the Supplement. The Little test for missing completely at random (MCAR) had a significant result, indicating that the MCAR assumption was generally invalid for the RWD.

Using multiple imputation to impute missing baseline confounder data, alectinib was similarly associated with a significantly lower risk of all-cause mortality in both trial data vs RWD and RWD vs RWD comparisons. The pooled adjusted HR comparing alectinib trial (180 patients) and ceritinib RWD (89 patients) was 0.59 (95% CI, 0.44 to 0.75; P < .001) (Table 3). In Kaplan-Meier curves, patients taking alectinib showed immediate improvement in OS after treatment initiation compared with those taking ceritinib and had significantly longer median survival times (29.1 months; 95% CI, 21.1 months to not estimable) compared with those taking ceritinib (14.9 months; 95% CI, 9.1 to 35.0 months). Sample adjusted Kaplan-Meier curves are shown in Figure 1.

Consistent with results from the alectinib trial group, patients in the alectinib RWD group had a significantly lower risk of all-cause death compared with patients in the ceritinib RWD group (HR, 0.46; 95% CI, 0.29 to 0.63; P < .001) (Table 3). Alectinib RWD were associated with significantly longer median survival time of 42.6 months (95% CI, 24.4 months to not estimable) compared with ceritinib RWD (17.8 months; 95% CI, 11.2 to 26.5 months). Adjustment for 11 additional baseline covariates, including comorbidities, all metastases, insurance status, and a separate Asian category for race (see the eAppendix in the Supplement for the full list), as a part of sensitivity analyses resulted in an HR of 0.60 (95% CI, 0.38-0.82). OS among alectinib trial and alectinib RWD was not significantly different (HR, 1.19; 95% CI, 0.83-1.56).

QBA of Unmeasured Confounding

To quantify the sensitivity of the adjusted worst-case HR from multiple imputation of 0.59 (95% CI, 0.44-0.75) to residual confounding, we computed the strength of associations of a hypothetical unmeasured binary confounder with mortality and treatment that would be required to nullify or reverse the observed beneficial effect of alectinib. We applied the method to the result with the point estimate closest to 1, which compared alectinib trial data with ceritinib RWD (HR, 0.59; 95% CI, 0.44-0.75). Results showed that the unmeasured confounder would need to be simultaneously highly correlated with mortality and imbalanced between treatment groups to reverse our findings (eFigure 3 in the Supplement). The E-value corresponding to HR of 0.59 (approximate risk ratio [RR], 0.69) was 2.24, representing confounder-mortality and confounder-treatment correlations needing to be simultaneously greater than 2.24 on the RR scale to move the HR point estimate to 1 and reverse our conclusions. This E-value was consistent with previous work using this data.17 At the 95% CI limit of 0.75, the E-value was 1.74, representing the strength of residual confounding required to move the 95% CI to be intersecting with 1, and rendering results nonsignificant.

For comparison, the largest imbalance in data was observed for prior chemotherapy (RR, 1.73; mortality RR, approximately 1.10). ECOG PS was the most prognostic variable (mortality RR, approximately 2.1; association with treatment RR, 1.19). Smoking status, another important confounder in NSCLC,18 had an association with treatment RR of 1.03 and mortality RR of 1.87.

QBA for Missing Data

The large amount of missing data for ECOG PS in the ceritinib group was a major source of concern for HTA appraisal of the previously submitted indirect comparison of alectinib trial and ceritinib RWD. To identify the distribution of missing ECOG PS in the ceritinib RWD that would be required to nullify or reverse the HR of 0.59 estimated missing at random (MAR) using multiple imputation, we performed a tipping point analysis by assuming that ECOG PS among ceritinib-treated patients in RWD could have been worse than expected under MAR (ie, missing not at random). To simulate worse-than-expected ECOG PS in the ceritinib RWD, we incorporated δ-shifts with multiple imputation.19,20 As shown in eFigure 4 in the Supplement, negative values for δ-shifted ECOG PS in the ceritinib group to progressively poorer distributions, with progressively greater number of patients in the ceritinib group excluded as they were shifted to ECOG PS greater than 2 (an exclusion criterion for our study).

Tipping points for loss of statistical significance were not identifiable when comparing alectinib trial data with ceritinib RWD because of the inability to achieve balance beyond δ = −3 (Figure 2), which corresponded to a mean shift of 1.04 (first and third quartiles, 0.8 and 1.3) for imputed ECOG PS. At δ = −3, 0 individuals had an imputed ECOG PS of 0, which was 37% lower than observed among those with nonmissing ECOG PS and 40% lower than expected under MAR. Instead, we used the RWD vs RWD comparison, where balance was easier to achieve when changing ECOG distributions.

For comparison of alectinib RWD with ceritinib RWD, the tipping point was identified at δ = −5 (Figure 2). A δ = −5 represents 148-fold odds of estimated ECOG PS in the ceritinib group being poorer than expected under MAR, or a mean shift of 1.63 to the estimated ECOG PS under MAR. For example, when δ = −5, a patient in the ceritinib RWD group with missing ECOG PS estimated to have a score of 0 conditional on their covariates, will instead be assigned a mean ECOG PS of 1.63. A δ = −5 corresponded to 0 patients with ECOG PS of 0, an approximately 20% decrease in number of patients with ECOG PS of 1 in the ceritinib RWD group and an additional 22 patients (22.5%) with ECOG PS shifted to greater than 2, who were excluded. HR point estimates plateaued between 0.75 and 0.85 with progressively worse ECOG PS shifts and did not reach 1 (Figure 2), indicating that our results could not be reversed under any scenario under missing not at random for ECOG PS, only lose their statistical significance.

In addition to ECOG PS, the distribution of comorbidities and metastases may have differed between trial data and RWD because of differences in surveillance and recording between trial and RWD. Sensitivity analyses found that our findings were robust to the assumption that, as a result of underreporting, CNS metastases and comorbidities actually had a 40% greater prevalence among those without a recorded CNS metastasis or comorbidity than reported in the ceritinib RWD.

Discussion

In settings where a randomized trial is infeasible or cannot be conducted in a timely manner, RWD can provide external control groups for head-to-head comparisons of treatment effectiveness.21 Real-world evidence can also supplement results from clinical trials and support reimbursement decisions. Despite their utility, bias in nonrandomized studies can be a major concern and is one reason for their relative nonacceptance by HTA agencies. As a result, there is a need for sensitivity analyses of bias in nonrandomized studies to support their findings. This comparative effectiveness study is a follow-up on previous published work9 comparing OS for alectinib trial vs ceritinib RWD in patients with crizotinib-refractory, ALK-positive NSCLC. We compared alectinib trial data and ceritinib RWD with updated data and included an additional comparison of ceritinib RWD with post–Food and Drug Administration approval alectinib RWD to support results from the trial data vs RWD comparison. Finally, we quantified the strength of residual confounding and missing data assumptions required to nullify our conclusions of a beneficial effect for alectinib for this indication.

Baseline patient characteristics between alectinib trial data and ceritinib RWD groups showed considerable imbalance in age and prior therapy. This is consistent with prior work using these data9 and may be due to differences in patient selection in a clinical trial vs real-world practices captured in an electronic health record–derived database. There are likely additional unmeasured complex systematic differences in patient selection, variable capturing, coding, and recording (construct validity) that exist between trial data and RWD and that we have not addressed in this study. For example, the accuracy of ECOG PS is likely measured with differing accuracy between trial settings and routine care settings, leading to differential misclassification, and clinical equivalence of ECOG in the 2 data sources may be limited.22 These scenarios could also be modeled using the methods presented within this study.

As an additional line of evidence, we compared alectinib RWD and ceritinib RWD groups, where patient data between treatment groups were likely better harmonized. As expected, we found fewer imbalances across baseline patient characteristics between the 2 RWD groups. ECOG PS was a key imbalance in both cases: ceritinib-treated patients in RWD had worse baseline ECOG PS than alectinib-treated patients in both the trial data and RWD. Although random chance may underlie this imbalance, it is also possible that patients with a more favorable performance status were preferentially channeled23 to alectinib because of factors associated with clinical pathways and quality of care, such as perceived efficacy, toxicity, cost, ease of prescribing, and reimbursement.

After adjustment for baseline confounders, alectinib-treated patients had a lower risk of all-cause mortality than those treated with ceritinib. This statistically significant beneficial effect was consistent for both trial data vs RWD and RWD vs RWD comparisons using multiple imputation of missing baseline confounder data under MAR. HRs from complete-case analysis were also consistent with results from multiple imputation, suggesting that our findings were robust to assumptions of data MAR and MCAR. HRs in this study were similar to those in the study by Davies et al,9 which estimated a mortality HR of 0.65 (95% CI, 0.48-0.88) comparing alectinib trial and ceritinib RWD, and were almost identical to their HR from imputation of 0.58 (95% CI, 0.44-0.76). Differences between our study and that of Davies et al9 could be associated with the availability of additional patients and longer follow-up for some patients in the Flatiron Health database in our analysis, and our further harmonization of the definition of therapy lines between trial data and RWD.

For bias analysis, we first quantified the strength of residual confounding needed to reverse our findings. We found that for a hypothetical highly prognostic binary confounder to fully explain away a HR of 0.59 (95% CI, 0.44-0.75), it needed to be highly imbalanced between treatment groups. Our assumption of a binary confounder, as opposed to a continuous or polytomous one, was to simplify interpretability without loss of generality. On the basis of our estimates of correlation with mortality and imbalance between treatment groups for important measured prognostic variables such as ECOG PS and smoking history, we do not expect that sufficiently large systematic differences in unmeasured prognostic variables to reverse our findings were plausible.

We next explored the extent to which missing ECOG PS data would need to deviate from MAR to explain away our findings of a beneficial effect for alectinib. We focused on ECOG PS because it was highlighted by HTA agencies in their appraisal over concerns about the large amount of missing values and its high correlation with OS, but also because it may not be uniformly recorded across clinical practices and trials. We tested the assumption that patients with missing ECOG PS in the ceritinib RWD may have skewed toward poorer values than expected using multiple imputation under MAR, which may have resulted in confounding. To do this, we estimated HR and 95% CI over a range of distributions of ECOG PS in the ceritinib group by shifting progressively more patients to worse ECOG PS than would be estimated given other measured covariates. Because this shift in ECOG PS worsened the imbalance between alectinib trial data and ceritinib RWD and further reduced sample sizes as increasingly more patients were shifted to ECOG PS greater than 2 and, thus, were excluded, we were unable to identify tipping points when comparing alectinib trial data with ceritinib RWD because of the inability to achieve balance beyond δ = −3.

For RWD vs RWD comparisons, where balance was easier to achieve, the tipping point for loss of significance was δ = −5; no tipping point was identified for reversal of conclusions of a beneficial effect for alectinib because HR point estimates plateaued at approximately 0.8. The worst-case tipping point of δ = −3 corresponds to 20-fold greater odds of a poorer ECOG PS than expected under MAR in the ceritinib RWD group. More specifically, this corresponded to a mean ECOG PS shift of 1.04, and 0 individuals with ECOG PS of 0 among those missing baseline ECOG. This was 37% lower than observed among those with a nonmissing baseline ECOG PS, and 40% lower than expected under MAR. We expect that such an ECOG PS distribution in the ceritinib RWD by random chance is implausible. We also do not expect that large systematic differences in missingness mechanisms between treatment groups, which could have given rise to such data, should exist.

Strengths and Limitations

The strengths of this study include a nationally representative RWD sample and comparisons of both trial data and RWD for alectinib. As with any nonrandomized comparison, there is the possibility of unmeasured confounding. Although this study includes a rigorous assessment of bias due to unmeasured confounding and missing ECOG PS and underrecorded comorbidities and metastases, there may be additional multifactorial sources of bias that could have affected the reported estimates. However, given that comparisons of trial data vs RWD and RWD vs RWD produced consistent findings and that findings of a beneficial effect of alectinib from this study appear to be consistent with a retrospective analysis of progression-free survival and objective response in a cohort of patients in Taiwan for the same indication,24 overall conclusions from this study are expected to be robust. Besides OS, there may be other patient-important measures of effectiveness (eg, toxicity) that were not examined in this study. However, prior retrospective analyses suggest that alectinib may be associated with fewer adverse events than ceritinib.24 Note that unbiased estimation of treatment effects using weighting depends on correct specification of the propensity model. Although a doubly robust estimator was used to reduce dependence on this assumption, it is not guaranteed to provide unbiased estimates if both the propensity and outcome models were misspecified.

Conclusions

In conclusion, our results were consistent with prior studies and showed that alectinib was associated with longer OS compared with ceritinib in patients with crizotinib-refractory, ALK-positive NSCLC. Application of QBA methods showed that our findings were robust to plausible sources of bias from unmeasured confounding and distributional assumptions about missing baseline ECOG PS and underrecording of comorbidities. There may be additional sources of bias that we have not addressed in this study, such as from misclassification of OS, which are especially relevant for comparison of single-group trial data and RWD. We propose that transparent and rigorous quantitative evaluation of uncertainty under a range of scenarios and sources of bias, and a discussion of their plausibility, should accompany submissions of nonrandomized studies to aid regulatory and reimbursement decision-making.

Back to top
Article Information

Accepted for Publication: July 20, 2021.

Published: October 7, 2021. doi:10.1001/jamanetworkopen.2021.26306

Open Access: This is an open access article distributed under the terms of the CC-BY-NC-ND License. © 2021 Wilkinson S et al. JAMA Network Open.

Corresponding Author: Sreeram Ramagopalan, PhD, Global Access, F. Hoffmann–La Roche, Grenzacherstrasse 124, Basel CH-4070, Switzerland (sreeram.ramagopalan@roche.com).

Author Contributions: Drs Ramagopalan and Gupta had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Wilkinson, Mackay, Arora, Thorlund, Wasiak, Ray, Ramagopalan.

Acquisition, analysis, or interpretation of data: Wilkinson, Gupta, Scheuer, Arora, Thorlund, Ray, Ramagopalan, Subbiah.

Drafting of the manuscript: Wilkinson, Gupta, Arora, Thorlund, Ray.

Critical revision of the manuscript for important intellectual content: Wilkinson, Scheuer, Mackay, Arora, Thorlund, Wasiak, Ramagopalan, Subbiah.

Statistical analysis: Wilkinson, Gupta, Thorlund, Ray.

Obtained funding: Arora, Ramagopalan.

Administrative, technical, or material support: Mackay, Arora, Wasiak, Ray, Subbiah.

Supervision: Arora, Thorlund, Wasiak, Ramagopalan.

Conflict of Interest Disclosures: Dr Wilkinson reported being employed by and holding stock in Roche. Dr Gupta reported receiving funding from Roche during the conduct of the study. Dr Scheuer reported receiving personal fees from Roche, receiving shares from Roche as an employee during the conduct of the study, and reported being an employee of and receiving shares from Novartis outside the submitted work. Mr Mackay reported receiving funding from Roche during the conduct of the study. Dr Arora reported receiving grants from Roche during the conduct of the study. Dr Ray reported receiving nonfinancial support from and being an employee of F. Hoffmann–La Roche, Ltd. Dr Ramagopalan reported receiving personal fees from Roche during the conduct of the study. Dr Subbiah reported receiving grants from Roche outside the submitted work; receiving research grants from Roche/Genentech, Eli Lilly/Loxo Oncology, Bayer, GlaxoSmithKline, Nanocarrier, Vegenics, Celgene, Northwest Biotherapeutics, Berghealth, Incyte, Fujifilm, D3, Pfizer, Multivir, Amgen, Abbvie, Alfa-sigma, Agensys, Boston Biomedical, Idera Pharma, Inhibrx, Exelixis, Blueprint Medicines, Altum, Dragonfly Therapeutics, Takeda, National Comprehensive Cancer Network, National Cancer Institute Cancer Therapy Evaluation Program, University of Texas MD Anderson Cancer Center, Turning Point Therapeutics, Boston Pharmaceuticals, Novartis, Pharmamar, and Medimmune; serving in advisory board or consultant positions with Helsinn, Incyte, QED Pharma, Daiichi-Sankyo, Signant Health, Novartis, Eli Lilly/Loxo Oncology, Relay Therapeutics, and Medimmune; receiving travel funds from Pharmamar, Incyte, ASCO, and ESMO; and receiving other support from Medscape, all outside the submitted work. No other disclosures were reported.

Funding/Support: This study was funded by F. Hoffmann–La Roche Ltd.

Role of the Funder/Sponsor: The funder was involved in all aspects of the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Hall  PS.  Real-world data for efficient health technology assessment.   Eur J Cancer. 2017;79:235-237. doi:10.1016/j.ejca.2017.04.003PubMedGoogle ScholarCrossref
2.
Oortwijn  W, Sampietro-Colom  L, Trowman  R.  How to deal with the inevitable: generating real-world data and using real-world evidence for HTA purposes—from theory to action.   Int J Technol Assess Health Care. 2019;35(4):346-350. doi:10.1017/S0266462319000400PubMedGoogle ScholarCrossref
3.
Hernán  MA, Hernández-Díaz  S, Robins  JM.  A structural approach to selection bias.   Epidemiology. 2004;15(5):615-625. doi:10.1097/01.ede.0000135174.63482.43PubMedGoogle ScholarCrossref
4.
Roberts  MH, Ferguson  GT.  Real-world evidence: bridging gaps in evidence to guide payer decisions.   Pharmacoecon Open. 2021;5(1):3-11. doi:10.1007/s41669-020-00221-yPubMedGoogle ScholarCrossref
5.
Malone  DC, Brown  M, Hurwitz  JT, Peters  L, Graff  JS.  Real-world evidence: useful in the real world of US payer decision making? how? when? and what studies?   Value Health. 2018;21(3):326-333. doi:10.1016/j.jval.2017.08.3013PubMedGoogle ScholarCrossref
6.
O’Donnell  JC, Le  TK, Dobrin  R,  et al.  Evolving use of real-world evidence in the regulatory process: a focus on immuno-oncology treatment and outcomes.   Future Oncol. 2021;17(3):333-347. doi:10.2217/fon-2020-0591PubMedGoogle ScholarCrossref
7.
Lash  TL, Fox  MP, Cooney  D, Lu  Y, Forshee  RA.  Quantitative bias analysis in regulatory settings.   Am J Public Health. 2016;106(7):1227-1230. doi:10.2105/AJPH.2016.303199PubMedGoogle ScholarCrossref
8.
Lash  TL, Fox  MP, Fink  AK.  Applying Quantitative Bias Analysis to Epidemiologic Data. Springer Science & Business Media; 2011.
9.
Davies  J, Martinec  M, Delmar  P,  et al.  Comparative effectiveness from a single-arm trial and real-world data: alectinib versus ceritinib.   J Comp Eff Res. 2018;7(9):855-865. doi:10.2217/cer-2018-0032PubMedGoogle ScholarCrossref
10.
IQWiG. IQWiG reports: commission No. A17-19—Alectinib (non-small cell lung cancer). May 2, 2017. Accessed September 3, 2021. https://www.iqwig.de/download/a17-19_alectinib_extract-of-dossier-assessment_v1-0.pdf?rev=185028
11.
CADTH. Alecensaro for locally advanced or metastatic non-small lung cancer (second line): details. CADTH project number PC0114-000. August 25, 2017. Accessed September 3, 2021. https://www.cadth.ca/alecensaro-locally-advanced-or-metastatic-non-small-cell-lung-cancer-second-line-details
12.
Ou  S-HI, Ahn  JS, De Petris  L,  et al.  Alectinib in crizotinib-refractory ALK-rearranged non–small-cell lung cancer: a phase II global study.   J Clin Oncol. 2016;34(7):661-668. doi:10.1200/JCO.2015.63.9443PubMedGoogle ScholarCrossref
13.
Shaw  AT, Gandhi  L, Gadgeel  S,  et al.  Alectinib in ALK-positive, crizotinib-resistant, non-small-cell lung cancer: a single-group, multicentre, phase 2 trial.   Lancet Oncol. 2016;17(2):234-242. doi:10.1016/S1470-2045(15)00488-XPubMedGoogle ScholarCrossref
14.
Allan  V, Ramagopalan  SV, Mardekian  J,  et al.  Propensity score matching and inverse probability of treatment weighting to address confounding by indication in comparative effectiveness research of oral anticoagulants.   J Comp Eff Res. 2020;9(9):603-614. doi:10.2217/cer-2020-0013PubMedGoogle ScholarCrossref
15.
Vandenbroucke  JP, von Elm  E, Altman  DG,  et al; STROBE Initiative.  Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration.   PLoS Med. 2007;4(10):e297. doi:10.1371/journal.pmed.0040297PubMedGoogle Scholar
16.
Schabath  MB, Cress  WD, Muñoz-Antonia  T.  Racial and ethnic differences in the epidemiology and genomics of lung cancer.   Cancer Control. 2016;23(4):338-346. doi:10.1177/107327481602300405PubMedGoogle ScholarCrossref
17.
Sammon  CJ, Leahy  TP, Gsteiger  S, Ramagopalan  S.  Real-world evidence and nonrandomized data in health technology assessment: using existing methods to address unmeasured confounding?   J Comp Eff Res. 2020;9(14):969-972. doi:10.2217/cer-2020-0112PubMedGoogle ScholarCrossref
18.
Gemine  RE, Ghosal  R, Collier  G,  et al; LungCast Investigators.  Longitudinal study to assess impact of smoking at diagnosis and quitting on 1-year survival for people with non-small cell lung cancer.   Lung Cancer. 2019;129:1-7. doi:10.1016/j.lungcan.2018.12.028PubMedGoogle ScholarCrossref
19.
Leacy  FP, Floyd  S, Yates  TA, White  IR.  Analyses of sensitivity to the missing-at-random assumption using multiple imputation with delta adjustment: application to a tuberculosis/HIV prevalence survey with incomplete HIV-status data.   Am J Epidemiol. 2017;185(4):304-315. doi:10.1093/aje/kww107PubMedGoogle Scholar
20.
Rezvan  PH, Lee  KJ, Simpson  JA.  Sensitivity analysis within multiple imputation framework using delta-adjustment: application to Longitudinal Study of Australian Children.   Longitudinal Life Course Studies. 2018;9(3):259-278. doi:10.14301/llcs.v9i3.503Google ScholarCrossref
21.
Thorlund  K, Dron  L, Park  JJH, Mills  EJ.  Synthetic and external controls in clinical trials: a primer for researchers.   Clin Epidemiol. 2020;12:457-467. doi:10.2147/CLEP.S242097PubMedGoogle ScholarCrossref
22.
Webster-Clark  M, Jonsson Funk  M, Stürmer  T.  Single-arm trials with external comparators and confounder misclassification: how adjustment can fail.   Med Care. 2020;58(12):1116-1121. doi:10.1097/MLR.0000000000001400PubMedGoogle ScholarCrossref
23.
Lobo  FS, Wagner  S, Gross  CR, Schommer  JC.  Addressing the issue of channeling bias in observational studies with propensity scores analysis.   Res Social Adm Pharm. 2006;2(1):143-151. doi:10.1016/j.sapharm.2005.12.001PubMedGoogle ScholarCrossref
24.
Kuo  CS, Tung  PH, Huang  AC,  et al.  A retrospective study of alectinib versus ceritinib in patients with advanced non-small-cell lung cancer of anaplastic lymphoma kinase fusion in whom crizotinib treatment failed.   BMC Cancer. 2021;21(1):309. doi:10.1186/s12885-021-08005-1PubMedGoogle ScholarCrossref
×