Comparison of Placebos and Propensity Score Adjustment in Multiple Sclerosis Nonrandomized Studies | Demyelinating Disorders | JAMA Neurology | JAMA Network
[Skip to Navigation]
Sign In
Figure.  Forest Plot Reporting the Unadjusted and the Adjusted Hazard Ratio for Placebo B vs Placebo A
Forest Plot Reporting the Unadjusted and the Adjusted Hazard Ratio for Placebo B vs Placebo A

Placebo A used the AFFIRM trial, whereas placebo B combined the DEFINE-CONFIRM trials. The untrimmed inverse probability weighting (IPW) included all the patients, whereas the trimmed IPW excluded patients with a propensity score (PS) with no overlap between the 2 compared arms. HR indicates hazard ratio; MSM, marginal structural model.

1.
Trojano  M, Tintore  M, Montalban  X,  et al.  Treatment decisions in multiple sclerosis—insights from real-world observational studies.   Nat Rev Neurol. 2017;13(2):105-118. doi:10.1038/nrneurol.2016.188PubMedGoogle ScholarCrossref
2.
Polman  CH, O’Connor  PW, Havrdova  E,  et al; AFFIRM Investigators.  A randomized, placebo-controlled trial of natalizumab for relapsing multiple sclerosis.   N Engl J Med. 2006;354(9):899-910. doi:10.1056/NEJMoa044397PubMedGoogle ScholarCrossref
3.
Gold  R, Kappos  L, Arnold  DL,  et al; DEFINE Study Investigators.  Placebo-controlled phase 3 study of oral BG-12 for relapsing multiple sclerosis.   N Engl J Med. 2012;367(12):1098-1107. doi:10.1056/NEJMoa1114287PubMedGoogle ScholarCrossref
4.
Fox  RJ, Miller  DH, Phillips  JT,  et al; CONFIRM Study Investigators.  Placebo-controlled phase 3 study of oral BG-12 or glatiramer in multiple sclerosis.   N Engl J Med. 2012;367(12):1087-1097. doi:10.1056/NEJMoa1206328PubMedGoogle ScholarCrossref
5.
Kalincik  T, Brown  JWL, Robertson  N,  et al; MSBase Study Group.  Treatment effectiveness of alemtuzumab compared with natalizumab, fingolimod, and interferon beta in relapsing-remitting multiple sclerosis: a cohort study.   Lancet Neurol. 2017;16(4):271-281. doi:10.1016/S1474-4422(17)30007-8PubMedGoogle ScholarCrossref
6.
Brown  JWL, Coles  A, Horakova  D,  et al; MSBase Study Group.  Association of initial disease-modifying therapy with later conversion to secondary progressive multiple sclerosis.   JAMA. 2019;321(2):175-187. doi:10.1001/jama.2018.20588PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    EXPAND ALL
    Propensity Score Adjustment in Multiple Sclerosis Nonrandomized Studies: a Neurologist's Perspective
    Luca Prosperini, MD, PhD | S. Camillo-Forlanini Hospital, MS Center and Neurology Unit, Rome, Italy
    I have read with great interest the research letter by Signori et al. and I fully agree that propensity score-based comparisons of non-randomized observational data should be interpreted with great caution. The Authors have elegantly demonstrated the bias generated by the violation of the positivity assumption in the context of PS-based adjustment that have been frequently used in multiple sclerosis (MS) studies.
    However, I guess, the question is… why there is an increasing number of real world studies comparing different treatment strategies in MS?
    The answer probably is that “…in the era of precision medicine, the aim in the
    clinic is no longer to (simply) prescribe an available treatment to a given patient, naive or nonresponder, but to prescribe the best possible treatment for that given patient, with a very specific set of clinical and paraclinical characteristics” (Tur C et al. Neurology 2019; 93:1-17).
    Multicenter, randomized, controlled clinical trials are undoubtedly the gold standard to obtain evidence-based data on the relative efficacy and safety profile of MS treatments. Unfortunately, head-to-head clinical trials are scarce and indirect comparisons of placebo-controlled trials are also biased by multiple factors, including (but not limited to) difference in study design, targeted population, baseline patients’ characteristics and behaviour of placebo groups.
    Consequently, important clinical questions would remain unanswered without real world studies.
    To give you some examples.
    (1) Is escalation to monoclonal antibodies better than switching to different immunomodulant agents? In April 2012, the SURPASS study (ClinicalTrials.gov Identifier: NCT01058005) was stopped prematurely by the sponsor due to slow patient enrolment. In the following years, observational PS-based studies suggested that escalation is better than switching in patients with unsatisfactory response to platform therapies (Prosperini et al. Mult Scler 2012; 18:64-71; Brown et al. JAMA 2019; 321:175-187).
    (2) Is Natalizumab superior to Fingolimod on MS-related brain tissue damage? In May 2016, the REVEAL study (ClinicalTrials.gov Identifier: NCT02342704) was stopped prematurely by the sponsor for business decision. Later, observational PS-based studies suggested the superiority of Natalizumab over Fingolimod on conventional magnetic resonance imaging markers (Barbin et al. Neurology 2016; 86:771-8; Baroncini D et al. Mult Scler 2016; 22:1315-26).
    (3) Which is the best oral drugs? In October 2019, the PRAG-MS study (ClinicalTrials.gov Identifier: NCT03345940), supported by PCORI, was terminated due to slow enrolment rate. More recently, observational PS-based studies aimed to compare the currently available oral drugs (Kalincik et al. J Neurol Neurosurg Psychiatry 2019; 90: 458-468; Ontaneda D et al. Mult Scler Relat Disord 2019; 27:101-111).
    My impression is that pharmaceutical companies have no interest in carrying out head-to-head trials which instead would provide extremely relevant clinical answers. In the absence of these data, we will continue to rely on real world studies, which although only hypotheses-generating and less robust, nevertheless provide relevant insights in the complex therapeutic management of MS.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Views 1,791
    Citations 0
    Research Letter
    April 20, 2020

    Comparison of Placebos and Propensity Score Adjustment in Multiple Sclerosis Nonrandomized Studies

    Author Affiliations
    • 1Section of Biostatistics, Department of Health Sciences, University of Genova, Genova, Italy
    • 2Biogen International GmbH, Baar, Switzerland
    • 3Biogen Inc, Cambridge, Massachusetts
    • 4IRCCS Ospedale Policlinico San Martino, Genova, Italy
    JAMA Neurol. 2020;77(7):902-903. doi:10.1001/jamaneurol.2020.0678

    In the last decade, methods based on propensity scores (PSs) have been frequently used in multiple sclerosis (MS) studies comparing disease-modifying treatments in nonrandomized observational settings.1 Propensity score adjustment was applied even in situations when all the necessary conditions for its applicability were not satisfied. The PS adjustment can reduce the intrinsic selection bias of nonrandomized studies only if all the confounders are measurable and are at least minimally overlapped between the treatment groups. Sometimes the calendar period or the geographical region of study conduction can be completely nonoverlapping between the compared groups. In such cases, in the causal inference jargon, the positivity assumption, requiring that the probability to receive any of the treatments in all the PS strata is higher than 0, is violated. We will show, with a practical example, the extent of failure of PS adjustment when the positivity assumption is violated.

    Methods

    We merged the placebo arm (placebo A) of the AFFIRM study (published in 20062) testing natalizumab vs placebo and the pooled placebo group (placebo B) from the DEFINE and CONFIRM studies (published in 20123,4) testing dimethyl-fumarate vs placebo. The DEFINE and CONFIRM trials were approved by central and local ethics committees and conducted in accordance with International Conference on Harmonisation Good Clinical Practice guidelines and the Declaration of Helsinki.

    We assessed the association of placebo A vs placebo B with the time to 6-month confirmed disability progression (CDP) as defined in the study reports,2-4 preferring the 6-month rather than the 3-month confirmation (primary end point of the randomized trials) to mimic what happens in observational studies. The PS was calculated by a logistic regression model that included all the baseline covariates common to the 3 trials. Cohen standardized mean differences were calculated between the 2 placebo groups in the original samples and after matching or weighting.

    We used first a Greedy 5-to-1-digit, 1:1-matching algorithm in which matched pairs are randomly selected from all possible pairs with equal PSs. Second, we applied an inverse probability weighting (IPW) approach analyzing all patients first (untrimmed IPW) and then including only patients with a PS overlapping between arms (trimmed IPW). Finally, we applied marginal structural models (MSMs) to account for differences in withdrawal patterns before CDP. The association of the 2 placebos was compared by a Cox model. Stata, version 16 (StataCorp) was used and statistical significance was set at P <.05.

    Results

    The baseline differences were consistently reduced by the PS adjustments. The unadjusted analysis indicated the superiority of placebo B over placebo A in association with CDP (hazard ratio, 0.64; 95% CI, 0.47-0.88; P = .006). The superiority of placebo B vs placebo A was confirmed and reinforced after adjustment using IPW and MSM methods (Figure). Only the PS matching 1:1 showed a nonsignificant difference between the 2 placebo arms, mainly because of the reduced number of patients after the matching (hazard ratio, 0.78; 95% CI, 0.53-1.16; P = .22). The analysis on 3-month CDP gave identical results.

    Discussion

    In this study, we showed the superiority of a placebo over another placebo using data from randomized clinical trials, adjusting the comparison with various PS methods and MSMs. Despite the efficiency of these techniques in mitigating the baseline differences of the compared cohorts (the 1:1 PS matching generally displayed the lowest Cohen standardized mean differences but the largest potentially informative losses), the superiority of placebo in DEFINE-CONFIRM vs the placebo in AFFIRM did not decrease when adjusting for PS. In this setting, PS cannot adjust for the different period of the trials’ conduction. It is well known in the epidemiological community that PS methods in the presence of a positivity assumption violation cannot guarantee the adjustment of the treatment selection bias. Nevertheless, the positivity assumption violation is present in most of the observational studies published in the MS field, in which the treatment cohorts do not overlap in geographical areas5,6 or periods (as in the present example). Therefore, the results of such comparisons must be interpreted with caution.

    Back to top
    Article Information

    Corresponding Author: Maria Pia Sormani, PhD, Department of Health Sciences, University of Genoa, Via Pastore 1, Genova, Italy (mariapia.sormani@unige.it).

    Published Online: April 20, 2020. doi:10.1001/jamaneurol.2020.0678

    Author Contributions: Drs Signori and Pellegrini had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Signori, Pellegrini, Carmisciano, de Moor, Sormani.

    Acquisition, analysis, or interpretation of data: Signori, Pellegrini, Bovis, de Moor, Sormani.

    Drafting of the manuscript: Signori, Pellegrini, Bovis, Sormani.

    Critical revision of the manuscript for important intellectual content: Signori, Pellegrini, Carmisciano, de Moor, Sormani.

    Statistical analysis: Signori, Pellegrini, Bovis, de Moor, Sormani.

    Administrative, technical, or material support: Pellegrini, de Moor.

    Supervision: Pellegrini, Sormani.

    Conflict of Interest Disclosures: Dr Pellegrini reported that he is a Biogen employee and owns stocks of my company. Dr Bovis reported personal fees from Novartis and Eisai outside the submitted work. Dr Sormani reported personal fees from Biogen during the conduct of the study. No other disclosures were reported.

    Funding/Support: This study was supported by Biogen.

    Role of the Funder/Sponsor: Biogen had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    References
    1.
    Trojano  M, Tintore  M, Montalban  X,  et al.  Treatment decisions in multiple sclerosis—insights from real-world observational studies.   Nat Rev Neurol. 2017;13(2):105-118. doi:10.1038/nrneurol.2016.188PubMedGoogle ScholarCrossref
    2.
    Polman  CH, O’Connor  PW, Havrdova  E,  et al; AFFIRM Investigators.  A randomized, placebo-controlled trial of natalizumab for relapsing multiple sclerosis.   N Engl J Med. 2006;354(9):899-910. doi:10.1056/NEJMoa044397PubMedGoogle ScholarCrossref
    3.
    Gold  R, Kappos  L, Arnold  DL,  et al; DEFINE Study Investigators.  Placebo-controlled phase 3 study of oral BG-12 for relapsing multiple sclerosis.   N Engl J Med. 2012;367(12):1098-1107. doi:10.1056/NEJMoa1114287PubMedGoogle ScholarCrossref
    4.
    Fox  RJ, Miller  DH, Phillips  JT,  et al; CONFIRM Study Investigators.  Placebo-controlled phase 3 study of oral BG-12 or glatiramer in multiple sclerosis.   N Engl J Med. 2012;367(12):1087-1097. doi:10.1056/NEJMoa1206328PubMedGoogle ScholarCrossref
    5.
    Kalincik  T, Brown  JWL, Robertson  N,  et al; MSBase Study Group.  Treatment effectiveness of alemtuzumab compared with natalizumab, fingolimod, and interferon beta in relapsing-remitting multiple sclerosis: a cohort study.   Lancet Neurol. 2017;16(4):271-281. doi:10.1016/S1474-4422(17)30007-8PubMedGoogle ScholarCrossref
    6.
    Brown  JWL, Coles  A, Horakova  D,  et al; MSBase Study Group.  Association of initial disease-modifying therapy with later conversion to secondary progressive multiple sclerosis.   JAMA. 2019;321(2):175-187. doi:10.1001/jama.2018.20588PubMedGoogle ScholarCrossref
    ×