Time-Related Biases in Nonrandomized COVID-19–Era Studies Using Real-world Data | Hematology | JAMA Oncology | JAMA Network
[Skip to Navigation]
Sign In
Views 3,101
Citations 0
Invited Commentary
June 17, 2021

Time-Related Biases in Nonrandomized COVID-19–Era Studies Using Real-world Data

Author Affiliations
  • 1Flatiron Health, Inc, New York, New York
  • 2University of Illinois at Chicago, College of Pharmacy
  • 3Boston Medical Center, Boston, Massachusetts
JAMA Oncol. 2021;7(8):1175-1177. doi:10.1001/jamaoncol.2021.1715

The urgent response to the COVID-19 pandemic has highlighted the importance of diverse, real-world data sources, such as electronic health records, insurance claims, and patient registries, to further inform evidence-based care amid an evolving public health crisis. Real-world data have the potential to provide a wealth of rapid, actionable information and inform ongoing work to evaluate the effectiveness and safety of potential therapies, vaccines, or diagnostics for COVID-19.

Having cancer was recognized early as a potential risk factor for severe illness associated with COVID-19 given the increased likelihood of having a weakened immune system and frequent multimorbidity observed among patients with cancer. Patients with hematologic cancer often have abnormal or depleted levels of immune cells that produce antibodies against viruses and may be at higher risk of prolonged infection and death from COVID-19 relative to patients with solid tumors.

In this issue of JAMA Oncology, Thompson and colleagues1 performed a retrospective cohort study evaluating the association of convalescent plasma therapy with mortality among hospitalized patients with COVID-19 with hematologic cancer in the COVID-19 and Cancer Consortium (CCC19; https://ccc19.org/), a multi-institution collaboration of more than 120 cancer centers and health care organizations. The presumed therapeutic mechanism of action for convalescent blood products is related to the antibodies to SARS-CoV-2 that are passively transferred to the recipient. In the 1918 influenza pandemic, a clinically important reduction in mortality was inferred for patients with Spanish influenza pneumonia who received convalescent blood products; notably, none of the studies in a meta-analysis supporting this therapeutic benefit were blinded, randomized, or placebo-controlled trials.2 Access to convalescent plasma therapeutics is expanding, and emerging evidence3 indicates that transfusion of plasma with high anti–SARS-CoV-2 immunoglobulin G antibody levels is associated with a lower risk of death in hospitalized patients with COVID-19. Thompson et al1 fill a major knowledge gap, as most studies on convalescent plasma or hyperimmune immunoglobulin have few patients with cancer, particularly hematologic cancer.4 Here we review how several important aspects of time-related bias were considered in this study, including time-dependent confounding and selection bias.

In the present analysis of patients with hematologic cancer hospitalized with COVID-19,1 30-day mortality was compared between 143 patients treated with convalescent plasma and 823 untreated controls. The investigators took multiple approaches to account for bias owing to the nonrandomized assignment of convalescent plasma treatment. A traditional approach using multivariable Cox proportional hazards models comparing all convalescent plasma–treated patients to 823 unmatched patients included covariate adjustment for multiple a priori confounders: age, sex, race/ethnicity, hematologic cancer type, current cancer status, cancer treatment timing, Eastern Cooperative Oncology Group performance status, obesity, presence of type 2 diabetes mellitus, hypertension, kidney comorbidities, pulmonary comorbidities, receipt of cytotoxic chemotherapy within 3 months of COVID-19 diagnosis, and trimester of diagnosis. Based on this approach, a statistically significant lower hazard of mortality within 30 days of hospitalization was found (hazard ratio [HR], 0.60; 95% CI, 0.37-0.97). A secondary analytic approach using propensity score matching was used in this study to address the imbalance of measured confounding factors at the time of hospitalization. Propensity score methods are a powerful tool for addressing confounder imbalances in which the likelihood of receiving convalescent plasma is predicted on the basis of measured confounders, then groups of treated and untreated patients matched 1:1 with comparable propensity scores are compared. These methods can have varying degrees of effectiveness in reducing bias relative to traditional multivariable adjustment depending on the nature of the event of interest (eg, disease-specific survival vs adverse events)5 and the number of measured confounders and events observed.6 The slightly greater benefit found in the propensity score–matched analysis (HR, 0.52; 95% CI, 0.37-0.92) implies that this more robust control of confounding accounted for an underlying higher risk of death in patients who ultimately received convalescent plasma.

Several noteworthy limitations to a conventional time-to-event analysis in this setting were also present. Treatment exposure was defined as having received convalescent plasma at any time during the COVID-19 hospitalization regardless of the time between hospitalization and plasma administration. This is contrary to clinical trial protocols for hospitalized patients with COVID-19 where this administration date is typically defined within a certain period (eg, ≤72 hours since hospital admission).7 Furthermore, it was unclear if at the time of hospitalization patients required supplemental oxygen, maintained adequate oxygen levels, or were on mechanical ventilation. This information is critical in determining an unbiased estimation of treatment benefit on overall survival benefit, as patients would need to survive a period of time after their index follow-up time (ie, hospitalization date in this study) to have the opportunity to undergo the plasma treatment. This potential bias, sometimes referred to as length bias or survival bias, may result in survival function estimates that are higher than would be obtained in prospective study.8 In a prospective study of this kind, long-surviving patients would not be preferentially sampled. Under the assumption of independent delayed entry, hazard of death after a patient’s entry time does not depend on the patient’s entry time to the cohort, and risk set adjustment provides an unbiased estimation of the survival function. In this case, a risk set adjustment would have treated patients at risk of survival function estimation only after they have satisfied the requisite inclusion criteria, even if that is after the applicable index date.

Separate from these selection biases was the potential influence of misclassified follow-up time for patients who received convalescent plasma, or immortal time bias. For the current study, the categorization of patients into groups that did or did not receive convalescent plasma at any time during their hospitalization is problematic because their eventual exposure could have happened days or weeks after the start of hospitalization (but before the outcome of 30-day mortality), and all of the preceding time would be counted in the analysis as “immortal” exposed time. In this scenario, even despite robust confounding control measures such as propensity score matching, which accounts for measured imbalanced characteristics between comparison groups at baseline, a persistent potential bias of effect estimates exists that differentially favors survival in exposed patients. Suissa9 described 2 methods for mitigating the influence of immortal time bias in observational studies: (1) treating exposures as time varying, in which patients are considered unexposed in the analysis up until the time treatment is received; and (2) starting follow-up time (ie, index time 0) for both exposed and unexposed patients after a fixed exposure assessment window. At a minimum, these approaches or similar methods should be adopted in future analyses of the CCC19 registry data to evaluate the robustness of the therapeutic benefits of convalescent plasma reported by Thompson et al.1

Finally, the investigators describe limitations in their study design arising from the temporality of other meaningful treatments and clinical exposures after the administration of convalescent plasma. For example, other treatments that occur both before and after convalescent plasma treatment, such as corticosteroids (eg, dexamethasone) and broad-spectrum antivirals (eg, remdesivir), could affect whether convalescent plasma treatment is initiated or continued and have known associations with mortality in COVID-19. The investigators also performed stratified analyses of more severely ill patients that restricted estimates to patients requiring mechanical ventilation, conditioning inclusion in these subanalyses on a future event that occurs later during follow-up. This leaves the possibility that the substantial survival benefit estimated among patients who required mechanical ventilation (HR, 0.20; 95% CI, 0.10-0.50) is biased away from the null. Analytic approaches to account for these other treatments could include using time-varying covariates in regression models or time-dependent confounding control through the use of marginal structural models.10 On the other hand, other COVID-19 treatment subsequent to convalescent plasma administration reflects clinical practice and could be in the causal pathway with 30-day mortality. Thus, the findings reported here without such adjustment offer insights on convalescent plasma treatment occurring in current real-world conditions. As the number of patients treated in the CCC19 registry grows, further rigorous sensitivity analyses and analytic approaches should be pursued.

This study by Thompson et al1 is an important starting point to guide emerging clinical evidence on the use of convalescent blood products in patients with hematologic cancer with severe COVID-19 illness. Despite the potential limitations from time-related biases described here and other residual confounding from unmeasured factors in this nonrandomized study, the consistent directionality and significance of the effect estimates reported is encouraging. Determining a causal treatment effect of convalescent plasma will ultimately require an appropriately designed clinical trial. These findings from the CCC19 consortium strongly indicate that conducting such a convalescent trial that includes patients with cancer, especially those with hematologic cancer, is well motivated. It is important to recognize, given the rarity of some hematologic cancer and historical barriers to clinical trial entry, high-quality real-world data should continue to play a crucial role in fully understanding the benefits of COVID-19 treatments in patients with cancer. Alongside comprehensive COVID-19 registries, other technologies, such as machine learning and natural language processing, which have advanced the utilization and curation of unstructured data, such as physicians’ notes, should be leveraged to meet the urgent need for clinical evidence on COVID-19 therapies.

Back to top
Article Information

Corresponding Author: Gregory S. Calip, PharmD, MPH, PhD, Flatiron Health, Inc, 233 Spring St, New York, NY 10013 (gregory.calip@flatiron.com).

Published Online: June 17, 2021. doi:10.1001/jamaoncol.2021.1715

Conflict of Interest Disclosures: Dr Calip reported receiving grants from Pfizer outside the submitted work. Drs Calip, Miksad, and Sarkar report current employment with Flatiron Health, Inc, which is an independent subsidiary of the Roche group, and stock ownership in Roche. Drs Miksad and Sarkar also reported equity ownership in Flatiron Health, Inc (initiated before acquisition by Roche in 2018).

Thompson  MA, Henderson  JP, Shah  PK,  et al; for the COVID-19 and Cancer Consortium.  Association of convalescent plasma therapy with survival in patients with hematologic cancers and COVID-19.   JAMA Oncol. Published online June 17, 2021. doi:10.1001/jamaoncol.2021.1799Google Scholar
Luke  TC, Kilbane  EM, Jackson  JL, Hoffman  SL.  Meta-analysis: convalescent blood products for Spanish influenza pneumonia: a future H5N1 treatment?   Ann Intern Med. 2006;145(8):599-609. doi:10.7326/0003-4819-145-8-200610170-00139 PubMedGoogle ScholarCrossref
Joyner  MJ, Carter  RE, Senefeld  JW,  et al.  Convalescent plasma antibody levels and the risk of death from COVID-19.   N Engl J Med. 2021;384(11):1015-1027. doi:10.1056/NEJMoa2031893 PubMedGoogle ScholarCrossref
Chai  KL, Valk  SJ, Piechotta  V,  et al.  Convalescent plasma or hyperimmune immunoglobulin for people with COVID-19: a living systematic review.   Cochrane Database Syst Rev. 2020;10:CD013600. doi:10.1002/14651858.CD013600.pub3PubMedGoogle Scholar
Kaiser  P, Arnold  AM, Benkeser  D,  et al.  Comparing methods to address bias in observational data: statin use and cardiovascular events in a US cohort.   Int J Epidemiol. 2018;47(1):246-254. doi:10.1093/ije/dyx179 PubMedGoogle ScholarCrossref
Arbogast  PG, Ray  WA.  Performance of disease risk scores, propensity scores, and traditional multivariable outcome regression in the presence of multiple confounders.   Am J Epidemiol. 2011;174(5):613-620. doi:10.1093/aje/kwr143 PubMedGoogle ScholarCrossref
Safety, tolerability, and efficacy of anti-spike (S) SARS-CoV-2 monoclonal antibodies for hospitalized adult patients with COVID-19. ClinicalTrials.gov identifier: NCT04426695. Accessed April 23, 2021. https://clinicaltrials.gov/ct2/show/NCT04426695
Chubak  J, Boudreau  DM, Wirtz  HS, McKnight  B, Weiss  NS.  Threats to validity of nonrandomized studies of postdiagnosis exposures on cancer recurrence and survival.   J Natl Cancer Inst. 2013;105(19):1456-1462. doi:10.1093/jnci/djt211 PubMedGoogle ScholarCrossref
Suissa  S.  Immortal time bias in pharmaco-epidemiology.   Am J Epidemiol. 2008;167(4):492-499. doi:10.1093/aje/kwm324PubMedGoogle ScholarCrossref
Robins  JM, Hernán  MA, Brumback  B.  Marginal structural models and causal inference in epidemiology.   Epidemiology. 2000;11(5):550-560. doi:10.1097/00001648-200009000-00011 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words