Understanding Observational Treatment Comparisons in the Setting of Coronavirus Disease 2019 (COVID-19) | Infectious Diseases | JAMA Cardiology | JAMA Network
[Skip to Navigation]
May 5, 2020

Understanding Observational Treatment Comparisons in the Setting of Coronavirus Disease 2019 (COVID-19)

Author Affiliations
  • 1Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, North Carolina
  • 2Duke Clinical Research Institute, Durham, North Carolina
  • 3Assistant Editor, JAMA Cardiology
  • 4Northwestern University Feinberg School of Medicine, Chicago, Illinois
  • 5Editor, JAMA Cardiology
  • 6Deputy Editor, JAMA Cardiology
JAMA Cardiol. Published online May 5, 2020. doi:10.1001/jamacardio.2020.1874

With the emergence of coronavirus disease 2019 (COVID-19) as a global pandemic, individuals with preexisting chronic health conditions such as hypertension, diabetes, and cardiovascular disease have been identified as particularly vulnerable.1 These patients are also more likely than the general population to be taking angiotensin-converting enzyme inhibitors (ACEIs) and angiotensin II receptor blockers (ARBs). As the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus that causes COVID-19 gains entry into cells via binding to the angiotensin-converting enzyme 2 (ACE2) receptor,2,3 concerns have been raised that these therapies might facilitate the transmission of the virus or affect outcomes adversely.4-6 Given that ACEI and ARB therapies are known to provide benefit for the underlying conditions treated, stopping ACEI/ARB therapy carries risks. Moreover, local inactivation of the renin-angiotensin-aldosterone system may have protective effects against the development and progression of acute lung failure.3 In the absence of clinical evidence of benefit or risk of ACEIs/ARBs, current societal statements recommend against discontinuing these drugs other than for standard clinical indications.7 Robust clinical data are needed to clarify the effect of ACEIs/ARBs on SARS-CoV-2 infection.

Mehta et al8 address this question in a cohort of 18 472 patients who were tested for COVID-19 between March 8 and April 20, 2020, at 2 centers in the Cleveland Clinic Health System. Among them, 2285 (12.4%) were either taking an ACEI or ARB. After overlap weighting based on a propensity score,9 no association was observed between ACEI/ARB status and testing positive (odds ratio [OR], 1.09; 95% CI, 0.87-1.37). These results, in a contemporary sample, support the current recommendations of professional societies to continue ACEI or ARB therapy7 because there was no signal that these agents increase susceptibility to contracting COVID-19. However, secondary analyses conducted in a smaller sample of 1735 individuals who tested positive for COVID-19 showed a significant association between ACEI/ARB treatment and hospitalization (OR, 1.93; 95% CI, 1.38-2.71) and need for care in an intensive care unit (ICU) (OR, 1.64; 95% CI, 1.07 - 2.51), suggesting greater severity of disease in those who tested positive. Despite statistical significance, these secondary results need to be interpreted with caution. To better understand the strengths and weaknesses of each analysis, we review them with respect to key considerations in comparative effectiveness research, including confounding, generalizability, selection bias, treatment misclassification, and precision (Table).

Table.  Key Considerations in Comparative Effectiveness Research Applied to Study of Mehta et al8
Key Considerations in Comparative Effectiveness Research Applied to Study of Mehta et al

As in many observational comparisons, which lack randomization to balance out patient characteristics across treatment groups, patients taking ACEIs/ARBs in the study by Mehta et al8 had much higher prevalence of coronary artery disease, heart failure and cardiovascular risk factors, as well as chronic obstructive lung disease. To address this problem of confounding, the authors used overlap weighting based on a propensity score that included age, sex, obesity, diabetes, coronary artery disease, chronic lung disease, hypertension, and heart failure. After adjustment, there were no differences in the measured comorbidities, on average, between groups taking ACEIs/ARBs and groups not taking ACEIs/ARBs. To be successful, this analysis should include all important confounders. Confounders are variables associated with both ACEI/ARB treatment and outcome, ie, a positive COVID-19 test result. Factors associated with ACEI/ARB treatment assignment include cardiovascular disease or risk factors, chronic kidney disease, and other comorbidities. Factors associated with becoming infected with COVID-19 are less known. All people are susceptible to novel viruses, and transmission is strongly related to social interaction. Social distancing decreases transmission rates, particularly in vulnerable populations that are encouraged to self-isolate. Therefore, factors that identify vulnerable populations10 may affect COVID-19 transmission indirectly through behavior. According to the US Centers for Disease Control and Prevention, vulnerable groups include older age (>65 years), serious heart conditions, chronic kidney disease, and other serious health conditions.10 Thus, the primary analysis of the study by Mehta et al8 is appropriately adjusted for important confounders, with the possible exception of chronic kidney disease (a variable related to ACEI/ARB treatment and high risk in the setting of COVID-19).

In the secondary analysis, the confounders are variables associated with both ACEI/ARB treatment and outcome, ie, hospitalization and ICU admission. As before, cardiovascular disease or risk factors and other comorbidities may be associated with receiving ACEIs/ARBs. In addition, these factors may directly affect the outcomes of hospitalization and ICU admission with older, frail, and sick patients having worse outcomes. An indirect effect is also plausible, in which the presence of cardiovascular disease and other comorbidities lowers the threshold on the part of referring clinicians to hospitalize and move to the ICU those individuals considered to be at higher risk than the general population. The secondary analysis of Mehta et al8 was adjusted for age and cardiovascular disease and risk factors, but residual and/or unmeasured confounding is likely, and decisions of clinicians to hospitalize selected patients cannot be accounted for.

Until testing for SARS-CoV-2 becomes widespread, studies such as the one by Mehta et al8 are inherently limited to a select group of individuals who are tested. This selection process affects generalizability and can induce bias in treatment comparisons. Generalizability refers to how broadly the conclusions can be applied, that is, to what target population?11 The protocol for testing in the study by Mehta et al8 prioritized older patients and those with cardiovascular disease or risk factors, end-stage kidney disease, and the other conditions identified as high risk by the Centers for Disease Control and Prevention.10 While not generalizable to all patients in the US, the results of this study are applicable to a relatively high-risk target population, for whom any incremental risk owing to ACEI/ARB would be important to identify. The secondary analysis is further conditioned on testing positive for COVID-19.

Selection of individuals based on testing can also create bias in the comparison of treatments. In the primary analysis, selection bias will occur if there are factors associated with ACEI/ARB treatment that also make testing more likely (selection). This is similar to confounding, except that the bias arises owing to common causes of ACEI/ARB and testing, rather than common causes of ACEI/ARB and outcome. Among the reasons for testing (listed above), cardiovascular disease or risk factors and kidney disease are most likely to be associated with ACEI/ARB use. Selection bias owing to these factors is addressed by propensity score overlap weighting. Importantly, COVID-19 symptoms are also likely to induce testing. These should not be included in adjustment because they are part of the outcome (having COVID-19). In the secondary analysis, selection bias may occur if there are factors associated with ACEI/ARB treatment and either phase of selection: (1) testing and (2) positive diagnosis. Again, the list of comorbidities is the same as above and they are adjusted for in the propensity score (except for kidney disease). In general, until there is widespread and generalized testing in unselected populations, studies conducted in patients tested for COVID-19 should adjust for patient comorbidities that affect testing in addition to traditional confounders.

Treatment status can be defined in various ways, particularly as patients switch over time and in response to COVID-19. Treatment misclassification occurs when the available treatment status is inaccurate for an alternative, superior definition. Mehta et al8 define treatment based on whether ACEIs/ARBs were recorded in the electronic medical record at the time of testing for SARS-CoV-2. One advantage to this approach is that it may reflect recent use, during the period of potential viral exposure. For the primary analysis, this corresponds closely to the proposed causal mechanism, associated with upregulation of ACE2. However, the secondary analysis evaluates outcomes after a positive test result. For this analysis, it is unclear whether patients continue to take ACEIs/ARBs after being tested, particularly after hospitalization. Generally, studies of treatment during COVID-19 should carefully consider whether treatment changes are relevant to the scientific hypothesis.

The primary analysis of Mehta et al8 achieves good precision (ie, reasonably narrow confidence intervals) owing to the large sample size and the use of propensity score overlap weighting. Overlap weighting has been shown to maximize precision among a large class of propensity score methods.9,12 Precision is important to this analysis because the conclusion of no association is best justified by narrow confidence intervals around an odds ratio of 1.0. The secondary analysis has lower precision (ie, wider confidence intervals) owing to a smaller sample size.

Mehta et al8 provide important clinical data to support current treatment recommendations regarding use in ACEIs/ARBs in the current COVID-19 pandemic. Despite the limitations of observational data and the unique challenges of conducting studies in the setting of COVID-19, the primary analysis is consistent with good observational research. Future research is needed to replicate these findings as testing becomes more widespread and/or additionally adjust for factors related to testing that were not available here. The authors’ interpretation of the secondary outcomes analysis is appropriately restrained. The observed associations are likely “real” but are likely explained by confounding and should not be inferred as causal.

Back to top
Article Information

Corresponding Author: Laine Elliott Thomas, PhD, Biostatistics and Bioinformatics, Duke Clinical Research Institute, 200 Morris, Durham, NC 27701 (laine.thomas@duke.edu).

Published Online: May 5, 2020. doi:10.1001/jamacardio.2020.1874

Conflict of Interest Disclosures: Dr Pencina reports past grants from Sanofi/Regeneron and Amgen to his institution, lecture fees from Merck, and previously serving on the advisory board of Boehringer Ingelheim outside the submitted work. No other disclosures were reported.

Wu  Z, McGoogan  JM.  Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention.   JAMA. Published online February 24, 2020. doi:10.1001/jama.2020.2648PubMedGoogle Scholar
Zheng  YY, Ma  YT, Zhang  JY, Xie  X.  COVID-19 and the cardiovascular system.   Nat Rev Cardiol. 2020;17(5):259-260. doi:10.1038/s41569-020-0360-5PubMedGoogle ScholarCrossref
Vaduganathan  M, Vardeny  O, Michel  T, McMurray  JJV, Pfeffer  MA, Solomon  SD.  Renin–angiotensin–aldosterone system inhibitors in patients with COVID-19.   N Engl J Med. 2020;382(17):1653-1659. doi:10.1056/NEJMsr2005760PubMedGoogle ScholarCrossref
Sommerstein  R, Gräni  C.  Re: preventing a COVID-19 pandemic: ACE inhibitors as a potential risk factor for fatal COVID-19.   BMJ. Published online February 28, 2020. doi:10.1136/bmj.m810Google Scholar
Fang  L, Karakiulakis  G, Roth  M.  Are patients with hypertension and diabetes mellitus at increased risk for COVID-19 infection?   Lancet Respir Med. 2020;8(4):e21. doi:10.1016/S2213-2600(20)30116-8PubMedGoogle Scholar
Aronson  JK, Ferner  RE.  Drugs and the renin-angiotensin system in covid-19.   BMJ. 2020;369:m1313. doi:10.1136/bmj.m1313PubMedGoogle ScholarCrossref
American Heart Association. HFSA/ACC/AHA statement addresses concerns re: using RAAS antagonists in COVID-19. Accessed April 27, 2020. https://professional.heart.org/professional/ScienceNews/UCM_505836_HFSAACCAHA-statement-addresses-concerns-re-using-RAAS-antagonists-in-COVID-19.jsp
Mehta  N, Kalra  A, Nowacki  AS,  et al.  Association of use of angiotensin-converting enzyme inhibitors and angiotensin II receptor blockers with testing positive for coronavirus disease 2019 (COVID-19).   JAMA Cardiol. Published online May 5, 2020. doi:10.1001/jamacardio.2020.1855Google Scholar
Li  F, Morgan  KL, Zaslavsky  AM.  Balancing covariates via propensity score weighting.   J Am Stat Assoc. 2018;113(521):390-400. doi:10.1080/01621459.2016.1260466PubMedGoogle ScholarCrossref
Centers for Disease Control and Prevention. Coronavirus disease 2019 (COVID-19): people who are higher risk for severe illness. Updated April 15, 2020. Accessed April 27, 2020. https://www.cdc.gov/coronavirus/2019-ncov/need-extra-precautions/people-at-higher-risk.html
Thomas  L, Li  F, Pencina  M.  Using propensity score methods to create target populations in observational clinical research.   JAMA. Published online January 10, 2020. doi:10.1001/jama.2019.21558PubMedGoogle Scholar
Li  F, Thomas  LE, Li  F.  Addressing extreme propensity scores via the overlap weights.   Am J Epidemiol. 2019;188(1):250-257.PubMedGoogle Scholar