[Skip to Content]
[Skip to Content Landing]
Figure.
Flowchart of Patients in the Study
Flowchart of Patients in the Study

Cohort criteria included receiving a prescription for ramipril or telmisartan between January 2003 (start of available data) and September 2009 (prior to the Food and Drug Administration approval of the supplemental indication). ACE-I indicates angiotensin converting enzyme inhibitor; ARB, angiotensin receptor blocker; CAD, coronary artery disease; CHF, congestive heart failure; CVD, cerebrovascular disease; MI, myocardial infarction; PAD, peripheral arterial disease; PCI, percutaneous coronary intervention; T2DM, type 2 diabetes mellitus; and TIA, transient ischemic attack.

Table 1.  
Baseline Patient Characteristics Before Receiving Telmisartan or Ramiprila,b
Baseline Patient Characteristics Before Receiving Telmisartan or Ramiprila,b
Table 2.  
Incidence of the Composite End Point, Its Components, and the Risk of Angioedema
Incidence of the Composite End Point, Its Components, and the Risk of Angioedema
Table 3.  
Results From Secondary Analyses That Expanded the Population by Including Patients Who Had Used Angiotensin-Converting Enzyme Inhibitors or Angiotensin Receptor Blockers in the Past 180 Days
Results From Secondary Analyses That Expanded the Population by Including Patients Who Had Used Angiotensin-Converting Enzyme Inhibitors or Angiotensin Receptor Blockers in the Past 180 Days
1.
Kesselheim  AS, Avorn  J.  New “21st century cures” legislation.  JAMA. 2017;317(6):581-582.PubMedGoogle ScholarCrossref
2.
Sherman  RE, Anderson  SA, Dal Pan  GJ,  et al.  Real-world evidence—what is it and what can it tell us?  N Engl J Med. 2016;375(23):2293-2297.PubMedGoogle ScholarCrossref
3.
Schneeweiss  S.  Developments in post-marketing comparative effectiveness research.  Clin Pharmacol Ther. 2007;82(2):143-156.PubMedGoogle ScholarCrossref
4.
Fralick  M, Kesselheim  AS.  Periodic benefit-risk evaluation reports have substantial promise to guide patient care and should be made publicly available.  Pharmacoepidemiol Drug Saf. 2017;26(5):597-599.PubMedGoogle ScholarCrossref
5.
Ray  WA.  Evaluating medication effects outside of clinical trials: new-user designs.  Am J Epidemiol. 2003;158(9):915-920.PubMedGoogle ScholarCrossref
6.
Schneeweiss  S.  A basic study design for expedited safety signal evaluation based on electronic healthcare data.  Pharmacoepidemiol Drug Saf. 2010;19(8):858-868.PubMedGoogle ScholarCrossref
7.
Goodman  SN, Schneeweiss  S, Baiocchi  M.  Using design thinking to differentiate useful from misleading evidence in observational research.  JAMA. 2017;317(7):705-707.PubMedGoogle ScholarCrossref
8.
Suissa  S.  Immortal time bias in pharmaco-epidemiology.  Am J Epidemiol. 2008;167(4):492-499.PubMedGoogle ScholarCrossref
9.
Wang  B, Kesselheim  AS.  Characteristics of efficacy evidence supporting approval of supplemental indications for prescription drugs in United States, 2005-14: systematic review.  BMJ. 2015;351:h4679.PubMedGoogle ScholarCrossref
10.
Berndt  ER, Cockburn  IM, Grépin  KA.  The impact of incremental innovation in biopharmaceuticals: drug utilisation in original and supplemental indications.  Pharmacoeconomics. 2006;24(suppl 2):69-86.PubMedGoogle ScholarCrossref
11.
Yusuf  S, Teo  KK, Pogue  J,  et al; ONTARGET Investigators.  Telmisartan, ramipril, or both in patients at high risk for vascular events.  N Engl J Med. 2008;358(15):1547-1559.PubMedGoogle ScholarCrossref
12.
McCormick  N, Lacaille  D, Bhole  V, Avina-Zubieta  JA.  Validity of myocardial infarction diagnoses in administrative databases: a systematic review.  PLoS One. 2014;9(3):e92286.PubMedGoogle ScholarCrossref
13.
McCormick  N, Bhole  V, Lacaille  D, Avina-Zubieta  JA.  Validity of diagnostic codes for acute stroke in administrative databases: a systematic review.  PLoS One. 2015;10(8):e0135834.PubMedGoogle ScholarCrossref
14.
McCormick  N, Lacaille  D, Bhole  V, Avina-Zubieta  JA.  Validity of heart failure diagnoses in administrative databases: a systematic review and meta-analysis.  PLoS One. 2014;9(8):e104519.PubMedGoogle ScholarCrossref
15.
Schneeweiss  S, Avorn  J.  A review of uses of health care utilization databases for epidemiologic research on therapeutics.  J Clin Epidemiol. 2005;58(4):323-337.PubMedGoogle ScholarCrossref
16.
Austin  PC.  Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples.  Stat Med. 2009;28(25):3083-3107.PubMedGoogle ScholarCrossref
17.
Wang  SV, Verpillat  P, Rassen  JA, Patrick  A, Garry  EM, Bartels  DB.  Transparency and reproducibility of observational cohort studies using large healthcare databases.  Clin Pharmacol Ther. 2016;99(3):325-332.PubMedGoogle ScholarCrossref
18.
Fralick  M, Schneeweiss  S, Patorno  E.  Risk of diabetic ketoacidosis after initiation of an sglt2 inhibitor.  N Engl J Med. 2017;376(23):2300-2302.PubMedGoogle ScholarCrossref
19.
Kim  SC, Solomon  DH, Rogers  JR,  et al.  Cardiovascular safety of tocilizumab versus tumor necrosis factor inhibitors in patients with rheumatoid arthritis: a multi-database cohort study.  Arthritis Rheumatol. 2017;69(6):1154-1164.PubMedGoogle ScholarCrossref
20.
Temple  R, Ellenberg  SS.  Placebo-controlled trials and active-control trials in the evaluation of new treatments, part 1: ethical and scientific issues.  Ann Intern Med. 2000;133(6):455-463.PubMedGoogle ScholarCrossref
21.
Institute of Medicine of the National Academies. Ethical and scientific issues in studying the safety of approved drugs. The national academy of press website. May 2012. https://www.nap.edu/catalog/13219/ethical-and-scientific-issues-in-studying-the-safety-of-approved-drugs. Accessed 7 Jun 2017.
22.
Franklin  JM, Schneeweiss  S.  When and how can real world data analyses substitute for randomized controlled trials?  [published online August 24, 2017].  Clin Pharmacol Ther. 2017;24(August):•••. doi:10.1002/cpt.857PubMedGoogle Scholar
23.
Schneeweiss  S, Seeger  JD, Landon  J, Walker  AM.  Aprotinin during coronary-artery bypass grafting and risk of death.  N Engl J Med. 2008;358(8):771-783.PubMedGoogle ScholarCrossref
24.
Choi  N-K, Solomon  DH, Tsacogianis  TN, Landon  JE, Song  HJ, Kim  SC.  Comparative safety and effectiveness of denosumab versus zoledronic acid in patients with osteoporosis: a cohort study.  J Bone Miner Res. 2017;32(3):611-617.PubMedGoogle ScholarCrossref
25.
Patorno  E, Goldfine  A, Schneeweiss  S, Glynn  R, Liu  J, Kim  S. Safety of canagliflozin vs. other antidiabetic agents in routine care. Paper presented at: 77th Scientific Sessions of the American Diabetes Association; June 9, 2017; San Diego, California.
26.
Graham  DJ, Reichman  ME, Wernecke  M,  et al.  Stroke, bleeding, and mortality risks in elderly medicare beneficiaries treated with dabigatran or rivaroxaban for nonvalvular atrial fibrillation.  JAMA Intern Med. 2016;176(11):1662-1671.PubMedGoogle ScholarCrossref
27.
Gagne  JJ, Choudhry  NK, Kesselheim  AS,  et al.  Comparative effectiveness of generic and brand-name statins on patient outcomes: a cohort study.  Ann Intern Med. 2014;161(6):400-407.PubMedGoogle ScholarCrossref
28.
Wang  PS, Schneeweiss  S, Avorn  J,  et al.  Risk of death in elderly users of conventional vs. atypical antipsychotic medications.  N Engl J Med. 2005;353(22):2335-2341.PubMedGoogle ScholarCrossref
29.
Ray  WA, Chung  CP, Murray  KT, Hall  K, Stein  CM.  Atypical antipsychotic drugs and the risk of sudden cardiac death.  N Engl J Med. 2009;360(3):225-235.PubMedGoogle ScholarCrossref
30.
Finucane  FF, Madans  JH, Bush  TL, Wolf  PH, Kleinman  JC.  Decreased risk of stroke among postmenopausal hormone users: results from a national cohort.  Arch Intern Med. 1993;153(1):73-79.PubMedGoogle ScholarCrossref
31.
Blais  L, Desgagné  A, LeLorier  J.  3-Hydroxy-3-methylglutaryl coenzyme A reductase inhibitors and the risk of cancer: a nested case-control study.  Arch Intern Med. 2000;160(15):2363-2368.PubMedGoogle ScholarCrossref
32.
Wassertheil-Smoller  S, Hendrix  SL, Limacher  M,  et al; WHI Investigators.  Effect of estrogen plus progestin on stroke in postmenopausal women: the Women’s Health Initiative: a randomized trial.  JAMA. 2003;289(20):2673-2684.PubMedGoogle ScholarCrossref
33.
Dale  KM, Coleman  CI, Henyan  NN, Kluger  J, White  CM.  Statins and cancer risk: a meta-analysis.  JAMA. 2006;295(1):74-80.PubMedGoogle ScholarCrossref
34.
Ioannidis  JP, Haidich  AB, Pappa  M,  et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies.  JAMA. 2001;286(7):821-830.PubMedGoogle ScholarCrossref
35.
Gruber  S, Chakravarty  A, Heckbert  SR,  et al.  Design and analysis choices for safety surveillance evaluations need to be tuned to the specifics of the hypothesized drug-outcome association.  Pharmacoepidemiol Drug Saf. 2016;25(9):973-981.PubMedGoogle ScholarCrossref
36.
Moride  Y, Abenhaim  L.  Evidence of the depletion of susceptibles effect in non-experimental pharmacoepidemiologic research.  J Clin Epidemiol. 1994;47(7):731-737.PubMedGoogle ScholarCrossref
37.
Schneeweiss  S, Patrick  AR, Stürmer  T,  et al.  Increasing levels of restriction in pharmacoepidemiologic database studies of elderly and comparison with randomized trial results.  Med Care. 2007;45(10)(suppl 2):S131-S142.PubMedGoogle ScholarCrossref
38.
Setoguchi  S, Warner Stevenson  L, Stewart  GC,  et al.  Influence of healthy candidate bias in assessing clinical effectiveness for implantable cardioverter-defibrillators: cohort study of older patients with heart failure.  BMJ. 2014;348:g2866.PubMedGoogle ScholarCrossref
39.
D’Agostino  RB  Sr, Massaro  JM, Sullivan  LM.  Non-inferiority trials: design concepts and issues—the encounters of academic consultants in statistics.  Stat Med. 2003;22(2):169-186.PubMedGoogle ScholarCrossref
40.
Snapinn  SM.  Noninferiority trials.  Curr Control Trials Cardiovasc Med. 2000;1(1):19-21.PubMedGoogle ScholarCrossref
41.
Agoritsas  T, Merglen  A, Shah  ND, O’Donnell  M, Guyatt  GH,  et al.  Adjusted analyses in studies addressing therapy and harm users’ guides to the medical literature clinical scenario.  JAMA. 2017;317(7):748-759.PubMedGoogle ScholarCrossref
42.
Fonarow  GC.  Randomization—there is no substitute.  JAMA Cardiol. 2016;1(6):633-635.PubMedGoogle ScholarCrossref
43.
Freemantle  N, Marston  L, Walters  K, Wood  J, Reynolds  MR, Petersen  I.  Making inferences on treatment effects from real world data: propensity scores, confounding by indication, and other perils for the unwary in observational research.  BMJ. 2013;347:f6409.PubMedGoogle ScholarCrossref
44.
Ioannidis  JP.  Why most published research findings are false.  PLoS Med. 2005;2(8):e124.PubMedGoogle ScholarCrossref
45.
Anglemyer  A, Horvath  HT, Bero  L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. In: Bero L, ed. Cochrane Database of Systematic Reviews. Chichester, England: John Wiley & Sons; 2014:MR000034.
46.
Concato  J, Shah  N, Horwitz  RI.  Randomized, controlled trials, observational studies, and the hierarchy of research designs.  N Engl J Med. 2000;342(25):1887-1892.PubMedGoogle ScholarCrossref
47.
Hemkens  LG, Contopoulos-Ioannidis  DG, Ioannidis  JPA.  Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey.  BMJ. 2016;352:i493.PubMedGoogle ScholarCrossref
48.
Dahabreh  IJ, Sheldrick  RC, Paulus  JK,  et al.  Do observational studies using propensity score methods agree with randomized trials? a systematic comparison of studies on acute coronary syndromes.  Eur Heart J. 2012;33(15):1893-1901.PubMedGoogle ScholarCrossref
Views 4,975
Citations 0
Original Investigation
November 20, 2017

Use of Health Care Databases to Support Supplemental Indications of Approved Medications

Author Affiliations
  • 1Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts
  • 2Program on Regulation, Therapeutics, and Law (PORTAL), Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts
  • 3Eliot Phillipson Clinician-Scientist Training Program, University of Toronto, Toronto, Ontario, Canada
JAMA Intern Med. Published online November 20, 2017. doi:10.1001/jamainternmed.2017.3919
Key Points

Question  Can health care databases be used to confirm a supplemental indication that has been demonstrated in a randomized clinical trial for an approved medication?

Findings  This cohort study replicated the results of a randomized clinical trial that established the supplemental indication for telmisartan by using data from a US health care database (insurance claims data) available at the time that the supplemental indication was approved. Similar to the randomized clinical trial, our study revealed a decreased risk of angioedema with telmisartan compared with ramipril.

Meaning  In certain clinical scenarios, database studies may support supplemental effectiveness applications for already approved medications.

Abstract

Importance  Manufacturers of US Food and Drug Administration–approved prescription drugs often apply for additional indications based on randomized clinical trials. Real-world database analyses on a medication’s use and outcomes in routine settings of care might help to inform decision making regarding such supplemental indications.

Objective  To examine whether longitudinal data from a health care database can support the results of a randomized clinical trial that led to a supplemental indication for telmisartan.

Design, Setting, and Participants  This cohort study of patients newly prescribed telmisartan or ramipril used insurance claims data from a nationwide health care database from January 1, 2003, through September 30, 2009, to compare patient outcomes. This study replicated the inclusion and exclusion criteria used in the Ongoing Telmisartan Alone and in Combination with Ramipril Global End-point Trial (ONTARGET) and used propensity score matching to balance 74 patient characteristics. Data analysis was performed from February 15, 2017, to May 24, 2017.

Exposures  Telmisartan use vs ramipril use.

Main Outcomes and Measures  The primary outcome was a composite of myocardial infarction, stroke, or hospitalization for congestive heart failure.

Results  Of the 640 951 patients included in the study, 48 053 were newly prescribed ramipril (mean [SD] age, 68.29 [9.52] years; 31 940 male [66.5%]) and 4665 were newly prescribed telmisartan (mean [SD] age, 69.43 [9.60] years; 2413 male [51.7%]). After propensity score matching, a total of 4665 patients were newly prescribed telmisartan (mean [SD] age, 69.43 [9.60] years; 2413 [51.7%]), and 4665 patients were newly prescribed ramipril (mean [SD] age, 69.36 [9.67] years; 2343 male [50.2%]). As seen in ONTARGET, the composite risk of stroke, myocardial infarction, and hospitalization for congestive heart failure was similar for the 2 medications (hazard ratio, 1.0; 95% CI, 0.9-1.1). In addition, the study found that telmisartan was associated with a substantially decreased risk of angioedema (hazard ratio, 0.1; 95% CI, 0.03-0.56) compared with ramipril.

Conclusions and Relevance  Real-world data analyses of patients receiving routine care provided findings similar to those found in the randomized clinical trial that established telmisartan’s supplemental indication. In certain situations, database studies may support supplemental applications for effectiveness for already approved medications.

Introduction

In December 2016, the 21st Century Cures Act was signed into law in the United States.1 It contained a provision intended to promote real-world data studies of medication use and outcomes in routine clinical settings in US Food and Drug Administration (FDA) authorization of additional indications for already approved prescription drugs.1 Such data, with or without randomization, are drawn from health care use data, insurance claims, registry studies, and/or electronic health record systems in typical clinical settings of care.24 Although the FDA has long used such data to clarify the safety of medications, the data can seldom establish a drug’s effectiveness. Well-designed randomized clinical trials are the criterion standard for assessing whether a drug is efficacious because random treatment assignment and a controlled research environment can more readily support causal inferences.

In recent years, new methodologic approaches have improved the validity and reproducibility of nonrandomized data, including new-user designs,5 active comparators, propensity score (PS) matching, and controlling for disease risk scores.6,7 Other important aspects include assessing covariates before cohort entry (to avoid adjusting for intermediate variables) and defining cohort entry as the time when the patient first receives the exposure of interest (to decrease the possibility of immortal time bias).68

Can such analytic techniques confirm supplemental indications for already approved drugs? Approximately half of all drugs approved in the United States are later approved for supplemental indications, modifications to the initial indication, or expanded populations.9,10 Supplemental indications are typically identified on the basis of prospective clinical trials. To determine whether real-world data analyses can confirm a supplemental indication, we identified a supplemental approval amenable to study and applied the same inclusion and exclusion criteria and outcomes measurements that were used in the pivotal randomized clinical trial.

Methods

Our cohort study was conducted in commercially insured patients using the MarketScan health care database provided by Truven (January 1, 2003, through September 30, 2009). This nationwide database captures anonymized longitudinal, individual-level data on health care use, patient demographics, inpatient and outpatient diagnostic and procedural codes, and pharmacy dispensing of prescription drugs for more than 60 million commercially insured people in the United States. The study was approved by the institutional review board at Brigham and Women’s Hospital, including a waiver for informed consent, and a valid data licensing agreement was in place. All data were anonymized and deidentified.

Data Sources

To identify an experimental setting, we reviewed all supplemental applications to the FDA from 2005 to 2014 and their accompanying clinical trials.9 The supplemental indications were classified into 3 mutually exclusive categories: new indication (n = 138), modification (n = 86), and expansion (n = 66) (eAppendix in the Supplement).9 Of the 138 new indications, 108 (78.3%) of the pivotal clinical trials had a primary outcome that was not identifiable in US longitudinal health care databases (eg, pathology results, change in clinical scores, and radiologic tumor response), 12 (8.7%) did not have an active comparator, 4 (2.9%) were based on in-hospital medication administration (eg, postoperative nausea medication, anesthetic medications), and 14 (10.1%) were potentially replicable with the claims data available to us. Of the 14, we selected telmisartan a priori and did not analyze data for the other 13 (eAppendix in the Supplement).

The angiotensin receptor blocker (ARB) telmisartan (Micardis) was approved as an antihypertensive in 1998. In October 2009, it was approved supplementarily for cardiovascular risk reduction in patients 55 years or older who are unable to take angiotensin-converting enzyme inhibitors (ACE-Is) and have a high risk of major cardiovascular events. Telmisartan was an optimal case study for 3 reasons. First, the primary outcome in the pivotal supplemental indication trial could be accurately identified in health care use data. Second, the randomized clinical trial used an active comparator, the ACE-I ramipril (Altace), which would minimize confounding in cohort studies.3,7 Third, the inclusion criteria, exclusion criteria, and baseline patient characteristics were identifiable in claims data.

The trial that identified the supplemental indication for telmisartan for cardiovascular risk reduction, Ongoing Telmisartan Alone and in Combination with Ramipril Global End-point Trial (ONTARGET), was published in April 2008.11 ONTARGET’s primary objectives were to determine whether telmisartan was at least as effective as ramipril at reducing cardiovascular risk and to assess whether the combination of telmisartan and ramipril was more effective than ramipril alone. The trial was conducted across 733 centers in 40 countries between 2001 and 2008.11

Study Cohort

Potentially eligible patients must have had at least 6 months of continuous enrollment in a participating health plan before the date of cohort entry. Our inclusion and exclusion criteria mirrored those of ONTARGET.11 We included patients 55 years or older who filled a new prescription for telmisartan or ramipril (no fills for either drug or any other ACE-I or ARB during the prior 180 days). Cohort entry date was the first day of a prescription fill. As in ONTARGET, we included patients with a diagnosis of coronary artery disease, peripheral artery disease, cerebrovascular disease, or diabetes mellitus during the 180 days before cohort entry.

As in ONTARGET, we excluded patients with a limited life expectancy (ie, living in a hospice, palliative care facility, or a nursing home and those with cancer), liver disease, syncope or a recent myocardial infarction (within 2 days of cohort entry), transient ischemic attack (within 7 days of cohort entry), percutaneous transluminal coronary angiography (within 30 days of cohort entry), or hospitalization for congestive heart failure during the 180 days before cohort entry. Other exclusion criteria used in ONTARGET were not applied because they were not readily identifiable (known allergy to study medication, unable to tolerate study medication, hemodynamically significant primary valvular or outflow tract obstruction, uncorrected volume or sodium depletion, planned cardiac procedure, blood pressure >160/100 mm Hg despite treatment, significant renal artery stenosis, and angina in the absence of multivessel coronary artery disease) or rare (hereditary fructose intolerance, complex congenital heart disease, primary hyperaldosteronism, and heart transplant). We also excluded patients who previously received any ACE-I or ARB.

Study End Point

Our primary outcome was a composite of myocardial infarction, stroke, or hospitalization for congestive heart failure using the primary discharge diagnosis code for an inpatient visit (see eTable 1 in the Supplement for International Classification of Diseases, Ninth Revision codes). These definitions have satisfactory measurement characteristics; the positive predictive value for myocardial infarction was 93% or higher; stroke, 81% or higher; and congestive heart failure, 87% or higher.1214 Cardiovascular deaths were included in the composite outcome if they occurred during a hospitalization for myocardial infarction, stroke, or heart failure but not outside the hospital.

Statistical Analysis

Our primary analysis compared the rates of the composite end point among patients initiating treatment with telmisartan vs ramipril. Data were censored for patients when they discontinued use of their initial medication, switched to the comparator medication, experienced a study outcome, disenrolled from their health plan, or died, or on September 30, 2009.15 To address confounding, we adjusted for 74 patient characteristics, including demographics, comorbid conditions, concurrent medications, and health care use measures, using PS methods (Table 1 and eTable 2 in the Supplement). To balance patient characteristics, we used 1:1 PS matching with a caliper of 0.05 and did not perform further variable selection. We compared standardized differences to evaluate the level of balance achieved in patient characteristics after PS matching16 and used unstratified Cox proportional hazards regression to compute hazard ratios (HRs) and 95% CIs. We then performed a predefined secondary analysis that carried forward the exposure to the first-used medication for 365 days.6

To assess the robustness of our results, we also sought to confirm the well-established increased risk of angioedema for ramipril, expecting that rates of angioedema would be lower for telmisartan, as also demonstrated in ONTARGET. To further assess robustness, we replicated all study end points using a larger cohort derived from less stringent exclusion criteria by creating a cohort that allowed for past ACE-I or ARB use other than telmisartan or ramipril in the preceding 180 days. All analyses were conducted using the Aetion platform and R, version 3.1.2.5 (R Foundation for Statistical Computing), which has been previously validated for a range of studies17,18 and for predicting clinical trial findings.19

Results

We identified 640 951 patients who filled a prescription for ramipril or telmisartan from January 1, 2003, through September 30, 2009, and had a sufficient baseline enrollment period of at least 180 days. After applying study inclusion and exclusion criteria, 52 739 patients were included (Figure), of whom 48 053 were newly prescribed ramipril (mean [SD] age, 68.29 [9.52] years; 31 940 male [66.5%]) and 4665 were newly prescribed telmisartan (mean [SD] age, 69.43 [9.60] years; 2413 male [51.7%]) (a total of 21 patients did not begin follow-up). Patients prescribed ramipril were more likely to be male and have cardiac disease, whereas patients prescribed telmisartan were more likely to have hypertension, kidney disease, and previous transient ischemic attack or stroke and be prescribed a calcium channel blocker (Table 1). After PS matching 4665 telmisartan users (mean [SD] age, 69.43 [9.60] years; 2413 [51.7%]) to 4665 ramipril users (mean [SD] age, 69.36 [9.67] years; 2343 male [50.2%]), these differences were well balanced with standardized differences less than 0.1 (Table 1). Most frequencies of baseline characteristics were consistent with ONTARGET (eg, similar age, rates of hypertension, coronary artery disease, diabetes, and stroke), whereas some were not (ie, lower rates of angina, lower rates of smoking, less documented antiplatelet use, and more women included in our study) (eTable 2 in the Supplement). In the unmatched cohort, mean follow-up time was 232 days (interquartile range, 113-454 days) for the ramipril group and 188 days (interquartile range, 108-427 days) for the telmisartan group. The most common reason for censoring was treatment discontinuation, in 32 135 ramipril users (66.9%) and 3483 telmisartan users (74.7%).

In ONTARGET, the relative risk of the composite outcome of death from cardiovascular causes, myocardial infarction, stroke, or hospitalization for congestive heart failure was 1.01 (95% CI, 0.94-1.09), indicating no significant difference between telmisartan and ramipril. In our study, the PS-matched relative risk of the composite of myocardial infarction, stroke, or hospitalization for congestive heart failure was almost identical (HR, 0.99; 95% CI, 0.85-1.14) (Table 2).

Sensitivity Analysis

A sensitivity analysis using the last exposure to the first-used medication for 365 days without considering treatment discontinuation found that the primary end point occurred in 402 ramipril users (86 events per 1000 patients) and 363 telmisartan users (78 events per 1000 patients). This resulted in no significant difference in risk after PS matching (HR, 0.90; 95% CI, 0.77-1.04).

Validation Against a Known Causal Association

Among PS-matched individuals, there were 18 angioedema events in new users of ramipril (3.1 events per 1000 person-years) and 2 events in new users of telmisartan (0.4 events per 1000 person-years). A decreased risk (HR, 0.13; 95% CI, 0.03-0.56) of angioedema with telmisartan was also observed in ONTARGET (HR, 0.40; P = .01).

Robustness of Findings

In the cohort with less stringent exclusion criteria to allow for past ACE-I or ARB use apart from ramipril or telmisartan, we identified 8656 PS-matched new users of telmisartan and 8656 PS-matched new users of ramipril. In this cohort, there was a similar PS-matched relative risk of the composite of myocardial infarction, stroke, or hospitalization for congestive heart failure (HR, 0.97; 95% CI, 0.88-1.08) (Table 3). A decreased PS-matched risk of angioedema with telmisartan compared with ramipril (HR, 0.35; 95% CI, 0.17-0.71) was also revealed.

Discussion

Among patients newly prescribed telmisartan and ramipril before the FDA’s decision to approve a supplemental indication for telmisartan, we found results that were almost identical to those of the randomized clinical trial that led to telmisartan’s supplemental indication. We further identified and quantified the known causal association between ramipril and angioedema. This finding suggests that our data and analysis plan were sufficiently valid to detect known causal associations first identified in a prospective trial.20

This study is one of the largest to analyze real-world data to mirror a large randomized clinical trial that had established the clinical basis for a supplemental indication for a medication. In contrast to ONTARGET, which took approximately 7 years to complete and cost tens of millions of dollars, our study took approximately 12 weeks to implement for less than a hundredth of the cost. The fact that our case study bolstered the conclusions of a trial designed to identify a supplemental indication for a marketed medication and was done relatively efficiently using available data sets, rigorous epidemiologic methods, and modern software platforms supports the concept of conducting similar database analyses as part of routine practice for manufacturers submitting applications for supplemental indications to the FDA.21

Results concordant with the pivotal clinical trial can provide regulators with greater confidence in approving the indication, whereas discordant results could warrant deeper reexamination of the clinical trial or nonrandomized data. When results are discordant with the pivotal trial, an in-depth analysis of the trial and the nonrandomized study will be necessary to identify reasons for this discordance. These reasons can include issues related to study design, statistical analysis, and patient population. Additional research will be necessary to help navigate this scenario.22 Eventually, the FDA can develop empirically based guidance on when database analyses are useful in this context and when they are less reliable as a confirmatory source.

Validity of Nonrandomized Real-world Data Analyses

There have been examples of real-world data providing results before the randomized clinical trial was completed2325 and nonrandomized real-world studies that changed prescribing practices for which there will likely never be randomized clinical trial findings.2629 A common signal of quality among these studies and our current study was the use of a new-user, active-comparator design. This approach compares 2 groups of patients who newly start taking a medication and avoids comparing 2 groups with intrinsically discrepant risk profiles as would be found using a nonuser comparator or comparing new users with ongoing users. The new-user design with an active comparator allows a more homogeneous baseline population and was one of the main reasons why the observed baseline characteristics for our patients were similar even before matching. By design, approximately 80% of the 74 baseline characteristics were well balanced before PS matching, suggesting that unmeasured factors may be equally balanced. Similar results were observed in the recent new-user, active-comparator study by Graham et al26 that compared the safety and effectiveness of rivaroxaban with those of dabigatran.

By contrast, some nonrandomized real-world studies30,31 found results that differed from those in subsequent randomized clinical trials.3234 This difference can occur for many reasons, including incorrect study design implementation, reverse causation,35 immortal time bias,8 depletion of susceptibles,36 failure to identify important unmeasured confounding factors, or the inclusion of a different study population than was used in the clinical trial. In particular, comparators that use patients defined as those who did not fill a prescription (nonusers) may introduce treatment selection bias that may not be controllable with any statistical method.37,38

Studies such as ours require that inclusion and exclusion criteria and end points be adequately defined in a randomized clinical trial report and subsequently identifiable in the health care data set being studied. Many trials include study end points that are not recorded in claims data or electronic health care records (eg, rating scales used in trials of psychiatric medications) or not easily identifiable (eg, progression-free survival used in oncology trials) without requiring challenging natural language processing of free-text information. It would also be difficult to replicate results from randomized clinical trials that include different treatment modalities with substantially different risk-benefit profiles (eg, implantable cardioverter defibrillators compared with medical therapy) because of fundamental differences in risk profiles between the 2 populations.7,38

Pharmacoepidemiology analysis of data from nonrandomized, real-world health care databases can be used to support supplemental indications established in prospective randomized clinical trials of marketed medications. This is powerful because they represent outcomes in settings of typical care, rather than the highly controlled research environments of RCTs, and can be accomplished quickly and inexpensively. The analyses can also include subgroups of patients who are underrepresented in clinical trials, including elderly individuals, patients with many comorbidities, pregnant women, and other at-risk groups. In our study, for example, 50% of patients were women compared with approximately 26% in ONTARGET. Finally, such studies can evaluate a larger population of patients and can assess end points that trials are often underpowered to detect, such as rare adverse events.

Limitations

Our observed null finding might reflect limitations within our data set (eg, lack of out-of-hospital death data), duration of follow-up, or study design rather than a true observation. It is well established that noninferiority can appear to be present because of inadequate rigor or scale in any study, whether a randomized clinical trial or an observational analysis.39,40 However, this does not explain the increased risk of angioedema that we observed with ramipril but not telmisartan. Some authors41,42 have questioned the value of PS matching over traditional risk-adjusted regression analysis, neither of which guarantee full account for unmeasured confounding. However, our unadjusted primary, secondary, and sensitivity analyses did not change meaningfully after PS matching. Another limitation of our study was an inability to assess medication adherence beyond prescription filling, although this is generally seen as a valid measure of actual use.15

Conclusions

The FDA is currently considering how it will use nonrandomized, real-world data as part of supplemental indication applications.42,43 In the absence of large-scale empirical comparative analyses that identify the reasons for failure and success to replicate randomized controlled findings with real-world data analyses, we performed a case study that highlights some important considerations. Many context-specific questions about study design, confounding control, data quality, and outcome validity will need to be considered.4,6 Preregistering study designs and analysis plans and providing a publicly available summary of the results when available, similar to the current practice of randomized clinical trials, promotes ethical conduct of these studies.

Even well-designed analyses sometimes result in incorrect conclusions, and some randomized clinical trials may be inaccurate.44 Retrospective reviews of the literature34,4548 provide single summarizations of the differences between these 2 approaches but provide few insights on the validity of individual real-world data analyses. To establish a meaningful baseline, the FDA will need many sets of randomized clinical trials with prospectively designed, nonrandomized analyses to match the populations included in randomized clinical trials across a range of clinical questions, each investigated with a set of designs and methods following rigorous epidemiologic principles.

Regulators have a difficult task in providing specific rules for decision making in this maturing yet still developing and highly context-specific field. However, if done selectively and with principled methods, it might be feasible to use nonrandomized, real-world data to provide supportive evidence in establishing supplemental drug indications.

Back to top
Article Information

Corresponding Author: Michael Fralick, MD, The Program on Regulation, Therapeutics, and Law, Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, 1620 Tremont St, Ste 3030, Boston, MA 02120 (mif823@mail.harvard.edu).

Accepted for Publication: August 15, 2017.

Published Online: November 20, 2017. doi:10.1001/jamainternmed.2017.3919

Open Access: This article is published under the JN-OA license and is free to read on the day of publication.

Author Contributions: Dr Schneeweiss had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Fralick.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Fralick, Schneeweiss.

Obtained funding: Kesselheim, Avorn, Schneeweiss.

Administrative, technical, or material support: Fralick, Avorn.

Study supervision: Kesselheim, Avorn, Schneeweiss.

Conflict of Interest Disclosure: Dr Kesselheim reported receiving grants from the US Food and Drug Administration Office of Generic Drugs and Division of Health Communication (2013-2016) unrelated to the topic of this article. Dr Schneeweiss reported receiving support from grants from the US Food and Drug Administration and the Patient Centered Outcomes Research Institute; serving as a consultant to WHISCON, LLC, and Aetion Inc, a software manufacturer of which he also owns equity; and being a principal investigator of investigator-initiated grants to the Brigham and Women’s Hospital from Genentech, Bayer, and Boehringer Ingelheim (not directly related to the topic of this article). No other disclosures were reported.

Funding/Support: This study was funded by the Division of Pharmacoepidemiology and Pharmacoeconomics, Brigham and Women’s Hospital, and the Laura and John Arnold Foundation. Dr Fralick was funded by the University of Toronto Clinician Scientist Training Program, Toronto, Ontario, Canada. Dr. Kesselheim’s work is additionally supported by the Engelberg Foundation and the Harvard Program in Therapeutic Science.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Kesselheim  AS, Avorn  J.  New “21st century cures” legislation.  JAMA. 2017;317(6):581-582.PubMedGoogle ScholarCrossref
2.
Sherman  RE, Anderson  SA, Dal Pan  GJ,  et al.  Real-world evidence—what is it and what can it tell us?  N Engl J Med. 2016;375(23):2293-2297.PubMedGoogle ScholarCrossref
3.
Schneeweiss  S.  Developments in post-marketing comparative effectiveness research.  Clin Pharmacol Ther. 2007;82(2):143-156.PubMedGoogle ScholarCrossref
4.
Fralick  M, Kesselheim  AS.  Periodic benefit-risk evaluation reports have substantial promise to guide patient care and should be made publicly available.  Pharmacoepidemiol Drug Saf. 2017;26(5):597-599.PubMedGoogle ScholarCrossref
5.
Ray  WA.  Evaluating medication effects outside of clinical trials: new-user designs.  Am J Epidemiol. 2003;158(9):915-920.PubMedGoogle ScholarCrossref
6.
Schneeweiss  S.  A basic study design for expedited safety signal evaluation based on electronic healthcare data.  Pharmacoepidemiol Drug Saf. 2010;19(8):858-868.PubMedGoogle ScholarCrossref
7.
Goodman  SN, Schneeweiss  S, Baiocchi  M.  Using design thinking to differentiate useful from misleading evidence in observational research.  JAMA. 2017;317(7):705-707.PubMedGoogle ScholarCrossref
8.
Suissa  S.  Immortal time bias in pharmaco-epidemiology.  Am J Epidemiol. 2008;167(4):492-499.PubMedGoogle ScholarCrossref
9.
Wang  B, Kesselheim  AS.  Characteristics of efficacy evidence supporting approval of supplemental indications for prescription drugs in United States, 2005-14: systematic review.  BMJ. 2015;351:h4679.PubMedGoogle ScholarCrossref
10.
Berndt  ER, Cockburn  IM, Grépin  KA.  The impact of incremental innovation in biopharmaceuticals: drug utilisation in original and supplemental indications.  Pharmacoeconomics. 2006;24(suppl 2):69-86.PubMedGoogle ScholarCrossref
11.
Yusuf  S, Teo  KK, Pogue  J,  et al; ONTARGET Investigators.  Telmisartan, ramipril, or both in patients at high risk for vascular events.  N Engl J Med. 2008;358(15):1547-1559.PubMedGoogle ScholarCrossref
12.
McCormick  N, Lacaille  D, Bhole  V, Avina-Zubieta  JA.  Validity of myocardial infarction diagnoses in administrative databases: a systematic review.  PLoS One. 2014;9(3):e92286.PubMedGoogle ScholarCrossref
13.
McCormick  N, Bhole  V, Lacaille  D, Avina-Zubieta  JA.  Validity of diagnostic codes for acute stroke in administrative databases: a systematic review.  PLoS One. 2015;10(8):e0135834.PubMedGoogle ScholarCrossref
14.
McCormick  N, Lacaille  D, Bhole  V, Avina-Zubieta  JA.  Validity of heart failure diagnoses in administrative databases: a systematic review and meta-analysis.  PLoS One. 2014;9(8):e104519.PubMedGoogle ScholarCrossref
15.
Schneeweiss  S, Avorn  J.  A review of uses of health care utilization databases for epidemiologic research on therapeutics.  J Clin Epidemiol. 2005;58(4):323-337.PubMedGoogle ScholarCrossref
16.
Austin  PC.  Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples.  Stat Med. 2009;28(25):3083-3107.PubMedGoogle ScholarCrossref
17.
Wang  SV, Verpillat  P, Rassen  JA, Patrick  A, Garry  EM, Bartels  DB.  Transparency and reproducibility of observational cohort studies using large healthcare databases.  Clin Pharmacol Ther. 2016;99(3):325-332.PubMedGoogle ScholarCrossref
18.
Fralick  M, Schneeweiss  S, Patorno  E.  Risk of diabetic ketoacidosis after initiation of an sglt2 inhibitor.  N Engl J Med. 2017;376(23):2300-2302.PubMedGoogle ScholarCrossref
19.
Kim  SC, Solomon  DH, Rogers  JR,  et al.  Cardiovascular safety of tocilizumab versus tumor necrosis factor inhibitors in patients with rheumatoid arthritis: a multi-database cohort study.  Arthritis Rheumatol. 2017;69(6):1154-1164.PubMedGoogle ScholarCrossref
20.
Temple  R, Ellenberg  SS.  Placebo-controlled trials and active-control trials in the evaluation of new treatments, part 1: ethical and scientific issues.  Ann Intern Med. 2000;133(6):455-463.PubMedGoogle ScholarCrossref
21.
Institute of Medicine of the National Academies. Ethical and scientific issues in studying the safety of approved drugs. The national academy of press website. May 2012. https://www.nap.edu/catalog/13219/ethical-and-scientific-issues-in-studying-the-safety-of-approved-drugs. Accessed 7 Jun 2017.
22.
Franklin  JM, Schneeweiss  S.  When and how can real world data analyses substitute for randomized controlled trials?  [published online August 24, 2017].  Clin Pharmacol Ther. 2017;24(August):•••. doi:10.1002/cpt.857PubMedGoogle Scholar
23.
Schneeweiss  S, Seeger  JD, Landon  J, Walker  AM.  Aprotinin during coronary-artery bypass grafting and risk of death.  N Engl J Med. 2008;358(8):771-783.PubMedGoogle ScholarCrossref
24.
Choi  N-K, Solomon  DH, Tsacogianis  TN, Landon  JE, Song  HJ, Kim  SC.  Comparative safety and effectiveness of denosumab versus zoledronic acid in patients with osteoporosis: a cohort study.  J Bone Miner Res. 2017;32(3):611-617.PubMedGoogle ScholarCrossref
25.
Patorno  E, Goldfine  A, Schneeweiss  S, Glynn  R, Liu  J, Kim  S. Safety of canagliflozin vs. other antidiabetic agents in routine care. Paper presented at: 77th Scientific Sessions of the American Diabetes Association; June 9, 2017; San Diego, California.
26.
Graham  DJ, Reichman  ME, Wernecke  M,  et al.  Stroke, bleeding, and mortality risks in elderly medicare beneficiaries treated with dabigatran or rivaroxaban for nonvalvular atrial fibrillation.  JAMA Intern Med. 2016;176(11):1662-1671.PubMedGoogle ScholarCrossref
27.
Gagne  JJ, Choudhry  NK, Kesselheim  AS,  et al.  Comparative effectiveness of generic and brand-name statins on patient outcomes: a cohort study.  Ann Intern Med. 2014;161(6):400-407.PubMedGoogle ScholarCrossref
28.
Wang  PS, Schneeweiss  S, Avorn  J,  et al.  Risk of death in elderly users of conventional vs. atypical antipsychotic medications.  N Engl J Med. 2005;353(22):2335-2341.PubMedGoogle ScholarCrossref
29.
Ray  WA, Chung  CP, Murray  KT, Hall  K, Stein  CM.  Atypical antipsychotic drugs and the risk of sudden cardiac death.  N Engl J Med. 2009;360(3):225-235.PubMedGoogle ScholarCrossref
30.
Finucane  FF, Madans  JH, Bush  TL, Wolf  PH, Kleinman  JC.  Decreased risk of stroke among postmenopausal hormone users: results from a national cohort.  Arch Intern Med. 1993;153(1):73-79.PubMedGoogle ScholarCrossref
31.
Blais  L, Desgagné  A, LeLorier  J.  3-Hydroxy-3-methylglutaryl coenzyme A reductase inhibitors and the risk of cancer: a nested case-control study.  Arch Intern Med. 2000;160(15):2363-2368.PubMedGoogle ScholarCrossref
32.
Wassertheil-Smoller  S, Hendrix  SL, Limacher  M,  et al; WHI Investigators.  Effect of estrogen plus progestin on stroke in postmenopausal women: the Women’s Health Initiative: a randomized trial.  JAMA. 2003;289(20):2673-2684.PubMedGoogle ScholarCrossref
33.
Dale  KM, Coleman  CI, Henyan  NN, Kluger  J, White  CM.  Statins and cancer risk: a meta-analysis.  JAMA. 2006;295(1):74-80.PubMedGoogle ScholarCrossref
34.
Ioannidis  JP, Haidich  AB, Pappa  M,  et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies.  JAMA. 2001;286(7):821-830.PubMedGoogle ScholarCrossref
35.
Gruber  S, Chakravarty  A, Heckbert  SR,  et al.  Design and analysis choices for safety surveillance evaluations need to be tuned to the specifics of the hypothesized drug-outcome association.  Pharmacoepidemiol Drug Saf. 2016;25(9):973-981.PubMedGoogle ScholarCrossref
36.
Moride  Y, Abenhaim  L.  Evidence of the depletion of susceptibles effect in non-experimental pharmacoepidemiologic research.  J Clin Epidemiol. 1994;47(7):731-737.PubMedGoogle ScholarCrossref
37.
Schneeweiss  S, Patrick  AR, Stürmer  T,  et al.  Increasing levels of restriction in pharmacoepidemiologic database studies of elderly and comparison with randomized trial results.  Med Care. 2007;45(10)(suppl 2):S131-S142.PubMedGoogle ScholarCrossref
38.
Setoguchi  S, Warner Stevenson  L, Stewart  GC,  et al.  Influence of healthy candidate bias in assessing clinical effectiveness for implantable cardioverter-defibrillators: cohort study of older patients with heart failure.  BMJ. 2014;348:g2866.PubMedGoogle ScholarCrossref
39.
D’Agostino  RB  Sr, Massaro  JM, Sullivan  LM.  Non-inferiority trials: design concepts and issues—the encounters of academic consultants in statistics.  Stat Med. 2003;22(2):169-186.PubMedGoogle ScholarCrossref
40.
Snapinn  SM.  Noninferiority trials.  Curr Control Trials Cardiovasc Med. 2000;1(1):19-21.PubMedGoogle ScholarCrossref
41.
Agoritsas  T, Merglen  A, Shah  ND, O’Donnell  M, Guyatt  GH,  et al.  Adjusted analyses in studies addressing therapy and harm users’ guides to the medical literature clinical scenario.  JAMA. 2017;317(7):748-759.PubMedGoogle ScholarCrossref
42.
Fonarow  GC.  Randomization—there is no substitute.  JAMA Cardiol. 2016;1(6):633-635.PubMedGoogle ScholarCrossref
43.
Freemantle  N, Marston  L, Walters  K, Wood  J, Reynolds  MR, Petersen  I.  Making inferences on treatment effects from real world data: propensity scores, confounding by indication, and other perils for the unwary in observational research.  BMJ. 2013;347:f6409.PubMedGoogle ScholarCrossref
44.
Ioannidis  JP.  Why most published research findings are false.  PLoS Med. 2005;2(8):e124.PubMedGoogle ScholarCrossref
45.
Anglemyer  A, Horvath  HT, Bero  L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. In: Bero L, ed. Cochrane Database of Systematic Reviews. Chichester, England: John Wiley & Sons; 2014:MR000034.
46.
Concato  J, Shah  N, Horwitz  RI.  Randomized, controlled trials, observational studies, and the hierarchy of research designs.  N Engl J Med. 2000;342(25):1887-1892.PubMedGoogle ScholarCrossref
47.
Hemkens  LG, Contopoulos-Ioannidis  DG, Ioannidis  JPA.  Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey.  BMJ. 2016;352:i493.PubMedGoogle ScholarCrossref
48.
Dahabreh  IJ, Sheldrick  RC, Paulus  JK,  et al.  Do observational studies using propensity score methods agree with randomized trials? a systematic comparison of studies on acute coronary syndromes.  Eur Heart J. 2012;33(15):1893-1901.PubMedGoogle ScholarCrossref
×