[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 35.175.191.168. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Figure 1.
Kaplan-Meier Curve Showing the Cumulative Incidence of Atrial Fibrillation (AF)
Kaplan-Meier Curve Showing the Cumulative Incidence of Atrial Fibrillation (AF)

Includes the Vanderbilt University Medical Center Electronic Medical Record Cohort from 2005 to 2010. The shaded area represents 95% CI.

Figure 2.
Calibration Curve Depicting Predicted and Observed Cumulative Incidence of Atrial Fibrillation (AF)
Calibration Curve Depicting Predicted and Observed Cumulative Incidence of Atrial Fibrillation (AF)

Calculation uses the the Cohorts for Heart and Aging Research in Genomic Epidemiology–Atrial Fibrillation (CHARGE-AF) model in the Vanderbilt University Medical Center Electronic Medical Record Cohort and is restricted to 95% of individuals with predicted cumulative incidence of AF from 0 to 0.3. The diagonal line represents a hypothetical ideal curve where predicted and actual AF probability match perfectly for all levels of predicted risk, representing perfect calibration. The curve shows underprediction for individuals with a predicted probability of AF of 0 to 0.15 and overprediction for individuals with a predicted probability greater than 0.15. A histogram showing the distribution of predicted risk is shown.

Figure 3.
Scatterplot of Cumulative Predicted Probability of Atrial Fibrillation (AF) at 5 Years and Age
Scatterplot of Cumulative Predicted Probability of Atrial Fibrillation (AF) at 5 Years and Age

Although age is the most powerful predictor of incident AF, this figure demonstrates that a wide variability in the predicted probability of developing AF exists for any given age.

Table 1.  
CHARGE-AF Simple and Full Risk Model Coefficientsa
CHARGE-AF Simple and Full Risk Model Coefficientsa
Table 2.  
Baseline Patient Characteristics
Baseline Patient Characteristics
1.
Chugh  SS, Havmoeller  R, Narayanan  K,  et al.  Worldwide epidemiology of atrial fibrillation: a Global Burden of Disease 2010 Study.  Circulation. 2014;129(8):837-847.PubMedGoogle ScholarCrossref
2.
Go  AS, Hylek  EM, Phillips  KA,  et al.  Prevalence of diagnosed atrial fibrillation in adults: national implications for rhythm management and stroke prevention: the Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA) Study.  JAMA. 2001;285(18):2370-2375.PubMedGoogle ScholarCrossref
3.
Miyasaka  Y, Barnes  ME, Gersh  BJ,  et al.  Secular trends in incidence of atrial fibrillation in Olmsted County, Minnesota, 1980 to 2000, and implications on the projections for future prevalence.  Circulation. 2006;114(2):119-125.PubMedGoogle ScholarCrossref
4.
Friberg  J, Buch  P, Scharling  H, Gadsbphioll  N, Jensen  GB.  Rising rates of hospital admissions for atrial fibrillation.  Epidemiology. 2003;14(6):666-672.PubMedGoogle ScholarCrossref
5.
Hart  RG, Benavente  O, McBride  R, Pearce  LA.  Antithrombotic therapy to prevent stroke in patients with atrial fibrillation: a meta-analysis.  Ann Intern Med. 1999;131(7):492-501.PubMedGoogle ScholarCrossref
6.
Stewart  S, Hart  CL, Hole  DJ, McMurray  JJ.  A population-based study of the long-term risks associated with atrial fibrillation: 20-year follow-up of the Renfrew/Paisley study.  Am J Med. 2002;113(5):359-364.PubMedGoogle ScholarCrossref
7.
Krahn  AD, Manfreda  J, Tate  RB, Mathewson  FA, Cuddy  TE.  The natural history of atrial fibrillation: incidence, risk factors, and prognosis in the Manitoba Follow-Up Study.  Am J Med. 1995;98(5):476-484.PubMedGoogle ScholarCrossref
8.
Benjamin  EJ, Wolf  PA, D’Agostino  RB, Silbershatz  H, Kannel  WB, Levy  D.  Impact of atrial fibrillation on the risk of death: the Framingham Heart Study.  Circulation. 1998;98(10):946-952.PubMedGoogle ScholarCrossref
9.
Wolf  PA, Mitchell  JB, Baker  CS, Kannel  WB, D’Agostino  RB.  Impact of atrial fibrillation on mortality, stroke, and medical costs.  Arch Intern Med. 1998;158(3):229-234.PubMedGoogle ScholarCrossref
10.
Hamer  ME, Blumenthal  JA, McCarthy  EA, Phillips  BG, Pritchett  EL.  Quality-of-life assessment in patients with paroxysmal atrial fibrillation or paroxysmal supraventricular tachycardia.  Am J Cardiol. 1994;74(8):826-829.PubMedGoogle ScholarCrossref
11.
Le Heuzey  JY, Paziaud  O, Piot  O,  et al.  Cost of care distribution in atrial fibrillation patients: the COCAF study.  Am Heart J. 2004;147(1):121-126.PubMedGoogle ScholarCrossref
12.
Kim  MH, Johnston  SS, Chu  BC, Dalal  MR, Schulman  KL.  Estimation of total incremental health care costs in patients with atrial fibrillation in the United States.  Circ Cardiovasc Qual Outcomes. 2011;4(3):313-320.PubMedGoogle ScholarCrossref
13.
De Caterina  R, Husted  S, Wallentin  L,  et al; Coordinating Committee.  New oral anticoagulants in atrial fibrillation and acute coronary syndromes: ESC Working Group on Thrombosis-Task Force on Anticoagulants in Heart Disease position paper.  J Am Coll Cardiol. 2012;59(16):1413-1425.PubMedGoogle ScholarCrossref
14.
Calkins  H.  Catheter ablation to maintain sinus rhythm.  Circulation. 2012;125(11):1439-1445.PubMedGoogle ScholarCrossref
15.
ARIC Investigators.  The Atherosclerosis Risk in Communities (ARIC) Study: design and objectives. The ARIC investigators.  Am J Epidemiol. 1989;129(4):687-702.PubMedGoogle ScholarCrossref
16.
Fried  LP, Borhani  NO, Enright  P,  et al.  The Cardiovascular Health Study: design and rationale.  Ann Epidemiol. 1991;1(3):263-276.PubMedGoogle ScholarCrossref
17.
Feinleib  M, Kannel  WB, Garrison  RJ, McNamara  PM, Castelli  WP.  The Framingham Offspring Study. Design and preliminary data.  Prev Med. 1975;4(4):518-525.PubMedGoogle ScholarCrossref
18.
Harris  TB, Launer  LJ, Eiriksdottir  G,  et al.  Age, Gene/Environment Susceptibility-Reykjavik Study: multidisciplinary applied phenomics.  Am J Epidemiol. 2007;165(9):1076-1087.PubMedGoogle ScholarCrossref
19.
Hofman  A, Breteler  MMB, van Duijn  CM,  et al.  The Rotterdam Study: 2010 objectives and design update.  Eur J Epidemiol. 2009;24(9):553-572.PubMedGoogle ScholarCrossref
20.
Benjamin  EJ, Chen  PS, Bild  DE,  et al.  Prevention of atrial fibrillation: report from a National Heart, Lung, and Blood Institute workshop.  Circulation. 2009;119(4):606-618.PubMedGoogle ScholarCrossref
21.
Alonso  A, Krijthe  BP, Aspelund  T,  et al.  Simple risk model predicts incidence of atrial fibrillation in a racially and geographically diverse population: the CHARGE-AF consortium.  J Am Heart Assoc. 2013;2(2):e000102.PubMedGoogle ScholarCrossref
22.
Moons  KG, Kengne  AP, Grobbee  DE,  et al.  Risk prediction models, II: external validation, model updating, and impact assessment.  Heart. 2012;98(9):691-698.PubMedGoogle ScholarCrossref
23.
Roden  DM, Pulley  JM, Basford  MA,  et al.  Development of a large-scale de-identified DNA biobank to enable personalized medicine.  Clin Pharmacol Ther. 2008;84(3):362-369.PubMedGoogle ScholarCrossref
24.
Schildcrout  JS, Denny  JC, Bowton  E,  et al.  Optimizing drug outcomes through pharmacogenetics: a case for preemptive genotyping.  Clin Pharmacol Ther. 2012;92(2):235-242.PubMedGoogle ScholarCrossref
25.
Ritchie  MD, Denny  JC, Crawford  DC,  et al.  Robust replication of genotype-phenotype associations across multiple diseases in an electronic medical record.  Am J Hum Genet. 2010;86(4):560-572.PubMedGoogle ScholarCrossref
26.
Newton  KM, Peissig  PL, Kho  AN,  et al.  Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network.  J Am Med Inform Assoc. 2013;20(e1):e147-e154.PubMedGoogle ScholarCrossref
27.
Kho  AN, Hayes  MG, Rasmussen-Torvik  L,  et al.  Use of diverse electronic medical record systems to identify genetic risk for type 2 diabetes within a genome-wide association study.  J Am Med Inform Assoc. 2012;19(2):212-218.PubMedGoogle ScholarCrossref
28.
Xu  H, Stenner  SP, Doan  S, Johnson  KB, Waitman  LR, Denny  JC.  MedEx: a medication information extraction system for clinical narratives.  J Am Med Inform Assoc. 2010;17(1):19-24.PubMedGoogle ScholarCrossref
29.
Doan  S, Bastarache  L, Klimkowski  S, Denny  JC, Xu  H.  Integrating existing natural language processing tools for medication extraction from discharge summaries.  J Am Med Inform Assoc. 2010;17(5):528-531.PubMedGoogle ScholarCrossref
30.
Liu  M, Shah  A, Jiang  M,  et al.  A study of transportability of an existing smoking status detection module across institutions.  AMIA Annu Symp Proc. 2012;2012:577-586.PubMedGoogle Scholar
31.
Horton  NJ, Kleinman  KP.  Much ado about nothing: a comparison of missing data methods and software to fit incomplete data regression models.  Am Stat. 2007;61(1):79-90.PubMedGoogle ScholarCrossref
32.
Spiegelhalter  DJ.  Probabilistic prediction in patient management and clinical trials.  Stat Med. 1986;5(5):421-433.PubMedGoogle ScholarCrossref
33.
Alonso  A, Roetker  NS, Soliman  EZ, Chen  LY, Greenland  P, Heckbert  SR.  Prediction of atrial fibrillation in a racially diverse cohort: the Multi-Ethnic Study of Atherosclerosis (MESA).  J Am Heart Assoc. 2016;5(2):pii:e003077.PubMedGoogle ScholarCrossref
34.
Pfister  R, Brägelmann  J, Michels  G, Wareham  NJ, Luben  R, Khaw  KT.  Performance of the CHARGE-AF risk model for incident atrial fibrillation in the EPIC Norfolk cohort.  Eur J Prev Cardiol. 2015;22(7):932-939.PubMedGoogle ScholarCrossref
35.
Cappola  AR, Fried  LP, Arnold  AM,  et al.  Thyroid status, cardiovascular risk, and mortality in older adults.  JAMA. 2006;295(9):1033-1041.PubMedGoogle ScholarCrossref
36.
Kannel  WB, Dawber  TR, Kagan  A, Revotskie  N, Stokes  J  III.  Factors of risk in the development of coronary heart disease: six year follow-up experience: the Framingham Study.  Ann Intern Med. 1961;55:33-50.PubMedGoogle ScholarCrossref
37.
Gress  TW, Nieto  FJ, Shahar  E, Wofford  MR, Brancati  FL.  Hypertension and antihypertensive therapy as risk factors for type 2 diabetes mellitus: Atherosclerosis Risk in Communities Study.  N Engl J Med. 2000;342(13):905-912.PubMedGoogle ScholarCrossref
38.
Kannel  WB, McGee  DL.  Diabetes and cardiovascular disease: the Framingham Study.  JAMA. 1979;241(19):2035-2038.PubMedGoogle ScholarCrossref
39.
Kannel  WB, LeBauer  EJ, Dawber  TR, McNamara  PM.  Relation of body weight to development of coronary heart disease: the Framingham Study.  Circulation. 1967;35(4):734-744.PubMedGoogle ScholarCrossref
40.
Allen Maycock  CA, Muhlestein  JB, Horne  BD,  et al; Intermountain Heart Collaborative Study.  Statin therapy is associated with reduced mortality across all age groups of individuals with significant coronary disease, including very elderly patients.  J Am Coll Cardiol. 2002;40(10):1777-1785.PubMedGoogle ScholarCrossref
41.
Emdin  CA, Rahimi  K, Patel  A.  Lowering blood pressure in patients with diabetes—reply.  JAMA. 2015;313(21):2183-2184.PubMedGoogle ScholarCrossref
42.
Law  MR, Wald  NJ, Thompson  SG.  By how much and how quickly does reduction in serum cholesterol concentration lower risk of ischaemic heart disease?  BMJ. 1994;308(6925):367-372.PubMedGoogle ScholarCrossref
43.
National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III).  Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report.  Circulation. 2002;106(25):3143-3421.PubMedGoogle Scholar
44.
Pedersen  TR, Kjekshus  J, Berg  K,  et al; Scandinavian Simvastatin Survival Study Group.  Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S). 1994.  Atheroscler Suppl. 2004;5(3):81-87.PubMedGoogle ScholarCrossref
45.
The Long-Term Intervention with Pravastatin in Ischaemic Disease (LIPID) Study Group.  Prevention of cardiovascular events and death with pravastatin in patients with coronary heart disease and a broad range of initial cholesterol levels.  N Engl J Med. 1998;339(19):1349-1357.PubMedGoogle ScholarCrossref
46.
Gibbons  GH. Department of Health and Human Services. National Institutes of Health. Fiscal Year 2014 Budget Request. https://officeofbudget.od.nih.gov/pdfs/FY14/2014_Overall%20Appropriations.pdf. 2013. Accessed January 31, 2016.
47.
Peterson  JF, Bowton  E, Field  JR,  et al.  Electronic health record design and implementation for pharmacogenomics: a local perspective.  Genet Med. 2013;15(10):833-841.PubMedGoogle ScholarCrossref
48.
Zhang  Y, Zhang  P, Mu  Y,  et al.  The role of renin-angiotensin system blockade therapy in the prevention of atrial fibrillation: a meta-analysis of randomized controlled trials.  Clin Pharmacol Ther. 2010;88(4):521-531.PubMedGoogle ScholarCrossref
49.
Schneider  MP, Hua  TA, Böhm  M, Wachtell  K, Kjeldsen  SE, Schmieder  RE.  Prevention of atrial fibrillation by renin-angiotensin system inhibition: a meta-analysis.  J Am Coll Cardiol. 2010;55(21):2299-2307.PubMedGoogle ScholarCrossref
50.
Hill  ME, Rosenwaike  I.  The Social Security Administration’s Death Master File: the completeness of death reporting at older ages.  Soc Secur Bull. 2001-2002/2002;64(1):45-51.PubMedGoogle Scholar
Original Investigation
December 2016

Evaluation of a Prediction Model for the Development of Atrial Fibrillation in a Repository of Electronic Medical Records

Author Affiliations
  • 1Division of Cardiovascular Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee
  • 2Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, Tennessee
  • 3Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee
  • 4School of Biomedical Informatics, University of Texas Health Science Center at Houston
  • 5Department of Epidemiology, University of Washington, Seattle
  • 6Cardiac Arrhythmia Service, Massachusetts General Hospital, Boston
  • 7Framingham Heart Study, National Heart Lung and Blood Institute and Boston University, Framingham, Massachusetts
  • 8Department of Medicine, Boston University School of Medicine, Boston, Massachusetts
  • 9Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, Georgia
  • 10Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
  • 11Department of Biostatistics, Osaka University, Suita, Japan
  • 12Division of Cardiology, University of Illinois at Chicago
JAMA Cardiol. 2016;1(9):1007-1013. doi:10.1001/jamacardio.2016.3366
Key Points

Question  Can the atrial fibrillation (AF) risk prediction model developed by the Calibration of the Cohorts for Heart and Aging Research in Genomic Epidemiology–Atrial Fibrillation (CHARGE-AF) investigators be externally validated using a large repository of electronic medical records (EMRs)?

Findings  In this prediction model study of EMRs of 33 494 patients, 7.3% developed AF during a 5-year period. The CHARGE-AF model was a poor predictor, with underprediction of AF among low-risk individuals and overprediction of AF in high-risk individuals.

Meaning  Application of a risk model derived from prospective cohort studies to an EMR setting has inherent difficulties.

Abstract

Importance  Atrial fibrillation (AF) contributes to substantial morbidity, mortality, and health care expenditures. Accurate prediction of incident AF would enhance AF management and potentially improve patient outcomes.

Objective  To validate the AF risk prediction model originally developed by the Cohorts for Heart and Aging Research in Genomic Epidemiology–Atrial Fibrillation (CHARGE-AF) investigators using a large repository of electronic medical records (EMRs).

Design, Setting, and Participants  In this prediction model study, deidentified EMRs of 33 494 individuals 40 years or older who were white or African American and had no history of AF were reviewed and analyzed. The participants were followed up in the internal medicine outpatient clinics at Vanderbilt University Medical Center for incident AF from December 31, 2005, until December 31, 2010. Adjusting for differences in baseline hazard, the CHARGE-AF Cox proportional hazards model regression coefficients were applied to the EMR cohort. A simple version of the model with no echocardiographic variables was also evaluated. Data were analyzed from October 31, 2013, to January 31, 2014.

Main Outcomes and Measures  Incident AF. Predictors in the model included age, race, height, weight, systolic and diastolic blood pressure, treatment for hypertension, smoking status, type 2 diabetes, heart failure, history of myocardial infarction, left ventricular hypertrophy, and PR interval.

Results  Among the 33 494 participants, the median age was 57 (interquartile range, 49-67) years; 57% of patients were women, 43% were men, 85.7% were white, and 14.3% were African American. During the mean (SD) follow-up of 4.8 (0.9) years, 2455 individuals (7.3%) developed AF. Both models had poor calibration in the EMR cohort, with underprediction of AF among low-risk individuals and overprediction of AF among high-risk individuals (10th and 90th percentiles for predicted probability of incident AF, 0.005 and 0.179, respectively). The full CHARGE-AF model had a C index of 0.708 (95% CI, 0.699-0.718) in our cohort. The simple model had similar discrimination (C index, 0.709; 95% CI, 0.699-0.718; P = .70 for difference between models).

Conclusions and Relevance  Despite reasonable discrimination, the CHARGE-AF models showed poor calibration in this EMR cohort. This study highlights the difficulties of applying a risk model derived from prospective cohort studies to an EMR cohort and suggests that these AF risk prediction models be used with caution in the EMR setting. Future risk models may need to be developed and validated within EMR cohorts.

Introduction

Atrial fibrillation (AF), the most common sustained cardiac arrhythmia, is becoming increasingly prevalent in the Western world.1,2 The number of patients with AF in the United States is projected roughly to double by the year 2050, to an estimated 12 to 16 million persons.2,3 Atrial fibrillation is associated with significant morbidity,4,5 mortality,6-9 decreased quality of life,10 and increased health care expenditures.11,12 Developing strategies for the prediction and prevention of AF in high-risk individuals remains an underexplored and important area of research.13

In 2012, the Cohorts for Heart and Aging Research in Genomic Epidemiology–Atrial Fibrillation (CHARGE-AF) investigators developed and validated a risk model for prediction of incident AF.14 The model was developed using pooled data from prospective cohort studies, including the Atherosclerosis Risk in Communities Study,15 Cardiovascular Health Study,16 and Framingham Heart Study,17 and was validated in the Age Gene/Environment Susceptibility–Reykjavik study18 and Rotterdam Study19 cohorts. The model is especially well suited for primary care settings because it does not require laboratory or echocardiographic (ECG) variables.

Novel risk models should be validated (ie, evaluated in new settings) before they are incorporated into routine care.20 Electronic medical records (EMRs) are becoming ubiquitous in clinical practice, and 1 potential use for EMR repositories in etiologic research is to validate existing risk prediction models. In addition, risk models are unlikely to be widely used unless they can be incorporated into EMR systems. We therefore evaluated the CHARGE-AF risk model for incident AF in a large, deidentified EMR repository.

Methods
Study Population

The study population was selected from a deidentified version of the Vanderbilt University Medical Center EMR (hereinafter referred to as the Vanderbilt EMR). This resource, termed the Synthetic Derivative,21 consists of deidentified medical records of Vanderbilt University Medical Center, Nashville, Tennessee, inpatients and outpatients; as of December 31, 2015, it contained nearly 2.6 million individuals. The Synthetic Derivative consists of the deidentified version of the Vanderbilt EMR that has been judged by the Vanderbilt University institutional review board as falling under the designation of nonhuman subjects under the Common Rule (45 CFR Part 46); therefore, this study and other Synthetic Derivative research were deemed exempt by the Vanderbilt University institutional review board.

To ensure that they had adequate follow-up, individuals met criteria for a medical home model, in which they were followed up in a Vanderbilt University internal medicine clinic with at least 3 visits documented within a 24-month period.22 Other criteria for entry into the study included being 40 years or older, being self-identified as white or African American race, and no known history of AF as of December 31, 2005. Individuals were excluded if they had billing codes for AF or there was mention of AF in ECG impressions or structured problem lists as determined by natural language processing.23 Individuals were also excluded from the study if they had International Classification of Diseases, Ninth Revision (ICD-9), or Current Procedural Terminology (CPT) codes for heart transplant at the beginning of follow-up. Data studied included all inpatient and outpatient ICD-9 and CPT codes, ECGs, and problem lists, and manual review included all inpatient and outpatient records.

Outcome

The follow-up period for incident AF was from December 31, 2005, until December 31, 2010. Ascertainment of incident AF was accomplished by using a validated algorithm that incorporates natural language processing and billing codes, as previously described.23 This automated algorithm was optimized through multiple reiterations with sensitivity analyses with different cutoffs and manual review of medical records until a positive predictive value for AF of greater than 95% was achieved.23 Cases were defined by natural language processing of cardiologist-interpreted ECG impressions, 4 or more occurrences of ICD-9 codes for AF, or AF instances recorded in the problem list. To be classified as free of AF, an individual record could not contain mention of AF in the ECG impressions or structured problem lists or ICD-9 codes for AF or atrial flutter. The 4 ICD-9 codes used in the automated AF algorithm are the most commonly used codes for AF or atrial flutter. Individuals with 1 to 3 ICD-9 codes for AF or atrial flutter were excluded from the study cohort. In effect, these steps resulted in a more sensitive method for excluding AF at baseline and a more specific method for ascertaining incident AF during follow-up. We assessed the accuracy of the AF algorithm in the present study by manually reviewing the full EMR in a random set (enriched for incident AF by selecting roughly equal proportions of individuals with and without AF) of 200 incident AF cases and control individuals block randomized to case-control ordering, as is common for validating EMR phenotypes.24

Predictors

CHARGE-AF model predictors were ascertained from records available from January 1 to December 31, 2005. Sex, race, age, weight, height, body mass index, and systolic and diastolic blood pressure were directly extracted from structured fields in the Synthetic Derivative. History of myocardial infarction, heart failure, and type 2 diabetes were determined by using ICD-9 codes, incorporating laboratory values and medication records.25 Treatment for hypertension was assessed using a previously validated algorithm incorporating medication records in the Synthetic Derivative.26,27 This algorithm was previously shown to have sensitivity and positive predictive values of 88% and 93%, respectively. We obtained PR interval and left ventricular hypertrophy data from outpatient ECG reports. Current smoking status was determined by using an existing algorithm with a reported positive predictive value of 93% in the Vanderbilt EMR.28

Statistical Analysis

Baseline characteristics present at the beginning of the observation period are presented in terms of median (interquartile range) for continuous variables and frequencies with percentage for categorical variables. The number of nonmissing values for each variable is also given. Single imputation of missing values was performed using predictive mean matching.29 To assess the validity of using single rather than multiple imputation, we conducted 5 separate imputations. These resulted in almost identical C indices (eTable 1 in the Supplement).

To evaluate the CHARGE-AF model in the Vanderbilt Synthetic Derivative cohort, we used the Cox proportional hazards regression model derived in the CHARGE-AF study (Table 1).14 As in the original CHARGE-AF study, we compared discrimination using the full model and a simple model that did not incorporate ECG variables. Comparisons used the rcorrp.cens function in the R Hmisc package (R Foundation for Statistical Computing), which provides a test of whether one model gives predictions that are more concordant than the other in a way that preserves pairing of 2 predictions in the pair of patients under consideration.

We adjusted for differences in baseline hazard between the original CHARGE-AF data set and our cohort by constructing the linear predictor for each observation using the original coefficients from the CHARGE-AF model and the covariate values in the Vanderbilt Synthetic Derivative cohort data, with replacement of the mean covariate values to reflect the Vanderbilt data.20 We then examined the calibration and discrimination of the CHARGE-AF model in our cohort. To assess calibration, the observed risk for developing AF during the study period was plotted against the predicted risk.30 The generated curve was compared against a hypothetical ideal curve with a slope of 1 and intercept of 0. For discrimination, which measures the ability of the model to distinguish individuals who will develop AF from those who will not, we calculated the continuous-time C index with censoring.31,32 We also constructed a Kaplan-Meier curve showing cumulative incidence of AF in our cohort. Statistical analyses were conducted from October 31, 2013, to January 31, 2014, using R software (version 3.0.2; R Foundation for Statistical Computing). A 2-sided P ≤ .05 was considered statistically significant. The detailed statistical code is included in eTable 1 in the Supplement.

Results

Based on the prospectively defined criteria, 33 494 individuals were included in the analysis. Baseline characteristics for the study cohort are presented in Table 2. Median age was 57 (interquartile range, 49-67) years; 57% were women, 43% were men, 85.7% were white, and 14.3% were African American. During the mean (SD) follow-up of 4.8 (0.9) years, 2455 individuals (7.3%) developed AF. A Kaplan-Meier curve for the cumulative incidence of AF is shown in Figure 1.

The calibration curve plotting the predicted probability of AF-free survival and the observed AF-free survival in our cohort indicated that the model had poor fit (Figure 2). We found underprediction of AF for individuals with lower (<0.15) predicted probabilities. This group included most of the individuals in the cohort. We also found overprediction of AF for higher probabilities (≥0.15). The 10th and 90th percentiles for predicted probability of incident AF were 0.005 and 0.179, respectively. Across this range, the maximum calibration error was 0.0526 (5.3%).

The full CHARGE-AF model had a C index of 0.708 (95% CI, 0.699-0.718) in our cohort. The simple model, which did not include ECG predictors (PR interval and left ventricular hypertrophy), had discrimination similar to that of the full model (C index, 0.709; 95% CI, 0.699-0.718; P = .70 for difference between the models).

Because the primary study outcome, incident AF during the 5-year follow-up period, was ascertained by an automated algorithm, we conducted a manual review of individual EMRs to assess the accuracy of the algorithm. We selected 200 records at random after enrichment of the sample for incident AF. A manual medical record review revealed that 88 individuals had incident AF during the 5-year follow-up period (eTable 2 in the Supplement). The sensitivity of the AF algorithm was 96.5% (95% CI, 90.1%-98.8%); specificity, 94.8% (95% CI, 89.1%-97.6%); positive predictive value, 93.2% (95% CI, 85.9%-96.8%); and negative predictive value, 97.3% (95% CI, 92.4%-99.1%). The 6 individuals who were not identified as having incident AF by the automated algorithm but who had AF on manual medical record review were each identified as having the arrhythmia based on mention of it in the clinic notes, although AF billing codes were not present. It was less clear why 3 individuals were identified as having incident AF by the algorithm but not confirmed on manual review, although incomplete medical record review and incorrect billing codes are possible explanations.

Because increasing age is the strongest predictor of AF in the CHARGE-AF model, we evaluated whether the model predicts incident AF better than knowledge of age alone. We generated a Cox proportional hazards regression model for the development of AF at 5 years in our cohort with age as the only dependent variable, resulting in a C index of 0.684 (95% CI, 0.674-0.694). The C indices for the Vanderbilt University age-only model and the externally validated CHARGE-AF model could not be compared directly because the models are not nested. We also generated a scatterplot of age and cumulative predicted probability of AF at 5 years (Figure 3). This model showed a broad distribution of AF risk across the spectrum of ages, indicating that age alone is an imprecise predictor of AF risk.

Discussion

We evaluated the CHARGE-AF full and simple risk models in a large cohort of individuals within the Vanderbilt EMR repository (Synthetic Derivative). The full model had a C index of 0.708 (95% CI, 0.699-0.718). The simple CHARGE-AF model, which did not include ECG variables as predictors, had similar discrimination, with a C index of 0.709 (95% CI, 0.699-0.718). However, calibration for both models was poor in our cohort, indicating a failure of validation. Our study represents a novel use of an EMR repository to evaluate an existing AF risk model and illustrates the limitations of applying a model developed in prospective cohort studies to a real-world EMR context.

Our findings illustrate several potential uses for EMR repositories in biomedical research. First, EMR repositories could serve as an inexpensive and efficient complement to community cohort studies for the development of prediction models. Second, an EMR repository could be used as an independent cohort to externally evaluate an existing model, as we did here. In fact, risk models are unlikely to be widely used unless they can be incorporated into EMR systems. Finally, given that EMRs are integrated into clinical practice, prediction models could be incorporated into these systems to prospectively identify individuals at high risk for AF or other diseases, with the ultimate goal of developing individualized preventive strategies. Specifically, improved knowledge about individual AF risk might enable aggressive risk factor modification, more intensive screening, diligent evaluation at the first sign of symptoms, and modification of stroke risk.

The CHARGE-AF model has recently been tested in additional community cohorts. When applied to the Multi-Ethnic Study of Atherosclerosis cohort,33 the simple CHARGE-AF model had good discrimination (C index, 0.779; 95% CI, 0.744-0.814) but suboptimal calibration, with overprediction of AF in higher-risk individuals. When applied to more than 24 000 participants in the European Prospective Investigation of Cancer Norfolk cohort,34 the CHARGE-AF simple model again had good discrimination (C index, 0.81; 95% CI, 0.75-0.85) but also poor calibration. These studies, along with our current findings, illustrate the difficulty of applying a risk model to diverse populations, particularly in an EMR setting.

Large, prospective community cohort studies, such as the Atherosclerosis Risk in Communities Study, the Cardiovascular Health Study, Framingham Heart Study, Age Gene/Environment Susceptibility–Reykjavik study, and Rotterdam Study, have been instrumental in identifying risk factors for common diseases.35-39 In the case of coronary heart disease, the discoveries of these studies have been translated into strategies for primary and secondary prevention that have had important effects on cardiovascular morbidity and mortality.40-45 Electronic medical record repository studies might emerge as an important complement to prospective cohort studies. Although EMR repositories might have important shortcomings, including inadequate disease classifications and missing data, they also offer several attractive advantages that could be leveraged for etiologic research. Because data in EMR repositories are collected during routine clinical care, the cost of these studies is small relative to that of prospective cohort studies. Notably, the National Heart, Lung, and Blood Institute recently announced an initiative to use large EMR studies to enhance the clinical utility and reproducibility of clinical research while reducing costs.46 We propose that automated algorithms could be deployed within EMR repositories to prospectively identify and flag individuals at high risk for AF or other common diseases. These data could then be used to guide primary prevention strategies. Peterson et al47 pursued a similar strategy to identify and genotype individuals at risk for receiving medications that have pharmacogenetic variations in efficacy. Although no specific treatment for the primary prevention of AF has been established,13 angiotensin-converting enzyme inhibitors and angiotensin II receptor blockers have been associated with a decreased incidence of AF in post hoc analyses of randomized clinical trials and in retrospective cohort studies.48,49

Our study has several important limitations. One of these relates to data collection and ascertainment. Although all patient variables were entered into the Vanderbilt EMR prospectively, the nature of the EMR with data entered by multiple users might lead to more inaccuracies when compared with carefully curated prospective cohorts, such as those studied by the CHARGE-AF consortium. Despite a 12-month run-in period from January 1 until December 31, 2005, when individuals with incident AF were excluded from the final study cohort, we did not conduct rigorous screening (eg, ambulatory ECG monitoring) to exclude baseline AF. In addition, patients with AF may not seek medical attention for more than 12 months, and a longer run-in period may be needed. Predictor and outcome variables were extracted from the EMR repository using automated algorithms. Although many of these variables (eg, age, sex, race, height, weight, body mass index, blood pressure, and PR interval) are structured data fields within the repository, other predictors, such as type 2 diabetes and heart failure, depend on billing codes, laboratory or note data, and medication records, potentially resulting in important inaccuracies. The automated algorithm for assignment of incident AF relied primarily on natural language processing of ECG impressions, problem lists, and clinic notes but also used billing codes. We conducted a manual review of a sample of medical records that demonstrated good performance of the algorithm for ascertainment of AF status, with high sensitivity and specificity. Our AF algorithm with a high diagnostic accuracy that exceeds those used by the CHARGE-AF cohorts may explain in part the poor calibration of the risk prediction score in an EMR setting.

Because individuals in our study were not prospectively enrolled and followed up for incident AF, our analysis is prone to indication bias, wherein individuals who developed AF may have had more clinical encounters than those who did not. Because individuals were not followed up at prespecified intervals, our results might be influenced by loss of follow-up. In addition, the classification of incident AF might have been inaccurate if individuals sought care outside Vanderbilt University Medical Center such that their AF diagnoses were not captured in the Vanderbilt EMR.

Although the CHARGE-AF models had satisfactory discrimination in our cohort, calibration was poor. One potential reason for the poor calibration is important differences in the characteristics of the CHARGE-AF discovery cohorts and our cohort.14 A direct comparison of baseline characteristics between our EMR cohort and the CHARGE-AF cohorts is difficult because 5 separate community cohorts were used to formulate the CHARGE-AF model. However, overall, individuals in our cohort tended to be younger, heavier, more likely to be smokers, and more likely to use antihypertensive therapy and generally tended to be sicker than those enrolled in the CHARGE-AF cohorts. Because age is the most important predictor of AF, the inclusion of younger individuals in our study might account, at least in part, for the poor calibration of the CHARGE-AF model in our EMR cohort. We chose the age cutoff for inclusion in our study based on previous findings that the incidence of AF begins to increase rapidly after 40 years of age. Our goal was to be able to apply the CHARGE-AF model in a primary care setting, and we postulate that it is particularly important to identify relatively young patients at risk for AF because they might benefit most from preventative measures, although postulation remains to be proven in prospective studies. Additional potential causes for failure of the models to accurately predict the development of AF include loss to follow-up (including death), differences in how predictors were defined, and more structured surveillance for incident AF in the CHARGE-AF cohorts (greater sensitivity). Owing to the nature of our study, we were unable to include comprehensive data on death (ie, using the Social Security Death Index50) other that what was available in the EMR.

Conclusions

We evaluated the CHARGE-AF full and simple risk models in a large cohort of individuals without a history of AF at baseline in the Vanderbilt EMR repository. The models performed poorly in our EMR cohort, illustrating the difficulty of applying risk models developed within prospective cohort studies to a real-world EMR context. Risk models for the development of AF or other complex disorders are unlikely to be widely used in clinical care unless they can be incorporated into EMR systems. Risk models, therefore, should be derived from and validated in different EMR cohorts, with the goal of prospectively and automatically identifying individuals at high risk for AF and implementing personalized strategies for primary prevention.

Back to top
Article Information

Corresponding Author: Dawood Darbar, MD, Division of Cardiology, University of Illinois at Chicago, 840 S Wood St, Ste 920S (MC 715), Chicago, IL 61612 (darbar@uic.edu).

Accepted for Publication: July 30, 2016.

Published Online: October 12, 2016. doi:10.1001/jamacardio.2016.3366

Author Contributions: Drs Kolek and Darbar had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Kolek, Parvez, Ellinor, Denny, Roden, Darbar.

Acquisition, analysis, or interpretation of data: Kolek, Graves, M. Xu, Bian, Teixeira, Shoemaker, Parvez, H. Xu, Heckbert, Benjamin, Alonso, Denny, Moons, Shintani, Harrell.

Drafting of the manuscript: Kolek, Parvez, Shintani.

Critical revision of the manuscript for important intellectual content: Kolek, Graves, M. Xu, Bian, Teixeira, Shoemaker, H. Xu, Heckbert, Ellinor, Benjamin, Alonso, Denny, Moons, Harrell, Roden, Darbar.

Statistical analysis: Kolek, Graves, M. Xu, Bian, Shintani, Harrell.

Obtained funding: Roden, Darbar.

Administrative, technical, or material support: Teixeira, Shoemaker, H. Xu, Ellinor, Denny.

Study supervision: Parvez, Denny.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Moons reported receiving a financial contribution from projects from the Netherlands Organisation for Scientific Research. No other disclosures were reported.

Funding/Support: This work was supported by Clinical and Translational Science Award UL1TR000445 from the National Center for Advancing Translational Sciences, the Cohorts for Heart and Aging Research in Genomic Epidemiology Challenge, and grants 1RC1HL101056, 2R01HL092577, 2R01HL092217, and N01HC25195 from the National Heart, Lung, and Blood Institute.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Chugh  SS, Havmoeller  R, Narayanan  K,  et al.  Worldwide epidemiology of atrial fibrillation: a Global Burden of Disease 2010 Study.  Circulation. 2014;129(8):837-847.PubMedGoogle ScholarCrossref
2.
Go  AS, Hylek  EM, Phillips  KA,  et al.  Prevalence of diagnosed atrial fibrillation in adults: national implications for rhythm management and stroke prevention: the Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA) Study.  JAMA. 2001;285(18):2370-2375.PubMedGoogle ScholarCrossref
3.
Miyasaka  Y, Barnes  ME, Gersh  BJ,  et al.  Secular trends in incidence of atrial fibrillation in Olmsted County, Minnesota, 1980 to 2000, and implications on the projections for future prevalence.  Circulation. 2006;114(2):119-125.PubMedGoogle ScholarCrossref
4.
Friberg  J, Buch  P, Scharling  H, Gadsbphioll  N, Jensen  GB.  Rising rates of hospital admissions for atrial fibrillation.  Epidemiology. 2003;14(6):666-672.PubMedGoogle ScholarCrossref
5.
Hart  RG, Benavente  O, McBride  R, Pearce  LA.  Antithrombotic therapy to prevent stroke in patients with atrial fibrillation: a meta-analysis.  Ann Intern Med. 1999;131(7):492-501.PubMedGoogle ScholarCrossref
6.
Stewart  S, Hart  CL, Hole  DJ, McMurray  JJ.  A population-based study of the long-term risks associated with atrial fibrillation: 20-year follow-up of the Renfrew/Paisley study.  Am J Med. 2002;113(5):359-364.PubMedGoogle ScholarCrossref
7.
Krahn  AD, Manfreda  J, Tate  RB, Mathewson  FA, Cuddy  TE.  The natural history of atrial fibrillation: incidence, risk factors, and prognosis in the Manitoba Follow-Up Study.  Am J Med. 1995;98(5):476-484.PubMedGoogle ScholarCrossref
8.
Benjamin  EJ, Wolf  PA, D’Agostino  RB, Silbershatz  H, Kannel  WB, Levy  D.  Impact of atrial fibrillation on the risk of death: the Framingham Heart Study.  Circulation. 1998;98(10):946-952.PubMedGoogle ScholarCrossref
9.
Wolf  PA, Mitchell  JB, Baker  CS, Kannel  WB, D’Agostino  RB.  Impact of atrial fibrillation on mortality, stroke, and medical costs.  Arch Intern Med. 1998;158(3):229-234.PubMedGoogle ScholarCrossref
10.
Hamer  ME, Blumenthal  JA, McCarthy  EA, Phillips  BG, Pritchett  EL.  Quality-of-life assessment in patients with paroxysmal atrial fibrillation or paroxysmal supraventricular tachycardia.  Am J Cardiol. 1994;74(8):826-829.PubMedGoogle ScholarCrossref
11.
Le Heuzey  JY, Paziaud  O, Piot  O,  et al.  Cost of care distribution in atrial fibrillation patients: the COCAF study.  Am Heart J. 2004;147(1):121-126.PubMedGoogle ScholarCrossref
12.
Kim  MH, Johnston  SS, Chu  BC, Dalal  MR, Schulman  KL.  Estimation of total incremental health care costs in patients with atrial fibrillation in the United States.  Circ Cardiovasc Qual Outcomes. 2011;4(3):313-320.PubMedGoogle ScholarCrossref
13.
De Caterina  R, Husted  S, Wallentin  L,  et al; Coordinating Committee.  New oral anticoagulants in atrial fibrillation and acute coronary syndromes: ESC Working Group on Thrombosis-Task Force on Anticoagulants in Heart Disease position paper.  J Am Coll Cardiol. 2012;59(16):1413-1425.PubMedGoogle ScholarCrossref
14.
Calkins  H.  Catheter ablation to maintain sinus rhythm.  Circulation. 2012;125(11):1439-1445.PubMedGoogle ScholarCrossref
15.
ARIC Investigators.  The Atherosclerosis Risk in Communities (ARIC) Study: design and objectives. The ARIC investigators.  Am J Epidemiol. 1989;129(4):687-702.PubMedGoogle ScholarCrossref
16.
Fried  LP, Borhani  NO, Enright  P,  et al.  The Cardiovascular Health Study: design and rationale.  Ann Epidemiol. 1991;1(3):263-276.PubMedGoogle ScholarCrossref
17.
Feinleib  M, Kannel  WB, Garrison  RJ, McNamara  PM, Castelli  WP.  The Framingham Offspring Study. Design and preliminary data.  Prev Med. 1975;4(4):518-525.PubMedGoogle ScholarCrossref
18.
Harris  TB, Launer  LJ, Eiriksdottir  G,  et al.  Age, Gene/Environment Susceptibility-Reykjavik Study: multidisciplinary applied phenomics.  Am J Epidemiol. 2007;165(9):1076-1087.PubMedGoogle ScholarCrossref
19.
Hofman  A, Breteler  MMB, van Duijn  CM,  et al.  The Rotterdam Study: 2010 objectives and design update.  Eur J Epidemiol. 2009;24(9):553-572.PubMedGoogle ScholarCrossref
20.
Benjamin  EJ, Chen  PS, Bild  DE,  et al.  Prevention of atrial fibrillation: report from a National Heart, Lung, and Blood Institute workshop.  Circulation. 2009;119(4):606-618.PubMedGoogle ScholarCrossref
21.
Alonso  A, Krijthe  BP, Aspelund  T,  et al.  Simple risk model predicts incidence of atrial fibrillation in a racially and geographically diverse population: the CHARGE-AF consortium.  J Am Heart Assoc. 2013;2(2):e000102.PubMedGoogle ScholarCrossref
22.
Moons  KG, Kengne  AP, Grobbee  DE,  et al.  Risk prediction models, II: external validation, model updating, and impact assessment.  Heart. 2012;98(9):691-698.PubMedGoogle ScholarCrossref
23.
Roden  DM, Pulley  JM, Basford  MA,  et al.  Development of a large-scale de-identified DNA biobank to enable personalized medicine.  Clin Pharmacol Ther. 2008;84(3):362-369.PubMedGoogle ScholarCrossref
24.
Schildcrout  JS, Denny  JC, Bowton  E,  et al.  Optimizing drug outcomes through pharmacogenetics: a case for preemptive genotyping.  Clin Pharmacol Ther. 2012;92(2):235-242.PubMedGoogle ScholarCrossref
25.
Ritchie  MD, Denny  JC, Crawford  DC,  et al.  Robust replication of genotype-phenotype associations across multiple diseases in an electronic medical record.  Am J Hum Genet. 2010;86(4):560-572.PubMedGoogle ScholarCrossref
26.
Newton  KM, Peissig  PL, Kho  AN,  et al.  Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network.  J Am Med Inform Assoc. 2013;20(e1):e147-e154.PubMedGoogle ScholarCrossref
27.
Kho  AN, Hayes  MG, Rasmussen-Torvik  L,  et al.  Use of diverse electronic medical record systems to identify genetic risk for type 2 diabetes within a genome-wide association study.  J Am Med Inform Assoc. 2012;19(2):212-218.PubMedGoogle ScholarCrossref
28.
Xu  H, Stenner  SP, Doan  S, Johnson  KB, Waitman  LR, Denny  JC.  MedEx: a medication information extraction system for clinical narratives.  J Am Med Inform Assoc. 2010;17(1):19-24.PubMedGoogle ScholarCrossref
29.
Doan  S, Bastarache  L, Klimkowski  S, Denny  JC, Xu  H.  Integrating existing natural language processing tools for medication extraction from discharge summaries.  J Am Med Inform Assoc. 2010;17(5):528-531.PubMedGoogle ScholarCrossref
30.
Liu  M, Shah  A, Jiang  M,  et al.  A study of transportability of an existing smoking status detection module across institutions.  AMIA Annu Symp Proc. 2012;2012:577-586.PubMedGoogle Scholar
31.
Horton  NJ, Kleinman  KP.  Much ado about nothing: a comparison of missing data methods and software to fit incomplete data regression models.  Am Stat. 2007;61(1):79-90.PubMedGoogle ScholarCrossref
32.
Spiegelhalter  DJ.  Probabilistic prediction in patient management and clinical trials.  Stat Med. 1986;5(5):421-433.PubMedGoogle ScholarCrossref
33.
Alonso  A, Roetker  NS, Soliman  EZ, Chen  LY, Greenland  P, Heckbert  SR.  Prediction of atrial fibrillation in a racially diverse cohort: the Multi-Ethnic Study of Atherosclerosis (MESA).  J Am Heart Assoc. 2016;5(2):pii:e003077.PubMedGoogle ScholarCrossref
34.
Pfister  R, Brägelmann  J, Michels  G, Wareham  NJ, Luben  R, Khaw  KT.  Performance of the CHARGE-AF risk model for incident atrial fibrillation in the EPIC Norfolk cohort.  Eur J Prev Cardiol. 2015;22(7):932-939.PubMedGoogle ScholarCrossref
35.
Cappola  AR, Fried  LP, Arnold  AM,  et al.  Thyroid status, cardiovascular risk, and mortality in older adults.  JAMA. 2006;295(9):1033-1041.PubMedGoogle ScholarCrossref
36.
Kannel  WB, Dawber  TR, Kagan  A, Revotskie  N, Stokes  J  III.  Factors of risk in the development of coronary heart disease: six year follow-up experience: the Framingham Study.  Ann Intern Med. 1961;55:33-50.PubMedGoogle ScholarCrossref
37.
Gress  TW, Nieto  FJ, Shahar  E, Wofford  MR, Brancati  FL.  Hypertension and antihypertensive therapy as risk factors for type 2 diabetes mellitus: Atherosclerosis Risk in Communities Study.  N Engl J Med. 2000;342(13):905-912.PubMedGoogle ScholarCrossref
38.
Kannel  WB, McGee  DL.  Diabetes and cardiovascular disease: the Framingham Study.  JAMA. 1979;241(19):2035-2038.PubMedGoogle ScholarCrossref
39.
Kannel  WB, LeBauer  EJ, Dawber  TR, McNamara  PM.  Relation of body weight to development of coronary heart disease: the Framingham Study.  Circulation. 1967;35(4):734-744.PubMedGoogle ScholarCrossref
40.
Allen Maycock  CA, Muhlestein  JB, Horne  BD,  et al; Intermountain Heart Collaborative Study.  Statin therapy is associated with reduced mortality across all age groups of individuals with significant coronary disease, including very elderly patients.  J Am Coll Cardiol. 2002;40(10):1777-1785.PubMedGoogle ScholarCrossref
41.
Emdin  CA, Rahimi  K, Patel  A.  Lowering blood pressure in patients with diabetes—reply.  JAMA. 2015;313(21):2183-2184.PubMedGoogle ScholarCrossref
42.
Law  MR, Wald  NJ, Thompson  SG.  By how much and how quickly does reduction in serum cholesterol concentration lower risk of ischaemic heart disease?  BMJ. 1994;308(6925):367-372.PubMedGoogle ScholarCrossref
43.
National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III).  Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report.  Circulation. 2002;106(25):3143-3421.PubMedGoogle Scholar
44.
Pedersen  TR, Kjekshus  J, Berg  K,  et al; Scandinavian Simvastatin Survival Study Group.  Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S). 1994.  Atheroscler Suppl. 2004;5(3):81-87.PubMedGoogle ScholarCrossref
45.
The Long-Term Intervention with Pravastatin in Ischaemic Disease (LIPID) Study Group.  Prevention of cardiovascular events and death with pravastatin in patients with coronary heart disease and a broad range of initial cholesterol levels.  N Engl J Med. 1998;339(19):1349-1357.PubMedGoogle ScholarCrossref
46.
Gibbons  GH. Department of Health and Human Services. National Institutes of Health. Fiscal Year 2014 Budget Request. https://officeofbudget.od.nih.gov/pdfs/FY14/2014_Overall%20Appropriations.pdf. 2013. Accessed January 31, 2016.
47.
Peterson  JF, Bowton  E, Field  JR,  et al.  Electronic health record design and implementation for pharmacogenomics: a local perspective.  Genet Med. 2013;15(10):833-841.PubMedGoogle ScholarCrossref
48.
Zhang  Y, Zhang  P, Mu  Y,  et al.  The role of renin-angiotensin system blockade therapy in the prevention of atrial fibrillation: a meta-analysis of randomized controlled trials.  Clin Pharmacol Ther. 2010;88(4):521-531.PubMedGoogle ScholarCrossref
49.
Schneider  MP, Hua  TA, Böhm  M, Wachtell  K, Kjeldsen  SE, Schmieder  RE.  Prevention of atrial fibrillation by renin-angiotensin system inhibition: a meta-analysis.  J Am Coll Cardiol. 2010;55(21):2299-2307.PubMedGoogle ScholarCrossref
50.
Hill  ME, Rosenwaike  I.  The Social Security Administration’s Death Master File: the completeness of death reporting at older ages.  Soc Secur Bull. 2001-2002/2002;64(1):45-51.PubMedGoogle Scholar
×