[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
Purchase Options:
[Skip to Content Landing]
Figure 1.
Analytic Framework: Nontraditional Risk Factors in Cardiovascular Disease Risk Assessment
Analytic Framework: Nontraditional Risk Factors in Cardiovascular Disease Risk Assessment

Evidence reviews for the US Preventive Services Task Force (USPSTF) use an analytic framework to visually display the key questions that the review will address to allow the USPSTF to evaluate the effectiveness and safety of a preventive service. The questions are depicted by linkages that relate interventions and outcomes. Further details are available in the USPSTF procedure manual.16 ABI indicates ankle-brachial index; CAC, coronary artery calcium; CVD, cardiovascular disease; hsCRP, high-sensitivity C-reactive protein; MI, myocardial infarction.

aRisk factors: age, sex, blood pressure, levels of total and high-density lipoprotein cholesterol, smoking, diabetes, race/ethnicity.

Figure 2.
Literature Search Flow Diagram: Nontraditional Risk Factors in Cardiovascular Disease Risk Assessment
Literature Search Flow Diagram: Nontraditional Risk Factors in Cardiovascular Disease Risk Assessment

In total, the current review included 54 articles (43 studies); studies may appear in more than 1 key question (KQ). Reasons for exclusion: Aim: Study aim was not relevant. Setting: Study was not conducted in a country relevant to US practice, or not conducted in, recruited from, or feasible for primary care or a health care system. Population: Study was not conducted in adults without known cardiovascular disease. Outcomes: Study did not report required outcomes. Intervention: Intervention was out of scope. Study design: Study did not use an included design. Quality: Study was poor quality. Base model: Eligible base models had to include age, sex, systolic blood pressure, antihypertensive medication use, total cholesterol, high-density lipoprotein cholesterol, and current smoking status; eligible base models could not include additional risk factors. Comparator: Study did not include an eligible model for comparison. Ancillary: The article met inclusion criteria, but more complete data such as a larger sample or longer follow-up was abstracted from a different included article.

aJanuary 2007-May 19, 2017 (high-sensitivity C-reactive protein); January 2008-May 19, 2017 (coronary artery calcium score); January 2012-May 19, 2017 (ankle-brachial index).

Table 1.  
Evidence Landscape
Evidence Landscape
Table 2.  
Ranges of Overall, Event, and Nonevent NRI for Cardiovascular or Coronary Heart Disease Events in Studies Reporting Categorical NRI
Ranges of Overall, Event, and Nonevent NRI for Cardiovascular or Coronary Heart Disease Events in Studies Reporting Categorical NRI
Table 3.  
Selected Examples of Appropriate and Inappropriate Reclassification Using Current Treatment Thresholds (≥7.5% for the Pooled Cohort Equations, ≥10% for Framingham Risk Score)
Selected Examples of Appropriate and Inappropriate Reclassification Using Current Treatment Thresholds (≥7.5% for the Pooled Cohort Equations, ≥10% for Framingham Risk Score)
Table 4.  
Summary of Evidence
Summary of Evidence
1.
Benjamin  EJ, Blaha  MJ, Chiuve  SE,  et al.  Heart Disease and Stroke Statistics—2017 Update: a report from the American Heart Association.  Circulation. 2017;135(10):e146-e603. doi:10.1161/CIR.0000000000000485Google ScholarCrossref
2.
Rao Kondapally Seshasai  S, Kaptoge  S, Thompson  A,  et al; Emerging Risk Factors Collaboration.  Diabetes mellitus, fasting glucose, and risk of cause-specific death.  N Engl J Med. 2011;364(9):829-841. doi:10.1056/NEJMoa1008862Google ScholarCrossref
3.
Lewington  S, Clarke  R, Qizilbash  N, Peto  R, Collins  R; Prospective Studies Collaboration.  Age-specific relevance of usual blood pressure to vascular mortality: a meta-analysis of individual data for one million adults in 61 prospective studies.  Lancet. 2002;360(9349):1903-1913. doi:10.1016/S0140-6736(02)11911-8Google ScholarCrossref
4.
Lewington  S, Whitlock  G, Clarke  R,  et al; Prospective Studies Collaboration.  Blood cholesterol and vascular mortality by age, sex, and blood pressure: a meta-analysis of individual data from 61 prospective studies with 55,000 vascular deaths.  Lancet. 2007;370(9602):1829-1839. doi:10.1016/S0140-6736(07)61778-4Google ScholarCrossref
5.
Jha  P, Ramasundarahettige  C, Landsman  V,  et al.  21st-century hazards of smoking and benefits of cessation in the United States.  N Engl J Med. 2013;368(4):341-350. doi:10.1056/NEJMsa1211128Google ScholarCrossref
6.
Karmali  KN, Persell  SD, Perel  P, Lloyd-Jones  DM, Berendsen  MA, Huffman  MD.  Risk scoring for the primary prevention of cardiovascular disease.  Cochrane Database Syst Rev. 2017;3:CD006887.Google Scholar
7.
Box  GEP.  Robustness in the Strategy of Scientific Model Building. Fort Belvoir, VA: Defense Technical Information Center; 1979. doi:10.1016/B978-0-12-438150-6.50018-2Crossref
8.
US Preventive Services Task Force.  Statin use for the primary prevention of cardiovascular disease in adults: US Preventive Services Task Force recommendation statement.  JAMA. 2016;316(19):1997-2007. doi:10.1001/jama.2016.15450Google ScholarCrossref
9.
Stone  NJ, Robinson  JG, Lichtenstein  AH,  et al.  2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  J Am Coll Cardiol. 2014;63(25, pt B):2889-2934. doi:10.1016/j.jacc.2013.11.002Google ScholarCrossref
10.
US Preventive Services Task Force.  Aspirin use for the primary prevention of cardiovascular disease and colorectal cancer: US Preventive Services Task Force recommendation statement.  Ann Intern Med. 2016;164(12):836-845. doi:10.7326/M16-0577Google ScholarCrossref
11.
Helfand  M, Buckley  DI, Freeman  M,  et al.  Emerging risk factors for coronary heart disease: a summary of systematic reviews conducted for the U.S. Preventive Services Task Force.  Ann Intern Med. 2009;151(7):496-507. doi:10.7326/0003-4819-151-7-200910060-00010Google ScholarCrossref
12.
Lin  JS, Olson  CM, Johnson  ES, Whitlock  EP.  The ankle-brachial index for peripheral artery disease screening and cardiovascular disease prediction among asymptomatic adults: a systematic evidence review for the U.S. Preventive Services Task Force.  Ann Intern Med. 2013;159(5):333-341. doi:10.7326/0003-4819-159-5-201309030-00007Google ScholarCrossref
13.
Fowkes  FG, Murray  GD, Butcher  I,  et al; Ankle Brachial Index Collaboration.  Ankle brachial index combined with Framingham Risk Score to predict cardiovascular events and mortality: a meta-analysis.  JAMA. 2008;300(2):197-208. doi:10.1001/jama.300.2.197Google ScholarCrossref
14.
U.S. Preventive Services Task Force.  Using nontraditional risk factors in coronary heart disease risk assessment: U.S. Preventive Services Task Force recommendation statement.  Ann Intern Med. 2009;151(7):474-482. doi:10.7326/0003-4819-151-7-200910060-00008Google ScholarCrossref
15.
U.S. Preventive Services Task Force.  Screening for peripheral artery disease and cardiovascular disease risk assessment with the ankle-brachial index in adults: U.S. Preventive Services Task Force recommendation statement.  Ann Intern Med. 2013;159(5):342-348. doi:10.7326/0003-4819-159-5-201309030-00008Google ScholarCrossref
16.
US Preventive Services Task Force.  US Preventive Services Task Force Procedure Manual. Rockville, MD: US Preventive Services Task Force; 2015.
17.
Buckley  DI, Fu  R, Freeman  M, Rogers  K, Helfand  M.  C-reactive protein as a risk factor for coronary heart disease: a systematic review and meta-analyses for the U.S. Preventive Services Task Force.  Ann Intern Med. 2009;151(7):483-495. doi:10.7326/0003-4819-151-7-200910060-00009Google ScholarCrossref
18.
Alahdab  F, Wang  AT, Elraiyah  TA,  et al.  A systematic review for the screening for peripheral arterial disease in asymptomatic patients.  J Vasc Surg. 2015;61(3)(suppl):42S-53S. doi:10.1016/j.jvs.2014.12.008Google ScholarCrossref
19.
Kramer  CK, Zinman  B, Gross  JL,  et al.  Coronary artery calcium score prediction of all cause mortality and cardiovascular events in people with type 2 diabetes: systematic review and meta-analysis.  BMJ. 2013;346:f1654. doi:10.1136/bmj.f1654Google ScholarCrossref
20.
Peters  SA, Bakker  M, den Ruijter  HM, Bots  ML.  Added value of CAC in risk stratification for cardiovascular events: a systematic review.  Eur J Clin Invest. 2012;42(1):110-116. doi:10.1111/j.1365-2362.2011.02555.xGoogle ScholarCrossref
21.
Peters  SA, den Ruijter  HM, Bots  ML, Moons  KG.  Improvements in risk stratification for the occurrence of cardiovascular disease by imaging subclinical atherosclerosis: a systematic review.  Heart. 2012;98(3):177-184. doi:10.1136/heartjnl-2011-300747Google ScholarCrossref
22.
Saito  I, Maruyama  K, Eguchi  E.  C-reactive protein and cardiovascular disease in East Asians: a systematic review.  Clin Med Insights Cardiol. 2014;8(suppl 3):35-42.Google Scholar
23.
Schnell-Inderst  P, Schwarzer  R, Gohler  A,  et al.  Prognostic value, clinical effectiveness, and cost-effectiveness of high-sensitivity C-reactive protein as a marker for major cardiac events in asymptomatic individuals: a health technology assessment report.  Int J Technol Assess Health Care. 2010;26(1):30-39. doi:10.1017/S0266462309990870Google ScholarCrossref
24.
Shah  T, Casas  JP, Cooper  JA,  et al.  Critical appraisal of CRP measurement for the prediction of coronary heart disease events: new data and systematic review of 31 prospective cohorts.  Int J Epidemiol. 2009;38(1):217-231. doi:10.1093/ije/dyn217Google ScholarCrossref
25.
Cook  NR, Buring  JE, Ridker  PM.  The effect of including C-reactive protein in cardiovascular risk prediction models for women.  Ann Intern Med. 2006;145(1):21-29. doi:10.7326/0003-4819-145-1-200607040-00128Google ScholarCrossref
26.
Kerr  KF, Wang  Z, Janes  H, McClelland  RL, Psaty  BM, Pepe  MS.  Net reclassification indices for evaluating risk prediction instruments: a critical review.  Epidemiology. 2014;25(1):114-121. doi:10.1097/EDE.0000000000000018Google ScholarCrossref
27.
Goff  DC  Jr, Lloyd-Jones  DM, Bennett  G,  et al.  2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  J Am Coll Cardiol. 2014;63(25, pt B):2935-2959. doi:10.1016/j.jacc.2013.11.005Google ScholarCrossref
28.
Anderson  KM, Odell  PM, Wilson  PW, Kannel  WB.  Cardiovascular disease risk profiles.  Am Heart J. 1991;121(1, pt 2):293-298. doi:10.1016/0002-8703(91)90861-BGoogle ScholarCrossref
29.
Anderson  KM, Wilson  PW, Odell  PM, Kannel  WB.  An updated coronary risk profile: a statement for health professionals.  Circulation. 1991;83(1):356-362. doi:10.1161/01.CIR.83.1.356Google ScholarCrossref
30.
Wilson  PW, D’Agostino  RB, Levy  D, Belanger  AM, Silbershatz  H, Kannel  WB.  Prediction of coronary heart disease using risk factor categories.  Circulation. 1998;97(18):1837-1847. doi:10.1161/01.CIR.97.18.1837Google ScholarCrossref
31.
National Cholesterol Education Program.  Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report.  Circulation. 2002;106(25):3143-3421.Google Scholar
32.
D’Agostino  RB  Sr, Vasan  RS, Pencina  MJ,  et al.  General cardiovascular risk profile for use in primary care: the Framingham Heart Study.  Circulation. 2008;117(6):743-753. doi:10.1161/CIRCULATIONAHA.107.699579Google ScholarCrossref
33.
Wells  GA, Shea  B, O’Connell  D,  et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute website. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. Published 2014. Accessed April 8, 2015.
34.
Moons  KG, de Groot  JA, Bouwmeester  W,  et al.  Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.  PLoS Med. 2014;11(10):e1001744. doi:10.1371/journal.pmed.1001744Google ScholarCrossref
35.
Raggi  P, Cooil  B, Callister  TQ.  Use of electron beam tomography data to develop models for prediction of hard coronary events.  Am Heart J. 2001;141(3):375-382. doi:10.1067/mhj.2001.113220Google ScholarCrossref
36.
Kondos  GT, Hoff  JA, Sevrukov  A,  et al.  Electron-beam tomography coronary artery calcium and cardiac events: a 37-month follow-up of 5635 initially asymptomatic low- to intermediate-risk adults.  Circulation. 2003;107(20):2571-2576. doi:10.1161/01.CIR.0000068341.61180.55Google ScholarCrossref
37.
Apfel  CC, Kranke  P, Greim  CA, Roewer  N.  What can be expected from risk scores for predicting postoperative nausea and vomiting?  Br J Anaesth. 2001;86(6):822-827. doi:10.1093/bja/86.6.822Google ScholarCrossref
38.
Yeboah  J, Young  R, McClelland  RL,  et al.  Utility of nontraditional risk markers in atherosclerotic cardiovascular disease risk assessment.  J Am Coll Cardiol. 2016;67(2):139-147. doi:10.1016/j.jacc.2015.10.058Google ScholarCrossref
39.
Fowkes  FG, Murray  GD, Butcher  I,  et al.  Development and validation of an ankle brachial index risk model for the prediction of cardiovascular events.  Eur J Prev Cardiol. 2014;21(3):310-320. doi:10.1177/2047487313516564Google ScholarCrossref
40.
Kaptoge  S, Di Angelantonio  E, Pennells  L,  et al; Emerging Risk Factors Collaboration.  C-reactive protein, fibrinogen, and cardiovascular disease prediction.  N Engl J Med. 2012;367(14):1310-1320. doi:10.1056/NEJMoa1107477Google ScholarCrossref
41.
Arad  Y, Spadaro  LA, Roth  M, Newstein  D, Guerci  AD.  Treatment of asymptomatic adults with elevated coronary calcium scores with atorvastatin, vitamin C, and vitamin E: the St. Francis Heart Study randomized clinical trial.  J Am Coll Cardiol. 2005;46(1):166-172. doi:10.1016/j.jacc.2005.02.089Google ScholarCrossref
42.
Belch  J, MacCuish  A, Campbell  I,  et al.  The Prevention of Progression of Arterial Disease and Diabetes (POPADAD) trial: factorial randomised placebo controlled trial of aspirin and antioxidants in patients with diabetes and asymptomatic peripheral arterial disease.  BMJ. 2008;337:a1840. doi:10.1136/bmj.a1840Google ScholarCrossref
43.
Bos  D, Leening  MJ, Kavousi  M,  et al.  Comparison of atherosclerotic calcification in major vessel beds on the risk of all-cause and cause-specific mortality: the Rotterdam Study  [published online December 11, 2015].  Circ Cardiovasc Imaging. doi:10.1161/CIRCIMAGING.115.003843Google Scholar
44.
Chang  SM, Nabi  F, Xu  J,  et al.  Value of CACS compared with ETT and myocardial perfusion imaging for predicting long-term cardiac outcome in asymptomatic and symptomatic patients at low risk for coronary disease: clinical implications in a multimodality imaging world.  JACC Cardiovasc Imaging. 2015;8(2):134-144. doi:10.1016/j.jcmg.2014.11.008Google ScholarCrossref
45.
Chi  WC, Sylwestrzak  G, Barron  J, Kasravi  B, Power  T, Redberg  R.  Does CAC testing alter downstream treatment patterns for cardiovascular disease?  Am J Manag Care. 2014;20(8):e330-e339.Google Scholar
46.
Danesh  J, Wheeler  JG, Hirschfield  GM,  et al.  C-reactive protein and other circulating markers of inflammation in the prediction of coronary heart disease.  N Engl J Med. 2004;350(14):1387-1397. doi:10.1056/NEJMoa032804Google ScholarCrossref
47.
Elias-Smale  SE, Proenca  RV, Koller  MT,  et al.  Coronary calcium score improves classification of coronary heart disease risk in the elderly: the Rotterdam study.  J Am Coll Cardiol. 2010;56(17):1407-1414. doi:10.1016/j.jacc.2010.06.029Google ScholarCrossref
48.
Erbel  R, Mohlenkamp  S, Moebus  S,  et al.  Coronary risk stratification, discrimination, and reclassification improvement based on quantification of subclinical coronary atherosclerosis: the Heinz Nixdorf Recall Study.  J Am Coll Cardiol. 2010;56(17):1397-1406. doi:10.1016/j.jacc.2010.06.030Google ScholarCrossref
49.
Folsom  AR, Chambless  LE, Ballantyne  CM,  et al.  An assessment of incremental coronary risk prediction using C-reactive protein and other novel risk markers: the atherosclerosis risk in communities study.  Arch Intern Med. 2006;166(13):1368-1373. doi:10.1001/archinte.166.13.1368Google ScholarCrossref
50.
Fowkes  FG, Price  JF, Stewart  MC,  et al.  Aspirin for prevention of cardiovascular events in a general population screened for a low ankle brachial index: a randomized controlled trial.  JAMA. 2010;303(9):841-848. doi:10.1001/jama.2010.221Google ScholarCrossref
51.
Fudim  M, Zalawadiya  S, Patel  DK, Egolum  UO, Afonso  L.  Data on coronary artery calcium score performance and cardiovascular risk reclassification across gender and ethnicities.  Data Brief. 2016;6:578-581. doi:10.1016/j.dib.2016.01.002Google ScholarCrossref
52.
Geisel  MH, Bauer  M, Hennig  F,  et al.  Comparison of coronary artery calcification, carotid intima-media thickness and ankle-brachial index for predicting 10-year incident cardiovascular events in the general population.  Eur Heart J. 2017;38(23):1815-1822. doi:10.1093/eurheartj/ehx120Google ScholarCrossref
53.
Greenland  P, LaBree  L, Azen  SP, Doherty  TM, Detrano  RC.  Coronary artery calcium score combined with Framingham score for risk prediction in asymptomatic individuals.  JAMA. 2004;291(2):210-215. doi:10.1001/jama.291.2.210Google ScholarCrossref
54.
Hamer  M, Chida  Y, Stamatakis  E.  Utility of C-reactive protein for cardiovascular risk stratification across three age groups in subjects without existing cardiovascular diseases.  Am J Cardiol. 2009;104(4):538-542. doi:10.1016/j.amjcard.2009.04.020Google ScholarCrossref
55.
Hoffmann  U, Massaro  JM, D’Agostino  RB  Sr, Kathiresan  S, Fox  CS, O’Donnell  CJ.  Cardiovascular event prediction and risk reclassification by coronary, aortic, and valvular calcification in the Framingham Heart Study.  J Am Heart Assoc. 2016;5(2):e003144. doi:10.1161/JAHA.115.003144Google ScholarCrossref
56.
Holewijn  S, den Heijer  M, Kiemeney  LA, Stalenhoef  AF, de Graaf  J.  Combining risk markers improves cardiovascular risk prediction in women.  Clin Sci (Lond). 2014;126(2):139-146. doi:10.1042/CS20130178Google ScholarCrossref
57.
Kavousi  M, Desai  CS, Ayers  C,  et al.  Prevalence and prognostic implications of coronary artery calcification in low-risk women: a meta-analysis.  JAMA. 2016;316(20):2126-2134. doi:10.1001/jama.2016.17020Google ScholarCrossref
58.
Kavousi  M, Elias-Smale  S, Rutten  JH,  et al.  Evaluation of newer risk markers for coronary heart disease risk classification: a cohort study.  Ann Intern Med. 2012;156(6):438-444. doi:10.7326/0003-4819-156-6-201203200-00006Google ScholarCrossref
59.
Koenig  W, Lowel  H, Baumert  J, Meisinger  C.  C-reactive protein modulates risk prediction based on the Framingham score: implications for future risk assessment: results from a large cohort study in southern Germany.  Circulation. 2004;109(11):1349-1353. doi:10.1161/01.CIR.0000120707.98922.E3Google ScholarCrossref
60.
Lyngbaek  S, Marott  JL, Sehestedt  T,  et al.  Cardiovascular risk prediction in the general population with use of suPAR, CRP, and Framingham Risk Score.  Int J Cardiol. 2013;167(6):2904-2911. doi:10.1016/j.ijcard.2012.07.018Google ScholarCrossref
61.
Malik  S, Budoff  MJ, Katz  R,  et al.  Impact of subclinical atherosclerosis on cardiovascular disease events in individuals with metabolic syndrome and diabetes: the multi-ethnic study of atherosclerosis.  Diabetes Care. 2011;34(10):2285-2290. doi:10.2337/dc11-0816Google ScholarCrossref
62.
Messenger  B, Li  D, Nasir  K, Carr  JJ, Blankstein  R, Budoff  MJ.  Coronary calcium scans and radiation exposure in the multi-ethnic study of atherosclerosis.  Int J Cardiovasc Imaging. 2016;32(3):525-529. doi:10.1007/s10554-015-0799-3Google ScholarCrossref
63.
Mohlenkamp  S, Lehmann  N, Moebus  S,  et al.  Quantification of coronary atherosclerosis and inflammation to predict coronary events and all-cause mortality.  J Am Coll Cardiol. 2011;57(13):1455-1464. doi:10.1016/j.jacc.2010.10.043Google ScholarCrossref
64.
Mora  S, Glynn  RJ, Hsia  J, MacFadyen  JG, Genest  J, Ridker  PM.  Statins for the primary prevention of cardiovascular events in women with elevated high-sensitivity C-reactive protein or dyslipidemia: results from the Justification for the Use of Statins in Prevention: An Intervention Trial Evaluating Rosuvastatin (JUPITER) and meta-analysis of women from primary prevention trials.  Circulation. 2012;121(9):1069-1077. doi:10.1161/CIRCULATIONAHA.109.906479Google ScholarCrossref
65.
Murphy  TP, Dhangana  R, Pencina  MJ, D’Agostino  RB  Sr.  Ankle-brachial index and cardiovascular risk prediction: an analysis of 11,594 individuals with 10-year follow-up.  Atherosclerosis. 2012;220(1):160-167. doi:10.1016/j.atherosclerosis.2011.10.037Google ScholarCrossref
66.
Nielsen  AD, Videbech  P, Gerke  O,  et al.  Population screening for coronary artery calcification does not increase mental distress and the use of psychoactive medication.  J Thorac Imaging. 2012;27(3):202-206. doi:10.1097/RTI.0b013e31824752bdGoogle ScholarCrossref
67.
O’Malley  PG, Feuerstein  IM, Taylor  AJ.  Impact of electron beam tomography, with or without case management, on motivation, behavioral change, and cardiovascular risk profile: a randomized controlled trial.  JAMA. 2003;289(17):2215-2223. doi:10.1001/jama.289.17.2215Google ScholarCrossref
68.
Polak  JF, Szklo  M, O’Leary  DH.  Carotid intima-media thickness score, positive coronary artery calcium score, and incident coronary heart disease: the Multi-Ethnic Study of Atherosclerosis.  J Am Heart Assoc. 2017;6(1):e004612. doi:10.1161/JAHA.116.004612Google ScholarCrossref
69.
Polonsky  TS, McClelland  RL, Jorgensen  NW,  et al.  Coronary artery calcium score and risk classification for coronary heart disease prediction.  JAMA. 2010;303(16):1610-1616. doi:10.1001/jama.2010.461Google ScholarCrossref
70.
Price  JF, Tzoulaki  I, Lee  AJ, Fowkes  FG.  Ankle brachial index and intima media thickness predict cardiovascular events similarly and increased prediction when combined.  J Clin Epidemiol. 2007;60(10):1067-1075. doi:10.1016/j.jclinepi.2007.01.011Google ScholarCrossref
71.
Rana  JS, Cote  M, Despres  JP,  et al.  Inflammatory biomarkers and the prediction of coronary events among people at intermediate risk: the EPIC-Norfolk prospective population study.  Heart. 2009;95(20):1682-1687. doi:10.1136/hrt.2009.170134Google ScholarCrossref
72.
Rana  JS, Gransar  H, Wong  ND,  et al.  Comparative value of coronary artery calcium and multiple blood biomarkers for prognostication of cardiovascular events.  Am J Cardiol. 2012;109(10):1449-1453. doi:10.1016/j.amjcard.2012.01.358Google ScholarCrossref
73.
Ridker  PM, Danielson  E, Fonseca  FA,  et al.  Rosuvastatin to prevent vascular events in men and women with elevated C-reactive protein.  N Engl J Med. 2008;359(21):2195-2207. doi:10.1056/NEJMoa0807646Google ScholarCrossref
74.
Rodondi  N, Marques-Vidal  P, Butler  J,  et al.  Markers of atherosclerosis and inflammation for prediction of coronary heart disease in older adults.  Am J Epidemiol. 2010;171(5):540-549. doi:10.1093/aje/kwp428Google ScholarCrossref
75.
Rozanski  A, Gransar  H, Shaw  LJ,  et al.  Impact of coronary artery calcium scanning on coronary risk factors and downstream testing: the EISNER (Early Identification of Subclinical Atherosclerosis by Noninvasive Imaging Research) prospective randomized trial.  J Am Coll Cardiol. 2011;57(15):1622-1632. doi:10.1016/j.jacc.2011.01.019Google ScholarCrossref
76.
Salim  A, Tai  ES, Tan  VY,  et al.  C-reactive protein and serum creatinine, but not haemoglobin A1c, are independent predictors of coronary heart disease risk in non-diabetic Chinese.  Eur J Prev Cardiol. 2016;12:1339-1349. doi:10.1177/2047487315626547Google ScholarCrossref
77.
Sattar  N, Murray  HM, McConnachie  A,  et al.  C-reactive protein and prediction of coronary heart disease and global vascular events in the Prospective Study of Pravastatin in the Elderly at Risk (PROSPER).  Circulation. 2007;115(8):981-989. doi:10.1161/CIRCULATIONAHA.106.643114Google ScholarCrossref
78.
Schneider  HJ, Wallaschofski  H, Volzke  H,  et al.  Incremental effects of endocrine and metabolic biomarkers and abdominal obesity on cardiovascular mortality prediction.  PLoS One. 2012;7(3):e33084. doi:10.1371/journal.pone.0033084Google ScholarCrossref
79.
Seven  E, Husemoen  LL, Sehested  TS,  et al.  Adipocytokines, C-reactive protein, and cardiovascular disease: a population-based prospective study.  PLoS One. 2015;10(6):e0128987. doi:10.1371/journal.pone.0128987Google ScholarCrossref
80.
Shreibati  JB, Baker  LC, McConnell  MV, Hlatky  MA.  Outcomes after coronary artery calcium and other cardiovascular biomarker testing among asymptomatic Medicare beneficiaries.  Circ Cardiovasc Imaging. 2014;7(4):655-662. doi:10.1161/CIRCIMAGING.113.001869Google ScholarCrossref
81.
Velescu  A, Clara  A, Penafiel  J,  et al.  Adding low ankle brachial index to classical risk factors improves the prediction of major cardiovascular events: the REGICOR study.  Atherosclerosis. 2015;241(2):357-363. doi:10.1016/j.atherosclerosis.2015.05.017Google ScholarCrossref
82.
Wannamethee  SG, Welsh  P, Lowe  GD,  et al.  N-terminal pro-brain natriuretic peptide is a more useful predictor of cardiovascular disease risk than C-reactive protein in older men with and without pre-existing cardiovascular disease.  J Am Coll Cardiol. 2011;58(1):56-64. doi:10.1016/j.jacc.2011.02.041Google ScholarCrossref
83.
Welsh  P, Doolin  O, Willeit  P,  et al.  N-terminal pro-B-type natriuretic peptide and the prediction of primary cardiovascular events: results from 15-year follow-up of WOSCOPS.  Eur Heart J. 2013;34(6):443-450. doi:10.1093/eurheartj/ehs239Google ScholarCrossref
84.
Wilson  PW, Nam  BH, Pencina  M, D’Agostino  RB  Sr, Benjamin  EJ, O’Donnell  CJ.  C-reactive protein and risk of cardiovascular disease in men and women from the Framingham Heart Study.  Arch Intern Med. 2005;165(21):2473-2478. doi:10.1001/archinte.165.21.2473Google ScholarCrossref
85.
Wilson  PW, Pencina  M, Jacques  P, Selhub  J, D’Agostino  R  Sr, O’Donnell  CJ.  C-reactive protein and reclassification of cardiovascular risk in the Framingham Heart Study.  Circ Cardiovasc Qual Outcomes. 2008;1(2):92-97. doi:10.1161/CIRCOUTCOMES.108.831198Google ScholarCrossref
86.
Wong  ND, Gransar  H, Shaw  L,  et al.  Thoracic aortic calcium versus coronary artery calcium for the prediction of coronary heart disease and cardiovascular disease events.  JACC Cardiovasc Imaging. 2009;2(3):319-326. doi:10.1016/j.jcmg.2008.12.010Google ScholarCrossref
87.
Yeboah  J, Erbel  R, Delaney  JC,  et al.  Development of a new diabetes risk prediction tool for incident coronary heart disease events: the Multi-Ethnic Study of Atherosclerosis and the Heinz Nixdorf Recall Study.  Atherosclerosis. 2014;236(2):411-417. doi:10.1016/j.atherosclerosis.2014.07.035Google ScholarCrossref
88.
Yeboah  J, McClelland  RL, Polonsky  TS,  et al.  Comparison of novel risk markers for improvement in cardiovascular risk assessment in intermediate-risk individuals.  JAMA. 2012;308(8):788-795. doi:10.1001/jama.2012.9624Google ScholarCrossref
89.
Zhou  QM, Zheng  Y, Cai  T.  Subgroup specific incremental value of new markers for risk prediction.  Lifetime Data Anal. 2013;19(2):142-169. doi:10.1007/s10985-012-9235-3Google ScholarCrossref
90.
Pencina  MJ, D’Agostino  RB  Sr, Steyerberg  EW.  Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers.  Stat Med. 2011;30(1):11-21. doi:10.1002/sim.4085Google ScholarCrossref
91.
Nasir  K, Bittencourt  MS, Blaha  MJ,  et al.  Implications of coronary artery calcium testing among statin candidates according to American College of Cardiology/American Heart Association Cholesterol Management Guidelines: MESA (Multi-Ethnic Study of Atherosclerosis).  J Am Coll Cardiol. 2015;66(15):1657-1668. doi:10.1016/j.jacc.2015.07.066Google ScholarCrossref
92.
McEvoy  JW, Martin  SS, Blaha  MJ,  et al.  The case for and against a coronary artery calcium trial: means, motive, and opportunity.  JACC Cardiovasc Imaging. 2016;9(8):994-1002. doi:10.1016/j.jcmg.2016.03.012Google ScholarCrossref
93.
Cook  NR.  Use and misuse of the receiver operating characteristic curve in risk prediction.  Circulation. 2007;115(7):928-935. doi:10.1161/CIRCULATIONAHA.106.672402Google ScholarCrossref
94.
Damen  JA, Hooft  L, Schuit  E,  et al.  Prediction models for cardiovascular disease risk in the general population: systematic review.  BMJ. 2016;353:i2416. doi:10.1136/bmj.i2416Google ScholarCrossref
95.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2Google ScholarCrossref
US Preventive Services Task Force
Evidence Report
July 17, 2018

Nontraditional Risk Factors in Cardiovascular Disease Risk AssessmentUpdated Evidence Report and Systematic Review for the US Preventive Services Task Force

Author Affiliations
  • 1Kaiser Permanente Research Affiliates Evidence-based Practice Center, Center for Health Research, Kaiser Permanente, Portland, Oregon
JAMA. 2018;320(3):281-297. doi:10.1001/jama.2018.4242
Abstract

Importance  Incorporating nontraditional risk factors may improve the performance of traditional multivariable risk assessment for cardiovascular disease (CVD).

Objective  To systematically review evidence for the US Preventive Services Task Force on the benefits and harms of 3 nontraditional risk factors in cardiovascular risk assessment: the ankle-brachial index (ABI), high-sensitivity C-reactive protein (hsCRP) level, and coronary artery calcium (CAC) score.

Data Sources  MEDLINE, PubMed, and the Cochrane Central Register of Controlled Trials for studies published through May 22, 2017. Surveillance continued through February 7, 2018.

Study Selection  Studies of asymptomatic adults with no known cardiovascular disease.

Data Extraction and Synthesis  Independent critical appraisal and data abstraction by 2 reviewers.

Main Outcomes and Measures  Cardiovascular events, mortality, risk assessment performance measures (calibration, discrimination, or risk reclassification), and serious adverse events.

Results  Forty-three studies (N = 267 244) were included. No adequately powered trials have evaluated the clinical effect of risk assessment with nontraditional risk factors on patient health outcomes. The addition of the ABI (10 studies), hsCRP level (25 studies), or CAC score (19 studies) can improve both discrimination and reclassification; the magnitude and consistency of improvement varies by nontraditional risk factor. For the ABI, improvements in performance were the greatest for women, in whom traditional risk assessment has poor discrimination (C statistic change of 0.112 and net reclassification index [NRI] of 0.096). Results were inconsistent for hsCRP level, with the largest analysis (n = 166 596) showing a minimal effect on risk prediction (C statistic change of 0.0039, NRI of 0.0152). The largest improvements in discrimination (C statistic change ranging from 0.018 to 0.144) and reclassification (NRI ranging from 0.084 to 0.35) were seen for CAC score, although CAC score may inappropriately reclassify individuals not having cardiovascular events into higher-risk categories, as determined by negative nonevent NRI. Evidence for the harms of nontraditional risk factor assessment was limited to computed tomography imaging for CAC scoring (8 studies) and showed that radiation exposure is low but may result in additional testing.

Conclusions and Relevance  There are insufficient adequately powered clinical trials evaluating the incremental effect of the ABI, hsCRP level, or CAC score in risk assessment and initiation of preventive therapy. Furthermore, the clinical meaning of improvements in measures of calibration, discrimination, and reclassification risk prediction studies is uncertain.

Introduction

Cardiovascular disease (CVD) is the leading cause of death in the United States, accounting for about 1 in 3 deaths.1 The incidence of CVD is strongly associated with a set of traditional risk factors,2-5 which have been combined using multivariable risk assessment tools to estimate an individual’s risk for having a CVD event.6 However, these tools can underestimate or overestimate CVD risk.7 Inclusion of nontraditional biological and physiologic risk factors might improve the performance of tools. Given that risk estimates are used to guide preventive therapy such as aspirin and statins,6,8-10 improved risk assessment performance could result in improved CVD outcomes.

Previous research has established that abnormal values for the nontraditional risk factors of the ankle-brachial index (ABI), high-sensitivity C-reactive protein (hsCRP) level, and coronary artery calcium (CAC) score are significantly associated with morbidity and mortality.11-13 Moreover, this research has shown that these factors can improve the ability of models to distinguish between individuals who will and will not have events and to reclassify individuals into clinically actionable risk strata. However, previous systematic reviews were conducted before the release of newly recommended risk tools and treatment thresholds. Additionally, these reviews did not address the ability of nontraditional risk factors to improve on existing models’ overestimation or underestimation of risk.

Based on previous systematic reviews, the US Preventive Services Task Force (USPSTF) issued recommendations in 2009 and 2013, concluding that the evidence was insufficient to assess the benefits and harms of using nontraditional risk factors for cardiovascular risk assessment (I statements).14,15 This review sought to identify and appraise updated evidence on the benefits and harms of nontraditional risk factor assessment and therapy guided by the ABI, hsCRP level, and CAC score to support the USPSTF in updating its 2009 and 2013 recommendations.

Methods
Scope of Review

This review addressed 5 key questions (KQs) as shown in Figure 1. Methodological details including study selection, a list of excluded studies, data analysis methods, and additional subpopulation results, as well as detailed descriptions of all models, are available in the full evidence report at https://www.uspreventiveservicestaskforce.org/Page/Document/UpdateSummaryFinal/cardiovascular-disease-screening-using-nontraditional-risk-assessment.

Data Sources and Searches

MEDLINE, PubMed, and the Cochrane Central Register of Controlled Trials were searched through May 22, 2017, to identify literature published after 2 reviews for the USPSTF11,12,17 (eMethods in the Supplement). All studies in the prior reviews were also evaluated, as well as reference lists of other systematic reviews.18-24 ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform were searched for relevant ongoing trials. Since May 22, 2017, ongoing surveillance through article alerts and targeted searches of journals with a high impact factor and journals relevant to the topic was conducted to identify major studies published in the interim that may affect the conclusions or understanding of the evidence and therefore the related USPSTF recommendation. The last surveillance was conducted on February 7, 2018, and identified no additional relevant studies.

Study Selection

Investigators reviewed 22 707 unique citations and 483 full-text articles against a priori inclusion criteria (Figure 2; eTable 1 in the Supplement).

Studies of adults without known CVD that were published in English were eligible for inclusion. For each KQ, subpopulation analyses by sex, race/ethnicity, and diabetes were identified a priori. For KQ1, trials comparing traditional risk assessment with traditional risk assessment plus the ABI, hsCRP level, or CAC score that reported patient health outcomes (ie, CVD events, mortality, or both) were included. For KQ2, individual participant data meta-analyses, trials, and well-designed prospective cohort studies evaluating risk prediction in models with traditional risk factors (base model), compared with models additionally including the ABI, hsCRP level, or CAC score (extended model), were included. KQ2 studies were required to include a measure of calibration, discrimination, or reclassification (Box). Eligible base models were the Pooled Cohort Equations,27 the Framingham Risk Score,28-32 or models including the same variables: age, sex, systolic blood pressure, use of antihypertensive medication, total cholesterol level, high-density lipoprotein cholesterol level, and smoking status. Models were eligible with or without inclusion of race/ethnicity and diabetes status. Studies with additional variables in their base models were excluded, as this would have precluded isolation of the effect of the nontraditional risk factor of interest. Extended models that incorporated multiple nontraditional risk factors were excluded if the effect of newly added risk factors could not be isolated.

Box Section Ref ID
Box.

Definitions of Risk Prediction Measures Included for KQ2

Calibration refers to the agreement between observed and predicted outcomes. Calibration plots and observed to expected ratios were identified a priori as preferable measures of calibration because of their ability to indicate direction of miscalibration, capacity to compare across models. and intuitive graphic interpretation.

Discrimination is the ability to distinguish between individuals who will and will not have an event. This analysis uses the change in C statistic to measure discrimination. The C statistic is the probability that, for a randomly selected pair of individuals, 1 with disease and the other without, the person with disease will have the higher estimated disease probability according to the model.25

Reclassification reflects the ability of a new model to appropriately reassign people into different risk strata. The most commonly used measure in this review was the Net Reclassification Index (NRI), which captures the reclassification of individuals that occurs when traditional risk assessment is enriched with the ABI, hsCRP level, or CAC score, where clinically meaningful risk strata are typically defined by treatment thresholds. Reclassification to a higher risk category is considered appropriate when the individual has an outcome, and reclassification to a lower risk category is considered appropriate when an individual does not have an outcome. The NRI is the sum of differences in proportions of individuals moving up a risk category minus those moving down a risk category with a cardiovascular disease outcome (known as the event NRI), plus the proportion moving down a risk category minus those moving up a risk category without an outcome (known as the nonevent NRl).26

Abbreviations: ABI, ankle-brachial index; CAC, coronary artery calcium; hsCRP, high-sensitivity C-reactive protein; KQ, key question.

For KQ3 and KQ5, trials, prospective and retrospective cohort studies, and well-designed case-control studies examining harms of risk assessment or treatment guided by risk assessment were included. Harms included any serious adverse event requiring unexpected or unwanted medical attention resulting from risk assessment or harms from risk factor modification. For assessment of CAC score, radiation exposure from computed tomography (CT) was included as a potential harm. For KQ4, trials of treatment guided by nontraditional risk factor assessment in addition to traditional risk assessment vs no treatment or usual care that reported patient health outcomes (ie, CVD events, mortality, or both) were included.

Data Extraction and Quality Assessment

Two independent reviewers critically appraised the included studies using predefined criteria,16,33,34 with a third reviewer resolving disagreements (eTable 2 in the Supplement). Articles were rated as good, fair, or poor quality. In general, a good-quality study met all criteria. A fair-quality study did not meet, or it was unclear whether it met, at least 1 criterion but had no known important limitations that could invalidate its results. A poor-quality study had a single fatal flaw or multiple important limitations. Poor-quality studies were excluded from the review. Two studies, both for KQ2, were excluded as poor quality because of nonrepresentative sampling of patients, self-reported outcomes, limited duration of follow-up, and/or a small number of CVD events.35,36 One reviewer abstracted descriptive and outcome data from each included study into standardized evidence tables; a second checked for accuracy and completeness.

Data Synthesis and Analysis

Data for each KQ are summarized narratively according to nontraditional risk factor. Quantitative analyses were not conducted because of the limited number of studies for each KQ or methodological and clinical heterogeneity, including differences in outcomes and treatments evaluated. KQ2 results are stratified by calibration, discrimination, or reclassification (Box). A full description of selected measures for each domain and their limitations is available in eTable 3 in the Supplement.

Because there is no guidance in existing literature about how to characterize the magnitude or clinical meaning of changes in discrimination (KQ2),26 the following definitions were used for practical reasons. For changes in the C statistic, the term “large” is used to denote changes of 0.1 or greater, “moderate” for changes of 0.05 to 0.1, “small” for changes of 0.025 to 0.05, and “very small” for changes less than 0.025. C statistics range from 0.5 to 1.0; the 0.1 cutpoint for “large” was set because it represents 20% of the possible range. A change in C statistic of 0.025 approximates a 5% higher sensitivity when specificity is 50%.37

KQ2 analyses were additionally stratified by (1) type of model design—ie, published coefficient vs model development; (2) choice of the Pooled Cohort Equations or the Framingham Risk Score as the base model; and (3) prediction of global CVD outcomes, which include coronary and noncoronary events. Published coefficient model studies are risk prediction studies evaluating the added prognostic value of the ABI, hsCRP level, or CAC score that preserved the coefficients from the published Pooled Cohort Equations or Framingham Risk Score, which are readily available for use in clinical practice. This is in contrast to model development studies, which fit entirely new models with locally developed coefficients, such that these models are generally not available in the public domain for clinical use. Studies that used the original published coefficients of the Framingham Risk Score or the Pooled Cohort Equations were considered preferable to model development studies because of their applicability.

To help illustrate the clinical meaning of the Net Reclassification Index (NRI), event NRI, and nonevent NRI for each nontraditional risk factor, reclassification tables from selected studies were modified to correspond to current treatment thresholds and show reclassification in both absolute and relative terms. A 2016 Multi-Ethnic Study of Atherosclerosis (MESA) published coefficient analysis (which evaluated all 3 nontraditional risk factors using both Pooled Cohort Equations and Framingham Risk Score base models)38 and 2 individual patient data meta-analyses (available for the ABI39 and hsCRP level40) were selected based on the applicability of the models evaluated, the ability to compare across risk factors, the larger populations represented therein, and the reporting of reclassification tables allowing for this analysis. For the Framingham Risk Score–based analyses, which used 3 risk strata, the top 2 risk strata (10%-20% and >20%) were combined to conform to the current USPSTF recommendation to initiate preventive low-dose aspirin and statins based on a 10-year CVD risk of 10% or greater.

Results

In total, 43 studies reported in 54 publications (n = 267 244) were included (Figure 2).24,25,38-89 Thirty-eight unique trials or cohorts are represented in the included literature. For all KQs, additional descriptive and outcome data are available in the full report.

Benefits of Risk Assessment

Key Question 1. Compared with the Pooled Cohort Equations or Framingham risk factors alone, does risk assessment of asymptomatic adults using nontraditional risk factors lead to reduced incidence of cardiovascular events, mortality, or both?

Only 1 fair-quality trial evaluated the incremental value of nontraditional risk factor assessment on CVD events. The Early Identification of Subclinical Atherosclerosis by Non-invasive Imaging Research (EISNER) trial (n = 2137), conducted in the United States, randomized volunteers to undergo CT scanning for CAC scoring in addition to the Framingham Risk Score vs no CAC scoring before risk factor counseling.75 Participants were middle-aged adults with CVD risk factors but no known CVD or symptoms. This study found no statistically significant difference in myocardial infarction, mortality, or combined myocardial infarction and mortality at 4 years between the 2 groups; however, the trial did not have adequate sample size and length of follow-up to detect differences in these outcomes, as the primary outcome of this trial was a change in CVD risk factors and Framingham Risk Score at 4 years.

Performance Characteristics of Risk Assessment Measures

Key Question 2. Does use of nontraditional risk factors in addition to traditional risk factors to predict cardiovascular disease risk improve measures of calibration, discrimination, and risk reclassification?

Thirty-three studies reported in 43 articles evaluated the ABI, hsCRP level, and/or CAC score in addition to traditional cardiovascular risk assessment and reported 1 or more measures of calibration, discrimination, and/or risk reclassification. Ten studies evaluated the ABI, 25 studies evaluated hsCRP level, and 19 studies evaluated CAC score (Table 1). The evidence base for the ABI and hsCRP included large individual patient data meta-analyses39,40; in these cases, the meta-analyses are discussed as the central piece of evidence and other studies of individual cohorts are discussed in relation to these meta-analyses.

In general, participants included in studies represented a broad range of primary prevention populations to whom CVD risk assessment is applicable. Most individuals were recruited as part of population-based cohort studies; additional populations were from cohorts derived from treatment trials72,77,79,83,86 and 1 registry of individuals who received CAC scores.44 Some analyses excluded people with known diabetes or already taking statins.25,38,53,57,65,69,71,76,88

Most included studies have important limitations in their applicability to current clinical practice. Fourteen studies (32.6%) used the published coefficients of publicly available Framingham Risk Score or Pooled Cohort Equations models. Moreover, the “low,” “intermediate,” and “high” risk strata used in many studies are not consistent with those used in current practice, where a single threshold, such as 7.5% or 10% 10-year risk, is used for decision making to initiate preventive therapy.

Calibration

Limited data are available to inform whether the addition of the ABI (5 studies), hsCRP level (9 studies), or CAC score (8 studies) can improve agreement between predicted and observed events of risk assessment models. Across all 3 nontraditional risk factors, less than half of studies reported a measure of calibration (Table 1). Of these, only 224,25 reported the preferred measures of calibration plots and observed to expected ratios, and these are reported only for studies evaluating hsCRP level. These studies showed that the addition of hsCRP can improve calibration in some risk groups but worsen it in others; however, the small number of events—particularly in lower-risk groups—precludes definitive conclusions.

Discrimination
Ankle-Brachial Index

A large body of literature (10 studies; n = 79 583) consistently shows that the addition of the ABI to risk prediction models generally results in no to small improvement in a model’s ability to distinguish between individuals who will and will not have cardiovascular events (C statistic change of −0.006 to 0.036); however, improvements in discrimination can be large (eg, change of 0.112) when base models perform poorly (eTable 4 in the Supplement). Most models are not directly applicable to current practice because most are model development studies rather than published coefficient analyses, and evidence for the addition of the ABI to the Pooled Cohort Equations is sparse. The 1 study evaluating the Pooled Cohort Equations, a 2016 analysis of the MESA study38 showed moderate base model discrimination of 0.74 and no statistically significant change in discrimination of 0.01 when the ABI was added. The central piece of evidence for the addition of the ABI to the Framingham Risk Score is the individual patient data meta-analysis from the ABI Collaboration, which includes 11 421 individuals in the external validation data set. Both published coefficient and model development analyses were conducted, and results were stratified by sex.39 For published coefficient analyses, the base model performed poorly in women, with a C statistic of 0.578 (95% CI, 0.492 to 0.661), and the addition of the ABI to the model showed a large improvement in discrimination, with a change of 0.112. The base model for men showed better discrimination, with a C statistic of 0.672 (95% CI, 0.599 to 0.737) and a smaller change of 0.013 when the ABI was added to the model. In the individual patient data meta-analysis model development analyses, base model performance was better because the model was fit to the studied population. The base model C statistics were 0.788 (95% CI, 0.709 to 0.850) for women and 0.683 (95% CI, 0.611 to 0.748) for men. The resulting changes in discrimination were very small when the ABI was added to the model: 0.003 for women and 0.007 for men. Results for cohorts not included in the individual patient data meta-analysis were consistent with those included in the analysis.38,52,56,81,88

hsCRP Level

The body of evidence for hsCRP level is much larger (25 studies; n = 265 704) than for ABI and CAC but demonstrates less consistent findings (eTable 5 in the Supplement). The hsCRP literature is dominated by model development studies, which add the nontraditional risk factor to base models including Framingham Risk Score variables. The 1 study evaluating the addition of hsCRP to the Pooled Cohort Equations, the 2016 MESA analysis,38 showed moderate base model discrimination of 0.74 and no change in discrimination when hsCRP level was added to the model. The central piece of evidence for the addition of hsCRP level to the Framingham Risk Score is the Emerging Risk Factors Collaboration individual patient data meta-analysis, which involved 166 596 participants and 13 568 hard CVD events; this was a model development study with a moderate base model discrimination of 0.714.40 This analysis showed that the addition of hsCRP level increased discrimination by 0.0039 (95% CI, 0.0028 to 0.0050) for predicting a composite of fatal and nonfatal myocardial infarction and cerebrovascular accident. Exploratory subgroup analyses showed a very small, statistically significant improvement in men and no change in women; the P value for heterogeneity was less than .001. Results were inconsistent for studies not included in the individual patient data meta-analysis, ranging from a worsening of discrimination with the addition of hsCRP level24 to a small improvement59,71; however, estimates of even small improvement likely represent an upper bound because of study design limitations.

CAC Score

The evidence (18 studies; n = 60 486) shows that the addition of CAC score to traditional risk assessment models results in the largest improvement in discrimination of all nontraditional risk factors evaluated, with change in the C statistic ranging from 0.018 to 0.144 (eTable 6 in the Supplement); however, there is no available individual patient data meta-analysis allowing for robust sex-stratified analyses. Two studies evaluated published coefficient Pooled Cohort Equations models; however, both were analyses of the same cohort.38,51 The MESA analysis by Yeboah et al,38 which had moderate base model discrimination of 0.74, found a very small to small, statistically significant improvement of 0.02 with the addition of CAC score to the model. The MESA analysis by Fudim et al,51 which presented only sex-stratified results, demonstrated similar findings, although statistically significant only for men. Results for published coefficient Framingham Risk Score base models were statistically significant and showed a slightly higher magnitude of change. For example, the 2016 MESA analysis found a statistically significant improvement in discrimination of 0.04.38 Results for model development studies were generally consistent with results from published coefficient analyses.

Reclassification
Ankle-Brachial Index

A large but heterogeneous body of literature (9 studies; n = 46 979) showed that the addition of the ABI to risk prediction models can improve the appropriate reclassification of individuals into clinically meaningful risk strata; NRIs are at best less than 0.1 and are usually much smaller and often nonsignificant (Table 2). There was considerable variation in the definitions of risk strata used across studies, and only 1 study used the Pooled Cohort Equations, with a treatment threshold of 7.5% 10-year risk.38 In this study, the total NRI was 0.017 (95% CI, −0.031 to 0.058). The individual patient data meta-analysis published coefficient Framingham Risk Score analysis used 10-year risk strata of less than 10%, 10% to 19%, and 20% or greater.39 The NRIs based on these 3 strata are no longer as relevant because current treatment thresholds to initiate preventive therapies have been lowered to 7.5% or 10%, and reported NRI will capture movement between the middle and upper categories. Consistent with findings for discrimination, the addition of the ABI showed a larger improvement in women than in men. In contrast to the findings for the Pooled Cohort Equations, this improvement was statistically significant for the Framingham Risk Score base model: 0.096 (95% CI, 0.061 to 0.164) for women and 0.043 (95% CI, 0.008 to 0.076) for men. Event NRIs (the appropriate upward reclassification of individuals having CVD events) were larger than nonevent NRIs (the appropriate downward reclassification of individuals not having CVD events). However, the nonevent NRI for women was negative and statistically significant, meaning that more women without events were inappropriately reclassified to a higher risk stratum than appropriately reclassified to a lower risk stratum. Consideration of the event and nonevent NRI separately is important because the total NRI does not weight for the proportion of individuals who have or do not have events; in the included primary prevention populations, it is much more common not to have an event. Cohorts not included in the individual patient data meta-analysis showed a similar magnitude of reclassification with the addition of the ABI, with mixed statistical significance.

hsCRP Level

The body of evidence for the addition of hsCRP level to risk prediction models (15 studies; n = 115 686) shows inconsistent evidence for an improvement in the appropriate reclassification of individuals into clinically meaningful risk strata; evidence from the largest analysis suggests an overall NRI of no greater than 0.02 (Table 2). As with the evidence for the ABI, risk strata were defined variably among studies, with few using thresholds applicable to current practice. Additionally, many studies used 3 risk strata, which is no longer as relevant to current clinical practice. Only 1 study used the Pooled Cohort Equations with treatment threshold of 7.5% 10-year risk, and this was the same study that evaluated the addition of the ABI to the Pooled Cohort Equations.38 In this study, the total NRI was 0.024 (95% CI, −0.015 to 0.067). The IPD meta-analysis model development study (n = 72 574) used 10-year risk strata of less than 10%, 10% to 19%, and 20% or greater.40 The total NRI and event NRI were 0.0152 (95% CI, 0.0078 to 0.0227) and 0.0146 (95% CI, 0.0073 to 0.0219), respectively. However, the nonevent NRI was much smaller and nonsignificant (0.0006 [95% CI, −0.0009 to 0.0022]); about 92% of individuals in the individual participant data meta-analysis did not have events. Similar to results for discrimination, exploratory subgroup analyses by sex show that reclassification was greater for men than for women, but reclassification results were not significant for either sex individually. Two other studies offer some confirmatory evidence about a larger effect for men, although reporting inconsistencies preclude definitive conclusions.60,76 The results of cohorts not included in the individual patient data meta-analysis were also inconsistent with respect to magnitude and statistical significance. Predicted outcome, definitions of risk strata, and case mix did not explain differences among studies; however, these comparisons were limited by several concurrent sources of heterogeneity.

CAC Score

Of the 3 nontraditional risk factors evaluated in this review, the addition of CAC score to traditional risk assessment models has the strongest effect on appropriate reclassification (15 studies; n = 58 289). Results range from 0.084 (95% CI, 0.024 to 0.196) to 0.35 (95% CI, 0.11 to 0.58) and are consistently statistically significant; however, the nonevent NRI is negative in several studies (Table 2). In the absence of an individual patient data meta-analysis for CAC score, few reliable data are available to inform whether effect modification exists by sex. As for other nontraditional risk factors, studies used a variety of definitions of risk strata, which precludes definitive comparisons. The most applicable evidence is from the published coefficient Pooled Cohort Equations analysis,38 which reported an overall NRI of 0.119 (95% CI, 0.080 to 0.256). Numerically, the overall NRI was driven by the event NRI of 0.178 (0.080 to 0.256). The nonevent NRI was negative (−0.059 [95% CI, −0.075 to 0.030]), meaning that individuals not having events were inappropriately reclassified upward into higher risk and above a treatment threshold. While not statistically significant in this study, the negative nonevent NRI is statistically significant in other studies.55,58,68

Clinical Interpretation of Reclassification Across Nontraditional Risk Factors

The clinical significance of the NRI, event NRI, and nonevent NRI can be understood by illustrating the absolute number of people appropriately and inappropriately reclassified in terms of current treatment thresholds. Selected examples from included studies are provided in Table 3. In the example for the Pooled Cohort Equations, 76 people having CVD events were appropriately reclassified upward when CAC score was added to risk assessment, and 19 people who had events were inappropriately reclassified downward who had a CVD event—a net improvement of 57 individuals among the 320 having events, about 18 events per 100 people (reported event NRI of 0.178 [95% CI, 0.080 to 0.256]). However, in the primary prevention populations to which CVD risk assessment with the Pooled Cohort Equations or Framingham Risk Score applies, the majority of people will not experience a CVD event. With the addition of CAC score to risk assessment, 202 people not having events are appropriately reclassified downward, but 496 people are inappropriately reclassified upward—on net, a worsening of reclassification of 294 individuals of 4865 not having events—or about 6 events per 100 people (reported nonevent NRI of −0.059 [ 95% CI, −0.075 to 0.030]). Therefore, the NRI of 0.119 [95% CI, 0.080 to 0.256] does not convey that for CAC score a sizeable proportion of individuals who are not having events will now be considered for treatment. The addition of the ABI to the Framingham Risk Score in women showed a similar pattern.

Harms of Risk Assessment

Key Question 3. What are the harms of nontraditional risk factor assessment?

Evidence for the harms of nontraditional risk factor assessment was limited to 8 studies evaluating CAC score; no eligible studies evaluated the potential harms of the ABI or hsCRP level. Four studies reported radiation exposure for CT imaging to obtain CAC score,43,55,62,75 and 5 studies reported other potential adverse events from CAC score measurement such as psychological outcomes, adverse cardiovascular events, and health care utilization.45,66,67,75,80

Overall, the radiation exposure or effective radiation dose per CT examination is low: 2 mSv or less. Based on 2 studies (n = 1619)—a small randomized clinical trial (RCT) and a subsample from a population-based cohort—risk assessment with CAC score does not appear to cause short-term mental distress.66,67 Additionally, 2 studies (n = 11 364) using administrative claims data showed that risk assessment with CAC score did not appear to paradoxically increase CVD events.45,80 However, a small body of evidence with study design and applicability limitations shows mixed findings for the effect of CAC score on downstream health care utilization.45,75,80 Both the EISNER RCT and 1 cohort using administrative data found no statistically significant increase in cardiac imaging or revascularization procedures in patients receiving vs not receiving CAC screening in follow-up of 4 years or 6 months, respectively.45,75 However, 1 study using Medicare claims data found a greater number of subsequent cardiac imaging tests and revascularization in asymptomatic people who received CAC screening compared with those receiving hsCRP or lipid screening.80

Benefits and Harms of Risk Factor–Guided Treatment

Key Question 4. Does treatment guided by nontraditional risk factors, in addition to traditional risk factors, lead to reduced incidence of cardiovascular events, mortality, or both?

No trials compared treatment guided by nontraditional risk factors in addition to traditional risk factors vs no treatment or usual care. In the absence of this evidence, studies in which preventive therapies were guided by nontraditional risk factors alone, without formal traditional risk assessment, were included. Four such RCTs reported the outcomes of CVD events, mortality, or both.41,42,50,64,73 Two of these trials evaluated aspirin in individuals with an abnormal ABI, 1 trial evaluated high-intensity statins in those with an abnormal hsCRP level, and 1 trial evaluated moderate-intensity statins in those with an abnormal CAC score.

Two good-quality RCTs (n = 4626) in asymptomatic adults with an abnormal ABI—including 1 trial exclusively in participants with diabetes—did not find any statistically significant benefit for aspirin (100 mg daily) on reducing CVD outcomes or all-cause mortality compared with placebo after approximately 7 to 8 years of follow-up.42,50 However, neither trial used the conventional 0.90 threshold for an abnormal ABI; 1 defined an abnormal ABI as 0.95 or less and the other as 0.99 or less. One good-quality RCT (n = 17 802) in asymptomatic people with elevated hsCRP level (≥2.0 mg/L [19.05 nmol/L]) but normal low-density lipoprotein cholesterol level (<130 mg/dL [3.37 mmol/L]) found a significant relative reduction in CVD events for rosuvastatin (20 mg daily) compared with placebo (hazard ratio, 0.56 [95% CI, 0.46 to 0.69]) at approximately 2 years.73 One fair-quality trial (n = 1005) in asymptomatic people with elevated CAC score, defined as 80th percentile or greater for age and sex, and with low-density lipoprotein cholesterol levels less than 175 mg/dL (4.5 mmol/L), did not find any statistically significant benefit for atorvastatin (20 mg daily) on reducing CVD outcomes compared with placebo after a mean of 4.3 years of follow-up41; however, that study had a lower than expected number of events and was terminated early because of futility.

Key Question 5. What are the harms of treatment guided by nontraditional risk factors?

Three of the 4 RCTs included for KQ4 reported harms of aspirin or statins guided by nontraditional risk factor assessment.42,50,73 No other studies evaluating harms met inclusion criteria. Neither aspirin trial (n = 4626) found a statistically significant increase in hemorrhagic cerebrovascular accident after approximately 7 to 8 years of follow-up, although analyses are limited by a rare event rate.42,50 In the 1 trial reporting major bleeding, the association with low-dose aspirin approached statistical significance (hazard ratio, 1.71 [95% CI, 0.99 to 2.97]).50 The trial evaluating high-intensity statins in adults with elevated hsCRP levels (n = 17 802) found evidence of an increased incidence of diabetes in the treatment group compared with placebo after approximately 2 years (3.0% vs 2.4%, P = .01); however, it did not find evidence of increases in other serious adverse effects, including hemorrhagic cerebrovascular accident or myopathic events, for high-intensity statin therapy compared with placebo.

Discussion

Direct evidence from adequately powered clinical trials evaluating the incremental effect of the ABI, hsCRP level, or CAC score to improve health outcomes is lacking (Table 4; eTable 7 in the Supplement). This remains unchanged from the 2009 and 2013 USPSTF recommendations, which concluded that the evidence was insufficient to assess the benefits and harms of using nontraditional risk factors for cardiovascular risk assessment (I statements).14,15

Advances in the evaluation of risk prediction literature, as well as an accrual of new evidence since the previous recommendations, has allowed for more detailed analyses of risk prediction studies that offer indirect evidence for the use of nontraditional risk factors (Figure 1). A substantial body of evidence evaluates the ability of the ABI, hsCRP level, or CAC score to improve on the discrimination and reclassification of traditional risk assessment, but most of this evidence is not readily applicable to current clinical practice because it does not use published coefficients, rarely evaluates the addition of nontraditional risk factors in the context of the Pooled Cohort Equations, and seldom uses current risk thresholds for aiding treatment decisions. Regardless, CAC score shows the largest potential for improvement in discrimination and reclassification. Improvements for the ABI are larger when base models perform poorly, such as the Framingham Risk Score in women. Results for hsCRP level are inconsistent. Based on selected included studies, examination of the absolute number of people appropriately and inappropriately reclassified using current treatment thresholds (ie, 7.5% or 10% 10-year CVD risk) demonstrates that even with favorable NRI for CAC score and ABI, more people will be incorrectly reclassified to a higher risk category, thus needing treatment, vs correctly reclassified as needing or not needing treatment.

A 2015 analysis of data from the MESA study91 found inconsistent results. That analysis, while valid, was designed to answer a different question about risk stratification and was excluded from this review because of its different study design. Multiple methodologic differences between this analysis and the included MESA analysis38 (ie, differences in definitions of risk strata, differences in risk classification of individuals with diabetes, differences in comparisons of CAC score alone vs CAC score plus Pooled Cohort Equations, and recalibration of the Pooled Cohort Equations) prohibit direct comparisons between the 2 studies’ findings. Therefore, while CAC score appears to be the most informative nontraditional risk factor, there remains uncertainty around the overall clinical effect of the inappropriate upward reclassification of individuals and of resultant downstream testing, including from incidental findings. While experts have argued that large-scale clinical trials evaluating the effectiveness of CAC screening on patient health outcomes are not feasible,92 these studies may be necessary to address this remaining clinical uncertainty.

Despite the recent proliferation of risk prediction studies, few clarify whether the addition of the ABI, hsCRP level, or CAC score can improve the calibration of traditional risk assessment. The lack of reporting of preferred measures such as calibration plots and observed to expected ratios limits the ability to interpret the clinical meaning of improvements in calibration. Even when preferred measures are reported, confidence in their results is usually limited by small numbers of events, especially among low-risk groups. As treatment thresholds have decreased over time, the implications of calibration in lower-risk groups have become especially important. Historically, the performance of risk prediction models has been focused on discrimination,93 so the sparse reporting of calibration measures is not surprising and is consistent with the findings of other systematic reviews.94 The focus on discrimination is incomplete because the C statistic is a rank order statistic; therefore a model can discriminate well but still systematically underestimate or overestimate risk.95

Limitations

The evidence report has several limitations. The review focused on just 3 nontraditional risk factors—the ABI, hsCRP level, and CAC score. Many more have been identified in the literature; however, these risk factors were selected because other synthesized literature identified them as having the greatest clinical potential.21,27 Additionally, the predictive value of traditional risk factors such as levels of total or high-density lipoprotein cholesterol was taken as given, but some literature suggests that these, too, might be very small to small when assessed in terms of the C statistic.40 Given the large volume of studies included for KQ2, some explicit exclusions were made so as to focus on the most clinically relevant analyses. Additionally, comparisons across studies are difficult because of the heterogeneity of base models, model type, predicted outcome, and definitions of risk strata as well as limited reporting of confidence intervals, statistical significance, and separate reporting of event and nonevent NRIs. In addition, 1 of the risk assessment analysis stratifications was whether a study preserved published coefficients or developed a new model; because methods were often only briefly described, the categorization represents a best guess for many studies, and, in cases of uncertainty, the study was categorized as model development.

Conclusions

There are insufficient adequately powered clinical trials evaluating the incremental effect of the ABI, hsCRP level, or CAC score in risk assessment and initiation of preventive therapy. Furthermore, the clinical meaning of improvements in measures of calibration, discrimination, and reclassification risk prediction studies is uncertain.

Back to top
Article Information

Corresponding Author: Jennifer S. Lin, MD, MCR, Kaiser Permanente Research Affiliates Evidence-based Practice Center, Center for Health Research, Kaiser Permanente Northwest, 3800 N Interstate Ave, Portland, OR 97227 (jennifer.s.lin@kpchr.org).

Accepted for Publication: March 19, 2018.

Published Online: July 10, 2018. doi:10.1001/jama.2018.4242

Author Contributions: Dr Lin had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Lin, Evans, Coppola.

Acquisition, analysis, or interpretation of data: Lin, Evans, Johnson, Redmond, Smith.

Drafting of the manuscript: Lin, Evans, Redmond, Coppola.

Critical revision of the manuscript for important intellectual content: Lin, Evans, Johnson, Smith.

Statistical analysis: Johnson, Redmond, Smith.

Obtained funding: Lin.

Administrative, technical, or material support: Evans, Coppola.

Supervision: Lin, Evans.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: This research was funded under contract HHSA290201500007I, Task Order 2, from the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services, under a contract to support the USPSTF.

Role of the Funder/Sponsor: Investigators worked with USPSTF members and AHRQ staff to develop the scope, analytic framework, and key questions for this review. AHRQ had no role in study selection, quality assessment, or synthesis. AHRQ staff provided project oversight, reviewed the report to ensure that the analysis met methodological standards, and distributed the draft for peer review. Otherwise, AHRQ had no role in the conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, and approval of the manuscript findings. The opinions expressed in this document are those of the authors and do not reflect the official position of AHRQ or the US Department of Health and Human Services.

Additional Contributions: We gratefully acknowledge the following individuals for their contributions to this project: Todd Hannon, MLS, Janelle Guirguis-Blake, MD, and Katherine Essick, BS (Kaiser Permanente Center for Health Research); Robert Platt, PhD (McGill University); Justin Mills, MD, MPH (Agency for Healthcare Research and Quality [AHRQ]); Elisabeth Kato, MD, MRP (formerly at AHRQ); and current and former members of the US Preventive Services Task Force (USPSTF) who contributed to topic deliberations. USPSTF members, peer reviewers, and federal partner reviewers did not receive financial compensation for their contributions.

Additional Information: A draft version of this evidence report underwent external peer review from 5 content experts (Nancy Cook, ScD, Harvard T. H. Chan School of Public Health; Donald Lloyd-Jones, MD, Northwestern University Feinberg School of Medicine; Gerry Fowkes, MD, Usher Institute of Population Health Sciences and Informatics, University of Edinburgh; Matthew Budoff, MD, David Geffen School of Medicine at UCLA; Bruce Psaty, MD, PhD, University of Washington) and 1 federal partner (the National Institutes of Health). Comments from reviewers were presented to the USPSTF during its deliberation of the evidence and were considered in preparing the final evidence review.

Editorial Disclaimer: This evidence report is presented as a document in support of the accompanying USPSTF Recommendation Statement. It did not undergo additional peer review after submission to JAMA.

References
1.
Benjamin  EJ, Blaha  MJ, Chiuve  SE,  et al.  Heart Disease and Stroke Statistics—2017 Update: a report from the American Heart Association.  Circulation. 2017;135(10):e146-e603. doi:10.1161/CIR.0000000000000485Google ScholarCrossref
2.
Rao Kondapally Seshasai  S, Kaptoge  S, Thompson  A,  et al; Emerging Risk Factors Collaboration.  Diabetes mellitus, fasting glucose, and risk of cause-specific death.  N Engl J Med. 2011;364(9):829-841. doi:10.1056/NEJMoa1008862Google ScholarCrossref
3.
Lewington  S, Clarke  R, Qizilbash  N, Peto  R, Collins  R; Prospective Studies Collaboration.  Age-specific relevance of usual blood pressure to vascular mortality: a meta-analysis of individual data for one million adults in 61 prospective studies.  Lancet. 2002;360(9349):1903-1913. doi:10.1016/S0140-6736(02)11911-8Google ScholarCrossref
4.
Lewington  S, Whitlock  G, Clarke  R,  et al; Prospective Studies Collaboration.  Blood cholesterol and vascular mortality by age, sex, and blood pressure: a meta-analysis of individual data from 61 prospective studies with 55,000 vascular deaths.  Lancet. 2007;370(9602):1829-1839. doi:10.1016/S0140-6736(07)61778-4Google ScholarCrossref
5.
Jha  P, Ramasundarahettige  C, Landsman  V,  et al.  21st-century hazards of smoking and benefits of cessation in the United States.  N Engl J Med. 2013;368(4):341-350. doi:10.1056/NEJMsa1211128Google ScholarCrossref
6.
Karmali  KN, Persell  SD, Perel  P, Lloyd-Jones  DM, Berendsen  MA, Huffman  MD.  Risk scoring for the primary prevention of cardiovascular disease.  Cochrane Database Syst Rev. 2017;3:CD006887.Google Scholar
7.
Box  GEP.  Robustness in the Strategy of Scientific Model Building. Fort Belvoir, VA: Defense Technical Information Center; 1979. doi:10.1016/B978-0-12-438150-6.50018-2Crossref
8.
US Preventive Services Task Force.  Statin use for the primary prevention of cardiovascular disease in adults: US Preventive Services Task Force recommendation statement.  JAMA. 2016;316(19):1997-2007. doi:10.1001/jama.2016.15450Google ScholarCrossref
9.
Stone  NJ, Robinson  JG, Lichtenstein  AH,  et al.  2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  J Am Coll Cardiol. 2014;63(25, pt B):2889-2934. doi:10.1016/j.jacc.2013.11.002Google ScholarCrossref
10.
US Preventive Services Task Force.  Aspirin use for the primary prevention of cardiovascular disease and colorectal cancer: US Preventive Services Task Force recommendation statement.  Ann Intern Med. 2016;164(12):836-845. doi:10.7326/M16-0577Google ScholarCrossref
11.
Helfand  M, Buckley  DI, Freeman  M,  et al.  Emerging risk factors for coronary heart disease: a summary of systematic reviews conducted for the U.S. Preventive Services Task Force.  Ann Intern Med. 2009;151(7):496-507. doi:10.7326/0003-4819-151-7-200910060-00010Google ScholarCrossref
12.
Lin  JS, Olson  CM, Johnson  ES, Whitlock  EP.  The ankle-brachial index for peripheral artery disease screening and cardiovascular disease prediction among asymptomatic adults: a systematic evidence review for the U.S. Preventive Services Task Force.  Ann Intern Med. 2013;159(5):333-341. doi:10.7326/0003-4819-159-5-201309030-00007Google ScholarCrossref
13.
Fowkes  FG, Murray  GD, Butcher  I,  et al; Ankle Brachial Index Collaboration.  Ankle brachial index combined with Framingham Risk Score to predict cardiovascular events and mortality: a meta-analysis.  JAMA. 2008;300(2):197-208. doi:10.1001/jama.300.2.197Google ScholarCrossref
14.
U.S. Preventive Services Task Force.  Using nontraditional risk factors in coronary heart disease risk assessment: U.S. Preventive Services Task Force recommendation statement.  Ann Intern Med. 2009;151(7):474-482. doi:10.7326/0003-4819-151-7-200910060-00008Google ScholarCrossref
15.
U.S. Preventive Services Task Force.  Screening for peripheral artery disease and cardiovascular disease risk assessment with the ankle-brachial index in adults: U.S. Preventive Services Task Force recommendation statement.  Ann Intern Med. 2013;159(5):342-348. doi:10.7326/0003-4819-159-5-201309030-00008Google ScholarCrossref
16.
US Preventive Services Task Force.  US Preventive Services Task Force Procedure Manual. Rockville, MD: US Preventive Services Task Force; 2015.
17.
Buckley  DI, Fu  R, Freeman  M, Rogers  K, Helfand  M.  C-reactive protein as a risk factor for coronary heart disease: a systematic review and meta-analyses for the U.S. Preventive Services Task Force.  Ann Intern Med. 2009;151(7):483-495. doi:10.7326/0003-4819-151-7-200910060-00009Google ScholarCrossref
18.
Alahdab  F, Wang  AT, Elraiyah  TA,  et al.  A systematic review for the screening for peripheral arterial disease in asymptomatic patients.  J Vasc Surg. 2015;61(3)(suppl):42S-53S. doi:10.1016/j.jvs.2014.12.008Google ScholarCrossref
19.
Kramer  CK, Zinman  B, Gross  JL,  et al.  Coronary artery calcium score prediction of all cause mortality and cardiovascular events in people with type 2 diabetes: systematic review and meta-analysis.  BMJ. 2013;346:f1654. doi:10.1136/bmj.f1654Google ScholarCrossref
20.
Peters  SA, Bakker  M, den Ruijter  HM, Bots  ML.  Added value of CAC in risk stratification for cardiovascular events: a systematic review.  Eur J Clin Invest. 2012;42(1):110-116. doi:10.1111/j.1365-2362.2011.02555.xGoogle ScholarCrossref
21.
Peters  SA, den Ruijter  HM, Bots  ML, Moons  KG.  Improvements in risk stratification for the occurrence of cardiovascular disease by imaging subclinical atherosclerosis: a systematic review.  Heart. 2012;98(3):177-184. doi:10.1136/heartjnl-2011-300747Google ScholarCrossref
22.
Saito  I, Maruyama  K, Eguchi  E.  C-reactive protein and cardiovascular disease in East Asians: a systematic review.  Clin Med Insights Cardiol. 2014;8(suppl 3):35-42.Google Scholar
23.
Schnell-Inderst  P, Schwarzer  R, Gohler  A,  et al.  Prognostic value, clinical effectiveness, and cost-effectiveness of high-sensitivity C-reactive protein as a marker for major cardiac events in asymptomatic individuals: a health technology assessment report.  Int J Technol Assess Health Care. 2010;26(1):30-39. doi:10.1017/S0266462309990870Google ScholarCrossref
24.
Shah  T, Casas  JP, Cooper  JA,  et al.  Critical appraisal of CRP measurement for the prediction of coronary heart disease events: new data and systematic review of 31 prospective cohorts.  Int J Epidemiol. 2009;38(1):217-231. doi:10.1093/ije/dyn217Google ScholarCrossref
25.
Cook  NR, Buring  JE, Ridker  PM.  The effect of including C-reactive protein in cardiovascular risk prediction models for women.  Ann Intern Med. 2006;145(1):21-29. doi:10.7326/0003-4819-145-1-200607040-00128Google ScholarCrossref
26.
Kerr  KF, Wang  Z, Janes  H, McClelland  RL, Psaty  BM, Pepe  MS.  Net reclassification indices for evaluating risk prediction instruments: a critical review.  Epidemiology. 2014;25(1):114-121. doi:10.1097/EDE.0000000000000018Google ScholarCrossref
27.
Goff  DC  Jr, Lloyd-Jones  DM, Bennett  G,  et al.  2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.  J Am Coll Cardiol. 2014;63(25, pt B):2935-2959. doi:10.1016/j.jacc.2013.11.005Google ScholarCrossref
28.
Anderson  KM, Odell  PM, Wilson  PW, Kannel  WB.  Cardiovascular disease risk profiles.  Am Heart J. 1991;121(1, pt 2):293-298. doi:10.1016/0002-8703(91)90861-BGoogle ScholarCrossref
29.
Anderson  KM, Wilson  PW, Odell  PM, Kannel  WB.  An updated coronary risk profile: a statement for health professionals.  Circulation. 1991;83(1):356-362. doi:10.1161/01.CIR.83.1.356Google ScholarCrossref
30.
Wilson  PW, D’Agostino  RB, Levy  D, Belanger  AM, Silbershatz  H, Kannel  WB.  Prediction of coronary heart disease using risk factor categories.  Circulation. 1998;97(18):1837-1847. doi:10.1161/01.CIR.97.18.1837Google ScholarCrossref
31.
National Cholesterol Education Program.  Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report.  Circulation. 2002;106(25):3143-3421.Google Scholar
32.
D’Agostino  RB  Sr, Vasan  RS, Pencina  MJ,  et al.  General cardiovascular risk profile for use in primary care: the Framingham Heart Study.  Circulation. 2008;117(6):743-753. doi:10.1161/CIRCULATIONAHA.107.699579Google ScholarCrossref
33.
Wells  GA, Shea  B, O’Connell  D,  et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute website. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. Published 2014. Accessed April 8, 2015.
34.
Moons  KG, de Groot  JA, Bouwmeester  W,  et al.  Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.  PLoS Med. 2014;11(10):e1001744. doi:10.1371/journal.pmed.1001744Google ScholarCrossref
35.
Raggi  P, Cooil  B, Callister  TQ.  Use of electron beam tomography data to develop models for prediction of hard coronary events.  Am Heart J. 2001;141(3):375-382. doi:10.1067/mhj.2001.113220Google ScholarCrossref
36.
Kondos  GT, Hoff  JA, Sevrukov  A,  et al.  Electron-beam tomography coronary artery calcium and cardiac events: a 37-month follow-up of 5635 initially asymptomatic low- to intermediate-risk adults.  Circulation. 2003;107(20):2571-2576. doi:10.1161/01.CIR.0000068341.61180.55Google ScholarCrossref
37.
Apfel  CC, Kranke  P, Greim  CA, Roewer  N.  What can be expected from risk scores for predicting postoperative nausea and vomiting?  Br J Anaesth. 2001;86(6):822-827. doi:10.1093/bja/86.6.822Google ScholarCrossref
38.
Yeboah  J, Young  R, McClelland  RL,  et al.  Utility of nontraditional risk markers in atherosclerotic cardiovascular disease risk assessment.  J Am Coll Cardiol. 2016;67(2):139-147. doi:10.1016/j.jacc.2015.10.058Google ScholarCrossref
39.
Fowkes  FG, Murray  GD, Butcher  I,  et al.  Development and validation of an ankle brachial index risk model for the prediction of cardiovascular events.  Eur J Prev Cardiol. 2014;21(3):310-320. doi:10.1177/2047487313516564Google ScholarCrossref
40.
Kaptoge  S, Di Angelantonio  E, Pennells  L,  et al; Emerging Risk Factors Collaboration.  C-reactive protein, fibrinogen, and cardiovascular disease prediction.  N Engl J Med. 2012;367(14):1310-1320. doi:10.1056/NEJMoa1107477Google ScholarCrossref
41.
Arad  Y, Spadaro  LA, Roth  M, Newstein  D, Guerci  AD.  Treatment of asymptomatic adults with elevated coronary calcium scores with atorvastatin, vitamin C, and vitamin E: the St. Francis Heart Study randomized clinical trial.  J Am Coll Cardiol. 2005;46(1):166-172. doi:10.1016/j.jacc.2005.02.089Google ScholarCrossref
42.
Belch  J, MacCuish  A, Campbell  I,  et al.  The Prevention of Progression of Arterial Disease and Diabetes (POPADAD) trial: factorial randomised placebo controlled trial of aspirin and antioxidants in patients with diabetes and asymptomatic peripheral arterial disease.  BMJ. 2008;337:a1840. doi:10.1136/bmj.a1840Google ScholarCrossref
43.
Bos  D, Leening  MJ, Kavousi  M,  et al.  Comparison of atherosclerotic calcification in major vessel beds on the risk of all-cause and cause-specific mortality: the Rotterdam Study  [published online December 11, 2015].  Circ Cardiovasc Imaging. doi:10.1161/CIRCIMAGING.115.003843Google Scholar
44.
Chang  SM, Nabi  F, Xu  J,  et al.  Value of CACS compared with ETT and myocardial perfusion imaging for predicting long-term cardiac outcome in asymptomatic and symptomatic patients at low risk for coronary disease: clinical implications in a multimodality imaging world.  JACC Cardiovasc Imaging. 2015;8(2):134-144. doi:10.1016/j.jcmg.2014.11.008Google ScholarCrossref
45.
Chi  WC, Sylwestrzak  G, Barron  J, Kasravi  B, Power  T, Redberg  R.  Does CAC testing alter downstream treatment patterns for cardiovascular disease?  Am J Manag Care. 2014;20(8):e330-e339.Google Scholar
46.
Danesh  J, Wheeler  JG, Hirschfield  GM,  et al.  C-reactive protein and other circulating markers of inflammation in the prediction of coronary heart disease.  N Engl J Med. 2004;350(14):1387-1397. doi:10.1056/NEJMoa032804Google ScholarCrossref
47.
Elias-Smale  SE, Proenca  RV, Koller  MT,  et al.  Coronary calcium score improves classification of coronary heart disease risk in the elderly: the Rotterdam study.  J Am Coll Cardiol. 2010;56(17):1407-1414. doi:10.1016/j.jacc.2010.06.029Google ScholarCrossref
48.
Erbel  R, Mohlenkamp  S, Moebus  S,  et al.  Coronary risk stratification, discrimination, and reclassification improvement based on quantification of subclinical coronary atherosclerosis: the Heinz Nixdorf Recall Study.  J Am Coll Cardiol. 2010;56(17):1397-1406. doi:10.1016/j.jacc.2010.06.030Google ScholarCrossref
49.
Folsom  AR, Chambless  LE, Ballantyne  CM,  et al.  An assessment of incremental coronary risk prediction using C-reactive protein and other novel risk markers: the atherosclerosis risk in communities study.  Arch Intern Med. 2006;166(13):1368-1373. doi:10.1001/archinte.166.13.1368Google ScholarCrossref
50.
Fowkes  FG, Price  JF, Stewart  MC,  et al.  Aspirin for prevention of cardiovascular events in a general population screened for a low ankle brachial index: a randomized controlled trial.  JAMA. 2010;303(9):841-848. doi:10.1001/jama.2010.221Google ScholarCrossref
51.
Fudim  M, Zalawadiya  S, Patel  DK, Egolum  UO, Afonso  L.  Data on coronary artery calcium score performance and cardiovascular risk reclassification across gender and ethnicities.  Data Brief. 2016;6:578-581. doi:10.1016/j.dib.2016.01.002Google ScholarCrossref
52.
Geisel  MH, Bauer  M, Hennig  F,  et al.  Comparison of coronary artery calcification, carotid intima-media thickness and ankle-brachial index for predicting 10-year incident cardiovascular events in the general population.  Eur Heart J. 2017;38(23):1815-1822. doi:10.1093/eurheartj/ehx120Google ScholarCrossref
53.
Greenland  P, LaBree  L, Azen  SP, Doherty  TM, Detrano  RC.  Coronary artery calcium score combined with Framingham score for risk prediction in asymptomatic individuals.  JAMA. 2004;291(2):210-215. doi:10.1001/jama.291.2.210Google ScholarCrossref
54.
Hamer  M, Chida  Y, Stamatakis  E.  Utility of C-reactive protein for cardiovascular risk stratification across three age groups in subjects without existing cardiovascular diseases.  Am J Cardiol. 2009;104(4):538-542. doi:10.1016/j.amjcard.2009.04.020Google ScholarCrossref
55.
Hoffmann  U, Massaro  JM, D’Agostino  RB  Sr, Kathiresan  S, Fox  CS, O’Donnell  CJ.  Cardiovascular event prediction and risk reclassification by coronary, aortic, and valvular calcification in the Framingham Heart Study.  J Am Heart Assoc. 2016;5(2):e003144. doi:10.1161/JAHA.115.003144Google ScholarCrossref
56.
Holewijn  S, den Heijer  M, Kiemeney  LA, Stalenhoef  AF, de Graaf  J.  Combining risk markers improves cardiovascular risk prediction in women.  Clin Sci (Lond). 2014;126(2):139-146. doi:10.1042/CS20130178Google ScholarCrossref
57.
Kavousi  M, Desai  CS, Ayers  C,  et al.  Prevalence and prognostic implications of coronary artery calcification in low-risk women: a meta-analysis.  JAMA. 2016;316(20):2126-2134. doi:10.1001/jama.2016.17020Google ScholarCrossref
58.
Kavousi  M, Elias-Smale  S, Rutten  JH,  et al.  Evaluation of newer risk markers for coronary heart disease risk classification: a cohort study.  Ann Intern Med. 2012;156(6):438-444. doi:10.7326/0003-4819-156-6-201203200-00006Google ScholarCrossref
59.
Koenig  W, Lowel  H, Baumert  J, Meisinger  C.  C-reactive protein modulates risk prediction based on the Framingham score: implications for future risk assessment: results from a large cohort study in southern Germany.  Circulation. 2004;109(11):1349-1353. doi:10.1161/01.CIR.0000120707.98922.E3Google ScholarCrossref
60.
Lyngbaek  S, Marott  JL, Sehestedt  T,  et al.  Cardiovascular risk prediction in the general population with use of suPAR, CRP, and Framingham Risk Score.  Int J Cardiol. 2013;167(6):2904-2911. doi:10.1016/j.ijcard.2012.07.018Google ScholarCrossref
61.
Malik  S, Budoff  MJ, Katz  R,  et al.  Impact of subclinical atherosclerosis on cardiovascular disease events in individuals with metabolic syndrome and diabetes: the multi-ethnic study of atherosclerosis.  Diabetes Care. 2011;34(10):2285-2290. doi:10.2337/dc11-0816Google ScholarCrossref
62.
Messenger  B, Li  D, Nasir  K, Carr  JJ, Blankstein  R, Budoff  MJ.  Coronary calcium scans and radiation exposure in the multi-ethnic study of atherosclerosis.  Int J Cardiovasc Imaging. 2016;32(3):525-529. doi:10.1007/s10554-015-0799-3Google ScholarCrossref
63.
Mohlenkamp  S, Lehmann  N, Moebus  S,  et al.  Quantification of coronary atherosclerosis and inflammation to predict coronary events and all-cause mortality.  J Am Coll Cardiol. 2011;57(13):1455-1464. doi:10.1016/j.jacc.2010.10.043Google ScholarCrossref
64.
Mora  S, Glynn  RJ, Hsia  J, MacFadyen  JG, Genest  J, Ridker  PM.  Statins for the primary prevention of cardiovascular events in women with elevated high-sensitivity C-reactive protein or dyslipidemia: results from the Justification for the Use of Statins in Prevention: An Intervention Trial Evaluating Rosuvastatin (JUPITER) and meta-analysis of women from primary prevention trials.  Circulation. 2012;121(9):1069-1077. doi:10.1161/CIRCULATIONAHA.109.906479Google ScholarCrossref
65.
Murphy  TP, Dhangana  R, Pencina  MJ, D’Agostino  RB  Sr.  Ankle-brachial index and cardiovascular risk prediction: an analysis of 11,594 individuals with 10-year follow-up.  Atherosclerosis. 2012;220(1):160-167. doi:10.1016/j.atherosclerosis.2011.10.037Google ScholarCrossref
66.
Nielsen  AD, Videbech  P, Gerke  O,  et al.  Population screening for coronary artery calcification does not increase mental distress and the use of psychoactive medication.  J Thorac Imaging. 2012;27(3):202-206. doi:10.1097/RTI.0b013e31824752bdGoogle ScholarCrossref
67.
O’Malley  PG, Feuerstein  IM, Taylor  AJ.  Impact of electron beam tomography, with or without case management, on motivation, behavioral change, and cardiovascular risk profile: a randomized controlled trial.  JAMA. 2003;289(17):2215-2223. doi:10.1001/jama.289.17.2215Google ScholarCrossref
68.
Polak  JF, Szklo  M, O’Leary  DH.  Carotid intima-media thickness score, positive coronary artery calcium score, and incident coronary heart disease: the Multi-Ethnic Study of Atherosclerosis.  J Am Heart Assoc. 2017;6(1):e004612. doi:10.1161/JAHA.116.004612Google ScholarCrossref
69.
Polonsky  TS, McClelland  RL, Jorgensen  NW,  et al.  Coronary artery calcium score and risk classification for coronary heart disease prediction.  JAMA. 2010;303(16):1610-1616. doi:10.1001/jama.2010.461Google ScholarCrossref
70.
Price  JF, Tzoulaki  I, Lee  AJ, Fowkes  FG.  Ankle brachial index and intima media thickness predict cardiovascular events similarly and increased prediction when combined.  J Clin Epidemiol. 2007;60(10):1067-1075. doi:10.1016/j.jclinepi.2007.01.011Google ScholarCrossref
71.
Rana  JS, Cote  M, Despres  JP,  et al.  Inflammatory biomarkers and the prediction of coronary events among people at intermediate risk: the EPIC-Norfolk prospective population study.  Heart. 2009;95(20):1682-1687. doi:10.1136/hrt.2009.170134Google ScholarCrossref
72.
Rana  JS, Gransar  H, Wong  ND,  et al.  Comparative value of coronary artery calcium and multiple blood biomarkers for prognostication of cardiovascular events.  Am J Cardiol. 2012;109(10):1449-1453. doi:10.1016/j.amjcard.2012.01.358Google ScholarCrossref
73.
Ridker  PM, Danielson  E, Fonseca  FA,  et al.  Rosuvastatin to prevent vascular events in men and women with elevated C-reactive protein.  N Engl J Med. 2008;359(21):2195-2207. doi:10.1056/NEJMoa0807646Google ScholarCrossref
74.
Rodondi  N, Marques-Vidal  P, Butler  J,  et al.  Markers of atherosclerosis and inflammation for prediction of coronary heart disease in older adults.  Am J Epidemiol. 2010;171(5):540-549. doi:10.1093/aje/kwp428Google ScholarCrossref
75.
Rozanski  A, Gransar  H, Shaw  LJ,  et al.  Impact of coronary artery calcium scanning on coronary risk factors and downstream testing: the EISNER (Early Identification of Subclinical Atherosclerosis by Noninvasive Imaging Research) prospective randomized trial.  J Am Coll Cardiol. 2011;57(15):1622-1632. doi:10.1016/j.jacc.2011.01.019Google ScholarCrossref
76.
Salim  A, Tai  ES, Tan  VY,  et al.  C-reactive protein and serum creatinine, but not haemoglobin A1c, are independent predictors of coronary heart disease risk in non-diabetic Chinese.  Eur J Prev Cardiol. 2016;12:1339-1349. doi:10.1177/2047487315626547Google ScholarCrossref
77.
Sattar  N, Murray  HM, McConnachie  A,  et al.  C-reactive protein and prediction of coronary heart disease and global vascular events in the Prospective Study of Pravastatin in the Elderly at Risk (PROSPER).  Circulation. 2007;115(8):981-989. doi:10.1161/CIRCULATIONAHA.106.643114Google ScholarCrossref
78.
Schneider  HJ, Wallaschofski  H, Volzke  H,  et al.  Incremental effects of endocrine and metabolic biomarkers and abdominal obesity on cardiovascular mortality prediction.  PLoS One. 2012;7(3):e33084. doi:10.1371/journal.pone.0033084Google ScholarCrossref
79.
Seven  E, Husemoen  LL, Sehested  TS,  et al.  Adipocytokines, C-reactive protein, and cardiovascular disease: a population-based prospective study.  PLoS One. 2015;10(6):e0128987. doi:10.1371/journal.pone.0128987Google ScholarCrossref
80.
Shreibati  JB, Baker  LC, McConnell  MV, Hlatky  MA.  Outcomes after coronary artery calcium and other cardiovascular biomarker testing among asymptomatic Medicare beneficiaries.  Circ Cardiovasc Imaging. 2014;7(4):655-662. doi:10.1161/CIRCIMAGING.113.001869Google ScholarCrossref
81.
Velescu  A, Clara  A, Penafiel  J,  et al.  Adding low ankle brachial index to classical risk factors improves the prediction of major cardiovascular events: the REGICOR study.  Atherosclerosis. 2015;241(2):357-363. doi:10.1016/j.atherosclerosis.2015.05.017Google ScholarCrossref
82.
Wannamethee  SG, Welsh  P, Lowe  GD,  et al.  N-terminal pro-brain natriuretic peptide is a more useful predictor of cardiovascular disease risk than C-reactive protein in older men with and without pre-existing cardiovascular disease.  J Am Coll Cardiol. 2011;58(1):56-64. doi:10.1016/j.jacc.2011.02.041Google ScholarCrossref
83.
Welsh  P, Doolin  O, Willeit  P,  et al.  N-terminal pro-B-type natriuretic peptide and the prediction of primary cardiovascular events: results from 15-year follow-up of WOSCOPS.  Eur Heart J. 2013;34(6):443-450. doi:10.1093/eurheartj/ehs239Google ScholarCrossref
84.
Wilson  PW, Nam  BH, Pencina  M, D’Agostino  RB  Sr, Benjamin  EJ, O’Donnell  CJ.  C-reactive protein and risk of cardiovascular disease in men and women from the Framingham Heart Study.  Arch Intern Med. 2005;165(21):2473-2478. doi:10.1001/archinte.165.21.2473Google ScholarCrossref
85.
Wilson  PW, Pencina  M, Jacques  P, Selhub  J, D’Agostino  R  Sr, O’Donnell  CJ.  C-reactive protein and reclassification of cardiovascular risk in the Framingham Heart Study.  Circ Cardiovasc Qual Outcomes. 2008;1(2):92-97. doi:10.1161/CIRCOUTCOMES.108.831198Google ScholarCrossref
86.
Wong  ND, Gransar  H, Shaw  L,  et al.  Thoracic aortic calcium versus coronary artery calcium for the prediction of coronary heart disease and cardiovascular disease events.  JACC Cardiovasc Imaging. 2009;2(3):319-326. doi:10.1016/j.jcmg.2008.12.010Google ScholarCrossref
87.
Yeboah  J, Erbel  R, Delaney  JC,  et al.  Development of a new diabetes risk prediction tool for incident coronary heart disease events: the Multi-Ethnic Study of Atherosclerosis and the Heinz Nixdorf Recall Study.  Atherosclerosis. 2014;236(2):411-417. doi:10.1016/j.atherosclerosis.2014.07.035Google ScholarCrossref
88.
Yeboah  J, McClelland  RL, Polonsky  TS,  et al.  Comparison of novel risk markers for improvement in cardiovascular risk assessment in intermediate-risk individuals.  JAMA. 2012;308(8):788-795. doi:10.1001/jama.2012.9624Google ScholarCrossref
89.
Zhou  QM, Zheng  Y, Cai  T.  Subgroup specific incremental value of new markers for risk prediction.  Lifetime Data Anal. 2013;19(2):142-169. doi:10.1007/s10985-012-9235-3Google ScholarCrossref
90.
Pencina  MJ, D’Agostino  RB  Sr, Steyerberg  EW.  Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers.  Stat Med. 2011;30(1):11-21. doi:10.1002/sim.4085Google ScholarCrossref
91.
Nasir  K, Bittencourt  MS, Blaha  MJ,  et al.  Implications of coronary artery calcium testing among statin candidates according to American College of Cardiology/American Heart Association Cholesterol Management Guidelines: MESA (Multi-Ethnic Study of Atherosclerosis).  J Am Coll Cardiol. 2015;66(15):1657-1668. doi:10.1016/j.jacc.2015.07.066Google ScholarCrossref
92.
McEvoy  JW, Martin  SS, Blaha  MJ,  et al.  The case for and against a coronary artery calcium trial: means, motive, and opportunity.  JACC Cardiovasc Imaging. 2016;9(8):994-1002. doi:10.1016/j.jcmg.2016.03.012Google ScholarCrossref
93.
Cook  NR.  Use and misuse of the receiver operating characteristic curve in risk prediction.  Circulation. 2007;115(7):928-935. doi:10.1161/CIRCULATIONAHA.106.672402Google ScholarCrossref
94.
Damen  JA, Hooft  L, Schuit  E,  et al.  Prediction models for cardiovascular disease risk in the general population: systematic review.  BMJ. 2016;353:i2416. doi:10.1136/bmj.i2416Google ScholarCrossref
95.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2Google ScholarCrossref
×