[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 54.158.173.184. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Download PDF
Figure. Correlation between the summary relative risk in each meta-analysis (random effects) and the relative risk in the largest study. Five outliers with relative risk greater than 3 in the meta-analysis and/or the largest study are not shown. Axes are in logarithmic scale.

Figure. Correlation between the summary relative risk in each meta-analysis (random effects) and the relative risk in the largest study. Five outliers with relative risk greater than 3 in the meta-analysis and/or the largest study are not shown. Axes are in logarithmic scale.

Table 1. Description of 56 Eligible Meta-analyses
Table 1. Description of 56 Eligible Meta-analyses
Table 2. Observed and Expected Number of Significant Studies in the 29 Meta-analyses With a Significant Excess of Significant Studies
Table 2. Observed and Expected Number of Significant Studies in the 29 Meta-analyses With a Significant Excess of Significant Studies
Table 3. Observed and Expected Number of Significant Studies Across All Meta-analyses and in Subgroups
Table 3. Observed and Expected Number of Significant Studies Across All Meta-analyses and in Subgroups
1.
Hlatky MA, Greenland P, Arnett DK,  et al; American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council.  Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association [published correction appears in Circulation. 2009;119(25):e606].  Circulation. 2009;119(17):2408-2416PubMedArticle
2.
Wang TJ. Assessing the role of circulating, genetic, and imaging biomarkers in cardiovascular risk prediction.  Circulation. 2011;123(5):551-565PubMedArticle
3.
Gilstrap LG, Wang TJ. Biomarkers and cardiovascular risk assessment for primary prevention: an update.  Clin Chem. 2012;58(1):72-82PubMedArticle
4.
Ioannidis JP, Tzoulaki I. Minimal and null predictive effects for the most popular blood biomarkers of cardiovascular disease.  Circ Res. 2012;110(5):658-662PubMedArticle
5.
Wilson PW. Challenges to improve coronary heart disease risk assessment.  JAMA. 2009;302(21):2369-2370PubMedArticle
6.
Lloyd-Jones DM. Cardiovascular risk prediction: basic concepts, current status, and future directions.  Circulation. 2010;121(15):1768-1777PubMedArticle
7.
Tzoulaki I, Liberopoulos G, Ioannidis JP. Use of reclassification for assessment of improved prediction: an empirical evaluation.  Int J Epidemiol. 2011;40(4):1094-1105PubMedArticle
8.
Tzoulaki I, Liberopoulos G, Ioannidis JP. Assessment of claims of improved prediction beyond the Framingham Risk Score.  JAMA. 2009;302(21):2345-2352PubMedArticle
9.
Ioannidis JP, Tzoulaki I. What makes a good predictor? the evidence applied to coronary artery calcium score.  JAMA. 2010;303(16):1646-1647PubMedArticle
10.
Tzoulaki I, Siontis KC, Ioannidis JP. Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study.  BMJ. 2011;343:d6829PubMedArticleArticle
11.
Vasan RS. Biomarkers of cardiovascular disease: molecular basis and practical considerations.  Circulation. 2006;113(19):2335-2362PubMedArticle
12.
Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories.  Circulation. 1998;97(18):1837-1847PubMedArticle
13.
Greenland P, Alpert JS, Beller GA,  et al; American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines.  2010 ACCF/AHA guideline for assessment of cardiovascular risk in asymptomatic adults: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines.  Circulation. 2010;122(25):e584-e636PubMedArticle
14.
Apple FS, Wu AH, Mair J,  et al; Committee on Standardization of Markers of Cardiac Damage of the IFCC.  Future biomarkers for detection of ischemia and risk stratification in acute coronary syndrome.  Clin Chem. 2005;51(5):810-824PubMedArticle
15.
Naghavi M, Libby P, Falk E,  et al.  From vulnerable plaque to vulnerable patient: a call for new definitions and risk assessment strategies: part I.  Circulation. 2003;108(14):1664-1672PubMedArticle
16.
Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in systematic reviews.  Ann Intern Med. 1997;127(9):820-826PubMedArticle
17.
Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses.  BMJ. 2007;335(7626):914-916PubMedArticle
18.
Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test.  BMJ. 1997;315(7109):629-634PubMedArticle
19.
Sterne JA, Sutton AJ, Ioannidis JP,  et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.  BMJ. 2011;343:d4002PubMedArticleArticle
20.
Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings.  Clin Trials. 2007;4(3):245-253PubMedArticle
21.
Kavvoura FK, McQueen MB, Khoury MJ, Tanzi RE, Bertram L, Ioannidis JP. Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer's disease.  Am J Epidemiol. 2008;168(8):855-865PubMedArticle
22.
Ioannidis JP. Excess significance bias in the literature on brain volume abnormalities.  Arch Gen Psychiatry. 2011;68(8):773-780PubMedArticle
23.
Freedman LS. Tables of the number of patients required in clinical trials using the logrank test.  Stat Med. 1982;1(2):121-129PubMedArticle
24.
Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis.  Stat Med. 2000;19(22):3127-3131PubMedArticle
25.
Wormser D, Kaptoge S, Di Angelantonio E,  et al; Emerging Risk Factors Collaboration.  Separate and combined associations of body mass index and abdominal adiposity with cardiovascular disease: collaborative analysis of 58 prospective studies.  Lancet. 2011;377(9771):1085-1095PubMed
26.
Santos-Oliveira R, Purdy C, da Silva MP, dos Anjos Carneiro-Leão AM, Machado M, Einarson TR. Haemoglobin A1c levels and subsequent cardiovascular disease in persons without diabetes: a meta-analysis of prospective cohorts.  Diabetologia. 2011;54(6):1327-1334PubMedArticle
27.
van der Velde M, Matsushita K, Coresh J,  et al; Chronic Kidney Disease Prognosis Consortium.  Lower estimated glomerular filtration rate and higher albuminuria are associated with all-cause and cardiovascular mortality: a collaborative meta-analysis of high-risk population cohorts.  Kidney Int. 2011;79(12):1341-1352PubMedArticle
28.
Hansen TW, Li Y, Boggia J, Thijs L, Richart T, Staessen JA. Predictive role of the nighttime blood pressure.  Hypertension. 2011;57(1):3-10PubMedArticle
29.
Lee M, Saver JL, Huang WH, Chow J, Chang KH, Ovbiagele B. Impact of elevated cystatin C level on cardiovascular disease risk in predominantly high-cardiovascular-risk populations: a meta-analysis.  Circ Cardiovasc Qual Outcomes. 2010;3(6):675-683PubMedArticle
30.
Pierdomenico SD, Cuccurullo F. Prognostic value of white-coat and masked hypertension diagnosed by ambulatory monitoring in initially untreated subjects: an updated meta analysis.  Am J Hypertens. 2011;24(1):52-58PubMedArticle
31.
Sarwar N, Aspelund T, Eiriksdottir G,  et al.  Markers of dysglycaemia and risk of coronary heart disease in people without diabetes: Reykjavik prospective study and systematic review.  PLoS Med. 2010;7(5):e1000278.PubMedArticleArticle
32.
Matsushita K, van der Velde M, Astor BC,  et al; Chronic Kidney Disease Prognosis Consortium.  Association of estimated glomerular filtration rate and albuminuria with all-cause and cardiovascular mortality in general population cohorts: a collaborative meta-analysis.  Lancet. 2010;375(9731):2073-2081PubMedArticle
33.
Erqou S, Thompson A, Di Angelantonio E,  et al.  Apolipoprotein(a) isoforms and the risk of vascular disease: systematic review of 40 studies involving 58,000 participants.  J Am Coll Cardiol. 2010;55(19):2160-2167PubMedArticle
34.
Thompson A, Gao P, Orfei L,  et al; Lp-PLA(2) Studies Collaboration.  Lipoprotein-associated phospholipase A2 and risk of coronary disease, stroke, and mortality: collaborative analysis of 32 prospective studies.  Lancet. 2010;375(9725):1536-1544PubMedArticle
35.
Kaptoge S, Di Angelantonio E, Lowe G,  et al; Emerging Risk Factors Collaboration.  C-reactive protein concentration and risk of coronary heart disease, stroke, and mortality: an individual participant meta-analysis.  Lancet. 2010;375(9709):132-140PubMedArticle
36.
Vlachopoulos C, Aznaouridis K, Stefanadis C. Prediction of cardiovascular events and all-cause mortality with arterial stiffness: a systematic review and meta-analysis.  J Am Coll Cardiol. 2010;55(13):1318-1327PubMedArticle
37.
Di Angelantonio E, Chowdhury R, Sarwar N,  et al.  B-type natriuretic peptides and cardiovascular risk: systematic review and meta-analysis of 40 prospective studies.  Circulation. 2009;120(22):2177-2187PubMedArticle
38.
Di Angelantonio E, Sarwar N, Perry P,  et al; Emerging Risk Factors Collaboration.  Major lipids, apolipoproteins, and risk of vascular disease.  JAMA. 2009;302(18):1993-2000PubMedArticle
39.
Erqou S, Kaptoge S, Perry PL,  et al; Emerging Risk Factors Collaboration.  Lipoprotein(a) concentration and the risk of coronary heart disease, stroke, and nonvascular mortality.  JAMA. 2009;302(4):412-423PubMedArticle
40.
McGeechan K, Liew G, Macaskill P,  et al.  Meta-analysis: retinal vessel caliber and risk for coronary heart disease.  Ann Intern Med. 2009;151(6):404-413PubMedArticle
41.
Inaba Y, Bergmann SR. Prognostic value of myocardial metabolic imaging with BMIPP in the spectrum of coronary artery disease: a systematic review.  J Nucl Cardiol. 2010;17(1):61-70PubMedArticle
42.
Ryding AD, Kumar S, Worthington AM, Burgess D. Prognostic value of brain natriuretic peptide in noncardiac surgery: a meta-analysis.  Anesthesiology. 2009;111(2):311-319PubMedArticle
43.
Sarwar A, Shaw LJ, Shapiro MD,  et al.  Diagnostic and prognostic value of absence of coronary artery calcification [published correction appears in JACC Cardiovasc Imaging. 2010;3(10):1089].  JACC Cardiovasc Imaging. 2009;2(6):675-688PubMedArticle
44.
Kodama S, Saito K, Tanaka S,  et al.  Cardiorespiratory fitness as a quantitative predictor of all-cause mortality and cardiovascular events in healthy men and women: a meta-analysis.  JAMA. 2009;301(19):2024-2035PubMedArticle
45.
Sarwar N, Sattar N, Gudnason V, Danesh J. Circulating concentrations of insulin markers and coronary heart disease: a quantitative review of 19 Western prospective studies.  Eur Heart J. 2007;28(20):2491-2497PubMedArticle
46.
Lorenz MW, Markus HS, Bots ML, Rosvall M, Sitzer M. Prediction of clinical cardiovascular events with carotid intima-media thickness: a systematic review and meta-analysis.  Circulation. 2007;115(4):459-467PubMedArticle
47.
Flores-Mateo G, Navas-Acien A, Pastor-Barriuso R, Guallar E. Selenium and coronary heart disease: a meta-analysis.  Am J Clin Nutr. 2006;84(4):762-773PubMed
48.
Sattar N, Wannamethee G, Sarwar N,  et al.  Adiponectin and coronary heart disease: a prospective study and meta-analysis.  Circulation. 2006;114(7):623-629PubMedArticle
49.
Thompson A, Danesh J. Associations between apolipoprotein B, apolipoprotein AI, the apolipoprotein B/AI ratio, and coronary heart disease: a literature-based meta-analysis of prospective studies.  J Intern Med. 2006;259(5):481-492PubMedArticle
50.
Khan NA, Hemmelgarn BR, Tonelli M, Thompson CR, Levin A. Prognostic value of troponin T and I among asymptomatic patients with end-stage renal disease: a meta-analysis.  Circulation. 2005;112(20):3088-3096PubMedArticle
51.
Danesh J, Lewington S, Thompson SG,  et al; Fibrinogen Studies Collaboration.  Plasma fibrinogen level and the risk of major cardiovascular diseases and nonvascular mortality: an individual participant meta-analysis.  JAMA. 2005;294(14):1799-1809PubMedArticle
52.
Wheeler JG, Juzwishin KD, Eiriksdottir G, Gudnason V, Danesh J. Serum uric acid and coronary heart disease in 9,458 incident cases and 155,084 controls: prospective study and meta-analysis.  PLoS Med. 2005;2(3):e76PubMedArticleArticle
53.
Whincup PH, Danesh J, Walker M,  et al.  von Willebrand factor and coronary heart disease: prospective study and meta-analysis.  Eur Heart J. 2002;23(22):1764-1770PubMedArticle
54.
Homocysteine Studies Collaboration.  Homocysteine and risk of ischemic heart disease and stroke: a meta-analysis.  JAMA. 2002;288(16):2015-2022PubMedArticle
55.
Ottani F, Galvani M, Nicolini FA,  et al.  Elevated cardiac troponin levels predict the risk of adverse outcome in patients with acute coronary syndromes.  Am Heart J. 2000;140(6):917-927PubMedArticle
56.
Danesh J, Whincup P, Walker M,  et al.  Chlamydia pneumoniae IgG titres and coronary heart disease: prospective study and meta-analysis.  BMJ. 2000;321(7255):208-213PubMedArticle
57.
Danesh J, Whincup P, Walker M,  et al.  Low-grade inflammation and coronary heart disease: prospective study and updated meta-analyses.  BMJ. 2000;321(7255):199-204PubMedArticle
58.
Danesh J, Collins R, Peto R, Lowe GD. Haematocrit, viscosity, erythrocyte sedimentation rate: meta-analyses of prospective studies of coronary heart disease.  Eur Heart J. 2000;21(7):515-520PubMedArticle
59.
Padayachee L, Rodseth RN, Biccard BM. A meta-analysis of the utility of C-reactive protein in predicting early, intermediate-term, and long-term mortality and major adverse cardiac events in vascular surgical patients.  Anaesthesia. 2009;64(4):416-424PubMedArticle
60.
Ioannidis JP, Panagiotou OA. Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses.  JAMA. 2011;305(21):2200-2210PubMedArticle
61.
Dwan K, Altman DG, Arnaiz JA,  et al.  Systematic review of the empirical evidence of study publication bias and outcome reporting bias.  PLoS One. 2008;3(8):e3081PubMedArticleArticle
62.
Andre F, McShane LM, Michiels S,  et al.  Biomarker studies: a call for a comprehensive biomarker study registry.  Nat Rev Clin Oncol. 2011;8(3):171-176PubMedArticle
63.
Ioannidis JP, Tarone R, McLaughlin JK. The false-positive to false-negative ratio in epidemiologic studies.  Epidemiology. 2011;22(4):450-456PubMedArticle
64.
Ioannidis JP. Why most published research findings are false.  PLoS Med. 2005;2(8):e124PubMedArticleArticle
65.
Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors.  BMJ. 2005;330(7494):753PubMedArticleArticle
66.
Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.  JAMA. 2004;291(20):2457-2465PubMedArticle
Original Investigation
April 22, 2013

Bias in Associations of Emerging Biomarkers With Cardiovascular Disease

Author Affiliations

Author Affiliations: Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece (Drs Tzoulaki, Siontis, and Evangelou); Department of Epidemiology and Biostatistics, Imperial College London, London, United Kingdom (Dr Tzoulaki); Mayo School of Graduate Medical Education, College of Medicine, Mayo Clinic, Rochester, Minnesota (Dr Siontis); and Stanford Prevention Research Center, Departments of Medicine and Health Research and Policy, Stanford University School of Medicine, and Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California (Dr Ioannidis).

JAMA Intern Med. 2013;173(8):664-671. doi:10.1001/jamainternmed.2013.3018
Abstract

Importance Numerous cardiovascular biomarkers are proposed as potential predictors of cardiovascular risk.

Objective To evaluate whether there is evidence for biases favoring statistically significant results and inflating associations in this literature.

Design and Setting PubMed search for meta-analyses of cardiovascular biomarkers that are not part of the Framingham Risk Score.

Main Outcome Measures We estimated summary effects and between-study heterogeneity (considered “very large” for I2 > 75%). We evaluated whether large studies had significantly more conservative results than smaller studies (small-study effects) and whether there were too many studies with statistically significant results compared with what would be expected on the basis of the findings of the largest study in each meta-analysis.

Results Of 56 eligible meta-analyses, 49 had statistically significant results. Very large heterogeneity and small-study effects were seen in 9 and 13 meta-analyses, respectively. In 29 meta-analyses (52%), there was a significant excess of studies with statistically significant results. Only 13 of the statistically significant meta-analyses had more than 1000 cases and no hints of large heterogeneity, small-study effects, or excess significance. These included the associations of glomerular filtration rate and albumin to creatinine ratio in general and high-risk populations with cardiovascular disease mortality and of non–high-density lipoprotein cholesterol, serum albumin, Chlamydia pneumoniae IgG, glycosylated hemoglobin, nonfasting insulin, apolipoprotein B/AI ratio, erythrocyte sedimentation rate, and lipoprotein-associated phospholipase mass or activity with coronary heart disease.

Conclusions and Relevance Selective reporting biases may be common in the evidence on emerging cardiovascular biomarkers. Most of the proposed associations of these biomarkers may be inflated.

Numerous cardiovascular biomarkers are proposed as potential predictors of cardiovascular risk.13 Despite intensive research efforts, to date, few emerging cardiovascular biomarkers have shown clear improvements in predictive discrimination, reclassification, and/or calibration, and their clinical utility remains equivocal.46 Methodologic limitations in this field, including poor reporting, lack of validation, and unjustified claims of improved prediction over well-established risk scores, cast doubts on the validity and magnitude of reported effect sizes.4,710

Numerous meta-analyses have been performed to date aiming to summarize the predictive value of individual cardiovascular biomarkers. However, to our knowledge, there has been no recent effort to summarize the evidence from these meta-analyses on their associated limitations such as publication, selective analysis, and outcome reporting biases. Here we assembled for the first time a comprehensive systematic sample of meta-analyses that examine associations of emerging biomarkers with cardiovascular outcomes. We sought to evaluate whether there is evidence for biases in this literature favoring statistically significant and/or inflated results and how many and which of these cardiovascular biomarker effects have no hints of bias.

METHODS
IDENTIFICATION OF ELIGIBLE STUDIES

We assembled meta-analyses that examined any emerging biomarker, defined as any biological parameter11 other than those included in the Framingham Risk Score,12 in relation to cardiovascular disease (CVD), coronary heart disease (CHD), or cardiovascular mortality. Meta-analyses were selected as a tool to provide us with systematic summary evidence on each examined biomarker. All types of biomarkers were eligible (blood, urine, tissue, imaging, or physical measurement). We excluded meta-analyses of single common genetic variants because these markers have limited prognostic ability when examined in isolation, but multigene scores were eligible.

We used 3 different approaches to collect a comprehensive sample of biomarker meta-analyses indexed in MEDLINE (with no year restriction and last update as of January 30, 2012). First, we used the algorithm “(‘Biological marker’[MeSH Terms]) AND (cardiovascular OR coronary [Title/Abstract])” limited to meta-analyses, English language, and human studies. Second, we performed targeted MEDLINE searches for meta-analyses of 71 additional specific emerging biomarkers included in recent comprehensive reviews1315 using the same algorithm, but instead of applying the generic “Biological marker” term, we tracked the name of each biomarker (see eTable for full list of biomarkers searched; http://www.jamainternalmed.com). We perused the title and abstract of each of these citations, and potentially eligible articles were then retrieved in full text. Finally, we identified meta-analyses of individual participant data published by major consortia operating in the field (Emerging Risk Factor Collaboration, Fibrinogen Studies Collaboration, Ankle Brachial Index Collaboration, Homocysteine Studies Collaboration, and Chronic Kidney Disease Prognosis Consortium).

We included studies regardless of the baseline characteristics (clinical setting) of the examined populations. Whenever an article presented separate meta-analyses on more than 1 eligible biomarker, outcome, or type of clinical settings, those were kept separately. Meta-analyses were eligible regardless of whether the included studies used adjustment for some covariates or score (eg, the Framingham Risk Score) or unadjusted analyses. We excluded meta-analyses of randomized controlled trials assessing the change of a biomarker in relation to an outcome. When more than 1 meta-analysis examining the same biomarker and same outcome on the same clinical setting were identified, only the most recent one with eligible data was kept.

DATA EXTRACTION

Data extraction was performed independently by 2 investigators (K.C.S. and I.T.) and, in the case of discrepancies, consensus was reached. From each eligible meta-analysis, we recorded the first author, journal, year of publication, and number of studies in the meta-analysis, and we noted the biomarker, risk factors or score used for covariate adjustment and the outcome examined. The study-specific relative risk (RR) estimates (risk ratio, odds ratio, hazard ratio, or incident risk ratio, as reported by the meta-analysis authors) along with the corresponding 95% CIs and the number of cases in each study were extracted for each biomarker and outcome. The number of control participants in addition to the number of cases for each study were extracted when needed for the power calculation (see the Statistical Analysis subsection). When data on number of cases (or number of cases and controls when needed) were missing, meta-analyses were excluded and substituted with a previously published meta-analysis on the same biomarker, whenever available. Meta-analyses of cardiovascular outcomes typically assume that study-specific RRs (odds ratio, hazard ratio, and risk ratio) are similar and combine different RRs under this assumption. This assumption is a fair approximation whenever the disease incidence is low or modest. We, therefore, extracted the metric of the largest study (study with smallest variance) and assumed that the remaining RRs correspond to the same metric. Whenever data were provided with different adjustments, we preferred estimates that adjusted for the Framingham Risk Score or the model with Framingham Risk Score variables; if neither of these options was available, we preferred the model with the larger number of adjusting factors. When subgroups were presented, we extracted the data for each subgroup separately unless the study combined the results across all subgroups.

STATISTICAL ANALYSIS

For each meta-analysis, we estimated the summary effect size and its 95% CI using random effects models16 and calculated the I2 metric for heterogeneity. The I2 metric ranges from 0% to 100% and is the ratio of between-study variance to the sum of the within- and between-study variances. Values exceeding 50% or 75% are considered to represent “large” or “very large” heterogeneity, respectively. The 95% CIs of I2 estimates can be wide when there are few studies.17 Furthermore, we used the regression asymmetry test proposed by Egger et al18 and examined by Sterne et al.19P < .10 with more conservative effect in larger studies was considered evidence for small-study effects—that is, that the results of small studies differed from those of larger studies. Various biases or genuine heterogeneity may cause small-study effects.19

We applied the excess significance test to evaluate whether the observed number of studies (O) with statistically significant results (“positive” studies) in each meta-analysis is larger than their expected number (E).2022 We also summed the O and E across all meta-analyses. For each meta-analysis, E is the sum of the power estimates of the studies it includes. The estimated power depends on the plausible effect size. The true effect size for any meta-analysis is unknown. Herein, we assumed that the most plausible effect is given by the largest study. Different equations were used to estimate the power when the largest study reported a hazard ratio23 or odds ratio.24 Excess significance for single meta-analyses was claimed at P < .10 (1-sided P < .05 with O > E as previously proposed22).

Predefined subgroup analyses applied the excess significance test in subgroups of meta-analyses with or without large between-study heterogeneity, small-study effects, individual-level data, or nominally significant summary effects and by primary (general populations) vs secondary (high risk or populations with CVD) prevention and biomarker (biological fluid measurement or physical measurement/imaging).

Stata, version 10.1 (StataCorp), was used for statistical analyses. P values were 2-tailed.

RESULTS
ELIGIBLE STUDIES AND META-ANALYSES

Overall, 582 articles were searched and 35 articles corresponding to 56 meta-analyses were deemed eligible2559 (eFigure). Examining 42 unique biomarkers for the 56 meta-analyses, the median (range) number of studies was 12 (3-68) and number of events was 2459 (34-12 785). Meta-analyses pertained to a range of biomarkers and populations examined; most data corresponded to primary prevention (general populations, 42 meta-analyses) and biomarkers measured in biological fluids (37 meta-analyses). The outcome examined was CHD in 28 meta-analyses, CVD in 21, and CVD mortality or cardiac death in 7 (Table 1). Fourteen meta-analyses were of individual participants and 42 analyzed published literature. All but 4 meta-analyses reported RR estimates adjusted for a variety of other cardiovascular risk factors (Table 1).

SUMMARY EFFECTS

Overall, 49 (88%) of the eligible meta-analyses reported a nominally statistically significant summary result (Table 1). The largest study had statistically significant results in 41 meta-analyses. The Figure shows the estimates of the largest studies against the random effects meta-analysis estimates. The largest study's result was more conservative compared with the summary result in 44 meta-analyses (79%), and most of the largest studies suggested effects of small magnitude (Table 1). Only myocardial metabolic imaging had the reported RR estimate adjusted for cardiovascular risk factors exceeding 3.00 both in the meta-analysis and in the largest study. Brain natriuretic peptide in high-risk populations and coronary artery calcium also reported large effect sizes (RR > 3) in unadjusted analyses. Nonetheless, studies with adjusted analyses have also shown relatively high estimates for these biomarkers.42,43

HETEROGENEITY AND SMALL-STUDY EFFECTS

Twenty-six meta-analyses (46%) had large heterogeneity (I2 > 50%) and 9 (16%) had very large heterogeneity (I2 > 75%). Evidence for significant small-study effects was noted in 13 meta-analyses (23%). Meta-analyses of fibrinogen, selenium, and apolipoprotein B with CHD and brain natriuretic peptide, aortic pulse wave velocity, and cystatin C with CVD showed very large heterogeneity and evidence of small-study effects. The meta-analyses of coronary artery calcium and cardiorespiratory fitness with CVD and troponin with 30-day cardiac death in patients with acute coronary syndrome showed large heterogeneity (I2 = 50%-75%) and evidence of small-study effects (Table 1).

EXCESS SIGNIFICANCE

In 29 meta-analyses (52%) there was a significant excess of observed “positive” studies compared with those expected (Table 2). Table 3 shows aggregate data from all the meta-analyses and according to different subgroups. Among 919 studies included in 56 meta-analyses, 472 (51%) had nominally statistically significant results, while the expected number was 317. The difference between the observed and expected was significant (P < .001). The excess of significant findings was documented across all the examined subgroups (Table 3).

SIGNIFICANT BIOMARKER ASSOCIATIONS WITHOUT HINTS OF BIAS

Of the 56 meta-analyses, 18 (32%) had nominally statistically significant summary associations per random effects calculations and had no evidence of small-study effects (P ≥ .10), not very large heterogeneity (I2 ≤ 75%), and no evidence for excess significance.

Overall, 13 of the 18 associations (72%) were based on cumulative evidence of more than 1000 cardiovascular events. This included 9 biomarkers with associations with CHD in general populations (non–high-density lipoprotein cholesterol, serum albumin, Chlamydia pneumoniae IgG titers, apolipoprotein B/AI ratio, glycosylated hemoglobin, lipoprotein-associated phospholipase mass and activity, erythrocyte sedimentation rate, and nonfasting insulin) and 2 biomarkers (estimated glomerular filtration rate [eGFR] and albumin to creatinine ratio) with associations with CVD mortality in high-risk and general populations. Across these 13 associations, the RR per 1-SD increase (assuming the top vs bottom tertile comparison corresponds to approximately 2 SD) had a median (range) of 1.2 (1.1-1.5), excluding the eGFR and albumin to creatinine meta-analyses. The latter meta-analyses in general and in high-risk populations reported RR using linear splines with knots at different levels of biomarkers. The RR was greater than 2 only in extreme comparisons—for example, the meta-analysis between eGFR and CVD mortality in general populations reported an RR of 2.66 comparing eGFR of 15 mL/min/1.73 m2 (0.4% of the population) vs the reference category (95 mL/min/1.73 m2); the RR in less extreme categories was more conservative (RR = 1.40 for 60 vs 95 mL/min/1.73 m2).

COMMENT

A systematic evaluation of 56 meta-analyses of emerging cardiovascular biomarkers suggests that many results are prone to biases. We found strong evidence to suggest the effect of biomarkers is exaggerated because the largest studies—which one would expect to produce the most stable estimates—consistently showed smaller effects. In most meta-analyses, too many single studies had reported “positive” results compared with what would be expected on the basis of the results of the largest studies. This suggests that small studies with “negative” results remain unpublished or that their results are distorted during analysis and reporting to seem more prominent.

Bias from preferential reporting of positive findings or “positive analyses” has been postulated as a major problem in clinical investigation and biomarker research in particular.6062 Perhaps only a fraction of potentially available data are eventually published in this field. For example, the number of studies in the meta-analyses of triglycerides, body mass index, C-reactive protein, homocysteine, and apolipoprotein B, which all examined general populations and CHD, varies extensively: 68, 39, 31, 26, and 21 studies, respectively. These biomarkers are relatively inexpensive and easy to measure routinely; thus, they should be available in most epidemiologic data sets with cardiovascular outcomes. Differences in the number of published studies may result in part from prioritization for publication of positive results. In addition, some promising biomarker results may simply reflect false-positives owing to multiple testing of many biomarkers.63,64 Selective analysis reporting bias emerges when there are many analyses that can be performed and only some of them, the ones with the “best” results, are presented.65,66 Scientists may have examined a wide range of biomarkers in the same population, different outcomes, different cut-off points, and different covariate adjustments for a single biomarker but then report only 1 or a few of these analyses.

In our evaluation, the excess of significant findings was seen across all prespecified subgroups. A cluster of meta-analyses, including those of fibrinogen, selenium, and apolipoprotein B with CHD and brain natriuretic peptide and aortic pulse wave velocity with CVD, had very large heterogeneity and small-study effects. Of note, the Egger test is particularly difficult to interpret in the presence of prominent between-study heterogeneity.19 Genuine heterogeneity of the strength of association in diverse settings and populations may be confused for selective reporting bias. However, in all of these cases there was also a marked excess of studies with positive results. Therefore, the overall picture is more consistent with bias and suggests that the claimed effect sizes are inflated. The meta-analyses themselves sometimes addressed these issues on individual biomarkers, calculated heterogeneity, assessed presence of publication bias, and presented subgroup and meta-regression analyses to identify differences between studies.

Hints of bias cannot exclude that these biomarkers have any association with cardiovascular outcomes. It is difficult to differentiate whether the underlying effect is small or null and whether genuine heterogeneity exists. There are also several associations in the literature of cardiovascular biomarkers that did not show any evidence of biases. These included the association of eGFR and albumin to creatinine ratio with CVD mortality in general and high-risk populations as well as the association of non–high-density lipoprotein cholesterol, serum albumin, Chlamydia pneumoniae IgG, glycosylated hemoglobin, nonfasting insulin, erythrocyte sedimentation rate, apolipoprotein B/AI ratio, and lipoprotein-associated phospholipase with CHD. Most of these associations were weak in terms of the magnitude of the RR. Therefore, their ability to improve cardiovascular risk assessment might be limited when used in isolation.

Several caveats should be considered in interpreting our findings. First, both asymmetry and excess significance tests offer hints of bias, not proof thereof. The exact estimation of excess significance is influenced by the choice of plausible effect size and the calculation of power. We selected the largest study's effect as the plausible effect. These studies may have inherent biases themselves. Finally, we performed power calculations assuming that all studies within each meta-analysis reported the same RR metric. This is a common assumption in meta-analyses of CVD outcomes, which typically consider the different reported metrics to be similar assuming CVD incidence is less than 10%.

With acknowledgment of these caveats, our evaluation maps the status of the evidence on the associations between popular cardiovascular biomarkers with CVD outcomes. Similar conclusions have been drawn from investigations in other fields, such as Alzheimer disease and mental health conditions.21,22 Cardiovascular biomarkers may be as prone to bias as biomarkers in other fields. Single biomarkers with large effect sizes are probably rare. Thus, the much-awaited improvement in CVD risk prediction will require evaluation of composites of numerous biomarkers in large-scale consortia using standardized approaches to minimize biases.4

Back to top
Article Information

Accepted for Publication: November 5, 2012.

Published Online: March 25, 2013. doi:10.1001/jamainternmed.2013.3018

Correspondence: John P. A. Ioannidis, MD, DSc, Stanford Prevention Research Center, 1265 Welch Rd, Medical School Office Building, Room X306, Stanford, CA 94350 (jioannid@stanford.edu).

Author Contributions: Dr Ioannidis had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Tzoulaki, Siontis, and Ioannidis. Acquisition of data: Tzoulaki, Siontis, and Evangelou. Analysis and interpretation of data: All authors. Drafting of the manuscript: Tzoulaki, Siontis, and Ioannidis. Critical revision of the manuscript for important intellectual content: Siontis and Evangelou. Statistical analysis: All authors. Administrative, technical, and material support: Siontis. Study supervision: Ioannidis.

Conflict of Interest Disclosures: None reported.

REFERENCES
1.
Hlatky MA, Greenland P, Arnett DK,  et al; American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council.  Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association [published correction appears in Circulation. 2009;119(25):e606].  Circulation. 2009;119(17):2408-2416PubMedArticle
2.
Wang TJ. Assessing the role of circulating, genetic, and imaging biomarkers in cardiovascular risk prediction.  Circulation. 2011;123(5):551-565PubMedArticle
3.
Gilstrap LG, Wang TJ. Biomarkers and cardiovascular risk assessment for primary prevention: an update.  Clin Chem. 2012;58(1):72-82PubMedArticle
4.
Ioannidis JP, Tzoulaki I. Minimal and null predictive effects for the most popular blood biomarkers of cardiovascular disease.  Circ Res. 2012;110(5):658-662PubMedArticle
5.
Wilson PW. Challenges to improve coronary heart disease risk assessment.  JAMA. 2009;302(21):2369-2370PubMedArticle
6.
Lloyd-Jones DM. Cardiovascular risk prediction: basic concepts, current status, and future directions.  Circulation. 2010;121(15):1768-1777PubMedArticle
7.
Tzoulaki I, Liberopoulos G, Ioannidis JP. Use of reclassification for assessment of improved prediction: an empirical evaluation.  Int J Epidemiol. 2011;40(4):1094-1105PubMedArticle
8.
Tzoulaki I, Liberopoulos G, Ioannidis JP. Assessment of claims of improved prediction beyond the Framingham Risk Score.  JAMA. 2009;302(21):2345-2352PubMedArticle
9.
Ioannidis JP, Tzoulaki I. What makes a good predictor? the evidence applied to coronary artery calcium score.  JAMA. 2010;303(16):1646-1647PubMedArticle
10.
Tzoulaki I, Siontis KC, Ioannidis JP. Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study.  BMJ. 2011;343:d6829PubMedArticleArticle
11.
Vasan RS. Biomarkers of cardiovascular disease: molecular basis and practical considerations.  Circulation. 2006;113(19):2335-2362PubMedArticle
12.
Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories.  Circulation. 1998;97(18):1837-1847PubMedArticle
13.
Greenland P, Alpert JS, Beller GA,  et al; American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines.  2010 ACCF/AHA guideline for assessment of cardiovascular risk in asymptomatic adults: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines.  Circulation. 2010;122(25):e584-e636PubMedArticle
14.
Apple FS, Wu AH, Mair J,  et al; Committee on Standardization of Markers of Cardiac Damage of the IFCC.  Future biomarkers for detection of ischemia and risk stratification in acute coronary syndrome.  Clin Chem. 2005;51(5):810-824PubMedArticle
15.
Naghavi M, Libby P, Falk E,  et al.  From vulnerable plaque to vulnerable patient: a call for new definitions and risk assessment strategies: part I.  Circulation. 2003;108(14):1664-1672PubMedArticle
16.
Lau J, Ioannidis JP, Schmid CH. Quantitative synthesis in systematic reviews.  Ann Intern Med. 1997;127(9):820-826PubMedArticle
17.
Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses.  BMJ. 2007;335(7626):914-916PubMedArticle
18.
Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test.  BMJ. 1997;315(7109):629-634PubMedArticle
19.
Sterne JA, Sutton AJ, Ioannidis JP,  et al.  Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.  BMJ. 2011;343:d4002PubMedArticleArticle
20.
Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings.  Clin Trials. 2007;4(3):245-253PubMedArticle
21.
Kavvoura FK, McQueen MB, Khoury MJ, Tanzi RE, Bertram L, Ioannidis JP. Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer's disease.  Am J Epidemiol. 2008;168(8):855-865PubMedArticle
22.
Ioannidis JP. Excess significance bias in the literature on brain volume abnormalities.  Arch Gen Psychiatry. 2011;68(8):773-780PubMedArticle
23.
Freedman LS. Tables of the number of patients required in clinical trials using the logrank test.  Stat Med. 1982;1(2):121-129PubMedArticle
24.
Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis.  Stat Med. 2000;19(22):3127-3131PubMedArticle
25.
Wormser D, Kaptoge S, Di Angelantonio E,  et al; Emerging Risk Factors Collaboration.  Separate and combined associations of body mass index and abdominal adiposity with cardiovascular disease: collaborative analysis of 58 prospective studies.  Lancet. 2011;377(9771):1085-1095PubMed
26.
Santos-Oliveira R, Purdy C, da Silva MP, dos Anjos Carneiro-Leão AM, Machado M, Einarson TR. Haemoglobin A1c levels and subsequent cardiovascular disease in persons without diabetes: a meta-analysis of prospective cohorts.  Diabetologia. 2011;54(6):1327-1334PubMedArticle
27.
van der Velde M, Matsushita K, Coresh J,  et al; Chronic Kidney Disease Prognosis Consortium.  Lower estimated glomerular filtration rate and higher albuminuria are associated with all-cause and cardiovascular mortality: a collaborative meta-analysis of high-risk population cohorts.  Kidney Int. 2011;79(12):1341-1352PubMedArticle
28.
Hansen TW, Li Y, Boggia J, Thijs L, Richart T, Staessen JA. Predictive role of the nighttime blood pressure.  Hypertension. 2011;57(1):3-10PubMedArticle
29.
Lee M, Saver JL, Huang WH, Chow J, Chang KH, Ovbiagele B. Impact of elevated cystatin C level on cardiovascular disease risk in predominantly high-cardiovascular-risk populations: a meta-analysis.  Circ Cardiovasc Qual Outcomes. 2010;3(6):675-683PubMedArticle
30.
Pierdomenico SD, Cuccurullo F. Prognostic value of white-coat and masked hypertension diagnosed by ambulatory monitoring in initially untreated subjects: an updated meta analysis.  Am J Hypertens. 2011;24(1):52-58PubMedArticle
31.
Sarwar N, Aspelund T, Eiriksdottir G,  et al.  Markers of dysglycaemia and risk of coronary heart disease in people without diabetes: Reykjavik prospective study and systematic review.  PLoS Med. 2010;7(5):e1000278.PubMedArticleArticle
32.
Matsushita K, van der Velde M, Astor BC,  et al; Chronic Kidney Disease Prognosis Consortium.  Association of estimated glomerular filtration rate and albuminuria with all-cause and cardiovascular mortality in general population cohorts: a collaborative meta-analysis.  Lancet. 2010;375(9731):2073-2081PubMedArticle
33.
Erqou S, Thompson A, Di Angelantonio E,  et al.  Apolipoprotein(a) isoforms and the risk of vascular disease: systematic review of 40 studies involving 58,000 participants.  J Am Coll Cardiol. 2010;55(19):2160-2167PubMedArticle
34.
Thompson A, Gao P, Orfei L,  et al; Lp-PLA(2) Studies Collaboration.  Lipoprotein-associated phospholipase A2 and risk of coronary disease, stroke, and mortality: collaborative analysis of 32 prospective studies.  Lancet. 2010;375(9725):1536-1544PubMedArticle
35.
Kaptoge S, Di Angelantonio E, Lowe G,  et al; Emerging Risk Factors Collaboration.  C-reactive protein concentration and risk of coronary heart disease, stroke, and mortality: an individual participant meta-analysis.  Lancet. 2010;375(9709):132-140PubMedArticle
36.
Vlachopoulos C, Aznaouridis K, Stefanadis C. Prediction of cardiovascular events and all-cause mortality with arterial stiffness: a systematic review and meta-analysis.  J Am Coll Cardiol. 2010;55(13):1318-1327PubMedArticle
37.
Di Angelantonio E, Chowdhury R, Sarwar N,  et al.  B-type natriuretic peptides and cardiovascular risk: systematic review and meta-analysis of 40 prospective studies.  Circulation. 2009;120(22):2177-2187PubMedArticle
38.
Di Angelantonio E, Sarwar N, Perry P,  et al; Emerging Risk Factors Collaboration.  Major lipids, apolipoproteins, and risk of vascular disease.  JAMA. 2009;302(18):1993-2000PubMedArticle
39.
Erqou S, Kaptoge S, Perry PL,  et al; Emerging Risk Factors Collaboration.  Lipoprotein(a) concentration and the risk of coronary heart disease, stroke, and nonvascular mortality.  JAMA. 2009;302(4):412-423PubMedArticle
40.
McGeechan K, Liew G, Macaskill P,  et al.  Meta-analysis: retinal vessel caliber and risk for coronary heart disease.  Ann Intern Med. 2009;151(6):404-413PubMedArticle
41.
Inaba Y, Bergmann SR. Prognostic value of myocardial metabolic imaging with BMIPP in the spectrum of coronary artery disease: a systematic review.  J Nucl Cardiol. 2010;17(1):61-70PubMedArticle
42.
Ryding AD, Kumar S, Worthington AM, Burgess D. Prognostic value of brain natriuretic peptide in noncardiac surgery: a meta-analysis.  Anesthesiology. 2009;111(2):311-319PubMedArticle
43.
Sarwar A, Shaw LJ, Shapiro MD,  et al.  Diagnostic and prognostic value of absence of coronary artery calcification [published correction appears in JACC Cardiovasc Imaging. 2010;3(10):1089].  JACC Cardiovasc Imaging. 2009;2(6):675-688PubMedArticle
44.
Kodama S, Saito K, Tanaka S,  et al.  Cardiorespiratory fitness as a quantitative predictor of all-cause mortality and cardiovascular events in healthy men and women: a meta-analysis.  JAMA. 2009;301(19):2024-2035PubMedArticle
45.
Sarwar N, Sattar N, Gudnason V, Danesh J. Circulating concentrations of insulin markers and coronary heart disease: a quantitative review of 19 Western prospective studies.  Eur Heart J. 2007;28(20):2491-2497PubMedArticle
46.
Lorenz MW, Markus HS, Bots ML, Rosvall M, Sitzer M. Prediction of clinical cardiovascular events with carotid intima-media thickness: a systematic review and meta-analysis.  Circulation. 2007;115(4):459-467PubMedArticle
47.
Flores-Mateo G, Navas-Acien A, Pastor-Barriuso R, Guallar E. Selenium and coronary heart disease: a meta-analysis.  Am J Clin Nutr. 2006;84(4):762-773PubMed
48.
Sattar N, Wannamethee G, Sarwar N,  et al.  Adiponectin and coronary heart disease: a prospective study and meta-analysis.  Circulation. 2006;114(7):623-629PubMedArticle
49.
Thompson A, Danesh J. Associations between apolipoprotein B, apolipoprotein AI, the apolipoprotein B/AI ratio, and coronary heart disease: a literature-based meta-analysis of prospective studies.  J Intern Med. 2006;259(5):481-492PubMedArticle
50.
Khan NA, Hemmelgarn BR, Tonelli M, Thompson CR, Levin A. Prognostic value of troponin T and I among asymptomatic patients with end-stage renal disease: a meta-analysis.  Circulation. 2005;112(20):3088-3096PubMedArticle
51.
Danesh J, Lewington S, Thompson SG,  et al; Fibrinogen Studies Collaboration.  Plasma fibrinogen level and the risk of major cardiovascular diseases and nonvascular mortality: an individual participant meta-analysis.  JAMA. 2005;294(14):1799-1809PubMedArticle
52.
Wheeler JG, Juzwishin KD, Eiriksdottir G, Gudnason V, Danesh J. Serum uric acid and coronary heart disease in 9,458 incident cases and 155,084 controls: prospective study and meta-analysis.  PLoS Med. 2005;2(3):e76PubMedArticleArticle
53.
Whincup PH, Danesh J, Walker M,  et al.  von Willebrand factor and coronary heart disease: prospective study and meta-analysis.  Eur Heart J. 2002;23(22):1764-1770PubMedArticle
54.
Homocysteine Studies Collaboration.  Homocysteine and risk of ischemic heart disease and stroke: a meta-analysis.  JAMA. 2002;288(16):2015-2022PubMedArticle
55.
Ottani F, Galvani M, Nicolini FA,  et al.  Elevated cardiac troponin levels predict the risk of adverse outcome in patients with acute coronary syndromes.  Am Heart J. 2000;140(6):917-927PubMedArticle
56.
Danesh J, Whincup P, Walker M,  et al.  Chlamydia pneumoniae IgG titres and coronary heart disease: prospective study and meta-analysis.  BMJ. 2000;321(7255):208-213PubMedArticle
57.
Danesh J, Whincup P, Walker M,  et al.  Low-grade inflammation and coronary heart disease: prospective study and updated meta-analyses.  BMJ. 2000;321(7255):199-204PubMedArticle
58.
Danesh J, Collins R, Peto R, Lowe GD. Haematocrit, viscosity, erythrocyte sedimentation rate: meta-analyses of prospective studies of coronary heart disease.  Eur Heart J. 2000;21(7):515-520PubMedArticle
59.
Padayachee L, Rodseth RN, Biccard BM. A meta-analysis of the utility of C-reactive protein in predicting early, intermediate-term, and long-term mortality and major adverse cardiac events in vascular surgical patients.  Anaesthesia. 2009;64(4):416-424PubMedArticle
60.
Ioannidis JP, Panagiotou OA. Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses.  JAMA. 2011;305(21):2200-2210PubMedArticle
61.
Dwan K, Altman DG, Arnaiz JA,  et al.  Systematic review of the empirical evidence of study publication bias and outcome reporting bias.  PLoS One. 2008;3(8):e3081PubMedArticleArticle
62.
Andre F, McShane LM, Michiels S,  et al.  Biomarker studies: a call for a comprehensive biomarker study registry.  Nat Rev Clin Oncol. 2011;8(3):171-176PubMedArticle
63.
Ioannidis JP, Tarone R, McLaughlin JK. The false-positive to false-negative ratio in epidemiologic studies.  Epidemiology. 2011;22(4):450-456PubMedArticle
64.
Ioannidis JP. Why most published research findings are false.  PLoS Med. 2005;2(8):e124PubMedArticleArticle
65.
Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors.  BMJ. 2005;330(7494):753PubMedArticleArticle
66.
Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.  JAMA. 2004;291(20):2457-2465PubMedArticle
×