[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    1 Comment for this article
    Meta-analyses may underestimate the effects of clinical and demographic variables on brain structure
    Tomas Hajek, MD, PhD | Department of Psychiatry, Dalhousie University,
    We read with interest the recent manuscript by Dr. Ioannidis,1 in which he demonstrated an excess of positive results in structural neuroimaging literature. Obtaining a reasonably accurate estimate of the effect size is critical for the calculation of the expected number of significant results. Dr. Ioannidis used the actual effect size from meta- analyses as an optimistic estimate and ½ of this effect as a conservative estimate, in order to account for publication bias. However, it is also possible that results of a meta-analysis may in fact underestimate the true effect size. Reports of both increased and decreased volumes of a particular brain structure are relatively frequent in neuropsychiatric literature. Clinical or demographic factors likely underlie the different directions of findings. For example, whereas amygdala volumes are significantly decreased among pediatric/adolescent bipolar patients, they tend to be increased among adults with bipolar disorders, yielding no difference when data from both age groups are combined.2 Similarly duration of illness3,4 and lithium treatment4 may exert opposing effects on hippocampal volumes in patients with mood disorders. All available meta-analyses, which combined bipolar subjects with varying duration of illness and exposure to lithium, reported no changes in hippocampal volumes in bipolar disorders.5-8 In contrast studies, which controlled for lithium exposure did report such changes.4,9,10 In addition even the above mentioned, known confounding variables are often not well controlled for or evaluated in individual studies. Thus, instead of estimates of true effect size, meta-analyses provide a sum of the effects of clinical and demographic variables affecting brain volumes. When variables, which exert opposing effects on gray matter volumes are combined, the results may underestimate the real effect of each of the factors. Consequently, calculating the excess significance test based on effect size or ½ effect size from meta-analyses may in fact underestimate the expected number of positive studies and overestimate the excess of positive results. References:
    1. Ioannidis JP. Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry. 2011;68:773-780.
    2. Hajek T, Kopecek M, Kozeny J, Gunde E, Alda M, Hoschl C. Amygdala volumes in mood disorders - Meta-analysis of magnetic resonance volumetry studies. J Affect Disord. 2009;115:395-410.
    3. MacQueen GM, Campbell S, McEwen BS, Macdonald K, Amano S, Joffe RT, Nahmias C, Young LT. Course of illness, hippocampal function, and hippocampal volume in major depression. Proc Natl Acad Sci U S A. 2003;100:1387-1392.
    4. Hallahan B, Newell J, Soares JC, Brambilla P, Strakowski SM, Fleck DE, Kieseppa T, Altshuler LL, Fornito A, Malhi GS, McIntosh AM, Yurgelun-Todd DA, LaBar KS, Sharma V, MacQueen GM, Murray RM, McDonald C. Structural magnetic resonance imaging in bipolar disorder: an international collaborative mega-analysis of individual adult patient data. Biol Psychiatry. 2011;69:326-335.
    5. Arnone D, Cavanagh J, Gerber D, Lawrie SM, Ebmeier KP, McIntosh AM. Magnetic resonance imaging studies in bipolar disorder and schizophrenia: meta-analysis. Br J Psychiatry. 2009;195:194-201.
    6. Kempton MJ, Geddes JR, Ettinger U, Williams SC, Grasby PM. Meta- analysis, database, and meta-regression of 98 structural imaging studies in bipolar disorder. Arch Gen Psychiatry. 2008;65:1017-1032.
    7. McDonald C, Zanelli J, Rabe-Hesketh S, Ellison-Wright I, Sham P, Kalidindi S, Murray RM, Kennedy N. Meta-analysis of magnetic resonance imaging brain morphometry studies in bipolar disorder. Biol Psychiatry. 2004;56:411-417.
    8. Videbech P, Ravnkilde B. Hippocampal volume and depression: a meta-analysis of MRI studies. Am J Psychiatry. 2004;161:1957-1966.
    9. Bearden CE, Thompson PM, Dutton RA, Frey BN, Peluso MA, Nicoletti M, Dierschke N, Hayashi KM, Klunder AD, Glahn DC, Brambilla P, Sassi RB, Mallinger AG, Soares JC. Three-dimensional mapping of hippocampal anatomy in unmedicated and lithium-treated patients with bipolar disorder. Neuropsychopharmacology. 2008;33:1229-1238.
    10. Beyer JL, Kuchibhatla M, Payne ME, Moo-Young M, Cassidy F, MacFall J, Krishnan KR. Hippocampal volume measurement in older adults with bipolar disorder. Am J Geriatr Psychiatry. 2004;12:613-620.

    Conflict of Interest: None declared
    Original Article
    Aug 2011

    Excess Significance Bias in the Literature on Brain Volume Abnormalities

    Author Affiliations

    Author Affiliations: Clinical and Molecular Epidemiology Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, and Biomedical Research Institute, Foundation for Research and Technology–Hellas, Ioannina, Greece; Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, and Department of Medicine, Tufts University School of Medicine, and Department of Epidemiology, Harvard School of Public Health, Boston, Massachusetts; and Stanford Prevention Research Center, Stanford University School of Medicine, Stanford, California.

    Arch Gen Psychiatry. 2011;68(8):773-780. doi:10.1001/archgenpsychiatry.2011.28

    Context Many studies report volume abnormalities in diverse brain structures in patients with various mental health conditions.

    Objective To evaluate whether there is evidence for an excess number of statistically significant results in studies of brain volume abnormalities that suggest the presence of bias in the literature.

    Data Sources PubMed (articles published from January 2006 to December 2009).

    Study Selection Recent meta-analyses of brain volume abnormalities in participants with various mental health conditions vs control participants with 6 or more data sets included, excluding voxel-based morphometry.

    Data Extraction Standardized effect sizes were extracted in each data set, and it was noted whether the results were“positive” (P < .05) or not. For each data set in each meta-analysis, I estimated the power to detect atα = .05 an effect equal to the summary effect of the respective meta-analysis. The sum of the power estimates gives the number of expected positive data sets. The expected number of positive data sets can then be compared against the observed number.

    Data Synthesis From 8 articles, 41 meta-analyses with 461 data sets were evaluated (median, 10 data sets per meta-analysis) pertaining to 7 conditions. Twenty-one of the 41 meta-analyses had found statistically significant associations, and 142 of 461 (31%) data sets had positive results. Even if the summary effect sizes of the meta-analyses were unbiased, the expected number of positive results would have been only 78.5 compared with the observed number of 142 (P < .001).

    Conclusion There are too many studies with statistically significant results in the literature on brain volume abnormalities. This pattern suggests strong biases in the literature, with selective outcome reporting and selective analyses reporting being possible explanations.