Accuracy of Reports of Lifetime Mental and Physical Disorders: Results From the Baltimore Epidemiological Catchment Area Study

Importance  Our understanding of how mental and physical disorders are associated and contribute to health outcomes in populations depends on accurate ascertainment of the history of these disorders. Recent studies have identified substantial discrepancies in the prevalence of mental disorders among adolescents and young adults depending on whether the estimates are based on retrospective reports or multiple assessments over time. It is unknown whether such discrepancies are also seen in midlife to late life. Furthermore, no previous studies have compared lifetime prevalence estimates of common physical disorders such as diabetes mellitus and hypertension ascertained by prospective cumulative estimates vs retrospective estimates.

Objective  To examine the lifetime prevalence estimates of mental and physical disorders during midlife to late life using both retrospective and cumulative evaluations.

Design, Setting, and Participants  Prospective population-based survey (Baltimore Epidemiologic Catchment Area Survey) with 4 waves of interviews of 1071 community residents in Baltimore, Maryland, between 1981 and 2005.

Main Outcomes and Measures  Lifetime prevalence of selected mental and physical disorders at wave 4 (2004-2005), according to both retrospective data and cumulative evaluations based on 4 interviews from wave 1 to wave 4.

Results  Retrospective evaluations substantially underestimated the lifetime prevalence of mental disorders as compared with cumulative evaluations. The respective lifetime prevalence estimates ascertained by retrospective and cumulative evaluations were 4.5% vs 13.1% for major depressive disorder, 0.6% vs 7.1% for obsessive-compulsive disorder, 2.5% vs 6.7% for panic disorder, 12.6% vs 25.3% for social phobia, 9.1% vs 25.9% for alcohol abuse or dependence, and 6.7% vs 17.6% for drug abuse or dependence. In contrast, retrospective lifetime prevalence estimates of physical disorders ascertained at wave 4 were much closer to those based on cumulative data from all 4 waves. The respective prevalence estimates ascertained by the 2 methods were 18.2% vs 20.2% for diabetes, 48.4% vs 55.4% for hypertension, 45.8% vs 54.0% for arthritis, 5.5% vs 7.2% for stroke, and 8.4% vs 10.5% for cancer.

Conclusions and Relevance  One-time, cross-sectional population surveys may consistently underestimate the lifetime prevalence of mental disorders. The population burden of mental disorders may therefore be substantially higher than previously appreciated.

A common approach to estimating the lifetime prevalence of mental disorders in population surveys is to ask participants to retrospectively recall any episodes of illness they may have experienced during their entire lifetime.^1-8 Lifetime prevalence estimates, however, are potentially susceptible to recall bias and other memory distortions. With regard to depressive symptoms, for example, diminishing recall of past symptoms with time has been reported in a number of studies.^9-12 Retrospective evaluation may thus substantially underestimate or overestimate the true lifetime prevalence of mental disorders. Recently, Moffitt et al¹³ examined the potential impact of recall bias in epidemiologic studies by comparing estimates based on 1-time retrospective reports and those based on reports over multiple interviews in the same cohort. They found that lifetime prevalence estimates based on cumulative reports were approximately 2 times higher than those based on retrospective data for a number of common mental disorders, including major depressive disorder (MDD), anxiety disorders, and alcohol or drug dependence. In another study, Regier et al¹⁴ found that lifetime prevalence estimates of major mental disorders based on the Epidemiologic Catchment Area (ECA) study obtained just 1 year apart (at waves 1 and 2) were 8% higher when estimates were based on combined data from both waves compared with wave 1 only. Likewise, Copeland et al¹⁵ reported underestimation of mental disorders at any given time as compared with prospective cumulative evaluation. Olino et al¹⁶ found higher lifetime prevalence estimates of several mental disorders ascertained prospectively compared with estimates based on siblings’ retrospective reports. The studies by Moffitt et al,¹³ Copeland et al,¹⁵ and Olino et al,¹⁶ however, were based on cohorts of children, adolescents, and young adults rather than adults of all ages. Furthermore, the study by Regier et al¹⁴ compared estimates based on 1 wave with those based on 2 waves conducted 1 year apart. To our knowledge, besides depression or depressive symptoms,^9-12 no previous studies have examined whether the lifetime prevalence estimates of common mental disorders based on retrospective reports in midlife to late life are underestimated when compared with cumulative reports based on multiple interviews conducted over a longer period. Furthermore, no previous studies have compared lifetime prevalence estimates of common physical disorders such as diabetes mellitus and hypertension ascertained by retrospective report vs cumulative reports over an extended period. Therefore, it is unknown whether lifetime prevalence estimates of common mental disorders in middle-aged or older adults or physical disorders are underestimated in 1-time retrospective evaluation as compared with cumulative evaluation using multiple interviews. Given the significant burden imposed by mental and physical disorders, it is important to examine how accurately surveys can estimate their prevalence. It is also important to study the accuracy of retrospective reports over the adult life span extending into middle and older age because individuals experiencing long intervals between episodes of illness may be especially prone to forgetting past health problems. Furthermore, among older adults, these errors may be exacerbated by cognitive decline.

We used data from the Baltimore ECA Follow-up Study waves 1 (1981) through 4 (2004-2005) to compare the lifetime prevalence of mental and physical disorders based on retrospective reports (from wave 4 only) vs cumulative reports (from all waves). The wave 4 sample consisted of middle-aged and older adults. We also assessed factors associated with the underreporting of lifetime history of disorders at wave 4.

The Baltimore ECA Follow-up Study is a longitudinal, population-based cohort of adults originally interviewed in 1981 (wave 1, n = 3481) and followed up in 1982 (wave 2, n = 2768), 1993 to 1996 (wave 3, n = 1920), and 2004 to 2005 (wave 4, n = 1071). The ECA study was designed to collect data on the prevalence and incidence of mental disorders in an adult community sample according to DSM-III criteria (for waves 1 and 2) or DSM-III-R criteria (for waves 3 and 4). Methods for the Baltimore ECA Follow-up Study have been described in detail elsewhere.¹⁷ Of the original 3481 participants originally interviewed in 1981, 2031 survived to 2004 to 2005 and 1071 participated in the wave 4 interview (53% of survivors). Of the 960 who did not participate in the wave 4 interview, 436 refused to participate and 524 were lost to follow-up. The primary reason for the loss of contact is change in residence. Those who refused to participate were older, were more likely to be white, and had lower education than those who participated. Those lost to follow-up were more likely to be nonwhite, to have lower education, and to have cognitive impairment at baseline than those who participated. Attrition analysis in the Baltimore ECA has been detailed previously.¹⁷ The study was approved by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board. All participants provided written informed consent.

At each wave, trained interviewers administered the Diagnostic Interview Schedule,¹⁸ a structured interview that yields psychiatric diagnoses based on DSM-III or DSM-III-R criteria. At waves 1 and 2, the Diagnostic Interview Schedule version III¹⁹ (based on DSM-III criteria) was used; at waves 3 and 4, the Diagnostic Interview Schedule version III-R²⁰ (based on DSM-III-R criteria) was used. At each wave, lifetime history of the following 6 mental disorders was evaluated and used for the present analysis: MDD, obsessive-compulsive disorder (OCD), panic disorder, social phobia, alcohol abuse or dependence, and drug (including cocaine, marijuana, stimulants, sedatives, and tranquilizers) abuse or dependence. These disorders were evaluated at all waves; schizophrenia and bipolar disorder were evaluated at waves 1 and 2 and simple phobia was examined at waves 1 through 3, but these mental disorders were not included in later waves and were therefore excluded from this study. In our study, lifetime diagnosis at a visit includes both the current episode and past episode(s) reported in the interview at that visit. Failure to recall lifetime mental disorders was defined as not meeting criteria for the lifetime history of the mental disorder at wave 4 despite reporting symptoms that met criteria for that disorder at 1 or more previous waves. Global cognitive function was assessed using the Mini-Mental State Examination (MMSE).²¹

At each wave, participants were also asked whether they had ever had the following physical illnesses: diabetes, hypertension, arthritis, stroke, and cancer (any type).

Sociodemographic characteristics included in the analyses were based on wave 4 reports and included age at wave 4 (≤49, 50-59, and ≥60 years), sex, race (white vs nonwhite, which included African American, Hispanic, Asian, Native American, and Pacific Islander), educational attainment (<12 vs ≥12 years), marital status (married vs not married), and employment status. Mental health service use (≥1 visit within the 6-month period prior to the wave 4 interview vs 0 visits within the past 6 months) was also recorded.

Lifetime prevalence was estimated using the assessments of lifetime disorders at each wave over waves 1 through 4 (prospective cumulative method) and compared with the lifetime prevalence based solely on the assessment of lifetime disorders at wave 4 (cross-sectional retrospective method). Cumulative prevalence of mental and physical disorders was defined as the sum of the participants who met the criteria at least once in previous and current interviews. Confidence intervals were calculated for each prevalence estimate. We first compared retrospective and cumulative lifetime prevalence of mental and physical disorders by the conservative method of examining the overlap of confidence intervals.²²

We also compared the 2 prevalence estimates using McNemar test, which is appropriate for comparing dichotomous responses within the same individual.²³ In further analyses, we also calculated the cumulative prevalence of mental disorders excluding participants who had an MMSE score lower than 24 to determine whether cognitive decline accounted for results. Multivariable logistic regression analysis was used to identify factors associated with failure to recall mental disorders at wave 4. Statistical analyses were carried out using SPSS version 20 statistical software (IBM Corp).

Because different versions of diagnostic criteria for mental disorders were used for waves 1 and 2 (DSM-III) and waves 3 and 4 (DSM-III-R), we compared (1) cumulative prevalence of mental disorders based on waves 1 and 2 and retrospective lifetime prevalence at wave 2 and (2) cumulative prevalence of mental disorders based on waves 3 and 4 and retrospective lifetime prevalence at wave 4, to determine whether differences in diagnostic criteria may have affected results.

Participants had a mean (SD) age of 35.4 (12.9) years at wave 1 and 58.9 (12.9) years at wave 4. Among 1071 participants, 397 (37.1%) were male, 662 (61.8%) were white, 374 (34.9%) were African American, and 33 (3.1%) were other races/ethnicities; 581 of the 1071 participants (54.2%) were married at the wave 4 interview. The mean (SD) educational attainment was 12.4 (2.8) years. Employment status at the wave 4 interview was reported by 1028 of the 1071 participants (96.0%); among these 1028 participants, 605 (58.9%) were employed, 196 (19.1%) were keeping house, 146 (14.2%) were retired, and 81 (7.9%) were unemployed or disabled. Among the 1071 participants, 947 (88.4%) completed the MMSE at wave 4. Participants’ mean (SD) MMSE score was 28.1 (3.1), and 50 participants (4.7%) had an MMSE score lower than 24. Of the 1071 participants, 129 (12.5%) had used a mental health service within 6 months prior to the wave 4 interview.

The lifetime prevalence estimates of mental and physical disorders according to the cross-sectional retrospective method are presented for each wave in Table 1. For the most part, the prevalence of mental disorders did not systematically vary across the years, even as respondents aged and had more years of exposure to the risk of disorder. In contrast, the prevalence of physical conditions increased with the aging of the cohort.

In analyses of mental disorders, the wave 4 lifetime prevalence estimates according to the cross-sectional retrospective method were markedly lower than estimates based on the prospective cumulative lifetime data from all 4 waves as shown by nonoverlapping 95% confidence intervals as well as the results of McNemar test (Table 2). The respective lifetime prevalence estimates ascertained by these 2 methods were 4.5% vs 13.1% for MDD, 0.6% vs 7.1% for OCD, 2.5% vs 6.7% for panic disorder, 12.6% vs 25.3% for social phobia, 9.1% vs 25.9% for alcohol abuse or dependence, and 6.7% vs 17.6% for drug abuse or dependence. The ratios of wave 4 lifetime prevalence estimates to cumulative prevalence estimates ranged from 2 to 12, indicating 2- to 12-fold differences among the 2 estimates.

In contrast, retrospective lifetime prevalence estimates of physical disorders ascertained at wave 4 were much closer to those based on cumulative data from all 4 waves. The ratios of wave 4 lifetime prevalence estimates to cumulative prevalence estimates ranged from 1.1 to 1.3. However, the 95% confidence intervals for the retrospective prevalence and cumulative prevalence of hypertension and arthritis did not overlap, indicating that retrospective lifetime prevalence was lower than cumulative lifetime prevalence for these disorders. These findings were corroborated by McNemar test results, which indicated significantly higher lifetime cumulative prevalence estimates across all physical disorders. The respective prevalence estimates ascertained by the 2 methods were 18.2% vs 20.2% for diabetes, 48.4% vs 55.4% for hypertension, 45.8% vs 54.0% for arthritis, 5.5% vs 7.2% for stroke, and 8.4% vs 10.5% for cancer (Table 2). The Figure presents cumulative prevalence and retrospective lifetime prevalence of mental and physical disorders at each wave.

Excluding participants with an MMSE score lower than 24 did not meaningfully change the results. After excluding these participants, the retrospective vs prospective lifetime prevalences respectively were 3.8% vs 12.8% for MDD, 0.7% vs 7.0% for OCD, 2.4% vs 6.6% for panic disorder, 12.3% vs 25.5% for social phobia, 14.5% vs 27.1% for alcohol abuse or dependence, and 6.7% vs 19.4% for drug abuse or dependence.

Among participants with a lifetime history based on prospective cumulative evaluation across 4 waves, 94 (67.1%), 70 (92.1%), 46 (63.8%), 142 (52.4%), 183 (66.1%), and 125 (66.4%) did not meet the criteria for MDD, OCD, panic disorder, social phobia, alcohol abuse or dependence, and drug abuse or dependence, respectively, at the wave 4 retrospective evaluation. Compared with participants aged 49 years or younger, those aged 60 years or older had greater odds of failing to recall lifetime history of MDD (odds ratio [OR] = 6.69; 95% CI, 1.54-29.06), panic disorder (OR = 14.42; 95% CI, 1.96-106.36), alcohol abuse or dependence (OR = 16.41; 95% CI, 5.56-48.48), and drug abuse or dependence (OR = 22.64; 95% CI, 2.20-232.96) (Table 3). Participants aged 50 to 59 years also had greater odds of failing to recall panic disorder (OR = 5.38; 95% CI, 1.43-20.21) and alcohol abuse or dependence (OR = 1.94; 95% CI, 1.05-3.60) at wave 4. Higher educational attainment (≥12 vs <12 years) correlated with greater risk of underreporting social phobia (OR = 3.07; 95% CI, 1.60-5.87), and married participants had greature odds of failure to recall alcohol abuse or dependence (OR = 1.85; 95% CI, 1.02-3.33). In contrast, visits to a mental health care professional within the 6 months prior to the wave 4 interview were associated with a lower likelihood of recall failure of OCD at wave 4 (OR = 0.04; 95% CI, 0.00-0.68). Being disabled or unemployed at wave 4 was negatively associated with recall failure of social phobia (OR = 0.32; 95% CI, 0.11-0.93). No other sociodemographic characteristics were associated with failure to recall past mental disorder episodes (Table 3).

In the comparison of lifetime prevalence of mental disorders using retrospective data from wave 2 with cumulative data from waves 1 and 2, retrospective data consistently underestimated the prevalence of all mental disorders studied; retrospective prevalence estimates ranged from 1.2 to 3.1 times lower than cumulative prevalence estimates (eTable 1 in Supplement). Similarly, estimates of lifetime prevalence of mental disorders obtained retrospectively at wave 4 were 1.6 to 2.7 times lower than estimates based on cumulative prevalence using data from waves 3 and 4 (eTable 2 in Supplement).

We found that estimates of the lifetime prevalence of mental disorders were 2 to 12 times lower when based on cross-sectional retrospective reports from a single interview compared with estimates based on cumulative reports from multiple interviews. This was true for the range of disorders we studied. This finding corroborates previous research comparing lifetime prevalence estimates based on 1 retrospective interview with those based on multiple interviews.^13,15,16 Past studies, however, were mainly based on samples of children, adolescents, and young adults. Findings from our study indicate that failure to recall lifetime episodes of mental disorders is not limited to this age range and extends to midlife, when mental disorders are most prevalent,²⁴ and to older adulthood.

The lifetime prevalence estimates of common physical disorders presented a strong contrast in that they did not differ meaningfully between the 2 ascertainment methods. The contrast between recall of mental and physical disorders is noteworthy and may be attributable to differences in assessment, age at onset, and course of these disorders. While ascertainment of mental disorders in our study was based on symptom criteria (ie, endorsement of a certain number of symptoms from various categories), ascertainment of physical illnesses was based on a participant’s report of presence vs absence of a particular physical disorder (eg, diabetes, hypertension). Furthermore, many of the mental disorders assessed in this study have an early age at onset and a course characterized by remission and relapse, whereas the medical conditions assessed are typically illnesses of middle and older age and tend to have a chronic, nonremitting course. These differences can partly explain variations in recall. The results of our logistic regression analyses support these conclusions; older participants were less likely to recall past psychiatric episodes, whereas participants with recent mental health care visits were more likely to recall past psychiatric episodes. Mental disorders have a lower point prevalence and 1-year prevalence in older age groups^1,2,25 and participants currently in treatment are more likely to meet criteria for current mental illness. Besides simply forgetting previous episodes, positive reframing of past episodes based on current circumstances may also lead to recall failure.²⁶ Finally, mental disorders are still associated with a substantial level of stigma,²⁷ potentially leading to less willingness to disclose psychiatric symptoms. The level of stigma may be evolving over calendar time, with less stigma now than earlier times, leading older individuals to be less likely to recall or report symptoms than younger individuals, contributing to the failure of recall.

To our knowledge, this is the first study that has compared the lifetime prevalence estimates of both mental and physical illnesses using retrospective and prospective approaches across the adult life span. The study also extends past research comparing estimates in adolescents and young adult samples^13,16 or limited to depression.^9-12 In our study, subjects were followed up from a mean (SD) age of 35.4 (12.9) to 58.9 (12.9) years and we evaluated several mental disorders. Nevertheless, our results are similar to those of the study by Moffitt et al,¹³ who showed higher lifetime prevalence estimates of psychiatric diagnoses across ages 18 to 32 years compared with retrospective assessments. Taken together, these studies raise doubts about the validity of lifetime prevalence estimates of mental disorders obtained in retrospective population surveys. Given the results of our study, the true lifetime prevalence of mental disorders may be considerably higher than those reported based on retrospective data—particularly in studies of middle-aged and older adults. The findings also suggest that surveys with a retrospective evaluation of mental disorders may not be appropriate for etiological research on the mental disorders as many individuals with lifetime history of such disorders would be misclassified as negative cases.

The estimates of lifetime prevalence of mental disorders ascertained by multiple interviews in this study are lower than those in the studies by Moffitt et al¹³ and Olino et al.¹⁶ For instance, the lifetime prevalence of MDD was estimated at 13.1% in our study compared with 41% in the study by Moffitt and colleagues. Several factors may be contributing to this discrepancy, including differences in age and racial distribution of the samples and assessment instruments. Alternatively, we may have missed a substantial amount of information related to mental disorders before age 35 years—the average age of the ECA sample at baseline. Thus, even the cumulative ECA data may have underestimated the true lifetime prevalence of many mental disorders, especially those with early ages at onset.

It might be plausible that the association of failure to recall past psychiatric symptoms with older age demonstrated in this study is due to the influence of cognitive decline because older age is one of the major risk factors of cognitive decline. However, excluding subjects with an MMSE score lower than 24 did not change our conclusions. In addition, we also entered MMSE score as a continuous variable and, separately, as a discrete variable (MMSE score <24 vs ≥24) in the regression analyses. In these analyses, MMSE scores did not predict recall failure of any mental disorders (data not shown). Therefore, the correlation of older age and recall failure of psychiatric symptoms may be due to factors other than significant cognitive decline. However, as reported in our previous study,¹⁷ cognitive impairment at baseline was associated with loss to follow-up; therefore, participants of this study may be less cognitively impaired than a random sample of the general population at the same age. In addition, because 12% of participants in this study did not complete the MMSE, it is possible that we have underestimated the impact of cognitive impairment on recall. On the other hand, more sensitive measures of cognitive impairment might reveal a contribution of cognitive decline to failure to recall past episodes of mental disorders.

In our study, lifetime prevalence estimates of mental disorders were defined by meeting the diagnostic criteria at any time during the lifetime. Therefore, any discrepancy between cumulative diagnoses and the wave 4 retrospective cross-sectional diagnoses would, by definition, indicate lack of accuracy of wave 4 diagnosis, suggesting that prevalence estimates of mental disorders in retrospective data may be less accurate than cumulative data. Recall of mental disorders can be affected by many factors such as aging, stigma, and interaction with a mental health care professional as discussed earlier. Also, it is possible that recency, chronicity, or severity of illness could affect recall, although we were not able to assess the impact of these latter factors in this study. However, unemployment and disability, which are associated with severity of mental health conditions, were associated with lower likelihood of failure to recall some mental disorders, as was recent mental health care contact, which may be associated with the recency of mental health problems. Because we used participants’ employment status at the wave 4 interview, it is possible that recent mental illness episodes at wave 4 would be associated more strongly with both recall of such episodes and unemployment or disability status at this wave.

Several limitations to this study should be noted. First, the ECA study used different versions of DSM criteria for waves 1 and 2 vs waves 3 and 4, although our sensitivity analysis similarly showed differences between wave 2 estimates and wave 1 and 2 cumulative estimates and between wave 4 estimates and wave 3 and 4 cumulative estimates. Second, it did not include all mental disorders. In particular, we were unable to assess several disorders that tend to have a recurrent or chronic course because they were not assessed in all waves. Likewise, we were not able to assess physical illnesses with early onset or an episodic nature, such as asthma, infectious diseases, or head injury. Future studies should include such episodic, remitting physical conditions to examine whether these attributes could explain differences in recall with mental disorders. Third, further analyses may be needed to compare participants who consistently recalled mental disorders in all 4 waves and those who did not; however, the modest sample size of this study did not permit such analyses. Fourth, because we limited our samples to participants who were interviewed at both wave 1 and wave 4, this selective attrition would impact representativeness and external validity of our results, although it would not impact the internal validity of our findings. Finally, as discussed earlier, we were unable to consider the duration, the severity, or the time since the last episode of mental and physical disorders despite past studies demonstrating an association between these factors and recall of past symptoms.^10-12 It is also important to examine whether mental or physical health crises differently affect recall of mental disorders and physical disorders, but health crisis data were not available in the ECA. All of these issues should be investigated in future studies.

The results of our study raise questions about the accuracy of lifetime prevalence estimates of mental disorders in general population surveys based on a single retrospective recall. When comparing cumulative reports over multiple waves and cross-sectional reports of medical conditions, self-reports of lifetime mental disorders may grossly underestimate the true lifetime prevalence of mental disorders.

Corresponding Author: Yoichiro Takayanagi, MD, PhD, Department of Mental Health, Bloomberg School of Public Health, The Johns Hopkins University, 624 N Broadway, Baltimore, MD 21205 (ytakayan@jhsph.edu).

Submitted for Publication: April 15, 2013; final revision received July 12, 2013; accepted August 15, 2013.

Published Online: January 8, 2014. doi:10.1001/jamapsychiatry.2013.3579.

Author Contributions: Dr Takayanagi had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Takayanagi, Gallo, Eaton, Mojtabai.

Acquisition of data: Roth, Gallo, Eaton.

Analysis and interpretation of data: Takayanagi, Spira, Eaton, Mojtabai.

Drafting of the manuscript: Takayanagi, Gallo, Mojtabai.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Takayanagi.

Obtained funding: Eaton.

Administrative, technical, or material support: Roth.

Study supervision: Spira, Gallo, Eaton, Mojtabai.

Conflict of Interest Disclosures: None reported.

Funding/Support: This work was supported by grant DA026652 from the National Institute on Drug Abuse. Dr Spira is supported by Mentored Research Scientist Development Award 1K01AG033195 from the National Institute on Aging.

Role of the Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.