Copyright 2005 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2005
National household surveys often rely on parents’ recall to assess children’s use of health care services. However, little is known about the accuracy of parental reporting of hospitalizations and emergency department (ED) use.
To assess the agreement between maternal reported and medical record acute health care data for children younger than 3 years and to determine if agreement between the 2 varies by maternal characteristics.
Design and Methods
Data were obtained from the national evaluation of the Healthy Steps for Young Children for 2937 families who completed parent interviews at 2 to 4 and 30 to 33 months and whose children’s medical records were abstracted. Services assessed included hospitalizations and ED visits since birth (2-4 and 30-33 months) and in the last 12 months (30-33 months). Absolute and beyond chance agreements were calculated. Results were stratified by maternal age (<20, 20-29, or ≥30 years), parity (first-time, second-time, or greater mother), income (<$20 000, $20 000-$49 999, or ≥$50 000), and the presence or absence of maternal depressive symptoms.
Absolute agreement was high for hospitalizations (≥90%) at both time points. It was high for ED use (>90%) only at 2 to 4 months. Beyond chance agreement was higher for hospitalizations than for ED use at 2 to 4 and 30 to 33 months. Beyond chance agreement declined with increased duration of recall and younger maternal age. No differences were found by other maternal characteristics.
Mothers have good recall for acute health care events during the first 3 years of their children’s lives. This finding suggests that mothers are a good source of information regarding children’s acute health care use.
Acute health care use is a common outcome measure in children’s health services research.1- 3 Knowledge about inpatient hospitalizations and emergency department (ED) visits informs researchers and policy makers about access to health care, quality of care, and health care use expenditures.2 Overuse of acute health care often reflects limited access to primary care, inefficient use of health care dollars, and, thus, wasted resources.2,3 Specifically, use of the ED for nonurgent care4,5 and ambulatory-care-sensitive hospitalizations6,7 are 2 indicators of reduced access to primary care. Thus, accurate information about acute health care use is important.
Use may be measured through administrative records, medical record audits, or surveys. Administrative data sources often are restricted to single payers or institutions and may not reflect usage experience of children with changing insurance and multiple sites of care.3 Although medical records are often considered the “gold standard” for measurement of health care use,1,8,9 there are several potential problems. They include the cost and time associated with abstracting medical records, especially in large studies; the use of multiple service providers by some patients necessitating access to records from multiple sources; and the reality that some visits, especially visits to specialists and to EDs, may not be reported in the primary care record.1,10
Survey data about health care use are less expensive to obtain and are likely to be more inclusive of all sources of health care use. Questions arise, however, as to whether these self-reports are accurate.1,10 The quality of this information depends on the ability to remember and to report accurately.
Many population-based surveys including the National Health Interview Survey, National Survey of American Families, Medical Expenditures Panel Survey, and Community Tracking Household Survey use parent-reported data to determine children’s use of health care. Because surveys collecting health information on the child depend on the ability of a parent to recall and report on a wide range of medical conditions with which they may have little familiarity, the quality of those reports is uncertain.8 Many pediatric studies are solely based on parents’ retrospective reports of the child’s health use.8,10- 13
Studies of the validity of parental reports of their children’s use of health care have mixed results. For immunization histories, results suggest that mothers are inadequate informants,13- 18 while studies on recall of children’s birth weight found good recall.19,20 Good recall was also found for receipt of mental health services,21 receipt of chronic care services,8 and occurrence of otitis media.11 Many previous studies of maternal recall, however, examined only a single event such as birth weight,19,20 vaccination,16,17 or otitis media.11 Moreover, to our knowledge, there are no recent studies that have examined parental reporting of their children’s acute health care use.
Most prior literature regarding agreement between self-report and medical records for health care use has focused on senior citizens.22,23 However, findings from studies that have examined recall in elderly populations may not pertain to children. Children are different because they are dependent on adults for their health care use, and adults are the proxy respondents used to collect children’s health care use data.3 Moreover, children tend to have fewer acute health care events than adults.3
In general, prior research has shown that reporting accuracy is affected by the characteristics of the child, acuity of the illness, type of health care service, and respondent characteristics.1,18,24,25 There may be systematic differences (underreporting or overreporting) in recall based on seriousness of health events, income, ages of the parent and child, number of siblings, gender, educational attainment of the parents, duration of recall, type of service used, and the relation of the respondent to the index subject. Kosa et al12 found that the younger the age of the child, the more accurate the health care service report by the parents. Daly et al11 found that the more severe the health condition, the more accurate were parental reports of the details of health care service experiences. Research also suggests that the frequency of child health care service encounters as well as their recency positively affects reporting accuracy.14,19 Hawe et al16 found that parents with lower educational levels and parents with more children in the household tend to provide less accurate reports of child health care service encounters.
Prior studies, however, have not examined whether maternal mental health influences the accuracy of reporting of children’s acute health care services use. Maternal depressive symptoms have been documented to have negative influences on parenting practices including health care seeking for their children.26- 31 It is possible that poor maternal mental health could lead to overreporting of a child’s receipt of acute health care services, thus, accounting for the observed findings. A meta analysis by Richters32 determined that depressed mothers reliably report on their children’s behavior problems; however, they did not examine the influence of maternal mental health on reporting on children’s acute health care use. Determining the accuracy of maternal reporting is needed to understand the previously reported associations between maternal depressive symptoms and children’s increased ED visits27- 29 and hospitalizations.30,31
The objective of this article is to assess the agreement between maternal reports and medical records for children from birth to 3 years of age for hospitalizations and ED use. The study also examines whether agreement varies with the type of acute care (hospitalizations vs ED visits), period of recall, or maternal demographic and health characteristics. We hypothesized that (1) maternal reporting for acute health care use is as complete a source of information as medical record data; (2) maternal recall is more complete for hospitalizations than it is for ED use; (3) mothers have greater recall of acute health care events when their child is 2 to 4 months old than when their child is 30 to 33 months old; and (4) mothers reporting depressive symptoms overreport acute health care use relative to mothers without depressive symptoms.
This study relied on data collected for the National Evaluation of Healthy Steps for Young Children. Healthy Steps for Young Children is a new model of pediatric practice that supplements routine pediatric care with child development specialists and enhanced developmental services for families of young children.33- 35 In the evaluation of the Healthy Steps for Young Children, a cohort of 5565 children was followed up from birth to age 3 years at 15 pediatric practice sites in the United States.35
The study sample included 2937 families whose children participated in the Healthy Steps for Young Children, whose parents completed 30- to 33-month interviews, remained at the practice throughout the interventions, and had their child’s medical records abstracted. Of the Healthy Steps for Young Children cohort of 5565 children, 3737 families had completed 30- to 33-month interviews. Of the 3737 families who had completed 30- to 33-month interviews, 764 were excluded because they were no longer at the practice at 30 to 33 months, and 36 had no medical records abstracted. Additionally, 177 did not complete 2- to 4-month interviews.
Institutional review board approval was obtained for this study from The Johns Hopkins Bloomberg School of Public Health Committee on Human Research, Baltimore, Md. Informed consent was obtained at the time of enrollment and reviewed prior to each interview.
Data for this study were obtained from enrollment questionnaires, parental interviews at 2 to 4 months and 30 to 33 months, and medical records.32,34 Parents completed a brief standardized questionnaire available in both English and Spanish at the time of enrollment into the Healthy Steps for Young Children.33 This form provided demographic characteristics including maternal age and parity. Structured parent telephone interviews were conducted when the children reached 2 to 4 months and 30 to 33 months of age; virtually all the respondents were mothers.34,35 These interviews provided additional demographic characteristics including household income. During the interviews, respondents reported on children’s use of acute health care services and maternal depressive symptoms.
Data from the interviews and the medical records included the dates of hospitalizations and ED visits (including urgent care center use).35 Medical records for each child were abstracted through 32 months of age. Using a standard precoded form, trained abstractors at each evaluation site audited the medical records of participating children after the children reached 32 months of age.36 The abstractors participated in 2 to 3 days of competency-based training in the protocol.36 Abstracted visit data included the dates of hospitalizations and ED visits or urgent care visits.36 Interrater reliability for child hospitalizations and ED visits was 0.75 and the percentage of agreement was 92% for hospitalizations and ED visits.36
Dichotomous variables were used to describe hospitalizations and ED use. Hospitalizations since birth were reported by mothers in the 2- to 4-month and 30- to 33-month surveys. Hospitalizations in the last 12 months were identified in the 30- to 33-month survey. Emergency department visits since birth were assessed at 2 to 4 months, and ED visits in the last 12 months were identified from the 30- to 33-month interviews.
Maternal age, parity, household income, and depressive symptoms were characterized to determine whether agreement varied by maternal characteristics. Maternal age in years was categorized as follows: less than 20, 20 through 29, and 30 and older. Parity indicated if the mother was a first-time parent or second-time or greater parent. Maternal income was categorized as having an income of $20 000 or less, between $20 000 and $49 999, and $50 000 or more.
Maternal depressive symptoms were assessed at both interviews using a modified 14-item version35 of the 20-item Center for Epidemiologic Studies–Depression (CES-D) scale.37 The CES-D has a high internal consistency, with α coefficients of 0.85 for general population samples and 0.90 for patient samples.37 A 14-item, abbreviated scale was developed for the Healthy Steps for Young Children evaluation by assessing redundant items using data on 600 parents in an evaluation of child abuse prevention programs in Pennsylvania.35 The correlation of the reduced version with the full scale exceeded 0.95. The α coefficient for the 14-item CES-D in the Healthy Steps for Young Children sample for mothers at 2 to 4 months was 0.85, similar to that for the total scale for general population samples.34 In the 14-item instrument, response categories remained the same as the 20-item scale and scores of 11 or more corresponded with the cutoff of 16 for the 20-item scale.
We first examined the distribution of demographic characteristics and maternal depressive symptoms. We then calculated the level of agreement between survey data and medical record data for hospital and ED use, by calculating the proportion of absolute agreement and beyond chance agreement using the Cohen κ statistic. Beyond chance agreement is the extent to which the observed agreement (absolute agreement) between 2 sources or informants exceeds that which would be expected by chance alone.38 In this study the observed agreement and beyond chance agreement are calculated between maternal report data and medical record data. Beyond chance agreement is expressed as the proportion of the maximum improvement that could occur beyond the agreement expected by chance alone that the sources/informants achieved.38 The degree of beyond chance agreement is interpreted according to Landis and Koch39: 0.00 to 0.20 (poor), 0.21 to 0.40 (fair), 0.41 to 0.60 (moderate), 0.61 to 0.80 (substantial), and 0.81 to 1.00 (almost perfect). Beyond chance agreement was calculated for both the 2- to 4-month and 30- to 33-month interview data. Results were stratified by maternal age (<20, 20-29, or ≥30 years), parity (first-time or second-time or greater mother), household income (<$20 000, $20 000-$49 999, or ≥$50 000), and presence (yes/no) of maternal depressive symptoms at 2 to 4 months and 30 to 33 months. Analyses were conducted using Stata software (version 7.0; Stata Corporation, College Station, Tex).
About half of the mothers were between 20 and 29 years of age and first-time parents, while 36% of the study sample was in the high-income category (Table 1). At 2 to 4 months and again at 30 to 33 months, 15.5% of the mothers reported depressive symptoms.
A comparison of the survey responses and medical record data at both the 2- to 4-month and 30- to 33-month survey is given in Table 2. Mothers reported higher numbers of hospitalizations at the 2- to 4-month survey but not at the 30- to 33-month survey than the number recorded in the medical records. There was higher reporting of ED visits in the parent interviews at both the 2- to 4-month and 30-to 33-month surveys than from medical records. Absolute agreement was high for hospitalizations (≥90%) at both time points and high for ED use (>91%) only at 2 to 4 months.
The κs calculated for hospitalizations since birth from the 2- to 4-month and 30- to 33-month interview data and medical record data showed substantial agreement. Similarly, substantial agreement was found for ED visits at 2 to 4 months. Beyond chance agreement for both hospitalizations and ED visits declined with increasing age of the children (Table 2).
Substantial beyond chance agreement was found for hospitalizations since birth at the 2- to 4-month survey regardless of maternal characteristics including age, depressive symptoms, income, or parity (Table 3). For hospitalizations since birth at the 2- to 4-month survey, κs increased with increasing age of the mother and higher parity. Substantial agreement was also seen for hospitalizations since birth at the 30- to 33-month survey for all maternal characteristics, although agreement was lower than that for hospitalizations since birth, as reported in the 2- to 4-month survey. Only moderate agreement was seen for hospitalizations in the last 12 months at the 30- to 33-month survey regardless of maternal characteristics. Beyond chance agreement between acute health care use and medical record information was similar for mothers who reported having depressive symptoms at 2 to 4 months or at 30 to 33 months and mothers with no depressive symptoms at these 2 time points.
Moderate to substantial beyond chance agreement was seen for ED use at the 2- to 4-month survey for all maternal characteristics, while beyond chance agreement was only fair to moderate for ED use in the last 12 months at the 30- to 33-month interviews. For ED use since birth at the 2- to 4-month survey, κs increased from moderate to substantial with increasing household income.
The purpose of this study was to determine the level of agreement between mothers’ reports and medical record data for children followed up from birth to 3 years old for hospitalizations and ED use. The study also examined whether agreement varied with the nature of the acute health care event (hospitalizations vs ED visits), period of recall, and maternal demographic characteristics.
Results of this study show substantial beyond chance agreement between maternal recall and medical record data for hospitalizations since birth at both time points and for ED visits since birth at the 2- to 4-month interviews. The discrepancies between parental reporting and the medical records in this study may be attributed to 2 factors—inaccurate recall and incomplete medical records. Although the medical record is intended to be the complete source of a child’s health care services use history, medical records may not have complete information for care provided at all sites. This may be due to children seeking care at other health care facilities or with other physicians, and the documentation of this care not being sent to the child’s primary care physician to be included in the medical record.3 Also ED visits may not be included in the medical record owing to separate information systems that exist within the same health care institution.
One hypothesis of our study is that maternal recall is greater for hospitalizations than it is for ED use. Our findings show substantial agreement was found for hospitalizations since birth at both the 2- to 4-month and the 30- to 33-month surveys while κs calculated for ED visits since birth at the 2- to 4-month survey also showed substantial agreement but not at the 30- to 33-month survey. These results suggest that parents have greater recall of hospitalizations than ED visits, likely because of the increased acuity of a hospitalization as well as the trauma of the child not being at home.21 Based on previous studies, it is reasonable to assume that recall depends on the seriousness of an event,10 supporting our hypothesis.
We were interested in examining if mothers had greater recall of acute health care events when their child was 2 to 4 months old than when their child was 30 to 33 months old. Analyses showed that beyond chance agreement decreased with increasing age of the child. Other studies suggest that recall depends on the recency of an event.10,14,19 The more recent the health care event (both hospitalizations and ED visits), the greater the parental reporting and recall relative to the medical record. Mothers in our sample seem to be likely to remember the more acute and recent health care event when the child was an infant. Other studies on parental recall of hospitalizations similarly suggest that acute health care events in an infant are imprinted in the parent’s memory while time may blur the details and events, and they may be confused with other episodes of health care use.1,19
Older mothers in our study had greater recall of hospitalizations than did younger mothers when their children were infants. This finding is contrary to those of Pless and Pless10 who found that younger mothers rather than older mothers recalled more accurately. In our study, mothers with higher incomes and more children also had greater recall of recent hospitalizations than mothers for whom this was their first child. This finding is also contrary to the findings from Pless and Pless10 who found that mothers with fewer children had better recall of hospitalizations. No differences in recall were seen by maternal characteristics for ED use and hospitalizations reported at the 30- to 33-month survey.
We hypothesized that mothers in the study reporting depressive symptoms overreport acute health care use more often than mothers without depressive symptoms. Recent studies have shown that maternal depressive symptoms are significantly associated with children’s acute health care use.27- 31 Bartlett et al27 reported that mothers with depressive symptoms were 30% more likely to report ED use than mothers without depressive symptoms. Similarly, Minkovitz et al29 demonstrated that maternal depressive symptoms at 2 to 4 months were associated with a 1.4 increased odds of ED visits in the preceding year among toddlers. Guttmann et al31 reported that maternal depressive symptoms were significantly associated with the risk of hospitalization in children aged 12 to 24 months. Similarly Flynn et al28 found that elevated depressive symptoms in mothers were related to both missed pediatric outpatient visits and greater use of pediatric ED services. Chung et al30 found that mothers with persistent depressive symptoms were nearly 3 times as likely to have their child hospitalized in the first year of life than mothers without depressive symptoms.
Sixteen percent of mothers in our study reported having depressive symptoms at 2 to 4 months and at 30 to 33 months. Beyond chance agreement was similar for mothers with and without depressive symptoms at both time points. Analysis of the frequency of children’s hospitalizations at both surveys indicated no differential reporting of acute health care use by depressive symptoms. Similar analysis of frequency of ED visits by depressive symptoms at both surveys suggests more complete reporting by mothers than the medical records. This is consistent with children likely receiving ED care at multiple sites and incomplete primary care records.
Despite the large sample size and the availability of repeated measurements of health care use from interviews and medical records, the study has a few limitations that may not make the results generalizable. Families in this sample participated in the study for approximately 3 years and had the same source of primary care for 30 months. Thus, mothers in this study are more likely to be aware of health care use, and their children may have more complete medical records than mothers whose children do not have a consistent source of medical care. Additionally, an evaluation of the severity of the health conditions for the hospitalizations and ED visits was not conducted to determine if maternal recall of their children’s hospitalizations and ED visits differed by acuity of the conditions. Also, we were unable to do a more refined analysis of the duration of recall as to when specifically hospitalizations and ED visits occurred.
The results of our study suggest that maternal reporting is a complete source of information of event occurrence from household surveys and that maternal reporting does not differ substantially from the information available from the medical records especially for acute health care events occurring when the child is an infant. Accuracy of information on acute health care events may be affected by duration of recall and maternal age but less so by parity, household income, and maternal depressive symptoms.
Correspondence: Cynthia S. Minkovitz, MD, MPP, Department of Population and Family Health Sciences, The Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe St, Baltimore, MD 21205 (firstname.lastname@example.org).
Accepted for Publication: October 7, 2004.
Funding/Support: This study was supported by grant R03 HS13053-01 from the Agency for Healthcare Research and Quality, Rockville, Md.
Acknowledgment: William Hou, MS, Department of Population and Family Health Sciences, The Johns Hopkins Bloomberg School of Public Health, constructed the analytic files and provided programming technical assistance.
D’Souza-Vazirani D, Minkovitz CS, Strobino DM. Validity of Maternal Report of Acute Health Care Use for Children Younger Than 3 Years. Arch Pediatr Adolesc Med. 2005;159(2):167-172. doi:10.1001/archpedi.159.2.167