Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
November 2001

Psychological Screening of Children for Participation in Nontherapeutic Invasive Research

Author Affiliations

From the College of Nursing (Drs McCarthy and Rubenstein) and Divisions of Pediatric Psychology (Dr Richman) and Pediatric Endocrinology (Dr Hoffman), Department of Pediatrics, College of Medicine, The University of Iowa, Iowa City. Dr Hoffman is now with the Department of Pediatrics, Columbus Children's Hospital, Columbus, Ohio.


Copyright 2001 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2001

Arch Pediatr Adolesc Med. 2001;155(11):1197-1203. doi:10.1001/archpedi.155.11.1197

Background  The need for children to participate in research has raised concerns about ethical issues surrounding their participation.

Objectives  To describe a protocol of preresearch psychological screening and postresearch outcomes and to present the results of the screening process for a nontherapeutic, invasive research study.

Design and Setting  Descriptive study carried out at The University of Iowa Hospitals and Clinics, Iowa City.

Participants  Twenty-eight children (mean age, 10.6 years) were screened, with 4 not completing the research study and another 4 unavailable for psychological follow-up.

Main Outcome Measures  Prescreening interviews with parent and child and screening measures of appropriate child cognitive abilities and behavior; postscreening parent and child questionnaires.

Results  Of the 4 children who did not complete the research study, 3 were identified with increased anxiety during the screening and were advised to not participate in the study. The primary motivator for participation was monetary reimbursement (14 parents [82%]; 15 children [75%]), followed by altruistic reasons (10 parents [59%]; 4 children [20%]). Before participating, none of the children reported concerns related to participating in the study. However, on follow-up, 9 (45%) of the children reported that they had had concerns before participating. Follow-up assessment showed that parents underestimated their children's concerns related to sexual development assessment and intravenous insertion.

Conclusions  Children with increased anxiety may not be appropriate participants in potentially anxiety-provoking research. Children's reports of concerns may change from preparticipation to postparticipation, and discrepancies may exist between parent and child reports of concerns with research participation. Further research is needed to ensure children's safe participation in research.

THERE IS A NEED for children to participate in research, as recognized by the recent National Institutes of Health research guidelines that require inclusion of children in studies, unless there are overriding reasons to exclude them.1 It has also been established by the US Department of Health and Human Services2 that children require additional protection, beyond adult guidelines, when involved as subjects in research. According to federal guidelines,2 a child may participate in research if informed consent (permission to participate) is obtained from the parent or legal guardian, the child assents (agrees), and the research falls into one of the risk categories established for research with children.1 Each institutional review board (IRB) sets the age at which a child should provide assent to participate; usually this is 7 years or older.3

A number of issues need to be clarified in following these government mandates and additional protections for children. One issue is that of a child's decision-making ability related to assent.4 For adults, elements of informed consent include the ability to understand, show evidence of reasonable choice, and hypothesize possible reasons for withdrawal.5,6 Chronological age does not necessarily guarantee a related degree of cognitive ability or decision-making skills. When child assent is considered,710 developmental status, not just age, must be assessed. This is not, however, just a matter of measuring a child's cognitive ability.4 There is also concern regarding the child's emotional state and the influence of social desirability to participate.8 For example, there is the dilemma of parents who volunteer and encourage their children to participate in research, raising concerns of whether the the child is giving informed assent or conforming to parental expectation.7,9

While there has been controversy over child participation in research,6,11,12 there is developing agreement by many researchers, ethicists, and IRBs that children should be allowed to participate in research, including nontherapeutic research, where there is no direct benefit to the child, if there is minimal risk or only minor increase in minimal risk to the child. However, there are ethical concerns related to the need to protect children participating in such research. As more children are involved in research studies, recognized procedures for identifying children who are appropriate for such research need to be developed. The purpose of this article is to describe a psychological screening protocol, developed at the request of a human subjects IRB, for prestudy screening of children involved in a nontherapeutic, invasive research study and to present results of the prestudy screening and follow-up data obtained on the children and parents who participated.


This study was developed in response to a request by an IRB related to a nontherapeutic, invasive research study involving children. The section that follows presents a description of the proposed nontherapeutic research study, the IRB's response, and the involvement of psychologists to assist with enrollment and follow-up monitoring of subjects.


The purpose of the research being reviewed by the IRB was to explore longitudinal changes in peripheral and hepatic insulin sensitivity in prepubertal and early pubertal children and to determine predictors for changes in insulin resistance and growth velocity. This study will be referred to as the Insulin Sensitivity Study.13 The subjects of the Insulin Sensitivity Study, carried out by one of us (R.P.H.), were to be 2 groups of healthy children, aged 8 to 13 years, one group classified as Tanner stage 1 for breast or genital development and the second group classified as Tanner stage 2. Each child would be admitted overnight 4 times, during 18 months, to the Clinical Research Center at the hospital. During each hospitalization, the child would be examined to identify Tanner stage of pubertal development, height, weight, and body fat. An intravenous line (IV) would be placed in each arm to have blood drawn for measurement of growth hormone, glucose, and other relevant variables, and for dextrose bolus. Subjects would receive $50 per hospital admission.

The IRB members questioned the impact on the children of overnight admissions to the Clinical Research Center and the accompanying tests. However, they believed it ikely that the study would yield important generalizable knowledge. Therefore, in an effort to minimize this potential minimal risk to the children in the proposed study, the IRB requested that pediatric psychologists (A.M.M. and L.C.R.) advise the investigator about procedures for identifying subjects appropriate for inclusion in the study and ensure that participants were followed up to determine any possible after-study effects.


We developed a brief psychological screening protocol, outlined in Table 1, to identify children who might be at risk for psychological distress by being in the study. Little information was available in the literature on screening children for involvement in nontherapeutic, invasive research. Therefore, on the basis of our experience, we designed a protocol that included interviewing the child and parent separately and screening the child for age-appropriate cognitive abilities and behavioral concerns. Instruments were included that are commonly used clinically and in research. The protocol took approximately 45 minutes to complete. Separate compensation for participation in the psychological screening sessions was not provided.

Table 1. 
Image not available
Psychological Screening Protocol*

The purpose of the interviews was to clarify both the child's and the parents' understanding of what was involved in participating in the study. The interviews were carried out on the day the families came to the medical center to discuss involvement in the study. The medical researcher met with the family; reviewed the study, including the psychological screening component;and obtained parental consent and child assent. Then the psychologist interviewed the parent and child separately. Each was asked to describe how he or she learned about the study, the motivation for participating, the parent's plans to spend the night during the child's hospitalizations, the child's experiences with hospitalization and separation from family, child developmental or academic concerns, and concerns the child or parent might have related to the study. The protocol for the study was reviewed to clarify that the participants knew what procedures would be carried out, that participation was voluntary, and that they might drop out of the study at any time.

Cognitive ability and behavioral concerns were screened after the interviews. To verify that the children had age-appropriate verbal intellectual ability to understand the basic aspects of the research project, cognitive skills were screened with the Wechsler Intelligence Scale for Children–III subtests, similarities and information.14 Ability to read the assent forms was confirmed by screening the children's reading ability with the Wide Range Achievement Test–3 to ensure that each child had reading skills at the third grade level or above.15 Once it was ascertained that the child showed adequate understanding, the child was screened for possible internalizing behavior problems, specifically increased anxiety and depression. The procedures required for the proposed study were thought to be potentially stressful for highly anxious children. The younger children (8-11 years) completed the State-Trait Anxiety Inventory for Children (STAIC)16 and the Children's Depression Inventory,17 and the older children (12-13 years) completed the Symptom Checklist-90-R.18 This information was supplemented by having parents complete the Pediatric Behavior Scale19 to identify both internalizing and externalizing problems. Before screening, we decided that if a child demonstrated problems in understanding the study protocol, showed diminished cognitive abilities, or scored greater than the 90th percentile on measures of anxiety, depression, or parent-reported behavioral concerns, then the child would not be considered an appropriate candidate to participate in the study. Two pediatric psychologists (A.M.M. and L.C.R.) individually reviewed data on each child, with the above criteria considered, and made recommendations to the family and the principal investigator of the research study (R.P.H.).

After completion of the Insulin Sensitivity Study, each child and a parent were contacted to complete a questionnaire describing their experiences. Families were mailed 2 follow-up questionnaires, one to be completed by the child who had been in the study and the other to be completed by a parent. A follow-up reminder was mailed to families who had not returned the questionnaires 3 weeks after the first mailing. The follow-up questionnaires included questions listed in Table 1. Likert questions on a scale of 1 to 5 were included to measure comfort with assessment of sexual development, IV insertions, blood draws, and staying overnight for each of the 4 hospital stays.


All descriptive statistics were generated by means of SAS statistical software.20 Because of the small sample size in our data set and the descriptive nature of our report, most statistical tests and measures of agreement were generated with StatXact4,21 a software package for analysis of small data sets. Cochran-Mantel-Haenszel methods were used to assess stratified data with nominal and ordinal categories.20 Exact nonparametric methods included Wilcoxon rank sum test for 2 independent samples, Fisher exact test for 2 independent samples, Page test for related samples and ordered categorical data, and Kruskal-Wallis methods for multiple independent samples and ordered categories.21 Magnitude of agreement was measured with Cohen κ for data with nominal categories and a weighted κ for data with ordered categories.22,23


Twenty-eight children, from 24 families, were screened for participation in the Insulin Sensitivity Study. There were 15 boys (54%) and 13 girls (46%), with a mean age of 10.6 years (SD, 1.8 years; range, 8-14 years) and a mean grade level of 5.2 (SD, 1.7; range, second to eighth). None of the children had repeated a grade, although 2 were receiving resource room assistance in school. All of the parents had at least a high school education, with 33% of the fathers and 38% of the mothers having a college degree or higher. Only 2 fathers participated.

Of the 28 children screened, 24 participated in the Insulin Sensitivity Study. Four children who were screened did not participate in the study, and 4 children who did participate were unavailable for follow-up of the psychological screening. This left 20 children and their parents from 17 families who completed the follow-up questionnaires.

Parent Interviews

The parents reported that they learned about the study primarily from newspapers (14 parents [58%]) and a friend or family member (11 [46%]). The 2 reasons most frequently given for parents wanting their child to participate in the research study were for the financial reward (13 parents [54%]) and altruistic reasons (10 [42%]). Most parents (17 [71%]; missing data in 7 [29%]) believed that they and their children were prepared to participate in the research and recognized the voluntary status of this research (23 parents [96%]). Generally, parents planned to stay with their children during the hospitalizations (17 [71%]).

Child Interviews

In separate interviews with the children, the children reported learning about the study from their parents. All of the children stated that they understood that participation in the study was voluntary, and none of the children reported concerns about being in the study.

Cognitive Screening

All of the children were within normal ranges on cognitive and reading testing. The mean standard score on the Wechsler Intelligence Scale for Children–III information subtest was 11.5 (SD, 1.4), and on the similarities subtest, 11.6 (SD, 2.0) (average subtest scores are 10). The mean standard score on the Wide Range Achievement Test–3 reading test was 99.9 (SD, 10.9) (average score is 100).

Anxiety and Depression Screening

Of the 28 children, 20 were in the age range to complete the STAIC and the Children's Depression Inventory. On the STAIC, the mean state score (current level of anxiety) was 26.8 (SD, 4.5), with 18 children scoring below the mean (less anxiety) and 2 children scoring more than 1 SD above the mean (more anxiety). The 2 children who were more than 1 SD above the mean were among the 4 children who did not complete the research study. The mean trait score (general level of anxiety) was 30.8 (SD, 6.8), with 15 children scoring below the mean, 1 scoring slightly above the mean, and 3 scoring more than 1 SD above the mean (trait data missing on 1 child). The 3 children who were more than 1 SD above the mean were children who did not complete the research study.

On the Children's Depression Inventory, the mean percentile score was 24.8 (SD, 26.3; range, 2-87). Of the 20 children, 18 were in the normal range and 2 were borderline to above (more depression) the normal range (more than +1 SD). The child with a borderline score was within the normal range on all other measures, related some of her concerns on this measure to her dislike of school, had a sibling participating in the study, and had successfully participated in a similar research study. She successfully completed participation in this study. The child with an elevated score also had elevated scores (+1 SD) on the STAIC and was among those who did not complete the study. Only 8 of the children were in the age range for the Symptom Checklist-90-R. All 8 children scored in the normal range.


Parents completed a Pediatric Behavior Scale for each child. On this instrument with 24 subscales, 13 of the 28 children were reported by their parents to be at or above the 90th percentile on at least 1 subscale. However, 11 of these children had elevated scores on only 1 or 2 scales, and typically these elevations were related to concerns with eating, clumsiness, arousal, or school issues. All 11 children were in the normal range on the self-report measures of anxiety and depression. Of the 2 remaining children, 1 child had a diagnosis of attention-deficit/hyperactivity disorder, was treated with methylphenidate hydrochloride, and had elevated scores on subscales related to attention-deficit/hyperactivity disorder and school. He was in the normal range on the self-report measure of anxiety and depression and successfully participated in the research study. The final child had elevated scores on scales related to anxiety, self-esteem, attention, and school; displayed increased anxiety on self-report; voiced concerns about needles; and was one of the children who did not complete the study. Interestingly, only 1 parent reported increased child anxiety.


As stated earlier, of the 28 children screened for participation in the Insulin Sensitivity Study, 24 actually participated and 4 did not. The 4 children who did not participate included 2 boys and 2 girls, aged 8 to 13 years with a mean age of 10.9 years (SD, 2.2 years) in grades 3, 4, 6, and 8. These 4 children did not differ on cognitive measures from the 24 children who did participate, but 3 of the 4 children had elevations on measures of anxiety. Means, SDs, medians, and ranges for anxiety scores and statistically significant differences are presented in Table 2.

Table 2. 
Image not available
Descriptive Statistics of State-Trait Anxiety Scores by Subgroups

The researcher and parents of the children with elevated anxiety scores were informed of the findings and cautioned regarding the child's participation in the study. The parents of the first child with increased anxiety initially decided to continue the child's participation in the Insulin Sensitivity Study. However, during the first hospitalization, the child became ill when the IV insertion was attempted and subsequently dropped from the study. When 2 other children were noted to have elevations on anxiety, depression, and/or a fear of needles the families were advised, with the support of the researcher, to not participate in the study and did not. The fourth child, who was in the normal range on all screening measures, dropped from the study because of difficulty starting the IV during the first hospitalization.


Twenty children (mean age, 10.8 years; SD, 1.6 years) and parents from 17 families completed the follow-up questionnaires. Three children who participated in the research but who were unavailable for follow-up scored significantly lower (less anxious) on the STAIC measure of anxiety than did those who completed the study and 3 of the 4 who dropped out. Descriptive statistics for the anxiety scores are presented in Table 2.

Where possible, follow-up data obtained on the 20 children and their 17 parents were compared (Table 3). The primary motivation for participation in the study was financial reimbursement, with 14 (82%) of the parents and 15 (75%) of the children reporting this as a reason for participating. The second reason given was an interest in contributing to medical knowledge; however, this was more of a factor for the parents (10 parents [59%]) than for the children (4 children [20%]). Another area of interest was the concerns children reported related to participating in the study. During the prestudy interview, none of the children reported concerns; however, after participation in the study, 9 (45%) of the children reported concerns, including worries about staying overnight, needles, physical assessment, side effects, and overall safety.

Table 3. 
Image not available
Parent and Child Follow-up Responses

Likert questions were included that asked the child and the parent to rate their level of comfort, on a scale of 1(very comfortable) to 5 (not at all comfortable) on 4 aspects of participation in the study for each of the 4 hospitalizations. Children rated their own comfort levels, while parents rated their perceptions of their children's comfort levels. The 4 aspects rated were Tanner stage assessment (assessment of sexual development carried out by a physician of the subject's sex), IV insertion, blood draws, and staying overnight. The median scores for the 4 aspects across the 4 hospitalizations ranged from 1.0 to 3.0 for the children and from 1.0 to 2.0 for the parents, indicating overall low levels of concern by both groups. Over time, levels of comfort improved in all areas as reported by both parents and children. The highest levels of concern for both groups were noted for Tanner staging and IV insertion. The parents' perceptions were that the IV insertion was the most difficult for their children. Although the children reported IV insertion as a concern, they reported that assessment of sexual development was the most concerning aspect of the study across all 4 hospitalizations. No differences were noted in levels of concern between children who had a parent stay with them during a hospitalization and children who did not.

Agreement between parents' and children's perceptions of comfort were assessed via κ and weighted κ coefficients for magnitude of agreement between parent and child pairs.22Table 4 shows the proportions of parents underestimating their child's discomfort for the 4 aspects of study participation across the 4 hospitalizations. For example, parents tended to rate children as being at 1 or 2 on the comfort scale, while the children's self ratings were higher, 2 or 3, indicating more discomfort. For assessment of sexual development, parents underestimated their child's concern for all 4 visits, from 50% of the time at the first visit to 29% of the time at the last visit. For assessment of IV insertion, parents tended to underestimate their child's concerns for the first few visits, but for the last visit, 5 of 15 parents overestimated their child's concern, 2 underestimated, and 8 agreed with their child's assessment of their level of concern. Parents were less likely to underestimate their child's concerns about blood draws and staying overnight, with proportions of underestimation ranging from 6% to 25% across the 4 hospitalizations for these 2 experiences. Except for 4 κ coeffecients that reflected the documented paradox of high κ, low agreement, or low κ, high agreement,24 κ coefficients for assessment of sexual development and IV insertions were less than 0.60, indicating fair to moderate agreement, while for blood draws and staying overnight they ranged from 0.64 to 0.85, indicating more substantial agreement in these areas.23

Table 4. 
Image not available
Proportion of Parents Underestimating Child's Discomfort*

Differences in children's perceptions of comfort based on their sex were also assessed. There were no significant sex differences in children's level of comfort responses across the 4 hospitalizations for overnight stays, blood draws, and IV insertion (Page test, Kruskal-Wallis test, and Cochran-Mantel-Haenszel methods). Although discomfort levels decreased for both sexes across all hospitalizations, girls and boys rated their comfort levels differently for the Tanner assessment. At visit 1, 5 boys and 1 girl reported discomfort levels of 4 or greater. For the second and third hospitalizations, 6 boys indicated ratings of 3 or greater but only 3 girls reported similar reactions. At the last visit, all of the girls reported comfort levels of 1 or 2, but 6 of 11 boys reported comfort levels of 3 or 4. At each visit, except the first, the distribution of comfort responses was significantly different between boys and girls (Fisher exact 2-tailed test, P<.05), with boys having greater discomfort.

Before starting the Insulin Sensitivity Study, 17 (71%) of the parents reported they planned to stay with their children during the hospitalizations required for the study. On follow-up, parents were asked whether they stayed during each of the hospitalizations. Of the 17 parents who responded to the follow-up questionnaire, 14 had planned on staying with their child overnight, but only 10 did stay during the first hospitalization. This number dropped during subsequent hospitalizations, with only 4 parents in this group staying overnight during the fourth hospitalization. Overall, across the 4 hospitalizations for 20 children, a parent stayed with their child 57% of the time. This ranged from a parent staying during 77% of the first hospitalizations to 42% by the fourth hospitalizations.

None of the children or parents reported emotional or behavioral sequelae after participation in the study that they attributed to being in the study. All of the parents stated that they would allow their children to participate in future research, and only 1 child reported not wanting to be in research in the future "if there are IVs."


This study within a study was undertaken at the request of the IRB to try to identify children potentially at risk from participating in nontherapeutic, invasive research. The psychological screening process piloted indicated that parents and children understood the voluntary nature of the study and the procedures involved in participating. Children demonstrated normal aptitude and achievement, supporting developmentally appropriate understanding of the processes involved. However, behavioral screening suggests that children with increased anxiety may be at risk in participating in this type of study. If trait anxiety is already high, potentially anxiety-provoking experiences such as hospitalization and invasive procedures may elevate state anxiety to a detrimental level. Therefore, monitoring anxiety ensures not placing a child who is already anxious in an anxiety-provoking situation needlessly. Although only 1 child who showed an increase in anxiety began to participate in the study and then dropped out, it is unknown what the fate of the other 2 children with increased anxiety, who were advised not to participate in the study and did not, might have been.

Financial reimbursement was the primary motivation for participation in this study. Interestingly, parental report that financial reimbursement was a reason for participation increased from 54% to 82% on follow-up questioning. Contributing to medical knowledge was a low motivator for the children, but a stronger motivator for the parents. Although money is typically used in our society to pay for services, these results raise 2 questions. First, is money too powerful a motivator? It is possible that children may be coerced to be in a study by parents to obtain the financial reimbursement. Children in lower socioeconomic groups may be particularly at risk for this form of coercion. Practices of monetary reimbursement for children may need to be reassessed. It may be that providing less money, providing the money before participation, or providing nonmonetary rewards may be less coercive to families. Second, are parents indirectly persuading their children to participate in research because of the parent's interest in doing something for the greater good (altruism by proxy)? While this may appear to be positive, it may not be in the child's self-interest.

Two assumptions often made in pediatric health care are that children are truthful when they provide information and that parents know their children and, therefore, are able to accurately represent them. In this study, none of the children reported concerns with being in the study before participation, but after completion of the study, 45% reported that they had concerns before participation. The children may not have felt comfortable with the investigators, or, on the basis of their prior experience, may not have understood the extent of participation. The discrepancies between parent and child report of concerns with various aspects of the study suggest that parents may not be able to accurately represent their child's feelings. Interestingly, the children, particularly the boys, were more distressed by assessment of sexual development than perceived by their parents. While parental right to provide consent for their child is not questioned, the extent to which parents are able to accurately recognize their child's concerns is questioned.

As noted earlier, it is generally accepted that children should be allowed to participate in nontherapeutic research. The majority of the children in this study appeared to successfully participate and voiced interest in participating in similar research in the future. However, concerns regarding the process of this involvement exist. This pilot study provides beginning information on this participation. While children appear to understand the research process, they may not be totally forthcoming in their concerns about participating. However, the unspoken concerns of children do not necessarily interfere with their positive participation in research. Researchers cannot assume that parents will be accurate in their assessment of their children's concerns. While children with increased anxiety may not be appropriate participants in potentially anxiety-provoking research, exclusion of these children may result in a sample selection bias that may be detrimental to some research projects. Clearly, there needs to be further research on child participation in research.

Back to top
Article Information

Accepted for publication August 2, 2001.

This study was supported by Genentech Foundation for Growth and Development, Charlottesville, Va, and grant NIH RR000059 from the Clinical Research Center, The University of Iowa, Iowa City.

Corresponding author and reprints: Ann Marie McCarthy, PhD, RN, PNP, College of Nursing, Nursing Bldg, Room 430, 50 Newton Rd, The University of Iowa, Iowa City, IA 52242 (e-mail:

Conrad  BHorner  S Issues in pediatric research: safeguarding the children. J Soc Pediatr Nurs. 1997;2163- 171Article
Department of Health and Human Services, Additional protection for children involved as subjects in research. 48 Federal Register. 9818 (1983) (codified at 45 CFR 46 subpart D)
Broome  MEStieglitz  KA The consent process and children. Res Nurs Health. 1992;15147- 152Article
Dorn  LDSusman  EJFletcher  JC Informed consent in children and adolescents: age, maturation and psychological state. J Adolesc Health. 1995;16185- 190Article
Susman  EJDorn  LDFletcher  JC Participation in biomedical research: the consent process as viewed by children, adolescents, young adults, and physicians. J Pediatr. 1992;121547- 552Article
Akers  JABell  SK Should children be used as research subjects? Nurs Forum. 1994;2928- 33Article
Committee on Bioethics, Informed consent, parental permission, and assent in pediatric practice. Pediatrics. 1995;95314- 317
McCabe  MA Involving children and adolescents in medical decision making: developmental and clinical considerations. J Pediatr Psychol. 1996;21505- 516Article
Thompson  RA Vulnerability in research: a developmental perspective on research risk. Child Dev. 1990;611- 16Article
Range  LMCotton  CR Reports of assent and permission in research with children: illustrations and suggestions. Ethics Behav. 1995;549- 66Article
Ramsey  P The enforcement of morals: nontherapeutic research on children. Hastings Cent Rep. 1976;621- 30Article
McCormick  RA Experimentation in children: sharing in sociality. Hastings Cent Rep. 1976;641- 46Article
Hoffman  RPVicini  PSivitz  WICobelli  C Pubertal adolescent male-female differences in insulin sensitivity and glucose effectiveness determined by the one compartment minimal model. Pediatr Res. 2000;48384- 388Article
Wechsler  D Wechsler Intelligence Scale for Children–III.  San Antonio, Tex Psychological Corp1991;
Wilkinson  G Wide Range Achievement Test–3: Administration Manual.  Wilmington, Del Wide Range Inc1993;
Spielberger  CD Manual for the State-Trait Anxiety Inventory for Children.  Palo Alto, Calif Consulting Psychologists Press1973;
Kovacs  M Children's Depression Inventory Manual.  North Tonawanda, NY Multi-Health Systems1992;
Derogatis  LR Symptom Checklist-90-R: Administration, Scoring and Procedures Manual.  Minneapolis, Minn National Computer Systems Inc1994;
Lindgren  SDKoeppel  GK Assessing child behavior problems in a medical setting: development of the Pediatric Behavior Scale. Adv Behav Assess Child Fam. 1987;357- 90
SAS Institute Inc, SAS/STAT User's Guide, Version 6. 1 and 24th ed. Cary, NC SAS Institute Inc1989;
Mehta  CPatel  N StatXact4 for Windows.  Cambridge, Mass CYTEL Software1998;
Cohen  J A coefficient of agreement for nominal scales. Edu Psychol Meas. 1960;2037- 46Article
Seigel  DGPodgor  MJRemaley  NA Acceptable values of kappa for comparison of two groups. Am J Epidemiol. 1992;135571- 578
Feinstein  ARCicchetti  DV High agreement but low kappa, I: the problems of two paradoxes. J Clin Epidemiol. 1990;43551- 558Article