Copyright 2000 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2000
The Questionnaire for Identifying Children With Chronic Conditions (QuICCC) is an instrument based on a conceptual noncategorical definition that uses parental responses to identify children with chronic conditions for epidemiological purposes.
To determine whether the QuICCC is sufficiently valid, sensitive, and specific to be used to identify individual children as having a chronic condition or disability; whether parents are accurate enough that their answers to QuICCC items can be accepted as valid; and what kinds of errors in classification occur when the QuICCC is used to identify children with chronic conditions.
The sample consisted of 424 children who were patients of 9 physicians in separate practice settings throughout New England. Each physician was briefly trained in the conceptual definition on which the QuICCC is based and then was asked to identify 25 children in his or her practice who met the definition and 25 children who did not meet the definition. The QuICCC was administered to the parents of these children by blinded interviewers via telephone. The QuICCC classification was compared with physician categorization. Discrepant cases were then followed up by asking physicians and parents to answer the original questions a second time.
Complete data were available on 379 (89.4%) of 424 children. There was agreement on 89% (κ=0.78). The sensitivity was 94%; specificity, 83%; positive predictive value, 86%; and negative predictive value, 92%. Of the 42 discordant cases, 30 parent reports on the QuICCC qualified the child as having a chronic condition when the physician classified the child as being without such a condition. Fewer (n=12) discrepancies occurred because physicians identified children with chronic conditions that the QuICCC failed to identify. When the questions were readministered at follow-up, physicians corrected errors in rating in 9 cases; mothers changed their answers in 5 instances. In 13 instances the issues were known to both parties and appeared to arise in the "gray zone" or boundary area, where there was disagreement over whether a particular child qualified using the theoretical definition. For 11 children identified as having a chronic condition only by the parent's responses to the QuICCC, physician report appeared to be inaccurate primarily due to the physician's lack of information. In 3 cases where the physician reported the child to have a chronic condition, but the parent did not, the physician appeared to be correct. Follow-up data were incomplete on 1 child.
These data support the validity of parent-generated information for the evaluation of health status. Although these findings should be replicated, this study suggests that the QuICCC may be applicable also as a screening tool for individual child identification, provided that several sources of error are considered.
THERE HAS been an increased interest in serving children with chronic conditions and disabilities since the Surgeon General's Report on family-centered care for these children in the mid 1980s.1 In addition, in the late 1980s, it was mandated that the Title V Maternal and Child Health block grant in each state use one third of funds on "children with special health care needs."2 Further, the recent evolution to managed health care by public and private purchasers has raised considerable concerns about how children with chronic conditions and disabilities will fare in these new arrangements.3,4 Increasingly, program administrators in state Title V and Medicaid programs, as well as those in managed care arrangements, are seeking ways to identify children with chronic conditions using criteria other than relying on a diagnosis. A method for identifying children across diagnostic groups is needed so that the prevalence and size of the group can be monitored over time, service programs and funds can be allocated to the eligible children, quality improvement efforts can be developed for this group and their care can be compared with the total child population, and special capitation rates can be developed for provision of health and related services.
Although there has been considerable interest in techniques for identifying children who have chronic health conditions and disabilities that do not rely on diagnostic lists,5- 8 none to date has been developed and tested for both epidemiological and clinical purposes. This article describes a study designed to assess whether the Questionnaire for Identifying Children With Chronic Conditions (QuICCC),9 a measure designed to identify children with chronic conditions for epidemiological purposes (such as prevalence estimates), can also can be used as a screening tool for individual children. If there is sufficient reliability, the QuICCC would have potential use in clinical and program settings.
The QuICCC is based on a noncategorical conceptual definition,7 and was designed and tested on several large populations to identify groups of children with chronic conditions, independent of diagnosis. It is administered to an adult caregiver (usually the parent) and takes about 8 minutes to collect data about all children younger than 18 years in a household. The QuICCC has strong psychometric properties in all the epidemiological studies in which it has been used.9
However, the QuICCC was not intended to be used to identify specific individual children, nor has it previously been evaluated for its adequacy for such use. The psychometric standards a tool has to meet to identify individual children accurately, or to be a screening tool, are somewhat stricter and more difficult to achieve than standards of validity and reliability for research tools applied to groups. Less error can be tolerated when identifying individuals than when assessing or comparing groups of individuals. This is because the consequences of misclassifying individuals can have major programmatic implications that classification of large groups does not. In addition to the measurement issues concerning the appropriateness of the QuICCC for screening purposes, there is some skepticism among health care professionals and policy makers about the reliability and validity of parental reports of child health on which the QuICCC is built.
This study was undertaken to address several issues: (1) Is the QuICCC sufficiently valid, sensitive, and specific to be used to identify individual children as having a chronic condition or disability? (2) Are parents accurate enough that their answers to QuICCC items can be accepted as valid? (3) What kinds of errors in classification occur when the QuICCC is used to identify children with chronic conditions?
The study was done collaboratively by the developers of the QuICCC at the Albert Einstein College of Medicine, Montefiore Medical Center, Bronx, NY, and New England SERVE, Boston, Mass. New England SERVE is a coalition of health care providers (ie, physicians, nurses, allied health professionals, and others) and families dedicated to the improvement of the care of children who have special health care needs.
The QuICCC is a measure based on the theoretical definition developed by Stein et al.7 According to this definition, a child has a chronic condition if he or she has a physiological, behavioral, or cognitive disorder that has lasted or is virtually certain to last for 1 year and produces 1 or more types of consequences: functional limitations, reliance on compensatory assistance for functioning, or increased service use or need compared with age-mates. The QuICCC incorporates all the above elements and was designed specifically to reflect this definition.9
There are no other operationalizations of the definition that would allow testing for convergent validity. Therefore, the only standard criterion against which to test the validity of the QuICCC is the definition itself. To do this, we recruited pediatric health care providers to identify some children in their practices who had a chronic health condition meeting the Stein et al definition and some who did not. Children's families were then interviewed by telephone using the QuICCC and the resulting classifications—parent's and physician's—were compared.
We were concerned that several factors might affect the accuracy of the classifications and prospectively designed the study to inventory and assess them. First, the health care providers who applied the definition might do so inconsistently, since most physicians have an intuitive sense of what they define as a chronic condition. To reduce this source of error, we conducted a training session for physicians on the QuICCC definition and designed a form that they completed on each child that required that they check off which of the 3 types of consequences the child was experiencing because of the condition. Second, parent and physician raters might have different information available to them about the individual child's health. In some cases, the parent may have more accurate information (eg, about learning or behavioral problems), and in other cases the physician may (eg, about the length of time an illness may last). Third, the child's health status might change between the time the physician rated the child and the parent was interviewed. Fourth, the parents and physicians might not disagree about the child's status, but may differentially interpret questions or their meaning. Fifth, some children may have conditions in the "gray zone," ie, they are being evaluated, they have a condition that may or may not last 1 year, or they have a disorder that could be considered a normal variant that is close to the "boundary" or threshold for having a chronic condition.
The sample consisted of 424 children who were patients of physicians in 9 separate practice settings throughout New England. New England SERVE recruited 9 physicians who were associated with its organization and have a commitment to improving the care of children with chronic conditions. The 9 included 8 pediatricians and 1 pediatric surgeon; 7 were men and 2 were women. All had been in practice for 10 years or more in a variety of community and hospital-based settings. Each physician was asked to identify 25 children in his or her practice who met the definition on which the QuICCC is based and 25 children who did not meet the definition. They were asked to select children who had a wide range of conditions and impairments with different levels of severity. We emphasized that they should pick children of diverse ages and heterogeneous family backgrounds (ie, race, income, educational level, and family structure). Physicians contacted parents of the children to obtain their permission for a member of the staff of New England SERVE to interview them by telephone about their child's health status. The physicians provided the requested information on a range of 25 to 50 patients.
Physicians were trained in a group to apply the definition on which QuICCC is based to individual children. Before training, all physician participants read the definition article by Stein et al.7 They each then attended a single 90-minute training session that explained the purpose of the study and the research procedures. At the beginning of the session, the physicians were given a 2-page list of descriptions of children (each only 1 sentence long) and asked independently to assess whether each of the 3 dozen children had a chronic condition. Subsequently, the group jointly reviewed the specific elements of the definition with 2 of the authors (R.E.K.S. and L.J.B.) of the definition. The physicians were asked to suspend their intuitive notion of who had a chronic condition for the purpose of the study and to try to follow the intent of the definition they had just reviewed, regardless of whether they agreed with it or its implications. The brief descriptive vignettes were reviewed again as a group, and there was discussion about how to apply this specific definition to them. At no point in the study were the physicians shown a copy of the actual QuICCC instrument, nor was there discussion of the content of the 39-question sequences contained in the QuICCC during either the training or data collection phases.
Each physician forwarded identifying information about the children and the contact information for the parents directly to the New England SERVE office. The physician's classification of each child as meeting or not meeting the conceptual definition of Stein et al7 was provided on a separate form and was forwarded directly to the New York research office for processing. The information was submitted considerably after the training session (submission range, 2-26 weeks). No diagnostic information about the children was obtained from either the physician or the parent.
The QuICCC instrument and some additional questions on family characteristics and background were administered to parents by telephone by 2 trained interviewers based at New England SERVE. Interviewers were blinded to the child's categorization by the physician. The coded data were sent to the New York research office for analysis.
As an additional step after the analysis of the data, we conducted a limited follow-up to improve our understanding of the sources of the discrepancies. Each time a discrepancy occurred between the physician's classification of a child based on the definition and the classification based on the parent-administered QuICCC, the family was recontacted. We asked them to give their consent to share their answers to the QuICCC questions with their child's physician. All but 3 families were successfully contacted, and all gave consent for this additional step. In the 3 instances in which we could not reach the parent for follow-up, we obtained additional information from the physicians, but did not share parent responses to the QuICCC with the physicians.
All contacted families whose children had been identified as having a chronic condition by the parent but not by the physician were readministered selected items of the QuICCC to determine whether their responses were still affirmative. They were also asked the basis for their affirmative responses by the interviewers who recorded their responses. In many instances this led to information about the diagnosis or nature of the condition. We did not readminister the QuICCC items to families whose children were identified by the physician, but not by the parent, since they had not had any affirmative responses in the past, and we had set the physician responses as our standard.
Physicians were then called to review all discrepant cases. They were told that we wanted to go over some of the information about their patients and were asked the same series of questions used in the original data collection: Does the child have a functional limitation? A compensatory dependency? Or an increased service use or need? Does the child meet the Stein et al7 definition? Again they were not asked any questions contained in the QuICCC items. After this information was obtained, we shared the parent's answers to individual QuICCC items with the physicians of consenting families to determine the nature of the differences in classification. Additional descriptive and diagnostic information was obtained in these discussions.
Data were analyzed using simple cross tabulations and κ was calculated to ascertain the level of agreement between the classification of the child as having or not having a chronic condition using QuICCC data obtained from the parents and the global categorization of the child by his or her physician. Sensitivity, specificity, and positive and negative predictive validity were calculated using the standard formulas.
Of the 424 children whose names were submitted to New England SERVE, complete information was obtained on 380 (89.6%); the remaining 44 could not be contacted despite multiple telephone calls. In 1 instance, 2 children with nearly identical names were confused: the physician classified one, but the New England SERVE office was given the telephone number of the other. When this case was dropped, it left a sample of 379 or 89.4% of the original sample.
The sample was preponderantly white (83%). Seventy-one percent were privately insured and 17% were receiving Medicaid. Six percent reported receiving Supplemental Security Income. Most parents were well educated, with 95% having at least a high school education and just under half having graduated from college.
Fifty-three percent of the children were identified by their physicians and 58% by the parent responses to the QuICCC as having a chronic condition. Figure 1 shows that there was agreement on 89% (39% without and 50% with chronic conditions), for a κ of 0.78. This is a level of agreement that is usually considered more than adequate for group identification, but falls short of the κ of 0.9 that is desired for individual identification. The sensitivity of the QuICCC was 94%; specificity, 83%; positive predictive value, 86%; and negative predictive value, 92%. Most of the disagreement occurred because parents' responses on the QuICCC qualified the child as having a chronic condition when physicians classified the child as not having a chronic condition (n=30); many fewer (n=12) discrepancies occurred because physicians identified children as having chronic conditions that the QuICCC failed to identify.
Comparison of physician categorization and Questionnaire for Identifying Children With Chronic Conditions (QuICCC)9 classification.
Physicians varied in the number of discrepancies among the patients they submitted (range, 0-9; x‒ = 4). All but 2 of the physicians had discrepancies in both directions. These 2 physicians had discrepancies only in 1 direction—they reported that the child did not have a chronic condition when the parent responses indicated the presence of a chronic condition.
Follow-up information about the 42 discrepancies is summarized in Table 1. When the questions were readministered at follow-up, physicians changed answers for 9 patients, reflecting measurement unreliability. In 1 case the physician insisted that the original answers on the physician form had clearly been "a mistake." In only 1 instance did the physician report that the revised classification was based on an actual change in the child's health status. In that case the physician asked the date of the parent interview and reported that a serious chronic condition had been diagnosed between the time the child had been categorized by the physician and the time the mother had been interviewed.
In 5 instances the parent changed answers when reinterviewed, which suggests another instance of measurement unreliability, although we cannot rule out changes in the child's health status. For each of the 11 children identified only by the parent's responses to the QuICCC as having a chronic condition, the physician on reinterview stated that he or she was unaware of the issues identified by the parent. Some of these issues were understandable, such as orthodonture, and school and behavioral problems, but others were unexpected, such as allergies and asthma requiring medication prescribed by a physician. For these discrepancies, the parent seemed to be correct and the discrepancy was a result of the physician's lack of information. In 3 cases where the physician reported a chronic condition, but the parent did not, the physician seemed to be correct. Two of these involved children who had chronic conditions that the physician follow-up interview revealed to have been entirely quiescent at the time of the parental interview. The third involved a caretaker whom the physician identified as "being in denial" of consequences of fetal alcohol syndrome.
For 13 children, discrepancies involved differences in whether the disease consequences met threshold for the definition. In 8 of these cases, the physician was aware of the condition (eg, mild asthma, headaches, or strabismus), but did not think that it met the definitional criteria, yet the parental responses to the QuICCC qualified the child. Conversely, in 5 cases, the physician judged a condition to meet the criteria for a chronic condition, based mostly on high use of physician services, but the parent considered it routine (eg, recurrent otitis, mild asthma, or recurrent gastrointestinal tract upsets) and the parental responses to the QuICCC did not identify the child.
Our review of the 42 discrepancies identified 20 errors on the part of the physicians, who were used as the standard criterion for testing the QuICCC. When these errors in our standard criterion were corrected, the κ was 0.88.
Despite our original concern that the physicians might be inconsistent in applying the definition and would use preconceived notions of who has a chronic condition to classify children rather than the Stein et al7 definition, this type of error accounted for only 12 (29%) of the 42 errors in classification. This is particularly impressive because there was as much as a 6-month lag period between the physician training session and the submission of the ratings of the children by the physicians. Thus, it does appear that a brief review of the definition and the vignettes can teach physicians who are interested in these classification issues to apply the definition consistently.
Among the remaining 30 discrepancies, 14 reflected differences in information known to the physicians and the parents. Of those, 11 seem to be caused by the inaccuracy of the physician's report. Parents are more likely than physicians to be aware of school-related difficulties and behavioral symptoms or physical symptoms that have lingered from a previously treated condition. In far fewer cases (n=3), it appeared that the physician was more likely to understand the long-term implications of some conditions than the parents, who either denied or wished to downplay a serious condition that was not necessarily in long-term remission. It is not surprising that there are instances in which each rater seems to have information that the other lacks. Overall, the parent-based report using the QuICCC was more likely than the physician-based report to generate valid data.
Third, in only 1 case did we document that a change in the health status of the child explained the discrepancy. However, we did not obtain information about the resolution of health conditions that may have contributed to parents changing positive answers to negative ones.
Many discrepancies (n=13) seemed to arise in the gray zone or boundary area around the definition where there is uncertainty over whether a particular child has a chronic condition. The gray zone reflects the problem that the construct, chronic condition, is a social notion and that health and illness occur on a spectrum. There is no absolute cutoff or threshold. The QuICCC is flexible enough to permit different thresholds to be set, depending on the purpose for identification. However, it was impractical to examine the validity of the QuICCC at multiple thresholds. Therefore, we set one general-purpose threshold, the one used in earlier published articles about the QuICCC, to test the validity of the QuICCC. Children in the gray zone around this threshold can be included for some purposes and excluded for others. When an exclusionary approach is used, we suggest that a back-up system be implemented so that children who are erroneously misclassified by the QuICCC as not meeting the threshold can appeal the decision.
In addition, it is important to have an alternate mechanism to supplement parental identification to identify children such as those 3 in whom the physician seemed to be "right." These children represent a small (<1%) but important misclassification. Inclusive approaches are particularly appropriate when further opportunities for review of information are possible.
In this study, performance of the QuICCC fell short of the ideal standards for a measure for individual use. However, our standard criterion (physician classification) seemed to have been in error in almost half of the discordant cases. When these errors are eliminated, the κ is almost at the desired level of 0.90. This level is the psychometric standard for stability of measurement at which an instrument is viewed as useful for individuals.10
There are several limitations to this study. First, we used a sample of convenience for both the patients and the physicians and this may limit to some extent the generalizability of the information that we generated. The physician participants were interested in issues of chronic conditions in children (although all but 1 saw a wide range of children in their practices). It is likely that they submitted ratings for children they either had seen recently or knew well. However, this was a needed advantage for the study to assure the physician ratings as a standard criterion were as accurate as possible. It is also possible that they picked children who were easier to classify than a general population of children or those with more severe conditions. It is unlikely that physician classification of all children in a practice would be as valid and reliable; this increases our confidence in the value and usefulness of parental report. However, it is important to replicate this research in different populations to be sure the findings are generalizable.
Another limitation is that we did not systematically reinterview parents or physicians where there was agreement, or reinterview parents when they did not report the presence of a chronic condition, even when the physician classified their child as having a chronic condition. It may be that in those 12 cases some parents would have changed some of their responses if given the opportunity. This might have increased agreement levels even further.
Despite these constraints, these data support the validity of parent-generated information for the evaluation of child health status. It is unclear whether such reliable and valid information would also be obtained in other samples of parents or if there was a benefit or gain attached to being identified. This requires further study. Overall though, in the absence of any incentive for or against being identified, the study does demonstrate that parent data compare favorably with physician data in classifying children, and may under certain circumstances be more accurate. Family caretakers of children with chronic conditions often have more complete information than primary care physicians (or any one physician) and should be perceived as very accurate respondents in most cases.
We previously reported that the QuICCC is adequate for epidemiologic use, including monitoring quality of care in groups of children with chronic conditions. Based on data presented in this article, it is possible that the QuICCC, although not originally designed for that purpose, may also be applicable as a screening tool for individual child identification. Potential uses for screening include at enrollment into a health plan or into any child health delivery system, or at any point as a trigger of more specific eligibility determination for specialized services, such as enhanced care coordination.
However, if these findings are replicated and if the QuICCC is used for screening of individual children, several sources of error must be considered. At a minimum, this requires the creation of appeal procedures for misclassified children. We recommend that if the QuICCC is used to screen, physicians should also be permitted to identify children as having a chronic condition independently of the parent report of the QuICCC, when they think a child has been misclassified. Also, for each application, specific decision rules for children in the gray zone need to be formulated, as these children account for a substantial proportion of discrepancies. We believe that, with the above-mentioned provisions, the QuICCC may be used as a screening tool for individual children for programmatic purposes.
Accepted for publication October 4, 1999.
This research was conducted as part of the "Ensuring Quality" project of New England SERVE, supported by grant MCJ-25R020 (through Title V of the Social Service Act) from the Maternal and Child Health Bureau, Department of Health Resources and Services Administration, Department of Health and Human Services, Washington, DC.
We also thank Margaret Driscoll and Nelagi Pinto for their help in data collection, the 9 New England physicians and the families who participated, and Sheila Camacho, MEd, for her assistance with data analysis.
Corresponding author: Ruth E. K. Stein, MD, Department of Pediatrics-Centennial 1, Albert Einstein College of Medicine, Montefiore Medical Center, 111 E 210th St, Bronx, NY 10467.
Stein REK, Bauman LJ, Epstein SG, Gardner JD, Walker DK. How Well Does the Questionnaire for Identifying Children With Chronic Conditions Identify Individual Children Who Have Chronic Conditions?. Arch Pediatr Adolesc Med. 2000;154(5):447-452. doi:10.1001/archpedi.154.5.447