MacCAT-CR indicates MacArthur Competence Assessment Tool for Clinical Research.
Hein IM, Troost PW, Lindeboom R, Benninga MA, Zwaan CM, van Goudoever JB, Lindauer RJL. Accuracy of the MacArthur Competence Assessment Tool for Clinical Research (MacCAT-CR) for Measuring Children’s Competence to Consent to Clinical Research. JAMA Pediatr. 2014;168(12):1147-1153. doi:10.1001/jamapediatrics.2014.1694
An objective assessment of children’s competence to consent to research participation is currently not possible. Age limits for asking children’s consent vary considerably between countries, and, to our knowledge, the correlation between competence and children’s age has never been systematically investigated.
To test a standardized competence assessment instrument for children by modifying the MacArthur Competence Assessment Tool for Clinical Research (MacCAT-CR), to investigate its reliability and validity, and to examine the correlation of its assessment with age and estimate cutoff ages.
Design, Setting, and Participants
This prospective study included children and adolescents aged 6 to 18 years in the inpatient and outpatient departments of allergology, gastroenterology, oncology, ophthalmology, and pulmonology from January 1, 2012, through January 1, 2014. Participants were eligible for clinical research studies, including observational studies and randomized clinical trials.
Competence judgments by experts aware of the 4 relevant criteria—understanding, appreciation, reasoning, and choice—were used to establish the reference standard. The index test was the MacCAT-CR, which used a semistructured interview format.
Main Outcomes and Measures
Interrater reliability, validity, and dimensionality of the MacCAT-CR and estimated cutoff ages for competence.
Of 209 eligible patients, we included 161 (mean age, 10.6 years; 47.2% male). Good reproducibility of MacCAT-CR total and subscale scores was observed (intraclass correlation coefficient range, 0.68-0.92). We confirmed unidimensionality of the MacCAT-CR. By the reference standard, we judged 54 children (33.5%) to be incompetent; by the MacCAT-CR, 61 children (37.9%). Criterion-related validity of MacCAT-CR scores was supported by high overall accuracy in correctly classifying children as competent against the reference standard (area under the receiver operating characteristics curve, 0.78). Age was a good predictor of competence on the MacCAT-CR (area under the receiver operating characteristics curve, 0.90). In children younger than 9.6 years, competence was unlikely (sensitivity, 90%); in those older than 11.2 years, competence was probable (specificity, 90%). The optimal cutoff age was 10.4 years (sensitivity, 81%; specificity, 84%).
Conclusions and Relevance
The MacCAT-CR demonstrated strong psychometric properties. In children aged 9.6 to 11.2 years, consent may be justified when competence can be demonstrated in individual cases by the MacCAT-CR. The results contribute to a scientific underpinning of regulations for clinical research directed toward children.
At present, more than 50% of drugs prescribed to children have not been tested in their age group.1 Prescribers often have no alternative but to extrapolate to children the doses of drugs registered for adults. Historically the protection of children from research was often translated as simply excluding them from trials.2 Research with this vulnerable population involves many unique ethical and legal considerations. Little is known about children’s capacities to meaningfully decide on research participation. To our knowledge, the correlation between competence and age in children has never been systematically investigated.
Strictly speaking, competence to consent denotes a legal status, representing an informed, free, self-determined choice based on understanding and rational reasons.3 Competence is task and context specific.3 Incompetence should, in principle, be determined by a court; however, good pragmatic reasons exist to continue the traditional practice of having clinicians determine patients’ competence.4 In clinical practice, competence is generally addressed as decision-making capacity.5 In this article we use the terms interchangeably; unless otherwise specified, we are referring to clinical assessment of capacity.5
The reliability of unstructured competence assessments has been poor because clinicians may not know which standard to apply. Age standards prescribed by law may have too much influence. Clinicians tend to judge a child as competent if the child’s decision conforms to their own ideas of the child’s best interest.6,7 Providing clinicians with the generally accepted legal standards for competence improves their judgments and increases interrater agreement.4 These legal standards embody the following 4 capacities: to communicate a choice, to understand the relevant information, to appreciate the medical consequences of the situation, and to reason about treatment choices. Clinicians aware of the relevant criteria should be able to assess a patient’s competence.8
The first objective of our study was to develop a standardized competence assessment instrument for children in clinical research, including drug trials. The MacArthur Competence Assessment Tool for Clinical Research (MacCAT-CR)5 has been evaluated as the best competence assessment instrument for adults. We modified the instrument to be more applicable for children. First, we assessed the reproducibility and validity of the child version of the MacCAT-CR. Second, we examined the dimensionality of the instrument. The MacCAT-CR is clearly not designed to provide an overall total score or a cutoff score because the instrument is structured according to the 4-capacities model of decision making. However, we hypothesized that when the scores on the 4 domains unexpectedly constitute a single trait or continuum of competence, estimation of cutoff scores above which competence is likely should be possible. This hypothesis may further objectify decisions regarding a child’s competence based on the modified instrument.
The international statutory age limits for deeming a child legally competent to give informed consent in clinical research vary widely, just as they vary within the United States and Europe.9 Legal interpretations at a local level may present an age limit below which children are considered by definition to be incompetent to act for themselves and may express their agreement as assent.10 Lower age limits vary from 7 to 18 years.9,10 Some jurisdictions present 2 age limits, and for children and adolescents between these limits, consent is required from the children and parents, provided the children are competent to decide. An upper age limit presenting an age above which adolescents are deemed adult in medical decision making varies from 15 to 18 years.9,10 To deal with discrepancies between jurisdictions, we studied the capacities of children and adolescents (hereinafter referred to as children) for consent regardless of their age and legal status. A final objective was to provide empirical evidence regarding the age cutoffs for presuming competence to consent in children.
From January 1, 2012, through January 1, 2014, we enrolled a cohort of 161 pediatric inpatients and outpatients eligible for clinical research from hospitals in Amsterdam, Rotterdam, and the Hague, the Netherlands (participating institutions are listed in Table 1). The inclusion criterion was age 6 to 18 years. We included children who decided to participate in the clinical research and children who refused. The only exclusion criterion was not speaking Dutch. The research design and methods have been described comprehensively elsewhere by Hein and colleagues.11
The study procedures were judged by the institutional review boards of the participating institutions, and all approved. Prior written consent was obtained from participants 12 years or older and all parents.
As the index test for competence assessment, we used the semistructured interview format MacCAT-CR developed by Appelbaum and Grisso in 2001.5 The MacCAT-CR measures the following 4 aspects of decision-making capacities that reflect the standards for competence in most jurisdictions: (1) understanding the disclosed information about the nature and procedures of the research; (2) reasoning in the process of deciding about participation, with a focus on abilities to compare alternatives in the light of their consequences; (3) appreciation of the effects of research participation on the patient’s own situation; and (4) expressing a choice about participation.5 The MacCAT-CR combines information disclosure with competence assessment in approximately 15 minutes. It provides subscale scores but does not offer a total score or a cutoff. The ratings provide the clinician with a structured overview.
Of a variety of operational translations of the concept of competence into assessment instruments, the MacCAT-CR receives the most empirical support.12 Studies in populations with dementia, mental disabilities, and psychiatric disorders show initial indications of validity13 and a high degree of reliability.14 The Dutch version of MacCAT-CR was modified for children with the approval of Paul S. Appelbaum, MD.11 Modification included the use of simple language to be understood by elementary school–aged children and added questions on the influence of social relationships (ie, in the reasoning domain, “How do you think your parents will feel about you participating or not participating? And how do you think your friends will feel about it?” has been added)11 (proprietary issues preclude publication of the version used). Face validity and feasibility were confirmed in a pilot study including 10 children aged 6 to 18 years (I.M.H., P.W.T., R.L., and R.J.L.L.; unpublished data, December 2011). We used competence judgments by clinicians aware of the relevant criteria to establish the reference standard, as is generally accepted best practice.8
Children and parents were informed about the clinical research in the conversation with the clinical researcher (including M.A.B. and C.M.Z.), who asked for informed consent. This conversation was videotaped and served as the basis for establishing the reference standard (see below). Within 2 weeks, an interviewer from a panel of experts (listed at the end of the article) administered a MacCAT-CR interview to the child. This interview was also videotaped and rated afterwards.
The panel of 14 experts (including I.M.H., P.W.T., M.A.B., C.M.Z., and R.J.L.L.) consisted of trained pediatricians, child psychiatrists, child psychologists, research nurses, a medical ethicist, and a jurist. The 3-hour training included instructions on competence judgment by the 4 relevant criteria and joint practicing through 3 videos of the conventional informed consent conversation together with instruction on rating the MacCAT-CR and joint rating of 2 videos of MacCAT-CR interviews. In addition, the expert panel practiced by judging 3 videos that were removed from the analysis to exclude bias through a learning effect.
Each member of the panel independently rated a number of conventional informed consent conversation videos and a number of MacCAT-CR interview videos that were presented in random order and reciprocally blinded. For establishing the reference standard, each video from the conventional informed consent conversation was rated by 2 different experts. Each MacCAT-CR interview video was rated by 3 different experts to assess the reproducibility. For all videos, the experts gave their judgment consisting of 1 of the following 4 categories: very likely competent, probably competent, probably incompetent, and very likely incompetent. We considered competence to be present when an expert gave a judgment of very likely or probably competent.
No general agreement exists about estimating a suitable sample size for the psychometric evaluation of multiple-item scales. Simulation studies for the related techniques of regression analysis indicate that a minimum of 10 to 15 observations per variable (item) are needed to obtain stable estimates.15 The 13 items of the MacCAT-CR therefore would require 130 to 190 observations. Judging from previous studies on MacCAT scales in adults with compromised decisional capacities,14,16- 18 our proposed sample size of 160 is justifiable.11
Reproducibility of the MacCAT-CR item scores for the sets of 3 raters was evaluated using weighted κ coefficients. Because 3 different raters scored the MacCAT-CR and because calculation of a weighed κ statistic for more than 2 raters is not possible, the multirater-weighted κ value for each item was calculated using intraclass correlations.19 Reproducibility of the MacCAT-CR domain scores was also evaluated using intraclass correlations (model 1, single measures).
We examined dimensionality by using internal consistency analysis on the mean scores of the 3 raters. We examined the item-rest correlations, also known as corrected item-total correlation, which refers to the correlation between the item score and the total sum of the remaining items, excluding the item. Item-rest correlations less than 0.30 indicate that the item does not sufficiently contribute to the reliability of the scale and may be discarded. If a positive relation could be demonstrated for all items on the subscale and total scale levels, factor analysis (principal components) with eigenvalues greater than 1 criterion and visual inspection of the scree plot were used to further examine dimensionality of the MacCAT-CR. Subsequently, we used item response theory methods, a specific extension of the Rasch measurement model,20 to confirm scale unidimensionality. Conditional maximum likelihood estimation methods were used to estimate the item and person (MacCAT-CR competence ability level) variables. Fit of the data to the unidimensional extended Rasch model was tested using item-oriented fit statistics (S tests)20 that examine observed and expected numbers with a given item score conditional on competence ability level as measured with the MacCAT-CR. Overall fit of all item scores to the unidimensional model was tested with the R1c statistic.20 For all fit statistics, P > .05 indicates fit to the unidimensional measurement model.
Rasch analysis was performed on the pooled MacCAT-CR scores using all interrater MacCAT-CR scores and on the scores of children who responded to all the MacCAT-CR items (excluding children with missing scores on ≥1 item) because the method requires a large number of observations, preferably more than 300. Because we used conditional maximum likelihood estimation methods, dependence between scores is not an issue; the method makes no assumptions on how the sample was selected or on the distribution of abilities in the sample.21
The overall accuracy of the MacCAT-CR total score (in case of unidimensionality) in classifying competence against the reference standard was quantified using receiver operator characteristics curve analysis. The area under the receiver operating characteristic curve (AUC) served as the validity coefficient; AUCs exceeding 0.70 are generally considered adequate. The optimal MacCAT-CR cutoff score and the accompanying sensitivity and specificity rates were determined using the Youden method, representing the score corresponding to the fewest false-positive and false-negative classifications.22 For a few trials in which some MacCAT-CR items were not applicable (ie, when no placebo was used in the trial), we computed adjusted scores, calculated as the obtained score divided by the fraction of items completed, for the analysis.
The validity of the current statutory cutoff ages was tested by the same method. In this estimation, we used age as the predictor of competence to consent.
Of the 209 children eligible for this study, 48 were excluded or were unavailable to participate for various reasons (Figure). Baseline characteristics of the 161 participants who entered analysis are listed in Table 1. Mean age was 11 (median, 10.0 [range, 6-17]; SD, 2.8; 25th to 75th percentiles, 9-12) years. Of the 6 children unable to complete the whole interview, 4 were 6 and 2 were 7 years of age. In the age range of 8 to 18 years, feasibility was very high. The interval between the reference test and MacCAT-CR exceeded 2 (maximum, 6) weeks on 9 occasions.
Reproducibility of the MacCAT-CR item scores for the sets of 3 raters, expressed in weighted κ values, are listed in Table 2. The intraclass correlation coefficient for the MacCAT-CR total score was 0.91; for the Understanding subscale, 0.92; for the Appreciation subscale, 0.84; for the Reasoning subscale, 0.68; and for the Choice subscale, 0.80. Weighted κ values for individual items ranged from 0.52 (item 11) to 0.82 (item 7).
The Cronbach α for the total MacCAT-CR scale considering all 13 items was 0.89. Item-rest total score correlations ranged from 0.47 to 0.84 (median, 0.65) (Table 2). The Cronbach α values for the subscales (Understanding, 0.82; Appreciation, 0.73; and Reasoning, 0.80) indicated good internal consistency.
Factor analysis showed that 1 component had an eigenvalue greater than 1, namely 6.8, explaining 53% of the total score variation. Factor loadings ranged from 0.54 to 0.89. The remaining 12 factors each explained 1% to 7% of the total variation. Subsequent Rasch analysis on the 21 individual MacCAT-CR question scores (n = 384) confirmed unidimensionality (R1c goodness of fit χ260, 60.7; P = .45).
By the reference standard, 54 children (33.5%) were judged incompetent; by the MacCAT-CR, 61 children (37.9%). Because unidimensionality of the MacCAT-CR was supported, we determined the overall accuracy of the MacCAT-CR total score in classifying competence against the reference standard as 0.78. The optimal cutoff for the MacCAT-CR total score above which competence is likely was 35 points. Accompanying sensitivity was 63%; specificity, 85%; positive predictive value, 79%; and negative predictive value, 72%.
Age was a good predictor of competence on the reference standard (AUC, 0.84) and on the MacCAT-CR (AUC, 0.90). The cutoff age with 90% sensitivity was 9.0 years for the reference standard and 9.6 years for the MacCAT-CR. The cutoff age with 90% specificity was 11.5 years for the reference standard and 11.2 years for the MacCAT-CR. The optimal cutoff age with the fewest false-positive and false-negative classifications on the MacCAT-CR was 10.4 years (sensitivity, 81%; specificity, 84%).
We have confirmed the accuracy of the MacCAT-CR for assessing children’s competence to consent. Results of this study in children aged 6 to 18 years provide evidence of reliability (reproducibility) and validity.
In children, our results suggest that MacCAT-CR scores on the 4 domains constitute a single trait or continuum of competence. This finding allows for estimating a cutoff score on the MacCAT-CR above which competence in children is likely. The demonstrated unidimensionality of the MacCAT-CR in children does not align with the adult literature stating that scores on subscales need to be weighted independently and that failure on one domain could translate into a judgment of incompetence.5,13
Competence assessed by the MacCAT-CR strongly correlates with age, with an overall accuracy of 0.90. The MacCAT-CR demonstrated 90% sensitivity for competence classification before 9.6 years and 90% specificity after 11.2 years. The range from 9.6 to 11.2 years of age constituted a changeover. This finding aligns with those of previous research to suggest that clinical competence to consent to medical interventions can be present in children younger than 12 years.23,24 However, the age categories resulting from our study do not coincide with age limits in many jurisdictions. Applying a format that uses the 3 age categories developed by Grisso and Vierling25 to our results would imply that asking for consent from children younger than 9.6 years might not be justified. Assuming that children older than 11.2 years cannot provide competent consent appears to have no ground. Children aged 9.6 to 11.2 years appear to be in a transition period; they develop important capacities but their maturity is not pervasive. In children around these ages, consent may be justified when competence can be demonstrated in individual cases.25 A tailored informed consent process and competence assessment by the MacCAT-CR can support these children in decision making; at the same time, the process could preempt the pitfall of imposing complex research decisions on children who are not able to make them. Generalizability of the study results is credible because the study population was heterogeneous.
This study has specific limitations. An assessment instrument is not meant to replace clinical judgment but to complement its accuracy. The expert judgment based on the videos might have been biased by the lack of all the required information in the unstructured conversations.
In light of the heterogeneity of the sample, interpretation of the data is subject to insufficiencies. Participants were enrolled in any type of clinical research, with expected differences in the complexity of the studies. This difference could render age limits resulting from our study less applicable to special patient groups. Further reporting on complexity, intelligence, prior trial experience, duration of illness, and other variables is under preparation (I.M.H., P.W.T., R.L., J.B.v.G., R.J.L.L., unpublished data, June 2014).
Duration between the reference and index tests exceeded the 48 hours planned originally for practical reasons. However, because the MacCAT-CR includes information disclosure, recall of information would not have been a problem.
Our modified version of the MacCAT-CR demonstrated accuracy in assessing children’s competence to consent; psychometric properties in children aged 6 to 18 years were strong. Results indicate a strong correlation of competence with age. In children younger than 9.6 years, competence is unlikely; in children older than 11.2 years, competence is probable. In children aged 9.6 to 11.2 years, consent may be justified when competence can be demonstrated in individual cases, and we suggest competence assessment by the MacCAT-CR for these children. The results provide objective starting points for policymakers and contribute to a scientific underpinning of parliamentary proposals to modify the regulations for clinical research in children.
Directions for future research include investigating children’s competence in different types of medical research situations and correlations of competence with other variables. More research on children’s competence in special research populations (ie, child and adolescent psychiatric and mentally disabled populations) is recommended. For treatment settings, we recommend research with the MacArthur Competence Assessment Tool for Treatment modified for children.26
Accepted for Publication: July 6, 2014.
Corresponding Author: Irma M. Hein, MSc, MD, Department of Child and Adolescent Psychiatry, Academic Medical Center, Meibergdreef 5, 1105 AZ Amsterdam, the Netherlands (email@example.com).
Published Online: October 13, 2014. doi:10.1001/jamapediatrics.2014.1694.
Author Contributions: Dr Hein had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Hein, Troost, Lindeboom, Lindauer.
Acquisition, analysis, or interpretation of data: Hein, Lindeboom, Benninga, Zwaan, van Goudoever.
Drafting of the manuscript: Hein, Troost, Lindeboom.
Critical revision of the manuscript for important intellectual content: Hein, Lindeboom, Benninga, Zwaan, van Goudoever, Lindauer.
Statistical analysis: Hein, Lindeboom, van Goudoever.
Obtained funding: Hein, Lindeboom, Lindauer.
Administrative, technical, or material support: Hein, Zwaan.
Study supervision: Benninga, Zwaan, van Goudoever, Lindauer.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported by the Netherlands Organization for Health Research and Development.
Role of the Funder/Sponsor: The funding institution had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: The expert panel included the following: Frits Boer, PhD, MD, Academic Medical Center, Amsterdam, the Netherlands; Chaim Huyser, PhD, MD, de Bascule, Amsterdam; Mini Hulscher, MD, Triversum, Alkmaar, the Netherlands; Martine de Vries, PhD, MD, Leiden University Medical Center, Leiden, the Netherlands; Inekee van der Vaart, BS, Erasmus Medical Center/Sophia Children’s Hospital, Rotterdam, the Netherlands; Marian Cavazza, BS, Erasmus Medical Center/Sophia Children’s Hospital; Alice Broersma, LLM, Accare, Assen; Bente Dijkman, MSc, Academic Medical Center; and Dannis van Vuurden, MD, PhD, VU University Medical Center, Amsterdam. The experts received no compensation. Lotte Gelens, MSc, and Marjolein Meester, MSc, assisted in patient inclusion during their employment as research assistants at Academic Medical Center. Jeroen van Setten, MSc, JVS Beheer en Advies BV, Amersfoort, the Netherlands, designed the Internet application, for which he was compensated.