Long-term follow-up of extremely preterm infants is essential because the proportion of infants who survive is increasing while neurodevelopmental impairment rates remain high.1 Psychometric tests, such as the Bayley Scales of Infant and Toddler Development, Third Edition (Bayley-III),2 are commonly used to measure early childhood outcomes in obstetric and neonatal randomized trials.3,4 However, the composite scores on psychometric tests can be inaccurate because of administrative and technical errors. To determine the magnitude of this problem, we performed a post hoc review of all errors detected by central source document verification of Bayley-III assessments during the 18-month follow-up of the Canadian Oxygen Trial (COT).5
Extremely preterm COT participants in 25 centers in 6 countries were randomly assigned to 2 oxygen saturation target ranges. The primary outcome at a corrected age of 18 to 21 months was death or survival with neurodevelopmental disability. Cognitive or language composite scores of less than 85 on the Bayley-III scales were included in the primary outcome.5 The research ethics boards of all clinical centers approved the protocol, and written informed consent was obtained from a parent or guardian of every study infant.
Experienced examiners administered the Bayley-III test and submitted copies of the source documents to the coordinating center. Between October 15, 2008, and August 15, 2012, trained staff (J.D. and L.C.) reviewed the source documents for accuracy, completeness, and agreement with the electronic database entries. Discrepancies were identified and corrected through a formal data clarification process with input from Bayley-III examiners at the respective study sites. A small number of queries (n = 36) could not be resolved directly with the clinical centers and were referred to an adjudication committee consisting of a developmental pediatrician (D.M.) and a neurodevelopmental consultant (K.P.).
In this study, 1 assessor (J.D.) reexamined all Bayley-III source documents and classified the errors that were identified during the original review process into 5 categories: calculation of the child’s corrected age, documenting or applying scoring rules, raw score addition, look-up of scaled scores and composite scores in normative tables, and electronic data entry. A 15% random sample was independently classified by a second assessor (L.C.) to ensure consistent classification of error types. Only 2 disagreements were found and resolved through consensus. The error categories are hierarchical in nature. An early mistake—for example, calculation of the corrected age—may affect all subsequent steps. We counted such errors only once, assigned them to the category in which they first occurred, and summarized the frequency of independent errors in each of the 5 categories.
During COT follow-up, the source documents for 936 of 954 (98.1%) Bayley-III assessments were submitted to the coordinating center. Eighteen children could not be evaluated because of severe developmental delay or autism. Of 936 source documents, 576 (61.5%) contained no errors. The remaining 360 (38.5%) contained at least 1 error, and the total number of independent errors was 387 (Table). None of the 25 clinical centers were completely error free. The best and worst center-specific error rates were 8 of 61 (13.1%) and 14 of 16 (87.5%), respectively. Had they not been detected during the original review process, 41 of 387 (10.6%) incorrectly reported composite scores would have changed the determination of the composite primary outcome in COT.
Experienced Bayley-III examiners made numerous administrative, scoring, and reporting errors in this international multicenter trial. Errors were detected and corrected in real time by comprehensive and rigorous central source document verification. The use of computer-assisted scoring software may reduce calculation and table look-up errors but cannot correct the effects of administrative or clerical errors that are made before data are entered into the program.
This study underscores the importance of diligent administration and recording of psychometric assessments. We recommend central source document verification of all psychometric tests that contribute to the primary outcome in large multicenter trials of perinatal and neonatal therapies.
Corresponding Author: Lorrie Costantini, BA, Department of Clinical Epidemiology and Biostatistics, McMaster University, Neonatal Trials Group, DBCVSRI Hamilton General Hospital Campus, 237 Barton St East, Hamilton, ON L8L 2X2, Canada (email@example.com).
Published Online: December 29, 2014. doi:10.1001/jamapediatrics.2014.2965.
Author Contributions: Mss Costantini and D’Ilario had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Costantini,Schmidt.
Acquisition, analysis, or interpretation of data: Costantini, D’Ilario, Moddemann, Penner.
Drafting of the manuscript: Costantini.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Costantini.
Administrative, technical, or material support: Costantini,D’Ilario, Moddemann, Penner.
Study supervision: Schmidt.
Conflict of Interest Disclosures: None reported.
Funding/Support: The Canadian Oxygen Trial was supported by grant MCT-79217 from the Canadian Institutes for Health Research (Drs Schmidt and Moddemann).
Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Costantini L, D'Ilario J, Moddemann D, Penner K, Schmidt B. Accuracy of Bayley Scores as Outcome Measures in Trials of Neonatal Therapies. JAMA Pediatr. 2015;169(2):188–189. doi:10.1001/jamapediatrics.2014.2965