Carroll AE, Bauer NS, Dugan TM, Anand V, Saha C, Downs SM. Use of a Computerized Decision Aid for Developmental Surveillance and ScreeningA Randomized Clinical Trial. JAMA Pediatr. 2014;168(9):815-821. doi:10.1001/jamapediatrics.2014.464
Developmental delays and disabilities are common in children. Research has indicated that intervention during the early years of a child's life has a positive effect on cognitive development, social skills and behavior, and subsequent school performance.
To determine whether a computerized clinical decision support system is an effective approach to improve standardized developmental surveillance and screening (DSS) within primary care practices.
Design, Setting, and Participants
In this cluster randomized clinical trial performed in 4 pediatric clinics from June 1, 2010, through December 31, 2012, children younger than 66 months seen for primary care were studied.
We compared surveillance and screening practices after adding a DSS module to an existing computer decision support system.
Main Outcomes and Measures
The rates at which children were screened for developmental delay.
Medical records were reviewed for 360 children (180 each in the intervention and control groups) to compare rates of developmental screening at the 9-, 18-, or 30-month well-child care visits. The DSS module led to a significant increase in the percentage of patients screened with a standardized screening tool (85.0% vs 24.4%, P < .001). An additional 120 records (60 each in the intervention and control groups) were reviewed to examine surveillance rates at visits outside the screening windows. The DSS module led to a significant increase in the percentage of patients whose parents were assessed for concerns about their child's development (71.7% vs 41.7%, P = .04).
Conclusions and Relevance
Using a computerized clinical decision support system to automate the screening of children for developmental delay significantly increased the numbers of children screened at 9, 18, and 30 months of age. It also significantly improved surveillance at other visits. Moreover, it increased the number of children who ultimately were diagnosed as having developmental delay and who were referred for timely services at an earlier age.
clinicaltrials.gov Identifier: NCT01351077
Developmental delays and disabilities are common in children. Research has indicated that intervention during the early years of a child’s life has a positive effect on cognitive development, social skills and behavior, and subsequent school performance.1- 3 To ensure early identification of developmental delays and disabilities, physician organizations, such as the American Academy of Pediatrics (AAP), have called on pediatricians to institute a standardized approach for the identification of developmental delays that includes developmental surveillance and screening (DSS).4- 6
Developmental screening refers to the administration of a standardized tool to aid in the identification of children at risk for a developmental disorder.5 Developmental surveillance, on the other hand, is “an ongoing process of monitoring the status of a child by gathering information about the child’s development and behavior from multiple sources, including skillful direct observation of the child’s behavior and elicitation of concerns from parents and relevant professionals.”4(p 5)
Significant efforts have been made to address the implementation of DSS practices within primary care settings.7 Despite such efforts, these processes are still not routinely implemented, and studies5,8- 11 have found that, when screening occurs, it often does not make use of standardized protocols or tools.
Numerous barriers exist to the successful implementation of DSS practices within the primary care setting, including lack of time and staff, logistical challenges in administering screening tools, inadequate reimbursement, and language barriers.10- 13 Computerized clinical decision support systems (CCDSS) are a promising strategy to overcome these barriers. Because CCDSS involve the use of integrated systems that routinely store and retrieve patient information, they can improve workflow by providing physicians with patient-specific recommendations based on integrated data at the time and place of a patient visit.14,15 Although previous attempts have been made to automate the process of developmental screening, these attempts have not proved successful because the decision support was not integrated with routine clinical care.14,15 The objective of this study was to determine whether CCDSS integrated with routine care could be an effective approach to improve standardized DSS within primary care practices.
For the purposes of this study, we used an existing CCDSS designed for use in pediatric primary care practices: the Child Health Improvement Through Computer Automation (CHICA) system. However, we refined CHICA to include the DSS algorithm published by the AAP in 2006 by creating the CHICA DSS module.5
We conducted a randomized clinical trial in 4 pediatric clinics. Two of these clinics functioned as our control sites and used the traditional CHICA system that did not include the DSS module. The other 2 clinics functioned as our intervention sites and used the enhanced CHICA system that included the DSS module. We then compared surveillance, screening, and diagnosis of developmental disorders between the control and intervention clinics. This study was approved by the institutional review board of the Indiana University School of Medicine, and informed consent was waived.
Although our intervention was aimed primarily at physicians, the unit of randomization was the primary care clinic, and the unit of analysis was the individual patient. The 4 clinics were matched based on the number of physicians practicing at each of the clinics before randomization and then block randomized as pairs. We chose to randomize by clinic because contamination was a major concern.
This study was performed from June 1, 2010, through December 31, 2012, in 4 primary care pediatric clinics in the Eskenazi Medical Group, the largest safety-net health system in Indianapolis, Indiana. These 4 clinics conducted approximately 88 000 pediatric patient visits during this time frame, with 84% of the visits being supported by Medicaid.
Our intervention was aimed primarily at physicians. However, the outcomes of interest for our study are patient based. Therefore, patients younger than 66 months were automatically placed into the control or intervention group based on which of the 4 clinics they attended (Figure 1). There were no additional inclusion or exclusion criteria.
In 2004, CHICA was developed to support the provision of well-child care and the care of a number of common chronic conditions.16,17 It is unique in that, at each visit, the system uses a library of Arden Syntax18,19 rules that gather data from the Regenstrief Medical Record System and CHICA record systems to prioritize and then select 20 health questions that are printed on a paper questionnaire or an electronic tablet for that family to complete in the waiting room.20 The paper questionnaire, called the Patient Screening Form, is scannable, and the answers to the questions are stored in the electronic health record. After the information is stored, a tailored worksheet is generated for the physician to use during the visit. The worksheet includes up to 6 alerts, also prioritized, each with up to 6 check-box responses with which the physician can document his or her response. The comprehensiveness of what CHICA covers makes it useful at every clinic visit. Further details about the CHICA system have been described elsewhere.16,18,20- 28
The CHICA system was operational at the 4 clinic sites before the initiation of this study. In June 2010, we added the DSS module to the CHICA system that was operating at the 2 intervention clinic sites. Before activating the DSS module in the intervention clinics, staff and physicians were informed that a new functionality was being added to the CHICA system, which is done any time new functionality is introduced to the system. Staff and physicians of the intervention and control clinics were not made aware that a clinical trial was under way.
The overall flow of the CHICA DSS module is shown in Figure 2 and mirrors the surveillance and screening algorithm for DSS recommended by the AAP in 2006.5 The CHICA DSS module had a number of components that we expected would improve the screening and diagnosis of developmental disorders that are not included in the traditional CHICA system:
Universal Screening: The CHICA DSS module was designed to provide universal screening at 3 target well-child care visits (9-, 18-, and 30-month visits) through the use of the Ages and Stages Questionnaire 3 (ASQ-3) , which was automatically printed when the patient checked in for these target visits.29 The ASQ-3 is a brief parent report measure with 19 age-specific questions that span the age range of 4 to 60 months.5,30 The 4 clinic study sites were familiar with and had licenses to use the ASQ-3 in clinical care.
Surveillance: On the patient screening form, parents were asked simple questions related to whether they had any concerns about their child’s development at every nontarget well-child care visit. If a parent responded affirmatively to any of the questions, the physician was notified on the physician worksheet, and a standardized screening tool was printed for use by the physician. The physician worksheet also provided a way for the physician to document the overall ratings of a child’s developmental status and specify developmental areas of concern.
Reassessment: The CHICA DSS module automatically tracked those children whose parents had concerns or who had borderline results on a previous ASQ-3 screening. For these children, the CHICA DSS module would generate a new ASQ-3 and prompt the physician to rescreen at subsequent visits, consistent with AAP guidelines.
Recommendations: On the basis of established guidelines,5 the CHICA DSS module prompted physicians to refer children with positive screening results for comprehensive evaluation and services.
Data collection began 6 months after the DSS module was activated at the intervention clinics. For the developmental screening portion of our study, 180 patient records were randomly pulled from the intervention clinics, and another 180 patient records were pulled from the control clinics, for a total of 360 patient records. These records were divided equally between the 9-, 18-, and 30-month well-child care visits. We made sure that patients selected for analysis at one visit were not included for the analyses at the other visits. An additional 120 patient records (60 from intervention clinics and 60 from control clinics) were randomly pulled from the 4 clinic sites for the developmental surveillance portion of the study. These records could be for any visit as long as it was not one of the target visits. Consequently, there was no overlap between the patient records reviewed for the developmental screening and developmental surveillance portions of the study.
Data were obtained from the electronic health record, the CHICA system, and manual paper record abstractions by trained research assistants. To assess the reliability of medical record abstraction, a random sample of 20% of the records were abstracted twice. The agreement on overall record abstraction was 89%, with a κ of 0.75.
For developmental screening, our primary outcome of interest was whether a standardized screening tool was administered at the target visits (ie, 9-, 18-, and 30-month visits). Other variables of interest for this portion of the study included the percentage of standardized screens with a positive result for a developmental delay at target visits, diagnosis of developmental delay or disorder at any point after target visit, and age at diagnosis.
For developmental surveillance, our primary outcome of interest was whether developmental concerns were elicited from parents at visits other than the target visits. Other data collected for this portion of the study included physician documentation of developmental concerns separate from parental concerns, diagnosis of developmental delay or disorder at any point after target visit, and age at diagnosis.
The primary outcome of interest was whether a child was screened for developmental delay at target visits. For sample size calculations, we used a rate of 25% in the control clinic. We wanted to have at least a 60% screening rate in the intervention clinics after implementation of the DSS module. Because patients were nested within clinics, we applied a conservative intracluster correlation estimate of 0.008, twice the expected intracluster correlation of 0.004. Although we did not anticipate much variability in the screening rate at the 4 study clinics, we used 4 different rates (12%, 15%, 18%, and 21%) to estimate intraclinic correlation. Using the χ2 test and setting α at .05, we found the probability of detecting a statistically significant difference in the proportion of children screened at the target visits (9-, 18-, and 6-month visits) between the intervention and the control group of 90%, with an effective sample size of 46 per group. However, the proposed sample size had more than 99% power to detect a 35% absolute difference in screening rates.
Patient characteristics in the surveillance and screening components of the study were compared between the control and intervention groups using χ2, Fisher exact, and t tests. Logistic and exact logistic regression models were used to assess associations between dichotomous outcomes and intervention after controlling for sex, race, and type of insurance. Point estimates and 95% CIs on treatment effect were reported. A linear mixed-effects model was used to compare age at diagnosis after controlling for the same confounding factors as in the previous models. We assumed the parents’ assessment of the child’s development was independent of the clinic; therefore, the clinic was not considered a cluster in modeling this outcome. However, all other models included the clinic as a cluster to incorporate dependency among responses from the same clinic, and the clinic was used as a random effect in the models. The linear mixed-effects model used a logit link function to model the dichotomous outcomes. Finally, a linear mixed-effect model that included clinic as a cluster was used to model the age at diagnosis for developmental delay. SAS statistical software (SAS Institute Inc) was used for all analyses, and P < .05 was considered statistically significant.
For the developmental screening portion of our study, the characteristics of the 360 study patients are given in Table 1. No significant differences were found between the intervention and control groups with respect to sex, type of insurance, and age. A significant difference was found between the 2 groups with respect to race based on differences in the clinic populations (P < .001). We controlled for race in all analyses.
The DSS module led to a significant increase in the percentage of patients screened with a standardized screening tool at the target visits (85.0% vs 24.4%, P < .001). The odds of being screened in the intervention group were 15.6 (95% CI, 6.9-35.7) times the odds for a child in the control group. If screening occurred, however, the rate of a positive screen result was similar between the groups (19.6% vs 18.2%, P = .57). This finding implies that the number of children at risk for developmental delay was similar between groups but that more children were picked up in the intervention group because of higher screening rates.
Although our study was not powered to detect differences in a full diagnosis of developmental delay, our results indicated that a diagnosis of developmental delay was not significantly more common in the intervention group (10.6% vs 6.7%, P = .52). However, the diagnosis of developmental delay was made earlier in the intervention group than in the control group (mean age at diagnosis, 17.2 vs 27.9 months; P < .001).
For the developmental surveillance portion of this study, characteristics of the 120 patients included are given in Table 2. Again, a significant difference was found in the distribution of race between the intervention and control groups (P < .001). We therefore controlled for race in all analyses.
The CHICA DSS module led to a significant increase in the percentage of parents who were asked about concerns regarding their children’s development outside the target visits (71.7% vs 41.7%, P = .04) (Figure 3). The odds of parents being asked about child development for children in the intervention group were 2.70 (95% CI, 1.05-6.84) times the odds for children in the control group. When concerns were assessed, more concern was noted in the control group than the intervention group, although the difference was not significant (9.3% vs 16.0%, P = .38). This finding suggests that parents were being assessed in the control group only when there was a higher likelihood of a positive concern. The intervention had no effect on whether physicians documented an assessment of developmental concerns (83.3% vs 81.7%, P = .52).
Once again, the study was not powered to detect differences in a full diagnosis of developmental delay. We found that a diagnosis of developmental delay occurred after positive surveillance in 20% of intervention children vs 8.3% of control children (P = .62). No significant difference was found between the 2 groups in terms of age at diagnosis.
This study found that the CHICA DSS module more than tripled the use of standardized tools to conduct screening at target visits (ie, 9, 18, and 30 months of age) as recommended by the AAP guideline. Guidelines also recommend that surveillance occur at all other ages,5 and the DSS module notably increased the rates at which parents were asked about developmental concerns. The surveillance aspect led to an increase in the number of children diagnosed as having developmental delay. The CCDSS intervention led to earlier detection of developmental delay (mean of approximately 8 months earlier). Because optimal outcomes of developmental delay depend on early detection, this finding is a critically important finding, although our study was not designed to detect changes in clinical outcomes.
This study has limitations that warrant consideration. Previously existing practice differences could account for some differences, although we have no reason to believe that such differences existed. Use of the clinic as the basis of randomization could also lead to biases. However, the limitations and advantages of randomizing at the patient, physician, and clinic levels led us to decide that the clinic served as the best point of intervention. Generally, all informatics interventions have limitations as well. Some researchers believe they make physicians more reliant on external sources for quality care or leave them exposed to legal risk if they ignore prompts. Such things were outside the scope of this study, however. Generalizability of a specific CCDSS is always an issue. Although most electronic medical records are not as sophisticated as the CHICA system, many systems would allow programming with the algorithms we used in this study. In addition, the CHICA system has been created with open-source tools and is available for installation into other systems.
Adherence to existing clinical practice guidelines can be difficult for pediatricians who are often overwhelmed with patients and constrained by time.31 Because of this, it is possible for them to fail to properly and fully screen and document every condition.32 The CHICA DSS module is unique in that it permits us to insert guideline-based care subtly into existing clinic practices, and by prioritizing when certain modules are used, the system ensures that physicians and patients are not overwhelmed. A holistic CCDSS, such as the CHICA system, has much potential for introducing better evidence-based care and chronic care management into busy practices.28,33- 35
Our study found that a CCDSS, such as the CHICA system, that performs surveillance and screens patients for the presence of developmental delay with a standardized screening tool, coupled with personalized, evidence-based prompts to physicians, significantly improved the rates of proper screening and time to diagnosis of children with developmental delay. It also offered hope that management and referral of such patients could be improved. Future research should focus on determining whether these improvements are clinically significant and can be replicated elsewhere and whether these results can be applied to other chronic conditions.
Use of a CCDSS to automate the screening of children for developmental delay significantly increased the numbers of children screened at 9, 18, and 30 months of age. It also significantly improved consistent surveillance at other ages. Moreover, it increased the number of children who ultimately were diagnosed as having developmental delay and referred for timely services at an earlier age. More work is needed to determine whether this system translates into improved outcomes for children.
Accepted for Publication: March 5, 2014.
Corresponding Author: Aaron E. Carroll, MD, MS, Children’s Health Services Research, Indiana University School of Medicine, 410 W 10th St, Health Information and Translational Sciences, Ste 4099c, Indianapolis, IN 46202 (email@example.com).
Author Contributions: Drs Carroll and Bauer had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Carroll, Anand, Downs.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Carroll, Saha, Downs.
Critical revision of the manuscript for important intellectual content: Bauer, Dugan, Anand, Saha, Downs.
Statistical analysis: Carroll, Saha, Downs.
Obtained funding: Carroll, Saha, Downs.
Administrative, technical, or material support: Carroll, Dugan, Anand, Downs.
Study supervision: Carroll, Anand, Saha, Downs.
Published Online: July 14, 2014. doi:10.1001/jamapediatrics.2014.464.
Conflict of Interest Disclosures: None reported.
Funding/Support: This project was supported by grant R01HS017939 from the Agency for Healthcare Research and Quality.
Role of the Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: The Child Health Informatics Research and Development Lab and the Pediatric Research Network group at the Indiana University School of Medicine collected data for this study.