Hospital Injury Encounters of Children Identified by a Predictive Risk Model for Screening Child Maltreatment Referrals: Evidence From the Allegheny Family Screening Tool | Child Abuse | JAMA Pediatrics | JAMA Network
[Skip to Navigation]
Sign In
Figure.  Selected Medical Encounter Rate (per 1000) for Children Referred for Alleged Maltreatment and Risk Scores (2010-2016)
Selected Medical Encounter Rate (per 1000) for Children Referred for Alleged Maltreatment and Risk Scores (2010-2016)
Table 1.  Descriptive Characteristics of Children Stratified by Injury Encounter
Descriptive Characteristics of Children Stratified by Injury Encounter
Table 2.  Correlation Between Risk Score and Hospital Encounter Rate
Correlation Between Risk Score and Hospital Encounter Rate
Table 3.  Odds Ratios for Children Stratified by a Predictive Risk Model, Including a Black vs Nonblack Subpopulation Analysis
Odds Ratios for Children Stratified by a Predictive Risk Model, Including a Black vs Nonblack Subpopulation Analysis
Table 4.  Encounter Rates for Children Stratified by a Predictive Risk Model, Including a Black vs Nonblack Subpopulation Analysis
Encounter Rates for Children Stratified by a Predictive Risk Model, Including a Black vs Nonblack Subpopulation Analysis
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    August 3, 2020

    Hospital Injury Encounters of Children Identified by a Predictive Risk Model for Screening Child Maltreatment Referrals: Evidence From the Allegheny Family Screening Tool

    Author Affiliations
    • 1Auckland University of Technology, Auckland, New Zealand
    • 2University of Southern California, Los Angeles
    • 3Carnegie Mellon University, Pittsburgh, Pennsylvania
    • 4Children’s Hospital of Pittsburgh, UPMC, Pittsburgh, Pennsylvania
    JAMA Pediatr. 2020;174(11):e202770. doi:10.1001/jamapediatrics.2020.2770
    Key Points

    Question  Do children reported as having experienced alleged maltreatment and classified by a predictive risk model to be at high risk of foster care placement show an increased risk of emergency department and inpatient hospitalizations for injuries?

    Findings  In this cohort study, children who scored in the highest 5% risk group by the predictive risk model were more likely to have a medical encounter for an injury during the follow-up period than low-risk children (ie, those in the bottom 50% of risk).

    Meaning  Our findings support that the Allegheny model, trained to predict foster care placement, was sensitive to medical encounters for injuries.

    Abstract

    Importance  Nearly 6 million children are reported as allegedly experiencing abuse or neglect in the US annually. Child protection agencies are increasingly turning to automated predictive risk models (PRMs) that mine information found in routinely collected administrative data and estimate a likelihood that an individual will experience some future adverse outcome.

    Objective  To test if a PRM used at the time of referral for alleged maltreatment, which automatically generates a risk stratification score indicating the relative likelihood of future foster care placement, is also predictive of injury hospitalization data.

    Design, Setting, and Participants  This retrospective cohort study based on a probabilistic association between child protection and hospital encounter data was conducted in Allegheny County, Pennsylvania, and at Children’s Hospital of Pittsburgh (Pittsburgh, Pennsylvania). Participants included children referred for alleged neglect or abuse in Allegheny County between April 1, 2010, and May 4, 2016.

    Exposures  Risk score generated from the PRM.

    Main Outcomes and Measures  Medical encounters (emergency department and inpatient hospitalizations) for any-cause injuries, suicide or self-inflicted harm injuries, and abuse injuries between 2002 and 2015 for children classified by the PRM to different risk levels at the time of a maltreatment referral. Cancer encounters were used as a placebo test.

    Results  Of 47 305 participants, 23 601 (49.9%) were girls, the mean (SD) age at referral was 8 (5.7) years, 28 211 (59.6%) were black, and 19 094 (40.4%) were nonblack. Children who scored in the highest 5% risk group by the PRM were more likely to have a medical encounter for an injury during the follow-up period than low-risk children (ie, those in the bottom 50% of risk). Specifically, among children referred for maltreatment and classified as highest risk, the rate of experiencing an any-cause injury encounter was 14.5 (95% CI, 13.1-15.9) per 100 compared with children who scored as low risk who had an any-cause injury encounter rate of 4.9 (95% CI, 4.7-5.2) per 100. For abuse-associated injury encounters, the rate for high-risk children was 2.0 (95% CI, 1.5-2.6) per 100 and that of low-risk children was 0.2 (95% CI, 0.2-0.3) per 100; for suicide and self-harm, the high-risk encounter rate was 1.0 (95% CI, 0.6-1.4) per 100 and that of low-risk children was 0.1 (95% CI, 0.1-0.1) per 100. There was no association between risk scores and cancer encounters.

    Conclusions and Relevance  Findings confirm that children reported for having experienced alleged maltreatment and classified by a PRM tool to be at high risk of foster care placement are also at increased risk of emergency department and in-patient hospitalizations for injuries.

    Introduction

    In the field of health and human services, there is growing interest in the use of automated decision-support tools, such as predictive risk models (PRMs), to support the targeting of public services. These tools use information found in routinely collected administrative data and estimate a likelihood that an individual will experience some future outcome based on prior individuals with similar data patterns. Likewise, new algorithmic tools are emerging in the field of criminal justice to classify recidivism risk with a goal of using data to make better and more standardized bail and other release decisions.1 The use of PRMs to support decisions made by child protective services (CPS) is also drawing significant academic and popular press attention.2-8 The difficulty of consistently and accurately screening and triaging large volumes of abuse and neglect referrals seems well suited to the use of risk stratification models.9 Yet, developing PRMs for such a use creates challenges. One such challenge is that PRMs by their very nature must be trained to look for data patterns that predict some type of harm. However, unambiguous measures of harm, such as death by physical abuse, are rare. As such, training a model to predict these statistically unlikely events presents substantial methodological barriers and requires much more data than are typically available in a local jurisdiction. Additionally, child protection systems have a broader mandate than simply preventing death (eg, protecting children from lesser forms of harm or stabilizing families so children can remain safely at home). Therefore, developing models to predict more frequent, system-observed outcomes, such as future foster care placement, may be a better model target in the context of available data and the relevance of the model to practices.

    However, efforts to train a risk model based on child protection outcomes should be questioned for at least 2 reasons. First, it may be that the system outcome available for model training purposes is systematically biased. For example, if black children are more frequently placed in foster care not because of actual risk of harm but because the system has applied a lower threshold for placement, then a risk stratification score built to predict this outcome could reflect and reinforce this unwarranted variation and bias. Second, and associated, notwithstanding the child welfare system’s broader mandates concerning child and family well-being, the system is held accountable first and foremost for the physical safety of children. If the system-generated outcome used to train the model for risk stratification purposes is not associated with more objective measures of child harm, then the use of the model could ultimately undermine child safety by directing attention and resources to the wrong cases.

    In the current analysis, we examined whether a risk model trained to predict future child protection events was sensitive to several external measures of child harm. Specifically, we tested whether the Allegheny Family Screening Tool (AFST), built to predict foster care placement at the time of a maltreatment referral, was sensitive to identifying children with a heightened risk of an emergency department (ED) visit or hospitalization because of injury. Our objective was to validate the risk of placement into foster care as a reasonable proxy for child harm and therefore a credible outcome for training risk stratification models for use by CPS systems. We also tested whether the correlation between the predicted risk generated by a PRM and rates of ED visits/hospitalization because of injury were consistent across racial groups.

    Methods

    We undertook a 2-stage process. We first used the same research data set that was used to construct the AFST and rebuilt the same predictive model, which was identical to the one being used. In the second part, we linked this data set with hospital encounter records. Access to AFST data was granted by the Allegheny County Department of Human Services under agreement number 222259/DHS contract ID 50626. Hospital data were accessed through a third-party trustee clinician who linked the data. This project was reviewed and approved as exempt by the Institutional Review Board at the University of Pittsburgh. Consent was waived because the study used precollected administrative data. Data involved in this project were deidentified and the research data set is not publicly available.

    PRM

    The AFST is a PRM used at the time a referral for suspected child abuse or neglect is received by Allegheny County, Pennsylvania. The AFST was built from structured fields captured in the county’s administrative health and human services records. A total of 83 311 referrals were drawn for children referred for alleged neglect or abuse in Allegheny County between April 1, 2010, and May 4, 2016. The observations were unique at the child-referral level. For example, if there were 3 children identified on a referral, the referral would be split into 3 observations associated with each of the children. There were 47 305 unique children included in this study’s data.

    Each observation was coded with a set of 673 features (predictive variables) describing the characteristics of the child, siblings, and other family members and the perpetrator of abuse. Variables included demographic characteristics of the family and perpetrator, allegations associated with the referral, child and maternal characteristics at the time of birth, and histories of interactions with the county’s CPS, social service, and criminal justice systems. There were 46 503 observations during which children were screened in for further investigation (it is these screened-in referrals that were used to train the model), with the data partitioned into a 90%/10% training (n = 40 531) and validation set (n = 5972). The reason we restricted the data to screened-in referrals is that the model10 was only built on screened-in referrals.

    We used a graph-based method to partition the data into these 2 sets11 that grouped all the children associated with a referral into 1 of the training or test partitions. The model was trained to risk-score referrals based on the likelihood of placement in foster care within 2 years of the screened-in referral. A least absolute shrinkage and selection operator (LASSO) regression method was used for modeling.12 Training data were split using 10-fold cross-validation with these folds also selected to maintain children associated with the same referral grouped into a single fold. The cross-validated model was trained to optimize for the area under the receiver operator curve. The model selected 118 variables as weighted predictors of the target outcome along with the intercept term and achieved an area under the curve of 0.78 on the validation set using the LASSO λ parameter at 1 SE from the best found optimal λ parameter value.

    We then used the PRM to score each child-referral observation based on the probability corresponding to a child’s predicted risk. Probabilities were then translated into scores from 1 to 20, with 1 reflecting the 5% of referrals with the lowest risk of future system involvement and 20 the 5% of referrals with the highest risk. These scores corresponded to the actual scores that are being used by the county. Overall, almost half of referrals are screened out by the county with no further action. As currently implemented in practice, the PRM score is incorporated into call screener discussions with supervisors when determining whether to screen in for an in-person investigation or to screen out with no further action.

    Medical Encounter Validation

    All children referred for suspected child abuse and neglect (maltreatment) during the observation window were linked with medical records from UPMC Children’s Hospital of Pittsburgh between February 2002 and December 2015 (the period available in the existing electronic records). UPMC Children’s Hospital is the sole provider of secondary care for children in the Allegheny County area. Records were linked by a third party (an honest broker) using a probabilistic matching method that incorporated the child’s first name, last name, date of birth, and Social Security number. A deidentified data set was then constructed and released to the research team.

    A total of 13 239 of the 47 305 unique children (28%) in our observations were matched to any ED or hospitalization record, referred to as a medical encounter throughout the rest of this article. We identified specific injury encounters based on the following International Classification of Diseases, Ninth Revision (ICD-9) codes: (1) all-cause injury (800-999); (2) suicide and self-inflicted injuries (E950-E959); and (3) abusive injury (E955). We also examined medical encounters because of cancer (140-165, 170-176, 179-239) as a placebo test to confirm that our model was not sensitive to a health condition that should be unassociated with child safety. We examined injury encounters (by cause) to validate the risk model using 2 different approaches.

    Highest Risk Score and Subsequent Injury Encounter

    We looked at all unique children in the data and coded the child’s risk level based on the highest risk score assigned during the analytic window. However, we only considered injury encounters occurring after this date.

    Randomly Selected Risk Score and Subsequent Injury Encounter

    We looked at all unique children in the data, randomly selected a referral and associated risk score for each child during the analytic window and coded a medical encounter as having occurred only if the selected referral date was before the injury encounter. We presented the correlation between medical encounter and risk score by graphing the rate of encounters against the risk score as well as calculating the correlation coefficient using Stata (version 14.2; StataCorp) command “pwcorr.”

    Next, we calculated the medical encounter rate if the child was defined as high risk. To define high risk, we took either (1) the highest-scored referral for each child or (2) a random referral for each child. For each of these referrals, we defined a child at high risk if their risk score was 20 (ie, in the top 5% most risky of all referrals). We also calculated the encounter rate for children who were assigned a score that placed them in the lowest 50% of referrals in the selected sample (with a score of 1-10).

    We excluded those who were neither high risk nor low risk from the analysis and calculated the unadjusted odds ratio using Stata (version 14.2; StataCorp). We also calculated (separately) the odds ratios for the subpopulation of black children and nonblack children using the same approach. To test whether these odds ratios were different for black and nonblack children, we estimated an interacted logistic regression. Statistical significance was set at P < .05.

    Results
    Injury Events

    In Table 1, we present the descriptive characteristics of children with at least 1 referral of maltreatment, stratified by injury encounters. Of the children who had a screened-in referral, 23 704 (50%) were boys and 28 211 (60%) were identified as black or African American in the CPS data. The mean (SD) age at the time of referral was 8.0 (5.7) years. Of the children who had an injury case, 4500 (54%) were boys and 3886 (47%) were black. Children who had injury encounters were also slightly older, with a mean (SD) age of 9.1 (4.8) years.

    In the Figure, we present the rate of injury-associated medical encounters and cancer encounters based on the 2 definitions provided: (1) the highest risk score before an injury encounter and (2) randomly selected risk score before an injury encounter. We also present data for children with a medical encounter associated with cancer. In Table 2, we present the correlation coefficient.

    Overall, the graph confirms the correlations in Table 2, that a high-risk score is strongly correlated with medical encounters for any-cause injury events. Across both definitions depicted in Figure, A, we see a clear association between any-cause injury encounters and risk ventile, with an increase in the gradient for those scoring 17 and higher. In Figure, B and C, we document similarly graded associations between the risk scores and suicide/self-inflicted injuries and abusive injuries. Cancer hospitalization does not show a similarly graded association, which is confirmed with the correlation coefficients of 0.53 and 0.44.

    Our findings are robust for the 2 approaches to defining the association between risk scores and injury events. As expected, restricting the medical encounters to those that occurred after the referral reduces the rate of encounters but does not alter the positive gradient.

    In Table 3 and Table 4, we report the encounter rates (per 100) and odds ratios for injury encounters among children with a referral risk score of 20 (ie, the top 5% most risky referrals) and those who scored low risk (ie, score 1-10). We report results for all children in addition to statistics for black children as well as nonblack children.

    For any-cause injury encounters, the encounter rate for high-risk children was 14.5 (95% CI, 13.1-15.9) per 100 compared with a rate of 4.9 (95% CI, 4.7-5.2) per 100 for low risk. For black children, the rate for the high-risk group was 16.1 (95% CI, 13.7-18.4) per 100 compared with 8.1 (95% CI, 7.6- 8.7) per 100 for low risk. While the overall encounter rates for specific injury causes of abuse and suicide/self-inflicted injury are lower, the contrast as measured by the odds ratio is larger. In contrast, the encounter rate for cancer is identical across the risk groups and the odds ratio is not statistically significantly different to 1.0. The sample size was too small to calculate the odds ratio for cancer by racial subgroups.

    A notable finding is the much higher association for nonblack children and abusive injury medical encounters. For example, for the highest risk score received before the medical encounter, the odds ratio of 12.70 (95% CI, 7.21-22.36) is compared with 4.04 for black children (95% CI, 2.36-6.90).

    Discussion

    Our results provide evidence that the risk scores generated by the AFST, while developed from a model trained to predict foster care placement, are highly sensitive to measures of injury harm as captured in medical encounter data. Using 2 specifications, we documented a strong, graded association between risk scores and medical injury encounters. This association was even more pronounced (see measured-by odds ratios) for the 2 forms of injuries most closely associated with child safety: abusive injuries and suicide/self-inflicted harm. Moreover, risk scores did not correlate with cancer-associated medical encounters, suggesting that the PRM underlying the AFST provides information about the risk of medical encounters closely associated with child safety rather than ED and hospital encounters more broadly.

    Findings from our validation effort showed a mixed picture with respect to the sensitivity of the risk score to different racial subpopulations. High-risk black and nonblack children had similar encounter rates for any-cause injury and specific injury hospitalization. However, the odds ratio for abuse-associated injuries was higher for nonblack children than black children, suggesting that the contrast is more marked for nonblack children. This is an area worthy of further investigation, as it could suggest that these sorts of tools are not equally sensitive for all subpopulations. Our findings suggest that even if certain high-risk referrals do not meet a legal threshold for investigation by CPS, these children and families may need services or support to ensure the safety and well-being of the child.

    Limitations

    Despite the strengths of using population-based, medical encounter records to externally validate a child protection risk model, our analysis is not without limitations. Our linkage reflects the full census of children referred for maltreatment who had an ED encounter or inpatient hospitalization at the children’s hospital regardless of family income or insurance type. However, a limitation is that the third party permitted to link the 2 data sources was limited to personal identifiers that had not been verified and contained unknown rates of clerical errors. Because there was no audit performed to assess the error rate of the match, our analytic data set contains an unknown number of false-positive and false-negative matches. Relatedly, our analysis provides a test of a risk model trained to predict foster care placement in Allegheny County, Pennsylvania. It is unknown to what extent findings would generalize to risk models trained to predict placement in other jurisdictions. Finally, assuming that foster care placement is protective, we would expect that high-risk children who are placed immediately will have experienced an intervention that reduces their risk. Additionally, to the extent that we include observations in our data that were used to train the model, there is a danger that the rate of placement would be unusually high in the highest risk group because of overfitting. For this reason, we calculated the relative risk of abuse-associated injuries following the CPS referral for those who were placed in foster care within 2 years of the referral and those who were not (using the highest risk score). We found that for those children who were placed within 2 years of the referral, the relative risk of a subsequent maltreatment-associated ED visit or hospitalization was 1.67 (95% CI, 0.86-3.26) while those who were not placed in foster care had a relative risk of 6.04 (95% CI, 4.54-8.05). Given that foster care placement appears to be protective, our estimates reported in Table 4 amount to lower-bound estimates of differences between high- and low-risk children. Future research will further explore this finding. We did not have a sufficient sample size to test the association between placement and suicide and self-harm outcomes.

    Conclusions

    As various types of PRMs built to predict CPS system outcomes become more widely adopted in child protection, it is critical that they are carefully assessed and validated using external data sources. In this study, we found evidence that the Allegheny County model, trained to predict foster care placement, was sensitive to medical encounters for injuries. The findings held across several validation definitions and suggest that models that can identify children at high risk of future foster care placement and a heightened risk of physical harm.

    Back to top
    Article Information

    Accepted for Publication: March 29, 2020.

    Corresponding Author: Rhema Vaithianathan, PhD, Centre for Social Data Analytics, Auckland University of Technology, 2-4 Wakefield St, WT Building, Room 1103, Auckland, New Zealand 1010 (rhema.vaithianathan@aut.ac.nz).

    Published Online: August 3, 2020. doi:10.1001/jamapediatrics.2020.2770

    Correction: This article was corrected on September 14, 2020, to fix errors in the rate of participants and Table numbering.

    Author Contributions: Dr Vaithianathan had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Vaithianathan, Putnam-Hornstein, Berger.

    Acquisition, analysis, or interpretation of data: Putnam-Hornstein, Chouldechova, Benavides-Prado, Berger.

    Drafting of the manuscript: Vaithianathan, Putnam-Hornstein, Benavides-Prado, Berger.

    Critical revision of the manuscript for important intellectual content: Putnam-Hornstein, Chouldechova, Benavides-Prado, Berger.

    Statistical analysis: Vaithianathan, Chouldechova.

    Administrative, technical, or material support: Berger.

    Supervision: Berger.

    Conflict of Interest Disclosures: Dr Chouldechova reported grants from Allegheny County during the conduct of the study. Drs Vaithianathan, Putnam-Hornstein, and Benavides Prado reported grants from Allegheny County Department of Human Services during the conduct of the study. No other disclosures were reported.

    Additional Contributions: Drs Vaithianathan, Putnam-Hornstein, and Benavides Prado were involved in the development of the AFST, which was funded by Allegheny County Department of Human Services.

    References
    1.
    Berk  R.  Criminal Justice Forecasts of Risk: A Machine Learning Approach. Springer Science & Business Media; 2012. doi:10.1007/978-1-4614-3085-8
    2.
    Vaithianathan  R, Maloney  T, Jiang  N.  Vulnerable Children—Can Administrative Data Be Used to Identify Children at Risk of Adverse Outcomes? Ministry of Social Development; 2012.
    3.
    Vaithianathan  R, Maloney  T, Jiang  N, Putnam-Hornstein  E.  Children in the public benefit system at risk of maltreatment: identification via predictive modeling.   Am J Prev Med. 2013;45(3):354-359. doi:10.1016/j.amepre.2013.04.022 PubMedGoogle ScholarCrossref
    4.
    Chouldechova  A, Benavides-Prado  D, Fialko  O, Vaithianathan  R.  A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions.   Proc Mach Learn Res. 2018;81:1-15. http://proceedings.mlr.press/v81/chouldechova18a/chouldechova18a.pdfGoogle Scholar
    5.
    de Haan  I, Connolly  M.  Another Pandora's box? Some pros and cons of predictive risk modeling.   Child Youth Serv Rev. 2014;47:86-91. doi:10.1016/j.childyouth.2014.07.016 Google Scholar
    6.
    Wilson  ML, Tumen  S, Ota  R, Simmers  AG.  Predictive modeling: potential application in prevention services.   Am J Prev Med. 2015;48(5):509-519. doi:10.1016/j.amepre.2014.12.003 PubMedGoogle Scholar
    7.
    Cuccaro-Alamin  S, Foust  R, Vaithianathan  R, Putnam-Hornstein  E.  Risk assessment and decision making in child protective services: predictive risk modeling in context.   Child Youth Serv Rev. 2017;79:291-298. doi:10.1016/j.childyouth.2017.06.027 Google Scholar
    8.
    Hurley  D. Can an algorithm tell when kids are in danger? Accessed June 30, 2020. https://www.nytimes.com/2018/01/02/magazine/can-an-algorithm-tell-when-kids-are-in-danger.html
    9.
    Coulton  CJ, Goerge  R, Putnam-Hornstein  E, de Haan  B. Harnessing big data for social good: a grand challenge for social work. Accessed June 30, 2020. https://grandchallengesforsocialwork.org/wp-content/uploads/2015/12/WP11-with-cover.pdf
    10.
    Vaithianathan  R, Kulick  E,, Putnam-Hornstein  E, Benavides-Prado  D. Allegheny Family Screening Tool: methodology, version 2. Accessed May 7, 2019 https://www.alleghenycountyanalytics.us/wp-content/uploads/2019/05/Methodology-V2-from-16-ACDHS-26_PredictiveRisk_Package_050119_FINAL-7.pdf
    11.
    Csardi  G, Nepusz  T.  The igraph software package for complex network research.   InterJournal Complex Systems. 2006;1695(5):1-9. https://pdfs.semanticscholar.org/1d27/44b83519657f5f2610698a8ddd177ced4f5c.pdfGoogle Scholar
    12.
    Friedman  J, Hastie  T, Tibshirani  R.  Regularization paths for generalized linear models via coordinate descent.   J Stat Softw. 2010;33(1):1-22. doi:10.18637/jss.v033.i01 PubMedGoogle Scholar
    ×