Comparing Public Safety Outcomes for Traditional Probation vs Specialty Mental Health Probation

Key Points

Question  Does specialty probation yield better public safety outcomes than traditional probation for people with mental illness?

Findings  In this longitudinal study that included 359 probationers with mental illness, specialty probation did not significantly reduce violence but substantially reduced rearrest rates. At 2 years, an estimated 28.6% of specialty probationers and 51.8% of traditional probationers had been rearrested.

Meaning  In the current era of criminal justice reform, specialty probation holds substantial promise as a method for reducing mass incarceration for people with mental illness.

Importance  Probation is a cornerstone of efforts to reduce mass incarceration. Although it is understudied, specialty probation could improve outcomes for the overrepresented group of people with mental illness.

Objective  To test whether specialty probation yields better public safety outcomes than traditional probation.

Design, Setting, and Participants  A longitudinal observational study with group matching on age, sex, race/ethnicity, probation time, and offense at 2 urban agencies that exemplify specialty and traditional probation. Enrollment began October 19, 2005; follow-up data were complete January 26, 2013. Participants were 359 diverse probationers with serious mental health problems and functional impairment. Probationers and officers were assessed 3 times (for probationers, 6-month retention, 315 of 359 [88%]; 12-month retention 304 of 359 [85%]) and follow-up records were obtained. Machine learning algorithms were combined with a targeted maximum likelihood estimation, a double robust estimator that accounts for associations between confounders and both treatment assignment and outcomes. Statistical analysis was conducted from January 1, 2016 to May 5, 2017.

Interventions  Specialty probationers were assigned to small, homogeneous caseloads supervised by experts. Prior data indicate that specialty officers had better relationships with probationers, participated more in probationers’ treatment, and relied more on positive compliance strategies than traditional officers.

Main Outcomes and Measures  Violence during 1 year, determined by probationer report, officer report, and records, and rearrest during a period of 2 to 5 years, according to Federal Bureau of Investigation records.

Results  Participants were 183 specialty (73.8% of 248 eligible) and 176 traditional (56.6% of 311 eligible) probationers (205 men and 154 women; mean [SD] age, 36.9 [10.6]). Although specialty probation had no significant effect on violence (odds ratio, 0.97; 95% CI, 0.69-1.36), the odds of rearrest were 2.68 times higher for traditional probationers than for specialty probationers (95% CI, 1.86-3.84; P < .001). At 2 years, estimated probabilities of rearrest were 28.6% for specialty probationers and 51.8% for traditional probationers. Survival analyses indicate that arrest effects endured up to 5 years.

Conclusions and Relevance  Although it did not specifically reduce violence, well-implemented specialty probation appears to be effective in reducing general recidivism. Reform efforts for people with mental illness could leverage probation—a ubiquitous and revitalized node of the justice system.

There is a well-recognized need for effective, specialized interventions for individuals with mental illness in the criminal justice system. In the United States, rates of mental illnesses such as major depression, bipolar disorder, and schizophrenia are 3 to 6 times higher in the criminal justice population than the general population.¹ Each year, approximately 2 million people with serious mental illness are booked into the nation’s jails.¹ These people typically stay longer in jail than do those without mental illness and, on release, are more likely to be reincarcerated.² “Jails spend 2 to 3 times more money on adults with mental illnesses…than on those without those needs, yet often do not see improvements to public safety or these individuals’ health.”³

Thus far, specialized intervention efforts have had limited success.² The human and fiscal costs of this problem have inspired Stepping Up, a national initiative led by the American Psychiatric Association Foundation, the National Association of Counties, and the Council of State Governments. Stepping Up structures counties’ efforts to develop cost-effective action plans that promote evidence-based alternatives to jail. To date, more than 300 counties have resolved to “step up” their efforts to reduce the number of people with mental illness in jail.⁴

Probation is an alternative to incarceration that could be well leveraged as part of efforts to reduce the number of people with mental illness in jail. Probation is the most common form of sentencing in the United States, with more than half of the correctional population supervised in the community,⁵ and probation has become a cornerstone of policies designed to reduce mass incarceration. In reform-oriented jurisdictions, surveillance-style probation is being replaced with balanced supervision approaches that include evidence-based practices shown to reduce recidivism.⁶ In part, this is because correctional services delivered in the community are lower in cost and more effective in reducing offenders’ recidivism than those delivered behind bars.^7,8

This larger context revitalizes specialty mental health probation, which has long been promoted for people with mental illness in the criminal justice system.^9,10 According to a national survey conducted more than a decade ago,¹¹ more than 130 agencies were implementing specialty probation, but only a subset manifested key characteristics that distinguish it from traditional probation, including the following: (1) small caseloads (<50 individuals) composed solely of people with mental illness (vs heterogeneous caseloads, with >100 individuals), (2) sustained officer training in mental health, (3) officer coordination of and direct involvement in probationers’ treatment, and (4) reliance on collaborative problem solving approaches. In prototypic specialty agencies, officers balance “control” (law enforcement) with “care” (social work) and stress linkage with psychiatric services as a key to reduction in recidivism.^12,13

Although key features of the specialty model have been clarified, research on whether specialty mental health probation works is especially limited. The most basic question is whether specialty probation reduces recidivism, given that the chief goal of justice agencies is to protect public safety.¹⁴ We could identify only 2 controlled studies of the effect of specialty probation on recidivism. First, in an unpublished evaluation (described in Skeem and Louden¹⁵) that is sparse on methodological detail, investigators randomly assigned 400 California probationers to specialty or traditional probation and found no difference in the groups’ rates of return to the local jail. Second, using administrative data from New Jersey, Wolff et al¹⁶ found a greater decrease in jail days for probationers placed on specialty mental health supervision than for probationers on traditional probation who received any mental health services, after controlling for basic demographics and type of offense. Although these results are promising, the follow-up was short (6 months) and the covariate set was limited, increasing the risk that results reflect unmeasured clinical and criminologic differences between groups.

Rigorous evidence about the effect of specialty probation is needed to inform efforts to step up justice reform for this group. In this article, we describe the results of a longitudinal, multimethod study designed to fill this gap in the literature. The aims of the study were to assess whether specialty probation reduces the likelihood of violence and arrest compared with traditional probation for probationers with mental illness. Outcomes were measured during follow-up periods of 1 to 5 years, based on multiple sources (self-report, collateral-report, and Federal Bureau of Investigation records). If our hypothesis that specialty probation improves public safety is supported, the study could elevate the status of probation itself—a ubiquitous and revitalized component of the justice system.

The results of a national survey on probation and mental health¹¹ were used to select 2 urban agencies that exemplified the specialty model (in Texas) and the traditional model (in California) but were similar in jurisdiction size, probationer characteristics, and mental health expenditures. Matched groups of probationers were recruited from the 2 agencies. Enrollment began October 19, 2005; follow-up data were complete January 26, 2013. Probationers and their supervising officers were assessed at 3 time points after probation placement, and official records were reviewed. Specifically, probationers were interviewed at baseline and were interviewed again 6 and 12 months later (6-month retention, 315 of 359 [88%]; 12-month retention 304 of 359 [85%]). Officers were surveyed on the same schedule. Records were coded to capture treatment involvement and arrests, after placement. The protocol was approved by the University of California, Irvine; Texas Department of Criminal Justice; and California Department of Corrections and Rehabilitation. All participants provided written informed consent.

Eligibility criteria included age 18 to 65 years, English speaking, on active probation, provision of voluntary and informed consent to participate, and identification as mentally ill (without mental retardation). The processes for identifying probationers with mental illness differed by site.

At the specialty site, probationers were referred to the specialty program by traditional officers (176 of 183 cases [96%]) or judges (7 or 183 cases [7%]) and received a diagnosis of a mental illness based on a psychological evaluation. Of 248 eligible probationers assigned to specialty caseloads, 183 (73.8%) enrolled. No significant differences were observed between enrollees and nonenrollees in age, sex, or self-reported race/ethnicity.

At the traditional site, researchers identified probationers with mental illness by asking officers to refer clients appropriate for specialty supervision to the study (ie, clients with known psychiatric problems, psychotropic medications, and/or hospitalizations) and adding validated mental health screening tools to the probation intake process.¹³ Research clinicians verified the presence of psychiatric problems, both for referred (141 of 176 [80%]) and screened-in (35 of 176 [20%]) participants.

Researchers enrolled traditional probationers who matched specialty probationers in sex, age, race/ethnicity, length of time on probation, and index offense type. Of 311 eligible and matched probationers, 176 (56.6%) enrolled. There were no significant differences between enrollees and nonenrollees in age, sex, or race/ethnicity.

Specialty and traditional probationers were similar across matching variables. As shown in the Table, probationers were ethnically diverse men and women with serious mental illness; their mean (SD) Colorado Symptom Index scores were at or near the cutoff score of 30 for psychiatric disability (specialty probationers, 30.1 [12.1]; traditional probationers, 25.8 [12.8]).¹⁷ Personality Assessment Inventory substance abuse subscale scores are consistent with the median for clinical samples,¹⁸ with no significant group differences.

The implementation of specialty probation and traditional probation was carefully assessed in this study. Results are detailed by Manchak et al.¹³ In brief, in the specialty agency, probationers were assigned to small caseloads composed exclusively of people with mental illness and supervised by officers with relevant expertise. The mean caseload size was 50 probationers for specialty officers (n = 15) and approximately 100 probationers for traditional officers (n = 87). Compared with traditional officers, specialty officers established higher-quality relationships with probationers, participated more directly in probationers’ treatment, and relied more heavily on positive compliance strategies than sanction threats. Specialty probationers were significantly more likely to receive mental health treatment (167 of 183 [91%] vs 106 of 176 [60%]) and integrated dual diagnosis treatment (62of 183 [34%] vs 26 of 176 [15%]) than were traditional probationers within 1 year, but they were no more likely to receive substance use treatment (51 of 183 [28%] vs 55 of 176 [31%]).¹³

Violence was operationalized based on the approach used by Steadman et al.¹⁹ At each assessment, probationers and their officers were independently asked whether the probationer had been involved in several categories of aggressive acts in the past 6 months. Official probation records were also reviewed. The most serious act per violent incident was coded. Violence was defined as physical battery, sexual assault, threat with a weapon in hand, or an assaultive act with a weapon. Because aggression tends to be underreported, violence was coded as having occurred when reported by any source. During the 1-year follow-up, 102 of 291 probationers (35.1%) were involved in violence.

Probationers’ Federal Bureau of Investigation reports were obtained to code the date and type²⁰ of arrests that occurred during a minimum 2-year follow-up period. The most serious charge per arrest was coded; arrests solely for probation violations were excluded, given our focus on public safety outcomes. During the 2-year follow-up, 150 of 354 probationers (42.4%) were arrested; of 274 for whom charge type data were available, 100 (36.5%) were listed as a person arrest, 69 (25.2%) a property arrest, 82 (29.9%) a drug arrest, and 23 (8.4%) minor offenses.

Because probationers had different lengths of follow-up, we also created variables to indicate how long probationers survived in the community without rearrest. These variables indicate the number of days without an arrest since the baseline interview (N = 354; mean [SD] follow-up period, 738.8 [448.4] days), excluding days incarcerated or hospitalized.

To accurately estimate the association between specialty probation and public safety outcomes, we controlled for covariates that theoretically predicted both treatment assignment (specialty vs traditional) and outcomes. Based on past research and empirical relationships, these covariates were selected from a pool of more than 100 baseline characteristics.¹³ The potential confounders were specified a priori using causal theory graphs.²¹ Missing values were imputed for continuous and categorical covariates using median imputation and resampling with replacement.

The main covariate set consisted of the 21 variables listed in the Table. This set characterizes a probationer’s demographics and socioeconomic status, history of criminal behavior and childhood abuse,²⁰ substance abuse, externalizing symptoms, and other psychiatric symptoms, based on the Personality Assessment Inventory,¹⁸ the Colorado Symptom Index,¹⁷ and the Global Assessment of Functioning.²² Measurement information is detailed by Manchak et al.¹³ These variables were selected as the main covariate set because the set is large enough to control for confounding but small enough to prevent picking up noise as the signal. Although our prediction algorithms use a final cross-validation step that prevents overfitting, our final estimates may be biased if there are too many covariates for our sample size. In the eAppendix in the Supplement, we describe 2 alternative covariate sets and the results they yielded (which parallel those reported).

Primary statistical analysis was conducted from January 1, 2016, to May 5, 2017. We used targeted maximum likelihood estimation (TMLE)²³ to estimate the association between specialty vs traditional probation and violence for 1 year and rearrest for 2 years. Targeted maximum likelihood estimation is a semiparametric estimator that depends on estimation of both the treatment mechanism (ie, probability of specialty assignment, given covariates) and the outcome (ie, violence or rearrest, given treatment and covariates). Targeted maximum likelihood estimation is doubly robust: it is consistent (unbiased as sample size grows) if either treatment assignment or the outcome is correctly estimated. Unlike alternative techniques, the estimator does not rely solely on a correctly specified outcome regression or treatment mechanism.²⁴

Both the outcome regression and treatment mechanism were estimated using the SuperLearner algorithm, an ensemble method that combines a library of data-adaptive machine learning algorithms and parametric models to build an estimator that performs as well or better than any algorithm in the library.²⁵ The library of methods helps one avoid potentially problematic assumptions (eg, linear associations), and a cross-validation step is included to avoid overfitting. In our application, the library included the 18 methods listed in the eAppendix in the Supplement.

Participants had different lengths of follow-up for rearrest, many of which were longer than the fixed 2-year follow-up. To make full use of each participant’s follow-up period, we used survival analyses to examine whether specialty probation preceded and increased the likelihood of longer “survival time” in the community without arrest. Specifically, we used the Kaplan-Meier method to estimate survival probabilities and median survival time by group.²⁶ Observations were weighted based on the propensity scores estimated by SuperLearner (ie, the estimated treatment assignment mechanism in TMLE). That is, participants whose conditional probability of receiving their probation type was low were upweighted, and those whose probability of receiving their probation type was high were downweighted, to approximate random assignment. Confidence intervals were calculated by adding and subtracting twice the SE of the estimate.

Analyses were performed using R, version 3.3.0 (GNU S), and statistical significance was set at α = .05. Targeted maximum likelihood estimation analyses were performed using the tmle package.²⁷ Estimation of the outcome regression and treatment mechanism was performed using SuperLearner.²⁸

Raw descriptive statistics indicate that specialty probationers and traditional probationers had similar observed rates of violence (54 of 149 [36.2%] vs 48 of 142 [33.8%] at 1 year) but not arrest (57 of 181 [31.5%] vs 93 of 173 [53.8%] at 2 years). These figures, however, are uncorrected for covariates.

SuperLearner is used to test and combine multiple models. As shown in the eAppendix in the Supplement, the highest weighted algorithms chosen by SuperLearner, shared across treatment assignment and outcome regressions, included generalized linear models, Bayesian generalized linear models, multivariate adaptive regression splines (“earth”), and classification and regression training (“caret”).

The results of TMLE estimation and inference indicate that specialty supervision has no significant association with violence. The odds of violence within 1 year are similar for traditional probationers and specialty probationers (odds ratio, 0.97; 95% CI, 0.69-1.36; P = .84). Figure 1 displays estimated probabilities of violence, which indicate a nonsignificant 0.76% difference between groups (95% CI, –8.21% to 6.69%; P = .84).

However, specialty probation is associated with a significant reduction in rearrest. Targeted maximum likelihood estimation indicates that the odds of rearrest within 2 years are 2.68 (95% CI, 1.86-3.84; P < .001) higher for probationers in traditional probation than those in specialty probation. As shown in Figure 1, the probability of rearrest within 2 years is 23.1% (95% CI, 14.9%-31.4%; P < .001) higher for probationers in traditional probation (51.8%) than specialty probation (28.6%).

Kaplan-Meier inverse propensity–weighted estimates plotted in Figure 2 show that specialty probationers (top curve) have longer times until the first arrest than do traditional probationers (bottom curve). The nonoverlapping 95% CIs indicate that group differences are maintained across time. At nearly 5 years (right side), the probability of having survived in the community without a rearrest is 62% for specialty probationers and 36% for traditional probationers.

Probation has become a cornerstone of policies designed to reduce mass incarceration in the United States. To inform efforts to step up justice reform for people with mental illness, we conducted a multimethod study to estimate the effect of specialty probation on public safety outcomes. We used TMLE, a double robust estimator, to approximate the causal effect of specialty probation on public safety outcomes. These estimates suggest that specialty probation had no significant effect on violence, but substantially reduced rearrest rates. Traditional probationers were 2.68 (95% CI, 1.86-3.84) times more likely to be rearrested within 2 years than were those on specialty mental health probation, which translates to a 23.1% decrease in arrest rates (28.6% specialty vs 51.8% traditional).

Because this study is the first, to our knowledge, to examine the effect of specialty probation on violence, it is difficult to contextualize our null finding for this outcome. During the 1-year follow-up, more than one-third of probationers were involved in violence, whether they were on specialty supervision or not. This lack of difference could be because specialty probation focuses on psychiatric service linkage, and mental illness is a weak predictor of violent recidivism.²⁹ It is more likely, however, that this lack of difference is because probation is not specifically designed to reduce violence.

Instead, the mission of probation is to protect general public safety. In keeping with that mission, our results indicate that specialty probation substantially reduced the likelihood of rearrest for any crime (ie, person, property, drug, and minor offenses). This finding is consistent with the finding by Wolff et al¹⁶ that specialty supervision had a short-term effect on the number of jail days, but it greatly extends that finding¹⁶ by focusing on arrests (which index public safety more than utilization) over a longer follow-up (years, not months). In the present study, estimated effects were meaningful and lasting; survival analyses indicate that nearly 5 years after placement, the probability of no rearrest was 62% among specialty probationers compared with only 36% for traditional probationers (Figure 2).

The positive effects observed in these studies are unlikely to generalize to nonprototypical agencies; results of a national survey suggest that as specialty agencies increase caseload sizes above a mean of 45, they begin to function like traditional agencies.¹¹ Therefore, agencies must allocate resources appropriately to permit high-fidelity implementation of specialty caseloads. In a future report, we will show that specialty probation is more cost-effective than traditional probation because costs of small caseloads are more than offset by savings in arrests and acute services.

In future work, we will also examine how specialty probation reduces rearrest rates. Possible mechanisms of specialty probation’s effectiveness include better correctional practices and symptom control. In this sample,¹⁴ we found that specialty officers managed half the caseload size of traditional officers and established higher-quality relationships with probationers, participated more directly in probationers’ treatment, and relied more heavily on positive compliance strategies.¹³

Participants were not randomly assigned to probation types; instead, specialty probationers were drawn from one jurisdiction, and traditional probationers were drawn from another, introducing potential confounds. To address this issue, we first used a powerful causal inference estimation approach (TMLE) and included a rich set of 21 covariates to adjust for confounders (eTable in the Supplement). As shown in the eAppendix in the Supplement, similar results emerged when we used an even more expansive set of covariates in sensitivity analyses. Second, we consulted Federal Bureau of Investigation comparison data,³⁰ which indicate that average arrest rates from 2007 to 2012 were slightly higher in the specialty jurisdiction than in the traditional jurisdiction (6% vs 3%), easing concern that the estimated reduction of arrest rates for specialty probationers is an artifact of local arrest practices. Together, these points (plus sample matching, precise measurement, and strong implementation) lend substantial confidence to our results, but they need to be replicated in an experiment.

Some hallmarks of specialty mental health probation, such as establishing firm, fair, and caring relationships with probationers, are general staples of evidence-based probation.^31,32 These study results provide direction for leveraging reform in general probation to help jurisdictions step up their efforts to reduce incarceration specifically for people with mental illness.

Corresponding Author: Jennifer L. Skeem, PhD, School of Social Welfare, Goldman School of Public Policy, University of California–Berkeley, 120 Haviland Hall, Berkeley, CA 94720 (jenskeem@berkeley.edu).

Accepted for Publication: April 23, 2017.

Published Online: August 9, 2017. doi:10.1001/jamapsychiatry.2017.1384

Author Contributions: Dr Skeem and Ms Montoya had full access to all the data in the study and take responsibility for the integrity of the data and accuracy of the data analysis.

Study concept and design: Skeem.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Skeem, Montoya.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Skeem, Montoya.

Obtained funding: Skeem.

Administrative, technical, or material support: Skeem, Manchak.

Study supervision: Skeem.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was funded by a grant from the John D. and Catherine T. MacArthur Research Network on Mandated Community Treatment.

Role of the Funder/Sponsor: The funding source had a role in the design and conduct of the study via a formal research network forum. The funding source had no role in the collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.

Additional Contributions: Data for this study were collected, entered, and cleaned by staff and volunteers in California and Texas who are affiliates of the Risk Reduction Lab (originally based at the University of California, Irvine; now based at the University of California, Berkley). They received no financial compensation.