Quality of life among full-service partnership (FSP) clients and homeless clients in outpatient programs (OPs). All differences are significant at P < .012.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Gilmer TP, Stefancic A, Ettner SL, Manning WG, Tsemberis S. Effect of Full-Service Partnerships on Homelessness, Use and Costs of Mental Health Services, and Quality of Life Among Adults With Serious Mental Illness. Arch Gen Psychiatry. 2010;67(6):645–652. doi:10.1001/archgenpsychiatry.2010.56
Copyright 2010 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2010
Chronically homeless adults with severe mental illness are heavy users of costly inpatient and emergency psychiatric services. Full-service partnerships (FSPs) provide housing and engage clients in treatment.
To examine changes in recovery outcomes, mental health service use and costs, and quality of life associated with participation in FSPs.
A quasi-experimental, difference-in-difference design with a propensity score–matched control group was used to compare mental health service use and costs of FSP with public mental health services. Recovery outcomes were compared before and after services use, and quality of life was compared cross-sectionally.
San Diego County, California, from October 2005 through June 2008.
Two hundred nine FSP clients and 154 clients receiving public mental health services.
Main Outcome Measures
Recovery outcomes (housing, financial support, and employment), mental health service use (use of outpatient, inpatient, emergency, and justice system services), and mental health services and housing costs from the perspective of the public mental health system.
Among FSP participants, the mean number of days spent homeless per year declined 129 days from 191 to 62 days; the probability of receiving inpatient, emergency, and justice system services declined by 14, 32, and 17 percentage points, respectively; and outpatient mental health visits increased by 78 visits (P < .001 each). Outpatient costs increased by $9180; inpatient costs declined by $6882; emergency service costs declined by $1721; jail mental health services costs declined by $1641; and housing costs increased by $3180 (P < .003 each). Quality of life was greater among FSP clients than among homeless clients receiving services in outpatient programs.
Participation in an FSP was associated with substantial increases in outpatient services and days spent in housing. Reductions in costs of inpatient/emergency and justice system services offset 82% of the cost of the FSP.
The lack of safe, affordable, and integrated housing remains a significant barrier to participation in both treatment and community life for persons with a serious mental illness (SMI).1 A recent point-in-time estimate indicated that 672 000 persons in the United States were homeless on a single night in January 2007; 58% of the homeless individuals were sheltered and 28% of those who were sheltered had SMI.2 Chronically homeless individuals may spend years or even decades living on the streets and in shelters, cycling through emergency departments, inpatient and crisis facilities, jails, and mental health and substance use programs.3 While the multiple service systems used by homeless persons with SMI provide various opportunities for engagement, research has shown that homeless persons with SMI are more likely to engage in a subset of treatment programs that are most responsive to their needs.4
Traditional housing placements often involve segregated, congregate (ie, group) settings, and programs typically require individuals to transition from more to less restrictive living situations based on their progress in treatment, including adherence to medication and sobriety.1 However, this sequential approach to housing and treatment and lack of consumer choice often fails to engage those with the most severe illnesses.1 Research has shown that people with SMI prefer to live independently in community settings5 and that consumer choice is an important predictor of clients' success in both retaining their housing and remaining in treatment.6
As an alternative to the traditional housing model, the Housing First model provides homeless individuals immediate access to permanent housing and access to both a treatment team and community supports that provide flexible, consumer-driven services.1 Studies of Housing First have found it to be effective at improving residential outcomes among homeless persons with SMI.6-10 Major supported housing initiatives, which share operational and philosophical similarities with Housing First with respect to housing and service delivery, have also had success by achieving residential stability for persons with SMI.11-14
However, less information is available on mental health service use and costs associated with supported housing in general and Housing First programs in particular. Culhane et al15 found that substantial cost savings were associated with a supported housing program in New York City; however, New York allocates substantially more resources than any other state to their mental health and housing programs, thus reducing the generalizability of the initiative. In another study, Rosenheck et al13 focused on a sample of veterans, again limiting generalizability. Gilmer et al16 found very modest cost increases among clients in supported housing in San Diego County, California, but were unable to include housing costs. Recently, Larimer et al17 estimated cost savings associated with Housing First for 95 persons with severe alcohol problems; their results are not likely to be generalizable to a population with SMI.
A recent policy experiment in California provides an opportunity to address these limitations by studying a new implementation of Housing First programs. On November 2, 2004, California voters approved Proposition 63, which was signed into law as the Mental Health Services Act (MHSA). Twenty-eight percent of MHSA funding is allocated to full-service partnerships (FSPs): Housing First programs that do “whatever it takes” to improve residential stability and mental health outcomes. The MHSA provides one of the largest natural experiments in mental health policy in recent history, and this natural experiment relies heavily on Housing First.
In this study, we examine changes in housing status, receipt of disability benefits, employment, mental health service use, and costs associated with participation in San Diego County's FSPs for adults from 1 year prior to entry into the program to 1 year postentry. Our perspective for the cost analysis is of the public mental health system. We also compare self-reported quality of life among FSP clients and homeless clients receiving services in outpatient programs.
Our hypotheses are that (1) enrollment in the FSP will increase the number of days in stable housing (thus, reducing the number of days homeless); (2) enrollment in the FSP will result in increased mental health outpatient service use and costs, and decreased inpatient, emergency, and justice system services and costs; and (3) FSP clients will report higher quality of life than homeless clients receiving services in outpatient programs.
The FSP programs implemented in San Diego County provide a combination of subsidized permanent housing and team-based services with a focus on rehabilitation and recovery. Clients are recruited through a combination of referrals and outreach from psychiatric hospitals, emergency departments, other mental health programs, county agencies, Institutes of Mental Disease, jails, shelters, rescue missions, and the street. Housing is provided in the community, where the client has legal tenancy rights and responsibilities. Clients are not required to participate in treatment to retain their housing, though they do need to meet at least monthly with a treatment team. Full-fidelity assertive community treatment teams (approximately 1 team per 100 persons), including psychiatrists, nurses, mental health professionals, employment specialists, peer specialists, and substance-abuse specialists, provide medication management, vocational services, substance abuse services, and other services to help consumers sustain community living at the highest level of functioning. Clients receive services in their homes, where they work, and in other settings in the community as identified by the individual as the most beneficial to them or where support is most needed. Crisis intervention services are available 24 hours a day, 7 days a week. The goal of the program is to provide individuals with housing and intensive community-based care to assist them in reaching their goals and living a life that is not defined by mental illness.
Data from the San Diego County Adult and Older Adult Mental Health Services encounter-based Management Information System were used to identify clients admitted to an FSP program between October 1, 2006, and December 31, 2007. We used propensity scores to identify a comparison group of homeless, severely mentally ill clients with similar demographic and clinical characteristics who were initiating services at the same time as clients of FSPs.18,19
Client information on diagnosis and living situation are gathered during an assessment when the client initiates services (ie, when they are admitted into a program). We gathered initial assessment data on all adults (aged ≥18 years) initiating services between October 1, 2006, and December 21, 2007, with a diagnosis of SMI (schizophrenia, bipolar disorder, or major depression) who were homeless. For clients who initiated services at multiple programs during this period, we randomly selected 1 assessment for matching. For FSP clients, we derived diagnosis from the initial FSP assessment. Previous analyses of these data have shown that for persons with SMI, diagnoses in case management and outpatient programs are fairly consistent, in that clients who visit multiple programs tend to receive the same diagnosis.20 Information on age, sex, and race/ethnicity was obtained from a static demographic file.
We used the Becker and Ichino21 algorithm, implemented in Stata, version 10, to estimate propensity scores, determine if the resulting propensity scores were balanced on observable characteristics, and identify a matched comparison group using nearest neighbor matching. We estimated a propensity score of FSP participation based on age, sex, race/ethnicity, living situation, and clinical diagnosis using a logistic regression model. We tested for and found no significant interactions between the predictor variables. We further assessed goodness of fit using a modified Hosmer-Lemeshow test and a Pregibon link test.22,23 The Becker and Ichino algorithm determined that resulting propensity scores were balanced on observable characteristics.21 We identified a matched comparison group using nearest neighbor matching.24 Control clients were sampled with replacement, resulting in fewer control clients than FSP participants. We have previously used this approach to identify a comparison group for an analysis of San Diego County's supported housing program.16
In 2006, the California Department of Mental Health convened a performance measurement advisory committee to develop the methods to measure the impact of the MHSA at the individual, system, and community levels. The first component of this statewide system is FSP client outcomes tracking. Client outcomes are recorded by clinicians or case managers using FSP assessment forms that are designed to gather historical, baseline, and follow-up data. Historical and baseline data are gathered at intake into the FSP using the Partnership Assessment Form. We used data from the Partnership Assessment Form to identify historical and baseline data on housing, financial support, and employment among FSP clients. A key event tracking (KET) form is completed whenever there are changes in important indicators. We used data from the KET form to collect current data on housing and employment. A quarterly assessment form is completed every 3 months and was used to collect sources of financial support, including disability benefits. We conducted an additional point-in-time survey of residential setting and financial support in 1 FSP to assess the convergent validity between case managers' knowledge of these outcomes and their status as concurrently reported in the FSP assessment data.
We summarized the number of days spent in various residential settings, including independent, congregate/residential, jail, emergency shelter, unsheltered homeless, and other/unknown. We additionally calculated residential costs. We derived the cost of independent and congregate settings using administrative data provided by housing managers. We estimated that independent living, a combination of 1-bedroom apartments and single-room occupancy units in residential hotels, costs $680 per person per month on average. Congregate living (a combination of sober living and more intensively supervised assisted living) costs $640 per month. Clients residing in these settings contributed, on average, 50% of residential costs, primarily through their Supplemental Security Income benefits. Additional program-level costs, including staff who provided housing management and on-site maintenance, pest control (primarily for bedbugs), upkeep (including furniture, repairs, and replacement), and telephone and utilities, were $779 per person per year.
The perspective for the cost analysis was the public mental health system. The FSP program is currently the only source of community-based, public mental health housing in San Diego County. Therefore, the relevant housing costs from this perspective include only the cost of subsidized housing and housing management for FSP enrollees in the period after enrollments, ie, no housing cost accrued to the public mental health system for FSP clients in the period before enrollment or control clients in either the before or after periods. We calculated residential costs in the postentry period as the sum of the days spent in each setting multiplied by 50% of each setting's daily residential cost (to account for the portion of the rent paid by clients), including the additional $779 per person annual costs as described previously.
Information on service use (including date of service, the program where the service was provided, procedure codes, and service minutes) was derived from the county encounter-based management information system. A limitation of encounter-based systems is that they not include information on the cost of services (in contrast, fee-for-service claims-based systems include information on the amounts paid for services). Rather than paying in a fee-for-service plan, San Diego County contracts extensively with nonprofit specialty mental health service providers. These contracted providers submit annual cost reports that use standard accounting practices to allocate the total costs of an entire program (including staff, supplies, administration, overhead, and capital costs) to each of the major types of services provided. For example, the cost report for an outpatient program will provide per-minute cost estimates for case management, medication management, and mental health services (including assessments, rehabilitation and recovery services, and individual, group, and family therapy). Costs per minute of service vary with the intensity of the service and costs of personnel. For example, medication management (provided by psychiatrists) is more expensive on a per-minute basis than case management (which might be provided by a social worker). Costs for emergency services are allocated to minutes of crisis intervention and stabilization. Costs for inpatient services are allocated to a 24-hour day. We calculated service costs by multiplying the units of each service (from the management information system) by the cost per unit of that service in 2007 (from the cost report). We previously developed this approach to assign costs to services for a study of the organization and financing of services in San Diego County.25
For FSP clients and the propensity score–matched control group, we aggregated outpatient, inpatient, emergency, and justice system mental health use and costs of services for 1 year preinitiation and up to 1 year postinitiation, defined at either the initial FSP assessment or the propensity score–matched initial assessment. Use and cost data were available from October 1, 2005, through June 30, 2008. Thus, clients had a full-year exposure to services in the preentry period and a minimum of 6 months of exposure in the postentry period. We capped exposure at 1 year to provide a consistent comparison with the prior year. The University of California–San Diego institutional review board and the San Diego County Mental Health Services Research Committee approved the use of these data for the purpose of this study in accordance with the privacy rule of the Health Insurance Portability and Accountability Act of 1996.
Housing status, receipt of disability benefits, and employment were analyzed before and after services enrollment, with indicator variables for the postentry period. Analyses of mental health services use and costs used a quasi-experimental, difference-in-difference design.26 A difference-in-difference design essentially uses each person as his or her own control, while accounting for possible confounding time trends through the use of a control group. The propensity score matching helped to ensure the validity of the key assumption of the difference-in-difference design, comparable secular time trends between the FSP and control groups.
We used negative binomial regression models to analyze the number of days spent in various residential settings and the numbers of case management, medication management, and therapy/rehabilitation services used.27-29 We used logistic regression to analyze the probabilities of receiving disability benefits, being employed, and receiving inpatient, emergency, or justice system services.
We analyzed mental health services costs using four 2-part models, in which we separately analyze costs for each category of service. The 2-part model is commonly used to estimate health care costs when the dependent variable is nonnegative and when its distribution is noticeably skewed and leptokurtic (with a heavy right-hand tail).30 Logistic regression was used to estimate the probability of any use of a service, and a generalized linear model regression (specifically based on a gamma distribution with a log link function) was used to estimate costs that were conditional on receiving at least 1 service among FSP and non-FSP clients based on standard tests for assessing alternative generalized linear models and transformed models.31-33 We assessed goodness of fit using a modified Hosmer-Lemeshow test and the Pregibon link test.22,23
In all models, age, sex, race/ethnicity, and clinical diagnosis were included as additional control covariates. In the before-and-after models, an indicator variable identified the postentry period. In the difference-in-difference models, indicator variables were included for participation in the FSP, for the postentry period, and for the interaction between the FSP and the postentry period. We adjusted for time at risk by including an exposure offset in the second part of the model.34
Incremental effects associated with the FSP were standardized to the underlying population characteristics; these effects were calculated in both parts of the model. We computed 3 sets of estimates from these regressions: before-and-after differences for FSP clients, before-and-after differences for control clients, and the difference between these estimated differences (difference-in-difference estimate). The control group difference provides an estimate of time trends. The experimental (FSP) difference captures the FSP effect and time trends. The difference in the differences removes the time-trend effect from the difference in the experimental group. Difference-in-difference estimates are presented in the article. Preentry, postentry, and difference estimates for FSP and control clients for mental health service use and costs are available from the authors upon request. Standard errors were calculated using the nonparametric bootstrapping method, and P values were computed using the percentile method from the empirical distributions of the results from 1000 replicates.35
Quality of life was compared among a cross-section of FSP clients and homeless clients receiving services in outpatient programs. San Diego County administers a biannual survey during 2-week periods in November and May to clients who are receiving services.36 We used the most recent survey (May 2009). We identified FSP clients from surveys returned from FSP programs; we used the management information system to identify homeless clients who responded to the survey in outpatient programs. The survey includes questions on patient satisfaction and quality of life. We focused on 21 questions that query respondents on their quality of life in 8 domains: general life satisfaction, living situation, leisure activities, daily activities, family, social relations, safety, and health. For example, the question for general life satisfaction is, “How do you feel about your life in general?” The answer choices were “terrible,” “unhappy,” “mostly dissatisfied,” “mixed,” “mostly satisfied,” “pleased,” or “delighted.” Answers were assigned numerical scores (range, 1-7) and were averaged across domains. F tests were used to assess significance of differences in means between FSP and homeless clients receiving services in outpatient programs.
Propensity score matching reduced the potential control group from 1582 to 154 clients. Propensity score matching improved the covariate balance on demographics and diagnosis. There were declines in absolute standardized differences for age (from 36 to 1); proportions of white (4 to 0), African American (16 to 4), and Hispanic (24 to 4) individuals; and the percentage receiving a diagnosis of schizophrenia (16 to 2). There were no significant differences in demographics or clinical diagnosis between FSP clients and the matched control group (n = 363, Table 1), even without a correction for multiple comparisons. Overall, the mean age was 44 years (SD, 10 years); 135 (37%) were female; 222 (61%) were white, 91 (25%) were African American, 35 (10%) were Hispanic, and 15 (5%) were of another race/ethnicity; 217 (60%) had an Axis I primary diagnosis of schizophrenia, 90 (25%) had bipolar disorder, and 56 (15%) had major depressive disorder. With the exception of 1.8 greater visits for therapy/rehabilitation among control clients (1.2 vs 3.0, P = .03), there were no significant differences between FSP and control clients in service use or costs in the preentry period (P > .17 each).
Our analysis of convergent validity among 288 clients in 1 FSP found high levels of agreement when a KET form had been submitted (171 [75%] had KET forms). Case manager reports of clients' residential setting had 81% agreement with current FSP assessment data when a KET form had been submitted, compared with 16% when a KET form had not been submitted. The Cohen κ statistic37 was 0.67, representing moderate agreement.38 Receipt of disability benefits had 91% agreement and employment had 77% agreement (κ cannot be calculated because there are only 2 outcomes). We believe that the discrepancy occurs because of timing; residential status and employment are reported as key events, while receipt of disability benefits is reported quarterly. Key events may not be uploaded to the system immediately (a program will typically upload forms in batches).
Table 2 shows the mean number of days spent in various living situations in the year before and year after enrollment into the FSP among 174 FSP clients with KET forms (83% had KET forms). The most dramatic shifts occurred in the number of days spent in either independent or congregate/residential living situations, which increased 99% from 74 to 147 days, and in days spent homeless, which declined 67% from 191 to 62 days (P < .001 each). With respect to disability benefits, the percent of clients receiving Supplemental Security Income or Social Security Disability Insurance rose from 53% to 70% (P = .001); there were no significant changes in employment (data not shown).
Differences in the 1-year standardized use of outpatient mental health services are shown in Table 3. Both FSP and non-FSP visits averaged 23 minutes (data not shown). There were sizeable increases in outpatient service use among FSP clients and much smaller increases among the propensity score–matched control group. The difference-in-difference estimates show that the FSP increased case management by 35 visits per year, medication management by 24 visits per year, therapy/rehabilitation by 19 visits per year, and total visits by 78 per year (P < .001 each).
Differences in the 1-year standardized probability of using inpatient, emergency, and justice system services are shown in Table 4. The probability of using these services among FSP clients declined by 11 percentage points for inpatient, 20 for emergency, and 20 for justice system services (P < .001 each). In contrast, the probability of using emergency services increased by 12 percentage points among control clients (P = .01). The difference-in-difference estimates show that the FSP reduced the use of inpatient services by 14 percentage points (P = .01), emergency services by 32 percentage points (P < .001), and justice system services by 17 percentage points (P < .001).
Differences in the 1-year standardized costs are shown in Table 5. The difference-in-difference estimates of the effect of the FSP on annual per person costs were $9180 for outpatient (P < .001), −$6882 for inpatient (P < .001), −$1721 for emergency services (P = .002), and −$1641 for mental health services received in jail (P < .001). The difference-in-difference estimate of the effect of the FSP on total service costs was statistically insignificant. Housing costs for FSP clients were $3180 in the postentry period. We calculated total costs by adding service costs among 209 clients to the average housing costs among 174 clients (assuming zero covariance). The difference-in-difference estimate of the effect of the FSP on total costs was not significant: $2116 (P = .45).
Mean values for self-reported quality of life among 161 FSP clients and 86 homeless clients receiving services in outpatient programs are shown in the Figure (the full survey is available from the authors upon request). The FSP clients had statistically significant more favorable responses than homeless clients in all domains (P < .012 each). The largest difference occurred in the living situation domain (mean numerical responses were 5.0 vs 3.4, the difference = 0.94 of a SD), followed by safety (difference, 0.54 SD), daily activities (difference, 0.51 SD), leisure (difference, 0.50 SD), health (difference, 0.46 SD), general life satisfaction (difference, 0.42 SD), social relationships (difference, 0.39 SD), and family relationships (difference, 0.36 SD).
We examined changes in recovery outcomes, mental health service use, and costs associated with participation in San Diego County's Housing First FSPs. We found that Housing First was associated with increased time spent in permanent, stable housing, including independent living arrangements. Time spent homeless was reduced by 129 days, or 68%. Although financial support increased, there were no changes in employment in the year after enrollment into the FSP. Positive life changes such as employment may take longer than a year to be affected. Participation in Housing First was associated with substantial increases in outpatient visits and declines in use of inpatient and emergency services. Reductions in inpatient and emergency service costs partially offset the increased outpatient and housing costs. As a result, the net cost of the program, $2780 per person per year, was substantially lower than actual program costs. Self-reported quality of life was greater among FSP clients than among homeless clients receiving services in outpatient programs.
The design of the study was observational, rather than based on a randomized controlled trial. Thus, our control group may have differed from FSP clients on unmeasured characteristics such as readiness to change. However, in a previous survey of 827 clients receiving mental health services in San Diego County, we found that among 91 who were homeless, only 6 (7%) were not seeking a change in their living situation.39 Nonequivalence between the intervention and propensity score–matched control group may have affected results that are sensitive to regression to the mean, such as days homeless and service use. However, the difference-in-difference design removes both unobserved, time-invariant differences between the 2 groups as well as unobserved, time-varying systemic changes between the before and after periods. The difference-in-difference estimator is the strongest of the observational study designs and is becoming widely used to evaluate interventions and natural policy experiments for which results from randomized controlled trials are often costly and take considerable time to obtain.16,17,26 Our approach provides timely analysis of an ongoing policy experiment that can inform policymakers in California about the success and cost of the FSPs.
The perspective for this analysis was the public mental health system and included costs of mental health services and housing. Our analysis did not include federal housing support such as Shelter Care Plus, Section 8, or federal administrative costs of income support. Our measure of justice system costs was limited to mental health services provided in jail and therefore did not include changes in incarceration costs or costs of crime that may have been reduced. Our analysis also did not include use or costs of physical health services or pharmaceuticals. Participation in the FSP may have affected use of (nonpsychiatric) hospitals and emergency departments, use of physician services, and adherence to medications. Although we were able to measure certain important aspects of recovery and quality of life, we did not have measures of emotional health such as anxiety, stress, confusion, or depression.
The net cost of the FSPs is higher than previous studies of supported housing among persons with SMI. Culhane et al15 found greater reductions in inpatient expenditures, perhaps due to either the greater cost of hospitalizations in New York City or a higher initial propensity to use inpatient services. They also found cost savings related to reduced emergency shelter use. Perhaps owing to a more welcoming outdoor environment in San Diego, FSP clients had a relatively low use of shelters, which did not change from the preentry to the postentry periods. These authors also found a much smaller increase in costs for outpatient services ($3273 vs $9180 in 2007 US dollars), suggesting that the FSPs provided outpatient care with a greater intensity. Our estimate of costs was closer to that of Rosenheck et al13 ($2707 vs $2780 per person annual total cost) and Gilmer et al16 ($218 vs −$1064 per person annual net service costs excluding housing).
Full-service partnerships were a major component of the MHSA and have been controversial owing to their expense and the opportunity cost of services that might be delivered more widely (albeit less intensively). We show that by providing housing and engaging and retaining clients using team-based services, FSPs reduce homelessness, increase use of outpatient services, and improve quality of life, while offsetting 82% of their cost by reducing the use of inpatient and emergency mental health services. Full-service partnerships are similar in structure and operation to other Housing First initiatives under way across the country. This study suggests that these programs may be similarly effective and cost neutral. However, additional research is needed to determine which components of FSPs and other Housing First programs are important predictors of clients' outcomes and costs.
Correspondence: Todd P. Gilmer, PhD, Department of Family and Preventive Medicine, University of California–San Diego, 9500 Gilman Dr, La Jolla, CA 92093-0622 (email@example.com).
Submitted for Publication: June 19, 2009; final revision received November 1, 2009; accepted November 9, 2009.
Author Contributions: Dr Gilmer had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Financial Disclosure: None reported.
Funding/Support: This study received financial support from the County of San Diego Health and Human Services Agency Adult and Older Adult Mental Health Services.
Previous Presentation: This paper was previously presented at the AcademyHealth Annual Research Meeting; June 26, 2009; Chicago, Illinois.
Additional Contributions: The authors gratefully acknowledge the County of San Diego Health and Human Services Agency Adult and Older Adult Mental Health Services for access to the management information systems and the Health Services Research Center, Department of Family and Preventive Medicine, University of California–San Diego for research support.