Mean proportion of participants employed competitively per month of study participation by treatment group. Groups are described in the “Interventions” subsection of the “Methods” section.
Mean proportion of participants employed 40 hours or more per month of study participation by treatment group. Groups are described in the “Interventions” subsection of the “Methods” section.
Mean number of dollars earned per month of study participation by treatment group. Groups are described in the “Interventions” subsection of the “Methods” section.
Cook JA, Leff HS, Blyler CR, Gold PB, Goldberg RW, Mueser KT, Toprac MG, McFarlane WR, Shafer MS, Blankertz LE, Dudek K, Razzano LA, Grey DD, Burke-Miller J. Results of a Multisite Randomized Trial of Supported Employment Interventions for Individuals With Severe Mental Illness. Arch Gen Psychiatry. 2005;62(5):505-512. doi:10.1001/archpsyc.62.5.505
National probability surveys indicate that most individuals with schizophrenia and other severe mental illnesses are not employed. This multisite study tested the effectiveness of supported employment (SE) models combining clinical and vocational rehabilitation services to establish competitive employment.
We randomly assigned 1273 outpatients with severe mental illness from 7 states in the United States to an experimental SE program or to a comparison or a services-as-usual condition, with follow-up for 24 months. Participants were interviewed semiannually, paid employment was tracked weekly, and vocational and clinical services were measured monthly. Mixed-effects random regression analysis was used to predict the likelihood of competitive employment, working 40 or more hours in a given month, and monthly earnings.
Cumulative results during 24 months show that experimental group participants (359/648 [55%]) were more likely than those in the comparison programs (210/625 [34%]) to achieve competitive employment (χ2 = 61.17; P<.001). Similarly, patients in experimental group programs (330/648 [51%]) were more likely than those in comparison programs (245/625 [39%]) to work 40 or more hours in a given month (χ2 = 17.66; P<.001). Finally, participants in experimental group programs had significantly higher monthly earnings than those in the comparison programs (mean, $122/mo [n=639] vs $99/mo [n=622]); t1259 = −2.04; P<.05). In the multivariate longitudinal analysis, experimental condition subjects were more likely than comparison group subjects to be competitively employed, work 40 or more hours in a given month, and have higher earnings, despite controlling for demographic, clinical, work history, disability beneficiary status, and study site confounders. Moreover, the advantage of experimental over comparison group participants increased during the 24-month study period.
The SE models tailored by integrating clinical and vocational services were more effective than services as usual or unenhanced services.
In the past several decades, research from a variety of fields has presented powerful evidence of the importance of employment to people with schizophrenia and other severe mental illnesses.1- 4 People with severe and persistent mental illnesses not only want to work2 but can successfully participate in the labor market in a variety of competitive jobs.3,4 Researchers also have explored the benefits of work to individuals with mental illness in terms of alleviation of poverty,5 therapeutic gain,3,6,7 skill acquisition,8 and improvement in quality of life.3 Society also benefits through reductions in the use of disability entitlements5,9 and the overall costs of care.10
At the same time, social and scientific developments have provided greater opportunities for people with psychiatric disabilities to enter and remain in the labor force.11 Strong patient and family advocacy movements have asserted the right to equal employment opportunities for people with mental illnesses.12 Federal legislation has mandated fair hiring and reasonable accommodation practices for people with disabilities,13 although protections have been curtailed by recent Supreme Court decisions.14 Advancements in mental health services, including the formulation and use of new psychopharmacological agents, have provided patients with more treatment options.15 Despite these developments, most people with psychiatric disabilities in the United States remain outside the labor force.16
Large-scale and nationally representative probability surveys such as the Health Care for Communities Study and the National Health Interview Survey Disability Supplement indicate that the unemployment rate among people with psychiatric disorders is 3 to 5 times higher than among those with no disorders.17,18 In these studies, proportions of people with severe mental illness who are out of the labor force (defined as not working and not looking for work) are substantially higher than these proportions among people with no mental disorder. For example, the 1994-1995 National Health Interview Survey Disability Supplement found that 61% of working age adults with mental health disabilities are out of the labor force, compared with only 20% of working-age adults in the general population.19
The early 1980s witnessed the development of a vocational rehabilitation model called supported employment (SE),20 used primarily for individuals with profound mental retardation and severe physical disabilities. This new model differed from previous approaches by emphasizing rapid job placement in socially integrated work settings followed by training and ongoing support with no time limits. In the intervening decades, SE has been refined for use with patients with severe mental illness. Reviews and meta-analyses of randomized controlled trials of these refined SE approaches have established them as an evidence-based practice in mental health treatment.21- 23 However, SE services for adults with psychiatric disabilities can be organized under a wide variety of service delivery models and can incorporate elements of a number of different program and agency types, calling for rigorous studies of implementation in this field.
This study tested the following 3 hypotheses: (1) experimental group participants receiving an SE intervention would show a greater likelihood than comparison group subjects of achieving competitive employment, 40 or more hours of work per month, and higher monthly earnings; (2) the difference between experimental and control group outcomes would increase over time; and (3) the experimental group would achieve superior outcomes despite the effects of participant characteristics including demographic features, clinical indicators, prior work history, comorbid physical/developmental disabilities, and receipt of disability income support.
The Employment Intervention Demonstration Program (EIDP) included 8 study sites located in Maryland, Connecticut, South Carolina, Pennsylvania, Arizona, Massachusetts, Maine, and Texas. Via the Cooperative Agreement funding mechanism, researchers, federal personnel, and patient representatives developed and implemented a common protocol and uniform data collection methods.24 This effort was led by a coordinating center (CC) based at the Department of Psychiatry, University of Illinois at Chicago, in partnership with the Human Services Research Institute in Cambridge, Mass. The original EIDP funding announcement (Request for Applications SM 94-09, Catalog of Federal Domestic Assistance 93.125) specified that the program’s governance body would consist of a steering committee composed of the principal investigator of the CC, the individual site principal investigators, federal staff, and consumer representatives. Decisions were made collaboratively, through consensus if possible and by a democratic vote when opinions were not unanimous.
The sample includes subjects with severe and persistent mental illness receiving outpatient psychiatric services. All subjects met criteria for severe and persistent mental illness based on diagnosis, duration, and level of disability as established by the federal Center for Mental Health Services25 along with the following inclusion criteria: 18 years or older at the time of study enrollment; willing and able to provide informed consent; and unemployed at the time of entry into the study. All sites recruited subjects from existing clinical populations via clinician referral, self-referral, and word of mouth. The Massachusetts site also used newspaper advertisements. Sites adhered to human subjects protection and confidentiality safeguards determined by their organizations’ institutional review boards. Across all sites, the eligible pool of subjects was estimated at 10 653; of this initial group, 2883 were approached for participation (these numbers exclude the Massachusetts site, which was unable to provide this information). Across all sites (including Massachusetts), a total of 1750 individuals consented to participate, 1655 completed baseline interviews, and 1648 were randomized. Reasons for consent but study nonparticipation included ineligibility, the patient’s decision to withdraw, and the researcher’s inability to locate the patient.
Of the 1648 participants, 1273 were included in this analysis. We excluded data from the Pennsylvania site (n = 182), which tested an intervention for already employed patients. As a result, Pennsylvania subjects did not meet the study inclusion criterion of unemployment, and the distribution of their outcome data was inappropriate for pooling with that of the remaining study sites. The remaining 193 excluded participants consisted of 65 at the Connecticut site, which used a second comparison condition (the only site that included 2 control conditions); 100 with no vocational outcome data; and 28 who were employed at the first study interview (ie, although all subjects reported no employment at the time of study recruitment and informed consent, some were later determined to have been employed at the time of the baseline interview). No EIDP participant was excluded from the analysis for any other reason, given its intent-to-treat design. Subjects were interviewed from February 22, 1996, through May 19, 2000, and all were monetarily compensated, with amounts determined by each site’s local economy (range, $10-$20 per interview).
All experimental conditions provided SE interventions consisting of (1) integrated services delivered by a multidisciplinary team that met 3 or more times per week to plan and coordinate employment interventions with case management and psychiatric treatment; (2) placement into competitive employment, defined as jobs paying at least minimum wage, in regular, socially integrated community settings, not reserved for individuals with disabilities, and held by patients rather than provider agencies; (3) development of jobs tailored to patients’ career preferences; (4) use of a job search process beginning immediately after program entry and moving as quickly as a patient desired; and (5) provision of ongoing vocational supports freely available throughout the entire study period rather than gradual withdrawal of support following successful employment.
Several of the study sites implemented preexisting SE models tailored for psychiatric populations such as Individual Placement and Support26 and the Program of Assertive Community Treatment.27 The Maryland and Connecticut sites tested the Individual Placement and Support model, which emphasizes team-based delivery of integrated vocational and mental health services that are responsive to recipient preferences and offer rapid job search for competitive employment followed by ongoing supports.28,29 The Massachusetts site tested the Program of Assertive Community Treatment vocational model, which offers a team-based approach integrating vocational and mental health services, delivered in clients’ natural environments using a shared caseload, with a team vocational specialist assessing, placing, training, and supporting service recipients in community-based jobs.30 The South Carolina site blended 2 established models, that is, the clinical approach of the Program of Assertive Community Treatment model and the employment services approach of the Individual Placement and Support model.
Other experimental conditions consisted of SE interventions defined by the 5 features described above along with enhancements developed specifically for this study. The Texas site added services that were designed to enrich clients’ natural support networks by strengthening existing social relationships, developing new and diverse relationships, and improving reciprocity in social networks. The Maine site combined SE with psychoeducational multifamily groups and a Mental Health Employer Consortium. This consortium formed partnerships among businesses and mental health and vocational service providers to share knowledge and resources and develop and support a diverse workforce including people with mental illnesses.31 The Arizona site delivered comprehensive case management and SE services through integrated service teams of psychiatrists, case managers, rehabilitation counselors, employment specialists, job developers, and benefits specialists, all located within a single administrative organization. Further information about the models tested at each site is available at the EIDP Web site (http://www.psych.uic.edu/eidp).
Under the cooperative agreement mechanism,24 sites were given leeway in choosing their experimental models and comparison conditions. Owing to ethical considerations, none of the sites used a no-treatment control condition. Because subjects entered the study with an interest in working, comparison condition subjects were not prevented from receiving vocational services outside of the study. Thus, as with many multisite studies, the comparison conditions varied somewhat from site to site. In 4 sites (Arizona, Connecticut, South Carolina, and Maryland), the comparison condition consisted of services as usual, defined by standard vocational rehabilitation services available in the local area. In 2 sites (Maine and Texas), a weaker version of the experimental condition served as the comparison (ie, no social network enhancements in Texas and no employer consortium in Maine). The Massachusetts site compared 2 preexisting vocational models, each hypothesized to be superior to the other for different vocational outcomes. For the outcomes analyzed herein, the Massachusetts site’s comparison condition was the International Center for Clubhouse Development program, which offers facility-based services through membership in a mutually supportive community that operates according to a work-ordered day, with clients and staff working together in Clubhouse-owned jobs (transitional employment placements) or independent competitive jobs.32,33 Within the nomenclature of randomized designs, this study is an “implementation effectiveness trial,”34 a randomized experiment designed to test the effectiveness of an intervention with established efficacy (SE) via implementation under real-world conditions of availability and patient acceptance. Such trials are not unusual in research on mental health care,35,36 given that randomized designs implemented at multiple sites in real-world environments, along with subjects’ ability to seek similar services outside study parameters, introduce confounds that can only be controlled statistically.
The multisite common protocol was a structured set of assessments administered at intake and every 6 months for the subsequent 24-month period. This included the Positive and Negative Syndrome Scale (PANSS),37 which was used to measure the severity of psychiatric symptoms across the following 3 categories: (1) positive symptoms or productive symptoms, such as hallucinations and delusions; (2) negative symptoms or deficit features, such as blunted affect and emotional withdrawal; and (3) general symptoms common across categories of psychopathology, such as disorientation, unusual thought content, and depression. At study enrollment, the Structured Clinical Interview for DSM-IV38 was administered at 2 sites, whereas case record diagnoses were extracted from clinical files at the remaining sites. Other data collected biannually included public disability income beneficiary status, self-rated level of functioning, marital status, and educational attainment. Labor force participation data regarding the nature of all paid work were collected weekly and included earnings, number of hours worked, job duties, eligibility for health care benefits and sick leave, and level of workplace integration. Sites also collected data regarding the types and amounts (in hours) of all vocational and clinical services received on a monthly basis. Further information about the EIDP common protocol and data collection methods is available on the EIDP Web site.39
Three vocational outcomes were selected for analysis, and each was considered to be a fundamentally different conceptualization of employment. The first, competitive employment, is defined as a job that pays minimum wage or higher; is located in a mainstream, socially integrated setting; is not set aside for people with disabilities; and is held independently (ie, not controlled by a service agency). The second variable, working for 40 or more hours in a single month, is an outcome used by the Centers for Medicare and Medicaid Services in their demonstration program, Demonstration to Maintain Independence and Employment, issued June 7, 2000 (Catalog of Federal Domestic Assistance 93.779). This outcome evaluates the intensity of employment in terms of a minimum number of hours worked during a 1-month period. The third outcome, monthly earnings from paid employment, is a standard labor force measure that captures the economic return to the individual from his or her labor force participation.
Given its critical importance in a multisite study, the CC conducted routine checks on data quality throughout the study. These included programmed logic checks of data at the time of submission to the CC to identify outlier or out-of-range values, conflicting subject reports within interviews, and disparities between patient self-reports and provider reports. At biannual meetings of the EIDP Steering Committee, the CC presented detailed reports on more than 150 data elements (excluding outcome variables to which sites remained blinded throughout the study), thus allowing the group to monitor and correct potential data problems as the study progressed. In addition, teleconference recalibration trainings for all EIDP PANSS interviewers were conducted from November 1, 1996, through January 31, 2000. These included monthly distribution of videotaped interviews that were rated by every interviewer and then submitted to the CC for computation of within- and cross-site interrater reliability measures. In addition, gold standard ratings of each tape were performed by a PANSS expert (PANSS coauthor Lewis A. Opler, MD), and these were distributed to all sites and discussed on mandatory conference calls led by the expert. An independent evaluation of the psychometric properties of the EIDP common protocol, commissioned by the CC, documented high levels of interrater and test-retest reliability.40
Outcome data were analyzed in multiple stages. First, the cumulative effect of the study condition on each of the dependent variables was tested in unadjusted bivariate analyses. Second, the longitudinal relationships between study condition and observed outcomes on each dependent variable were graphed across all 24 months of study participation. Third, a multivariate, longitudinal random-effects logistic regression analysis was conducted to test for differences between experimental and comparison conditions over time. A 2-level random intercepts model, controlling for study site as a fixed effect along with other covariates, was fit to the data. This approach was chosen because of the superiority of random regression models41 in addressing issues commonly found in longitudinal multisite data, including state dependency or serial correlations among repeated observations within individual participants; individual heterogeneity or varying propensities toward the outcomes of interest due to subjects’ predispositions and other unobserved influences; missing observations due to the fact that not all subjects completed all assessments; and inclusion of fixed covariates (sex, age, race/ethnicity, study site, prior work history, marital status, presence of coresident children younger than 18 years, level of education, baseline diagnosis, and baseline health comorbidity), along with time-varying covariates (positive, negative, and general symptoms, level of functioning, and disability income beneficiary status).
Demographics, employment histories, clinical status, and public disability income beneficiary status of study subjects are shown in Table 1. The success of randomization was confirmed by the absence of statistically significant differences at study baseline between experimental and comparison group subjects on education, work history, race/ethnicity, sex, marital status, coresident children, diagnosis of schizophrenia spectrum disorder, level of functioning, health comorbidity, disability income beneficiary status, age, and severity of positive, negative, and general symptoms.
Subjects were offered but were not required to accept services in this implementation effectiveness trial. Moreover, because the experimental condition consisted of different SE models, no studywide measure of service fidelity was available. Thus, as a manipulation check of program fidelity, we compared vocational and clinical services data for subjects in the 2 study conditions. Although 96% of the experimental group received vocational services (including vocational assessment, treatment planning and counseling, job placement, employment support, and vocational skills training), only 67% of those in the control conditions received these services. Conversely, clinical services (including psychiatric evaluation and diagnosis, medication evaluation and management, individual counseling, family or couples counseling, group counseling, case management, psychosocial rehabilitation/partial hospitalization, and emergency services) were received by identical proportions (99%) of experimental and comparison subjects (Table 2). Similarly, experimental group subjects received an average of 74 hours (median, 43 hours) of vocational services per person, whereas the comparison group received an average of 43 hours (median, 5 hours) of vocational services. The experimental and comparison groups received almost identical average amounts of clinical services, ie, 125 hours (median, 69 hours) and 124 hours (median, 47 hours), respectively.
There were no statistically significant differences in follow-up rates between the experimental and control conditions. Interviews were completed by 578 (89%) of the experimental condition and 546 (87%) of comparison condition subjects at the 6-month follow-up; by 547 (84%) of the experimental condition and 504 (81%) of comparison condition subjects at the 12-month follow-up; by 521 (80%) of the experimental condition and 481 (77%) of comparison condition subjects at the 18-month follow-up; and by 515 (79%) of the experimental condition and 478 (76%) of comparison condition subjects at the 24-month follow-up. Of 1273 participants, 824 (65%) completed 5 interviews, 173 (14%) completed 4, 122 (10%) completed 3, 111 (9%) completed 2, and the remaining 43 (3%) completed 1. Those completing 5 interviews were compared with all others regarding study condition and model covariates. The only significant differences were in sex and age; ie, 420 (51%) of completers were men compared with 258 (57%) of noncompleters (χ21 = 4.92; P = .03); and completers were an average of 1 year older than noncompleters (39 vs 38 years; t1270 = −2.24; P = .03).
Viewed longitudinally, higher proportions of experimental group subjects secured competitive employment and worked 40 or more hours in a given month during most of the 24-month study period; they also had significantly higher earnings (Figure 1, Figure 2, and Figure 3, respectively). Viewed cumulatively, experimental group participants (359/648 [55%]) were more likely than those in the comparison programs (210/625 [34%]) to achieve competitive employment (χ2 = 61.17; P<.001); experimental group subjects (330/648 [51%]) were more likely than those in the comparison group (245/625 [39%]) to work 40 or more hours in a given month (χ2 = 17.66 P<.001); and experimental group participants had significantly higher monthly earnings than the comparison groups (mean, $122/mo vs $99/mo); t1259 = −2.04; P = .04).
Multivariate analyses of longitudinal patterns of all 3 employment outcomes (Table 3) showed significant main effects for treatment group, with experimental group participants achieving significantly better outcomes. In this analysis, monthly earnings were grouped into 3 ordinal categories because of their nonnormal distribution: zero earnings; $1 to $199; and $200 or greater per month. The effects of time were similar for the experimental and comparison groups across all 3 outcomes, that is, the probability of competitive employment, working 40 or more hours per month, and higher monthly earnings increased significantly over time. In addition, the effects of time were nonlinear (as shown by the significant estimate for time2) for all 3 outcomes, with the probability of competitive employment, working 40 or more hours per month, and higher earnings increasing dramatically during the first 6 to 9 months of program participation and then leveling off for the remainder of the study. Thus, after adjusting for patient characteristics and study site, all 3 employment outcomes were more likely among the experimental group than comparison group subjects, and this difference increased over time.
A final analysis issue concerned the definition of competitive employment used in this study. Under this definition, transitional employment placements, such as those provided through the International Center for Clubhouse Development program, were not coded as competitive employment. Multisite analysis (not shown) using an alternative definition of competitive employment that included transitional employment placements did not substantially alter the results reported herein. The final report for the Massachusetts site of the EIDP43 demonstrates further details of the site-specific analysis using this alternative definition.
A major finding of this study was that, compared with services as usual or unenhanced comparison programs, the experimental programs had a significant and positive effect on each of the 3 employment outcomes, even when controlling for the effects of time, time × condition interaction, participant characteristics, and study site. In addition, the advantage of the experimental group participants increased over time relative to the comparison group. A number of programs were tested in the experimental condition and, as a group, they demonstrated consistently better results than their comparisons. Moreover, the experimental program models shared common characteristics, including a focus on integration of clinical and employment services delivered by providers working on the same treatment team, availability of ongoing job support vs gradual withdraw of services after successful employment, development of jobs consonant with patients’ career preferences, and an emphasis on rapid job placement into permanent, community-based, socially integrated, competitive employment.
The first major caveat to our results concerns the fact that the study’s subjects were not drawn from a national probability sample of individuals with severe and persistent mental illness, which limits the generalizability of the findings. Instead, the subjects consist of 7 cohorts of individuals seeking vocational rehabilitation services at 6 urban sites and 1 rural site in 7 states, 5 of which were on the East Coast, 1 in the South, and 1 in the southwest region of the United States. A second major caveat concerns the nature of the study as an implementation effectiveness trial. Because the experimental conditions tested were not identical, it was impossible to conduct a studywide assessment of program fidelity. Manipulation checks included an analysis of vocational and clinical services delivered to subjects in the 2 conditions, and fidelity assessments were completed independently by investigators regarding degree of implementation of the 5 essential components of SE, with results of both checks indicating high levels of adherence to the SE model. However, uniformity in experimental models would have allowed us to monitor fidelity more closely and to make more definitive statements about the quality of program implementation. Third, the comparison conditions did not represent a no-treatment control condition; thus, some of the subjects in these conditions received substantial amounts of vocational services. As such, study findings must be viewed with caution, since it is possible that the cross-site study underestimated the effects of SE relative to conditions in which far fewer employment services and supports were available, as is typical for most psychiatric outpatients attempting to return to work. On the other hand, it is also possible that, because of the large sample size, some differences achieved statistical significance no matter how inconsequential they were clinically or vocationally. Finally, it must also be acknowledged that a longer period of data collection (which was achieved by some sites that continued to collect data for 36 months or longer) might have revealed different findings than those attained at the end of the 24 months tracked in this study. All of these limitations suggest that caution should be applied to interpretations from study results. However, to date and to our knowledge, this is the most comprehensive, rigorously monitored, and thoroughly analyzed set of data collected using a uniform protocol that exhibits noteworthy validity and reliability across the largest number of sites ever examined in a study of SE interventions.
Study results build on prior evidence concerning best practices in vocational rehabilitation, but go further in demonstrating the effectiveness of SE on a larger scale, with more racially and ethnically diverse populations, at multiple geographic sites, and using different SE models. Given the diversity of sites and models tested, the results suggest that these programs have the potential to work anywhere, for a wide variety of service consumers. At the same time, it must be acknowledged that most of the study participants did not achieve success on each of the outcome measures, suggesting a need for continued enhancement of these models.
The findings of this study confirm the superiority of SE tailored for individuals with psychiatric disability over other approaches such as standard vocational rehabilitation services as usual. Given research cited earlier concerning the benefits of employment for individuals with psychiatric disabilities, this knowledge can be used to create the next generation of models that contribute to recovery and increased community integration of those with severe psychiatric disorders.
Correspondence: Judith Cook, PhD, Department of Psychiatry, University of Illinois at Chicago, 104 S Michigan Ave, Suite 900, Chicago, IL 60603 (firstname.lastname@example.org).
Submitted for Publication: May 13, 2004; final revision received August 5, 2004; accepted October 7, 2004.
Funding/Support: This study is part of the Employment Intervention Demonstration Program (EIDP), a multisite collaboration among 8 research demonstration sites, a Coordinating Center, and the Center for Mental Health Services/Substance Abuse and Mental Health Services Administration (CMHS/SAMHSA), Rockville, Md. This research was supported by cooperative agreement SM51820 from CMHS/SAMHSA.
Disclaimer: The views expressed herein are those of the authors and do not necessarily reflect the policy or position of any federal agency.