Racial/Ethnic Disparities in the Performance of Prediction Models for Death by Suicide After Mental Health Visits

Key Points

Question  Could implementation of suicide prediction models reinforce and worsen racial/ethnic disparities in care?

Findings  In this diagnostic/prognostic study, 2 prediction models for suicide within 90 days were developed and validated in a retrospective study of 13 980 570 outpatient mental health visits. Both models accurately predicted suicide risk for visits by White, Hispanic, and Asian patients, but performance was poor for visits by Black and American Indian/Alaskan Native patients and patients without race/ethnicity reported.

Meaning  This study suggests that implementation of either suicide prediction model would disproportionately benefit White, Hispanic, and Asian patients compared with Black and American Indian/Alaskan Native patients and patients with unrecorded race/ ethnicity.

Importance  Clinical prediction models estimated with health records data may perpetuate inequities.

Objective  To evaluate racial/ethnic differences in the performance of statistical models that predict suicide.

Design, Setting, and Participants  In this diagnostic/prognostic study, performed from January 1, 2009, to September 30, 2017, with follow-up through December 31, 2017, all outpatient mental health visits to 7 large integrated health care systems by patients 13 years or older were evaluated. Prediction models were estimated using logistic regression with LASSO variable selection and random forest in a training set that contained all visits from a 50% random sample of patients (6 984 184 visits). Performance was evaluated in the remaining 6 996 386 visits, including visits from White (4 031 135 visits), Hispanic (1 664 166 visits), Black (578 508 visits), Asian (313 011 visits), and American Indian/Alaskan Native (48 025 visits) patients and patients without race/ethnicity recorded (274 702 visits). Data analysis was performed from January 1, 2019, to February 1, 2021.

Exposures  Demographic, diagnosis, prescription, and utilization variables and Patient Health Questionnaire 9 responses.

Main Outcomes and Measures  Suicide death in the 90 days after a visit.

Results  This study included 13 980 570 visits by 1 433 543 patients (64% female; mean [SD] age, 42 [18] years. A total of 768 suicide deaths were observed within 90 days after 3143 visits. Suicide rates were highest for visits by patients with no race/ethnicity recorded (n = 313 visits followed by suicide within 90 days, rate = 5.71 per 10 000 visits), followed by visits by Asian (n = 187 visits followed by suicide within 90 days, rate = 2.99 per 10 000 visits), White (n = 2134 visits followed by suicide within 90 days, rate = 2.65 per 10 000 visits), American Indian/Alaskan Native (n = 21 visits followed by suicide within 90 days, rate = 2.18 per 10 000 visits), Hispanic (n = 392 visits followed by suicide within 90 days, rate = 1.18 per 10 000 visits), and Black (n = 65 visits followed by suicide within 90 days, rate = 0.56 per 10 000 visits) patients. The area under the curve (AUC) and sensitivity of both models were high for White, Hispanic, and Asian patients and poor for Black and American Indian/Alaskan Native patients and patients without race/ethnicity recorded. For example, the AUC for the logistic regression model was 0.828 (95% CI, 0.815-0.840) for White patients compared with 0.640 (95% CI, 0.598-0.681) for patients with unrecorded race/ethnicity and 0.599 (95% CI, 0.513-0.686) for American Indian/Alaskan Native patients. Sensitivity at the 90th percentile was 62.2% (95% CI, 59.2%-65.0%) for White patients compared with 27.5% (95% CI, 21.0%-34.7%) for patients with unrecorded race/ethnicity and 10.0% (95% CI, 0%-23.0%) for Black patients. Results were similar for random forest models, with an AUC of 0.812 (95% CI, 0.800-0.826) for White patients compared with 0.676 (95% CI, 0.638-0.714) for patients with unrecorded race/ethnicity and 0.642 (95% CI, 0.579-0.710) for American Indian/Alaskan Native patients and sensitivities at the 90th percentile of 52.8% (95% CI, 50.0%-55.8%) for White patients, 29.3% (95% CI, 22.8%-36.5%) for patients with unrecorded race/ethnicity, and 6.7% (95% CI, 0%-16.7%) for Black patients.

Conclusions and Relevance  These suicide prediction models may provide fewer benefits and more potential harms to American Indian/Alaskan Native or Black patients or those with undrecorded race/ethnicity compared with White, Hispanic, and Asian patients. Improving predictive performance in disadvantaged populations should be prioritized to improve, rather than exacerbate, health disparities.

Concern is increasing that the use of prediction models in health care will exacerbate existing disparities.^1,2 Clinical prediction models are frequently estimated using health records data, which reflect inequities in health care access, quality, and outcomes based on race/ethnicity, sexual orientation, gender identity, immigration status, and other aspects of patient identity. Given existing disparities, the presumption that historical care patterns reflect clinical need can perpetuate inequities.^3,4 Obermeyer et al⁵ investigated an algorithm to identify patients with complex care needs that used health care spending as a proxy for disease severity and found that it underestimated disease severity for Black patients, resulting in fewer services offered to Black patients and exacerbating inequitable allocation of health care resources. Barriers to accessing health care also impact the availability of data for model estimation. Prediction models will be more accurate for populations with better access to care.⁶

Prediction models that perform poorly within subpopulations may expose patients inaccurately identified as high risk to unnecessary or inappropriate care. One strategy to improve a prediction model’s true-positive rate within a subpopulation is to lower the threshold for intervention.³ However, if risk discrimination is poor, this strategy will also increase false-positive results. Costs, burdens, or adverse effects of overtreatment may also vary across populations. Marginalized communities that have historically received more invasive or intrusive interventions could be disproportionately harmed.

Suicide risk prediction models have been developed and validated in several settings and are now in regular use at the Veterans Health Administration, HealthPartners, and Kaiser Permanente.^7-10 Existing suicide prediction models are accurate on the population level, yet performance within subpopulations, including racial/ethnic subgroups, remains unexamined. The current study has several motivations for examining performance across race/ethnicity.¹¹ First, suicide mortality rates vary widely. Some disadvantaged populations have markedly lower rates—suicide death rates for Black, Hispanic, Asian, and Pacific Islander populations range from 6.7 to 6.9 per 100 000 people, whereas American Indian and Alaskan Native populations have a suicide death rate of 22.1 per 100 000 people.¹² Second, racial/ethnic inequities in access to high-quality care are particularly pronounced in mental health.¹³ Consequently, mental health diagnosis and treatment patterns (key predictors of suicidal behavior) vary by race/ethnicity because of persistent disparities in care received and subsequent differences in patient preferences.^13-21 Third, suicide death is rare; the number of suicide deaths observed even in large data sets may be small for low-prevalence racial/ethnic groups, resulting in insufficient information available to identify race/ethnicity-specific risk factors and precluding estimation of separate prediction models. Fourth, potential harms of both false-negative errors (failure to intervene before suicide) and false-positive errors (delivery of unnecessary or potentially intrusive interventions) may vary across racial/ethnic groups. In this study, we evaluated racial/ethnic disparities in the performance of a risk algorithm trained on health record data to predict suicide death in the 90 days after an outpatient mental health visit.

Seven health systems (HealthPartners; Henry Ford Health System; and the Colorado, Hawaii, Northwest, Southern California, and Washington regions of Kaiser Permanente) participated in this diagnostic/prognostic study. Each system provides comprehensive mental health and general medical care to a defined population, has complete capture of utilization via electronic health record and insurance claims data, and maintains a research data warehouse that conforms to the Health Care Systems Research Network’s Virtual Data Warehouse model.²² Responsible institutional review boards for each health care system granted waivers of consent to use deidentified records data for this research. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.

Our study sample of 1 433 543 patients 13 years or older covered all 13 980 570 outpatient visits to a mental health specialist between January 1, 2009, and September 30, 2017 (or latest date cause-of-death data were available), with follow-up through December 31, 2017 (eTable 1 in the Supplement). Data analysis was performed from January 1, 2019, to February 1, 2021.

Suicide predictors were extracted from health record and insurance billing information.²² Predictors included demographic characteristics (age, sex, race, ethnicity, and insurance type), comorbidities (measured by Charlson Comorbidity Index²³), mental health and substance use diagnoses, dispensed psychiatric medications, prior suicide attempts, prior mental health encounters (including emergency department and hospitalizations), and Patient Health Questionnaire 9 (PHQ-9) responses²⁴ (eTable 2 in the Supplement). Race and ethnicity were included as separate predictors. Clinical predictors were binary, indicating the presence of each diagnosis, prescription, or encounter in 3 overlapping periods before the visit: 90 days, 1 year, and 5 years. The PHQ-9 responses were summarized for the visit day and for the 90, 183, and 365 days before the visit. Separate predictors were defined for the first 8 items of the PHQ-9, a scale of depression symptom severity, and the ninth item, which specifically asks about suicidal ideation.^24,25

Race/ethnicity categories included White, Hispanic, Black, Asian, American Indian/Alaskan Native, Native Hawaiian/Pacific Islander, multiracial or other race, and unrecorded. Patients with Hispanic ethnicity were classified as such regardless of race for primary analyses; sensitivity analyses examined racial categories including Hispanic patients. Race/ethnicity data were routinely collected by self-report at clinic visits. Thus, race/ethnicity designations do not reflect biological or genetic underpinnings but instead represent patient-identified race/ethnicity in a health care setting. Patients may not have race/ethnicity recorded if they had fewer encounters during which this information was collected, they did not identify with any categories offered, or clinic staff did not inquire; available clinical data do not indicate why race/ethnicity data may be absent. Our analysis included patients with unrecorded race/ethnicity because they represent 6.1% of patients and 9.8% of suicides in the sample.

Suicide death within 90 days after a visit was defined by an International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) cause-of-death code for self-inflicted injury (codes X60–X84) or injury or poisoning with undetermined intent (codes Y10–Y34) in state mortality records.^26,27 Additional information on the data collection methods is in the eMethods in the Supplement.

The training set for model development contained all visits for a random sample of 50% of patients stratified by race/ethnicity and suicide death; visits for the remaining 50% of patients constituted the validation set.

Prediction models were estimated using logistic regression with LASSO variable selection²⁸ and random forest.²⁹ Logistic regression models considered prespecified interactions between predictors (including prior diagnoses, suicide attempts, and PHQ-9 responses) and race and ethnicity (eTable 2 in the Supplement). Random forest is a tree-based method that explores interactions between predictors (including those with race and ethnicity) in estimating probability of an outcome. Event rates were too low to estimate separate models for each racial/ethnic subgroup.

Tuning parameter values for prediction models were selected using 10-fold cross-validation in the training data set (eTable 3 in the Supplement). Optimal values for each model corresponded to the highest area under the curve (AUC).³⁰ Prediction models were estimated in the training set using selected tuning parameters and used to obtain predictions in the validation data set.

We evaluated prediction models’ performance in the overall validation set and within subgroups defined by race/ethnicity. The AUC measured model discrimination. Sensitivity (true-positive rate) was estimated for global and race/ethnicity-specific thresholds defined at the 99th, 95th, 90th, and 75th percentiles of predictions in the training data set. Receiver operator characteristic curves were examined for racial/ethnic subgroups.

We also evaluated positive predictive value (PPV), a popular performance measure, because it incorporates true- and false-positive results, and the Brier score, which evaluates the accuracy of predictions. However, because both measures are influenced by event rate, neither was informative for comparing predictive accuracy across racial/ethnic subgroups given the variability in suicide rates. Additional details on this and other analytic decisions are in the eMethods and eTables 4 and 5 in the Supplement.

Our analysis included 13 980 570 visits by 1 433 543 patients. Most visits were made by White patients (58% of visits), followed by patients who were Hispanic (24% of visits), Black (8% of visits), Asian (4% of visits), Native Hawaiian/Pacific Islander (1% of visits), and American Indian/Alaskan Native (0.7% of visits) (Table 1; eTable 6 in the Supplement). Patients without race or Hispanic ethnicity recorded made up 4% of visits, and 0.3% of visits were made by patients recorded as multiracial or other race. Prior suicide attempts and inpatient hospitalizations with mental health diagnosis were most common among visits for American Indian/Alaskan Native patients (5% had attempts and 26% had hospitalizations) and White patients (4% had attempts and 26% had hospitalizations) and least common for patients without race/ethnicity recorded (2% had attempts and 11% had hospitalizations).

A total of 768 suicide deaths, 722 indicated as self-inflicted and 46 with undetermined intent, were observed within 90 days after 3143 visits (eTable 7 in the Supplement). The number of suicides was greatest for White patients (542 deaths), followed by Hispanic patients (80 deaths), patients without race/ethnicity recorded (75 deaths), and Asian patients (37 deaths). Fewer than 20 suicides were observed among patients who were Black, Native Hawaiian/Pacific Islander, or American Indian/Alaskan Native or were reporting multiple or other races (eTable 8 in the Supplement). Many suicides were preceded by multiple visits in the prior 90 days (median, 2; interquartile range, 1-5 visits). Suicide rates were highest for visits by patients without race/ethnicity recorded (5.71 per 10 000 visits), followed by visits from patients who were Asian (2.99 per 10 000 visits), White (2.65 per 10 000 visits), multiracial or other race (2.28 per 10 000 visits), American Indian/Alaskan Native (2.18 per 10 000 visits), Native Hawaiian/Pacific Islander (1.63 per 10 000 visits), Hispanic (1.18 per 10 000 visits), and Black (0.56 per 10 000 visits).

Suicide predictions had high discrimination in the validation set for logistic regression and random forest models and variable performance across racial/ethnic subgroups (Table 2). The AUCs were highest for visits from White, Hispanic, and Asian patients and lowest for visits by Black and American Indian/Alaskan Native patients and those with unrecorded race/ethnicity. For example, the AUC of the logistic regression model was 0.828 (95% CI, 0.815-0.840) for White patients compared with 0.640 (95% CI, 0.598-0.681) for visits from patients without race/ethnicity data recorded.

Table 3 compares the sensitivity of global and race/ethnicity-specific thresholds defined at the 99th, 95th, 90th, and 75th percentiles of risk predictions. Sensitivities of global thresholds vary widely across racial/ethnic subgroups. For example, the 95th percentile threshold identifies nearly half of visits preceding suicide for White patients, whereas for American Indian/Alaskan Native and Black patients, it is no better than chance (a random 5% sample of visits). Sensitivity at global thresholds was improved in the random forest model for Asian patients but decreased for White patients and remained similar for other subgroups.

Defining race/ethnicity-specific thresholds improved sensitivity inconsistently for subgroups with poor sensitivity using global thresholds. For example, defining risk thresholds separately for visits by Black patients increased sensitivity of the random forest model at all percentiles examined (Table 3). For visits by American Indian/Alaskan Native patients and patients without race/ethnicity data recorded, however, sensitivity was lower or unchanged.

The Figure shows receiver operating characteristic curves for suicide risk predictions in each racial/ethnic group. For both models, if subgroup-specific thresholds were selected to equalize sensitivity across race/ethnicity, the false-positive rate would vary widely. For example, at 40% sensitivity, the false-positive rate from the logistic regression model is 4% for White patients, 9% for Black patients, and 28% for patients without race/ethnicity data recorded (eTable 11 in the Supplement).

Sensitivity analyses that evaluated prediction model performance in racial/ethnic groups including visits by Hispanic patients found similar results to primary analyses (eTables 12, 13, and 14 in the Supplement).

This diagnostic/prognostic study examined the performance of 2 prediction models for suicide death within 90 days after an outpatient mental health visit. Although both models performed well at the population level, prediction was poor within some racial/ethnic subgroups. Clinical implementation of either model to target suicide prevention would provide less potential benefit to patients who are Black, are American Indian/Alaskan Native, or have no race/ethnicity data recorded compared with White, Hispanic, and Asian patients because a smaller proportion of eventual suicides would be identified.

One strategy to increase sensitivity within racial/ethnic subgroups is to define separate thresholds for intervention based on the distribution of predictions within each group.³ In the current analysis, because risk discrimination was poor for visits by Black and American Indian/Alaskan Native patients and patients without race/ethnicity data recorded, improved sensitivity was accompanied by higher false-positive rates. To intervene on the same proportion of eventual suicides across racial/ethnic subgroups, the proportion of visits receiving intervention must be considerably higher for Black patients, American Indian/Alaskan Native patients, and patients without race/ethnicity data recorded, exposing low-risk individuals in these populations to unnecessary and possibly intrusive interventions.

Relative benefits and harms of suicide prevention interventions vary by race/ethnicity. Additional attention from a mental health care professional may increase access to beneficial services and likely presents limited harm but could cause stigmatization or discrimination and damage patient-practitioner therapeutic alliances, particularly for patients from marginalized communities already less likely to trust or engage with traditional mental health care.³¹ Identifying high-risk patients could also initiate a cascade of interventions, including involuntary psychiatric hospitalization and contact with law enforcement, which pose a greater risk of harm to populations collectively referred to as BIPOC (Black, Indigenous, and people of color).^32-34

Suicide prevention interventions should be studied to understand their benefits and harms, and these benefits and harms may differ by race/ethnicity. Preferences about interventions should be elicited from patients, and stakeholders from affected communities should be engaged in determining if, and how, to implement a prediction model. Statistical decisions about what threshold to use for recommending intervention and which performance measure is most important to examine are intricately linked with decisions about what types of intervention a prediction model should guide. For example, when the PPV of a prediction model is low in a particular subgroup, it is important that the accompanying intervention pose minimal potential harm to those with false-positive results.

Health records data may poorly predict suicide death in some racial/ethnic groups for several reasons. First, systemic barriers to accessing affordable, culturally competent mental health care reduce utilization,¹³ resulting in sparser records data for BIPOC populations. In this analysis, prior mental health diagnoses and encounters were lowest among patients without race/ethnicity data recorded—missing information was likely related to low utilization and fewer encounters during which clinic staff asked about race/ethnicity—and predictive accuracy was particularly low in this subgroup.

Second, BIPOC populations are less likely to be diagnosed and treated for mental health conditions because of practitioner bias^19,35 and institutionalized discrimination.^13,20,36 For example, BIPOC populations are less likely to be screened^37,38 or receive high-quality treatment¹³ for depression and more likely to experience discrimination in health care settings.¹⁵ Lacking access to culturally responsive and appropriate treatment, BIPOC populations may be more likely to prefer nonmedical treatment for mental health conditions.^14,15

Third, outcome misclassification could diminish performance of suicide prediction models in racial/ethnic subgroups.^39,40 Suicide deaths may be incorrectly identified as unintentional, accidental, or homicide and vice versa. Suicide misclassification rates may vary by race/ethnicity,⁴¹ with greater risk of misclassification in groups with higher rates of violent deaths now classified as accidents or assaults.

Fourth, the association of mental health conditions with suicidal behavior may vary by race/ethnicity because of cultural differences that influence suicide risk and protective factors.^42,43 Risk modification is typically accommodated by stratifying model estimation or by including many interactions. However, because suicide death is a rare event, statistical power to identify race/ethnicity-specific risk factors is often low. Even in this sample of more than 13 million mental health visits, including more than 1 million visits by Black patients and almost 100 000 visits by American Indian/Alaskan Native patients, there were not enough events to estimate separate prediction models for each racial/ethnic subgroup. This limited statistical power is exemplified by the finding that a random forest prediction model, which uses a data-driven estimation procedure to explore interactions, did not outperform a logistic regression model with prespecified interactions. The suicide prediction models estimated in this study reflect the association between predictors and events for larger racial/ethnic groups (ie, White non-Hispanic patients). Performance for smaller populations (ie, Black and American Indian/Alaskan Native patients) suffered, likely because the association between predictors and the outcome diverged from population-level trends in these subgroups, and information was insufficient to accurately estimate race/ethnicity-specific associations.

Currently available methods maximize predictive performance over the entire sample and do not consider performance within subgroups. Poor performance within smaller subgroups, for example, visits for American Indian/Alaskan Native patients, has little influence on overall performance because this group represents a small proportion of the entire sample. New methods that account for performance in subgroups are needed to reduce racial/ethnic inequities.

This study has limitations. Specifically, it examined suicide prediction for people engaged in outpatient mental health care in integrated health systems. This population is not representative of all people with mental health care needs and the settings where they receive care. BIPOC populations are less likely to benefit from prediction models deployed in this context because of barriers to mental health treatment, including low rates of insurance, patient mistrust after historical mistreatment of BIPOC populations, and a lack of culturally competent care.^{14,15,36,44,45} Addressing inequitable effects of clinical prediction models will require using data from a variety of settings and populations. To enable this, research networks with existing health records infrastructure and resources can partner with health care systems that serve more racially/ethnically diverse, lower-income, and uninsured or underinsured populations. Incorporating data sources beyond health records, including information on other social determinants of health⁴⁶ and traumatic or negative life events, including racial trauma,⁴⁷ could also increase suicide risk discrimination.^48,49

In addition, too few suicides were observed for multiracial patients to precisely estimate model performance. Multiracial identity was uncommon in our sample, suggesting that many multiracial people were included in monoracial categories or declined to self-report racial identity (eTables 8-10 in the Supplement).^50,51 Predictive performance for multiracial people is important because they report higher rates of mental illness than other racial/ethnic groups.⁵²

Increasing use of clinical prediction models has been met with justified concern about disparate performance and impact of algorithms trained on clinical data. This study found racial/ethnic differences in performance of 2 suicide prediction models that accurately identify high-risk patients in the overall population. These findings suggest that clinical implementation of either prediction model to guide suicide prevention interventions would pose less potential benefit and more harm to patients who are Black and American Indian/Alaskan Native or whose race/ethnicity is unrecorded compared with White, Hispanic, and Asian patients. Systemic barriers faced by Black and Indigenous people to accessing affordable, effective, and culturally responsive mental health care contribute to differences in model performance. Clinical use of the prediction models examined here would exacerbate these disparities.

Prediction models intended for clinical use should be assessed for performance in the patient populations in whom they will be used, including vulnerable groups in that population. Evaluating performance with precision may be challenging because of small sample size or low event rate, but local validation should be attempted regardless because performance differences may be detectable even in small samples. Health care system stakeholders, including practitioners, patients, and representatives from diverse communities, must carefully consider disparities in both benefits and harms posed when deciding whether and how to implement a prediction model. Researchers and health care systems are obligated to bridge the gap when potential inequitable impacts are uncovered, through either improving prediction model performance or ensuring effective, minimally harmful interventions for underserved populations. Otherwise, benefits of clinical prediction models will not be universally realized, and systemic modes of deprivation will be reinforced.

Finally, potential benefits and harms of using a prediction model within particular populations must be considered in the context of existing health inequities. BIPOC populations already face significant barriers to accessing mental health care and, as a result, have poorer outcomes. In this context, deploying a prediction model that provides less benefit to already underserved populations will widen this care gap. Health care systems should prioritize resources to reduce race/ethnicity-based inequities in care.

Accepted for Publication: February 22, 2021.

Published Online: April 28, 2021. doi:10.1001/jamapsychiatry.2021.0493

Corresponding Author: R. Yates Coley, PhD, Kaiser Permanente Washington Health Research Institute, 1730 Minor Ave, Ste 1600, Seattle, WA 98101 (rebecca.y.coley@kp.org).

Author Contributions: Dr Coley had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Coley, Simon, Cruz, Shortreed.

Acquisition, analysis, or interpretation of data: Coley, Johnson, Simon, Shortreed.

Drafting of the manuscript: Coley, Shortreed.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Coley, Johnson, Cruz, Shortreed.

Obtained funding: Coley.

Administrative, technical, or material support: Shortreed.

Supervision: Coley, Shortreed.

Conflict of Interest Disclosures: Dr Shortreed reported having received grants from Syneos Health, which represented a consortium of pharmaceutical companies carrying out US Food and Drug Administration–mandated studies regarding the safety of extended-release opioids outside the submitted work. No other disclosures were reported.

Funding/Support: Research reported in this work was supported by the Mental Health Research Network with funding from grants U19 MH092201 and U19 MH121738 from the National Institute of Mental Health. Dr Coley is also supported by grant K12HS026369 from the Agency for Healthcare Research and Quality.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health or the Agency for Healthcare Research and Quality.

Additional Contributions: Theresa Nguyen, LCSW, Chief Program Officer at Mental Health America, provided thoughtful feedback on the draft manuscript. Compensation was provided to Mental Health America for her time. She was not personally or directly compensated.