A, Presented as 3-year moving averages. B, Presented as 3-year moving averages. Persons who prior to dying by suicide had been diagnosed with different neurological disorders were included in the plot of each of these disorders. C, Adjusted for period, sex, age group, living status, region, socioeconomic status, physical comorbidity, psychiatric hospitalization prior to diagnosis of any neurological disorders, and deliberate self-harm prior to diagnosis of any neurological disorders. Error bars indicate 95%CIs. See eTable 4 in the Supplement for complete data estimates. The reference group represent persons not diagnosed with neurological disorders.
aAalen-Johansen estimates were calculated for the outcomes of suicide and other causes of death. Only the results pertaining to suicide are presented. For each disorder, a comparison group consisting of 2 matched persons who were of the same sex, born in the same year, and alive at the date of diagnosis of the matched individual with the examined neurological disorder was selected. For some disorders, only few suicide deaths were observed in the matched comparison group within the first 3 years after date of matching. This is, for instance, seen for Huntington disease where the line for the comparison group fades out.
bThe estimates for Huntington disease are depicted on a 0 to 0.01 scale while the other graphs are depicted on a 0 to 0.005 scale.
Aalen-Johansen estimates were calculated for the outcomes of suicide and other causes of death. Only the results pertaining to suicide are presented. For each disorder, a comparison group consisting of 2 matched persons who were of the same sex, born in the same year, and alive at the date of diagnosis of the matched individual with the examined neurological disorder was selected. For some disorders, few suicide deaths were observed in the matched comparison group within the first 3 years after date of matching.
eFigure 1. Adjusted incidence rate ratio according to time since first diagnosis of dementia
eTable 1. List of examined neurological disorders
eTable 2. Distribution of suicides, person-years and incidence rates for study population
eTable 3. Adjusted incidence rate ratio of suicide in relation to period
eTable 4. Adjusted incidence rate ratio of suicide in relation to time since first diagnosis
Customize your JAMA Network experience by selecting one or more topics from the list below.
Erlangsen A, Stenager E, Conwell Y, et al. Association Between Neurological Disorders and Death by Suicide in Denmark. JAMA. 2020;323(5):444–454. doi:10.1001/jama.2019.21834
Is there an association between neurological disorders and a higher risk of death by suicide?
In this retrospective cohort study that included 7 300 395 persons in Denmark from 1980 through 2016, there was a significantly higher rate of suicide among those with a diagnosed neurological disorder than all other persons (44.0 per 100 000 person-years vs 20.1 per 100 000 person-years, adjusted incidence rate ratio, 1.8).
In Denmark, having a diagnosis of a neurological disorder was associated with a small but statistically significant increased risk of death by suicide.
Neurological disorders have been linked to suicide, but the risk across a broad spectrum of neurological disorders remains to be assessed.
To examine whether people with neurological disorders die by suicide more often than other people and to assess for temporal associations.
Design, Setting, and Participants
Nationwide, retrospective cohort study on all persons 15 years or older living in Denmark, from 1980 through 2016 (N = 7 300 395).
Medical contact for head injury, stroke, epilepsy, polyneuropathy, diseases of myoneural junction, Parkinson disease, multiple sclerosis, central nervous system infections, meningitis, encephalitis, amyotrophic lateral sclerosis, Huntington disease, dementia, intellectual disability, and other brain diseases from 1977 through 2016 (n = 1 248 252).
Main Outcomes and Measures
Death by suicide during 1980-2016. Adjusted incidence rate ratio (IRRs) were estimated using Poisson regressions, adjusted for sociodemographics, comorbidity, psychiatric diagnoses, and self-harm.
Of the more than 7.3 million individuals observed over 161 935 233 person-years (49.1% males), 35 483 died by suicide (median duration of follow-up, 23.6 years; interquartile range, 10.0-37.0 years; mean age, 51.9 years; SD, 17.9 years). Of those, 77.4% were males, and 14.7% (n = 5141) were diagnosed with a neurological disorder, equivalent to a suicide rate of 44.0 per 100 000 person-years compared with 20.1 per 100 000 person-years among individuals not diagnosed with a neurological disorder. People diagnosed with a neurological disorder had an adjusted IRR of 1.8 (95% CI, 1.7-1.8) compared with those not diagnosed. The excess adjusted IRRs were 4.9 (95% CI, 3.5-6.9) for amyotrophic lateral sclerosis, 4.9 (95% CI, 3.1-7.7) for Huntington disease, 2.2 (95% CI, 1.9-2.6) for multiple sclerosis, 1.7 (95% CI, 1.6-1.7) for head injury, 1.3 (95% CI, 1.2-1.3) for stroke, and 1.7 (95% CI, 1.6-1.8) for epilepsy. The association varied according to time since diagnosis with an adjusted IRR for 1 to 3 months of 3.1 (95% CI, 2.7-3.6) and for 10 or more years, 1.5 (95% CI, 1.4 to 1.6, P < .001). Compared with those who were not diagnosed with a neurological disorder, those with dementia had a lower overall adjusted IRR of 0.8 (95% CI, 0.7-0.9), which was elevated during the first month after diagnosis to 3.0 (95% CI, 1.9-4.6; P < .001). The absolute risk of suicide for people with Huntington disease was 1.6% (95% CI, 1.0%-2.5%).
Conclusions and Relevance
In Denmark from 1980 through 2016, there was a significantly higher rate of suicide among those with a diagnosed neurological disorder than persons not diagnosed with a neurological disorder. However, the absolute risk difference was small.
Neurological disorders were estimated to account for 11.6% of the total global burden of disability-adjusted life-years and to represent the underlying cause for 16.5% of total global deaths in 2016.1 Population aging has led to an increasing prevalence of neurological disorders and the associated health care burden.1 Many neurological disorders affect people’s physical functioning and cognitive skills, potentially leading to behavioral and communication problems, with further consequences in terms of quality of life and risk of suicide.2
Population-based studies have associated head injury, stroke, epilepsy, and multiple sclerosis with suicide.3-6 Findings related to less prevalent neurological disorders, such as amyotrophic lateral sclerosis,7 Huntington disease,8 and Parkinson disease,8 were inconclusive due to small sample sizes, selection bias, and suboptimal comparison groups while not being adjusted for relevant confounders, such as physical and mental comorbidity.4,8,9 Meningitis, polyneuropathy,10 and Guillain-Barré have seemingly not been associated with suicidal behavior. Alzheimer disease and dementia might be associated with a lower the risk of suicide although findings have shown inconsistencies.11,12 For some disorders, an accentuated risk has been identified shortly after diagnosis and longer hospital stays.6,13,14
The aim of this study was to examine whether people with neurological disorders have higher suicide rates than people without these disorders. It was also investigated whether specific disorders, number of hospitalizations, time since first diagnosis, and earlier vs later time spans were predictive of higher rates.
The project was approved by the Danish Data Protection Agency via the Capital Region of Copenhagen (journal number RHP-2012-021). According to Danish legislation, informed consent was not required when using register linkage data.
Quiz Ref IDUsing a retrospective cohort study design, we conducted a nationwide, individual-level data linkage. Each person living in Denmark has a unique personal identifier,15 which enabled linkage of data from the Danish Civil Registration System with the National Patient Register (from 1977), the Psychiatric Central Research Register (from 1969), the Registry of Causes of Death, and labor market registers.
All persons aged 15 years or older who were alive and registered as living in Denmark from 1980 through 2016 were included.
Quiz Ref IDPersons diagnosed with neurological disorders were identified in the National Patient Register from 1977 to 2016. Diagnoses were recorded according to the International Classification of Diseases, Eighth Revision (ICD-8) until 1994, when the ICD-10 revision was implemented. The following disorders were identified: head injury, stroke, epilepsy, polyneuropathy, diseases of myoneural junction, Parkinson disease, multiple sclerosis, central nervous system infections, meningitis, encephalitis, amyotrophic lateral sclerosis, Huntington disease, dementia, intellectual disability, and other brain diseases (for ICD codes, see eTable 1 in the Supplement). All full-time admissions, outpatient visits, and emergency department contacts with a hospital were screened. In addition, we screened the Psychiatric Central Research Register to identify additional diagnoses of dementia or intellectual disability. Individuals were considered exposed from the date of diagnosis, including those diagnosed prior to the observation period and before the age of 15 years. People diagnosed with more than 1 neurological disorder were considered as having each of these disorders.
The primary outcome was death by suicide, defined as ICD-8 codes: E950-E959 and ICD-10 codes: X60-X84, Y87.0 recorded in the Danish Cause of Death Registry. A 92% agreement between official suicide records and expert assessments has been reported.16
Participants were observed between January 1, 1980, and December 31, 2016. Those individuals who turned 15 years or who immigrated were included on their birthdate or date of entry, respectively. People migrating out of the country or who died were censored on the respective date of these events.
We used a Poisson regression model to compare incidence rates of people with neurological disorders with those without disorders, expressed as adjusted incidence rate ratios (IRRs) with 95% CIs. Different disorders were assessed in separate models. In the Poisson regression model, covariates are assumed to enter additively, ie, without interactions. Because such interactions were expected to have little effect on the association between the exposure and the suicide rate,17 this assumption was not formally tested. We examined whether suicide rates differed with respect to severity of disorder, ie, cumulative number of hospital contacts for the examined disorder (none, 1, 2-3, >4 admissions), temporal closeness to diagnosis, ie, time since first diagnosis (<1, 3, 6 months, <1, <3, <5, <10, ≥10 years), and earlier vs later time spans (1980-1999 vs 2000-2016). Direct comparison between specific categories was performed with one category as the reference, for instance 1 vs more than 4 hospital contacts. Tests were 2-sided and with a P < .05 as the level of statistical significance.
Multiple regression models were adjusted for period (1980-1989, 1990-1999, 2000-2009, 2010-2016), sex (male, female), age group (15-29, 30-39, 40-49, 50-59, 60-69, 70-79, 80-89, ≥90 years), living status (alone, cohabiting, or married), region (Capital, Zealand, Southern Denmark, Central Denmark, North Denmark), socioeconomic status (working, unemployed, retired, or disability pension; student; missing data), physical comorbidity (defined as the Charlson comorbidity index score and categorized into18 none, 1-3, ≥4), psychiatric hospitalization prior to diagnosis of any neurological disorders (none, depression, other disorders), and deliberate self-harm prior to diagnosis of any neurological disorders (none, ≥1 episode of deliberate self-harm). All covariates were included as time-varying and updated either on the exact registered date of change or on a yearly basis. Missing values were noted for living status (<1% of total person-years) and socioeconomic status (8.4% of total person-years), of which the latter was handled as a separate category.
Due to the possibility that an observed association between neurological disorders and depression may not be specific to these disorders but could represent a general relationship between chronic illness and depression, a model with a negative comparison group was fitted. We therefore repeated the main analysis using rheumatoid arthritis (ICD-10 codes: M05-M06), which is characterized by presence of chronic pain, rather than the general population as the comparison group.
We calculated cumulative incidences for individuals first diagnosed during the observation period by using the Aalen-Johansen estimator to account for competing risks by deaths due to other causes.19 A comparison group was formed by matching each exposed individual with 2 persons of same sex and same birth year who were alive in the calendar year of first diagnosis and unexposed at the time of matching. Data management and analysis were conducted using SAS statistical software version 9.4 (SAS Institute Inc).
During the years 1980 through 2016, we observed 7 300 395 individuals (49.9% males), over 161 935 233 person-years (median follow-up, 23.6 years; interquartile range, 10.0-37.0 years) In all, 35 483 persons died by suicide (77.4% males, mean age, 51.9 years; SD, 17.9 years), of whom 14.7% (n = 5141) had a neurological disorder. The incidence rate among persons diagnosed with neurological disorders was 44.0 per 100 000 person-years while those not diagnosed had an incidence rate of 20.1 per 100 000 person-years. The absolute difference between incidence rates was 23.9 per 100 000 person-years.
Head injury, stroke, and epilepsy accounted for 4.5% (n = 1592), 3.5% (n = 1235), and 3.0% (n = 1048) of all suicide deaths, respectively. As seen in Figure 1, the incidence rate of people with neurological disorders and those without differed during the second part of life. Peaks in the incidence rate were noted for diseases of myoneural junction during youth and dementia during midlife.
People diagnosed with a neurological disorder had an adjusted IRR of 1.8 (95% CI, 1.7-1.8) compared with those not diagnosed (Table 1). The highest rate ratio was noted between the first and third months after being diagnosed (adjusted IRR per 100 000 person-years, 3.1; 95% CI, 2.7-3.6). After 10 years or more years, the adjusted IRR was 1.5 (95% CI, 1.4-1.6). The largest excess adjusted IRRs of suicide mortality were for Huntington disease (4.9, 95% CI, 3.1-7.7) and for amyotrophic lateral sclerosis (4.9, 95% CI, 3.5-6.9). For head injury, an adjusted IRR of 1.7 (95% CI, 1.6-1.7) was found while the corresponding values were 1.3 (95% CI, 1.2-1.3) for stroke and 1.7 (95% CI, 1.6-1.8) for epilepsy.
The adjusted IRR demonstrated elevated suicide rates of 1.4 (95% CI, 1.1-1.6) for intracerebral hemorrhage; 1.3 (95% CI, 1.1-1.6) for cerebral infarction; 1.3 (95% CI, 1.0-1.7) for subarachnoid hemorrhage; 1.7 (95% CI, 1.6-1.8) for polyneuropathy and peripheral neuropathy; 2.2 (95% CI, 1.2-4.1) for Guillain-Barré; 1.9 (95% CI, 1.8-2.1) for diseases of myoneural junction and muscle; 1.8 (95% CI, 1.5-2.1) for other brain disorders; 1.7 (95% CI, 1.5-1.9) for Parkinson disease; 2.2 (95% CI, 1.9-2.6) for multiple sclerosis; 1.6 (95% CI, 1.3-1.9) for central nervous system infection and 1.6 (95% CI, 1.2-2.0) for its specific subgroups of meningitis; and 1.7 (95% CI, 1.3-2.3) for encephalitis.
The adjusted IRR demonstrated lower suicide rates of 0.8 (95% CI, 0.7-0.9) for dementia, 0.2 (95% CI, 0.2-0.3) for Alzheimer disease, and 0.6 (95% CI, 0.5-0.8) for intellectual disabilities. However, the adjusted IRR for people with dementia during the first month after diagnosis was 3.0 (95% CI, 1.9-4.6; eFigure in the Supplement).
We found an increased suicide rate with an increasing cumulative number of hospital contacts for neurological conditions. The adjusted IRR for 1 admission was 1.7 (95% CI, 1.6-1.7); for 2 to 3 admissions, 1.8 (95% CI, 1.7-1.9); and for more than 4 admissions, 2.1 (95% CI, 1.9-2.2) (P < .001; Table 2). The adjusted IRR among those diagnosed with multiple sclerosis and who had 4 or more hospital contacts (3.2; 95% CI, 2.5-4.0) had a suicide rate twice as high as those with only 1 contact (1.6; 95% CI, 1.2-2.1). Similarly, an increase was noted for polyneuropathy, with an adjusted IRR of 2.5 (95% CI, 1.9-3.1) vs 1.6 (95% CI, 1.5-1.7; Table 2).
Over the study period, the suicide incidence rate for people with neurological disorders decreased from 78.6 per 100 000 person-years during the 1980-1999 years to 27.3 per 100 000 person-years during the 2000-2016 years. The suicide incidence rate for those without a disorder decreased from 26.3 to 12.7 during the same time spans (eTable 3 the Supplement). Comparing the same time spans, the adjusted IRR for suicide among those with dementia decreased from 2.4 (95% CI, 2.0-2.9) to 1.0 [reference] as it did for those with multiple sclerosis from 2.0 (95% CI, 1.5-2.8) to 1.0 [reference] (eTable 3 in the Supplement). The decline in the overall suicide rate over time did not affect the relative risk pattern.
While accounting for competing risks by other causes of death, steeper cumulative incidences were found for all neurological disorders, except Alzheimer disease, compared with those without these disorders (Figure 2 and Figure 3). Among those with amyotrophic lateral sclerosis, 4 out of 1000 had died by suicide within 3 years of being diagnosed, while their cumulative incidence (absolute risk) was 0.58 (95% CI, 0.38-0.87) over the 30-year follow-up (Table 3). For the subset of people diagnosed with Huntington disease, 1.62 (95% CI, 1.04-2.52) died by suicide, while the figure was 1.19 (95% CI, 1.11-1.27) for diseases of myoneural junction and muscle. Risks ranged between 0.8 and 0.9 (95% CI, 0.6-1.1) for head injury, polyneuropathy, encephalitis, and epilepsy.
Using people diagnosed with rheumatoid arthritis as a negative comparison group, those with any neurological disorder had a higher suicide rate (rheumatoid arthritis only, incident rate, 18.4 per 100 000 person-years; adjusted IRR, 1.0 (reference); for neurological disorder only, incident rate, 30.2 per 100 000 person-years, adjusted IRR, 1.4; 95% CI, 1.2-1.6, and for neither disorder, incident rate, 14.0 per 100 000 person-years, adjusted IRR, 0.9; 95% CI, 0.7-1.0).
Quiz Ref IDIn Denmark from 1980 through 2016, there was a small but significantly higher rate of suicide among those with a diagnosed neurological disorder than other persons without neurological disorders. Rates were highest among people diagnosed with Huntington disease and amyotrophic lateral sclerosis but also common disorders, such as head injury, stroke, and epilepsy were associated with higher suicide rates than for nonexposed individuals. An association between the number of hospital contacts and suicide was demonstrated for any disorder as well as for multiple sclerosis, head injury, and polyneuropathy. A temporal relation was noted between the time since first diagnosis of a neurological disorder and suicide, including dementia.
The findings for head injury supplement those of other studies.6,20 Stroke accounted for a substantial share of suicide deaths and while previous, less rigorously adjusted findings have suggested an association,21 these findings strengthen the possible causal link through the association to number of hospital contacts, which could be viewed as a proxy for illness severity. Recent treatment gains, thrombolysis and thrombectomy, and the more aggressive use of rehabilitative services after stroke, might have improved quality of life of patients over recent years. In addition, a stronger emphasis on rehabilitation has evolved over the studied period. Quiz Ref IDEarlier data linkage studies reported a higher adjusted IRR for epilepsy than those noted herein but failed to account for preceding mental disorders and history of deliberate self-harm.5,13 Associations with mental disorders, especially depression, are well known for epilepsy5,22 and are likely to exacerbate the risk of suicide. The finding of an association with the proxy for severity of epilepsy adds to the existing body of evidence.
The association between amyotrophic lateral sclerosis and suicide7 is supported by reports of frequent physician-assisted suicides.23 Previous findings regarding Huntington disease and suicide were limited by not accounting for possible confounders. Factors that may contribute to the relative and absolute excess risk found among patients with Huntington disease24 could be related to depression, possibly mediated by hyperactivity in the hypothalamic-pituitary-adrenal axis. The autosomal inheritance of Huntington disease might also be a contributing factor, either because of a shared neurobiological diathesis or from witnessing the course of the disease in one’s parent. Life-threatening disorders, such as pancreatic cancer, have also been linked to suicide.25 Earlier crude estimates for multiple sclerosis26,27 were reinforced by the association with the number of hospitalizations as a proxy for severity. Better health care management may have contributed to the decline in the suicide rate of people with multiple sclerosis over time.
Although findings relating suicide risks to central nervous system infections have been reported,28 the results regarding meningitis and encephalitis seem to be novel. Significantly elevated suicide rates were found for Parkinson disorder, polyneuropathy, and Guillain-Barré, which have not been shown previously despite an association of these disorders with mental disorders.8,29,30
Diagnoses of dementia, and especially Alzheimer disease, have been associated with lower rates of suicide.11 Earlier findings based on the same data source as what is reported herein showed a higher suicide risk.12 The implementation of community-based support for dementia after 1995 might have indirectly prevented suicides.31 Also, time of hospitalization may have transitioned to later stages of the disorder, which are characterized by more advanced cognitive decline and therefore reduced functional ability to complete a suicidal act. Both explanations would be supported by the observed reduction in the most recent period. The increased risk shortly after diagnosis of dementia remains noteworthy. Lower rates of suicidal behavior have been reported for men with intellectual disabilities and people with Down syndrome.13,32
The observed decline in the general suicide rate from 40.4 per 100 000 person-years in 1980 to 10.9 per 100 000 person-years in 2016 in Denmark has largely been attributed to means restriction, such as efforts to limit availability of firearms and particularly toxic medication.33 Because there were no initiatives directly addressing suicide among people with physical disorders, it seems unlikely that the decrease affected the relevance of neurological disorders as a predictor of suicide. It is possible that the improvements observed for dementia and multiple sclerosis may be related to improvements in treatment and intensified community-based support.
Quiz Ref IDThis study design cannot establish causality. However, there are plausible mechanisms that could link the examined disorders to suicide. First, being diagnosed may constitute a distressing life event9; second, disease-related psychological consequences, such as feelings of perceived burdensomeness, impaired self-image, limited social life, reduced financial security, anxieties, and dependency on others for care34; third, physical symptoms, such as communication difficulties, poor sleep, and pain35; fourth, psychiatric symptoms, including depression, alcohol misuse, and psychosis9,36-38; fifth, neurobiological consequences, such as aggression and reduced impulse control; and sixth, alterations in the integrity of neural circuitry, hypothalamic-pituitary-adrenal axis and central serotonin functioning.39 People with neurological disorders may also have easier access to toxic medication.
The absolute risk differences observed herein were small; hence, these findings do not necessarily warrant changing the management of treatment for individual patients with neurological diseases. As with all patients, physicians should be aware of the potential for depression, demoralization, and suicide.
The population-based setting, national coverage, and long follow-up were some of the strengths of this study. A complete coverage of all hospital-based diagnoses and no loss to follow-up are other strengths. The validity of hospital diagnoses from the National Patient Registry has been evaluated as good.40 For the examined disorders, it might be improved by assessment from a medical, hospital-based specialist using clinical tools, such as computed tomographic, magnetic resonance imaging, and positron-emission tomographic scans; arteriographic electroencephalographic, and neurophysiological tests; and spinal fluid analyses in accordance with international diagnostic criteria. The level of detail and precision, such as exact date of diagnosis and ICD codes, availability of relevant covariates for stratification, add strength to the examined associations. Although the number of hospitalizations could be a marker of comorbidity, the measure has previously been used as a marker of severity.6,28 Comorbidity was assessed using the Charlson comorbidity index, a validated tool for identifying chronic and severe disorders.
This study has several limitations. First, individuals diagnosed prior to 1977 were not included if no later hospital contact had taken place. Second, diagnoses given in primary care were not included.
Third, an underrecording of suicide deaths cannot be excluded. Fourth, persons with neurological disorders and other chronic disorders who are depressed may be more likely to have been diagnosed with depression than people with no chronic disorders and less frequent medical contact. If this occurred, and if the physician determining the cause of death was biased to attributing it to suicide if the person had carried a diagnosis of depression, some of the observed differences may have been artifactual. Fifth, adjusting for preexisting mental disorders could be viewed as overadjusting due to exiting mental comorbidity.36-38 Sixth, due to the large number of tests, type I errors cannot be excluded. These limitations are not assumed to bias the findings except in a conservative manner.
In Denmark between 1980 and 2016, there was a significantly higher rate of suicide among those with a diagnosed neurological disorder than persons not diagnosed with a neurological disorder. However, the absolute risk difference was small.
Corresponding Author: Annette Erlangsen, PhD, Danish Research Institute for Suicide Prevention, Mental Health Centre Copenhagen, Gentofte Hospitalsvej 15, 4, DK-2900 Hellerup, Denmark (firstname.lastname@example.org).
Accepted for Publication: December 14, 2019.
Author Contributions: Dr Erlangsen had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Erlangsen, Egon Stenager, Conwell, Hawton, Benros, Nordentoft, Elsebeth Stenager.
Acquisition, analysis, or interpretation of data: Erlangsen, Egon Stenager, Conwell, Andersen, Hawton, Benros, Elsebeth Stenager.
Drafting of the manuscript: Erlangsen, Hawton.
Critical revision of the manuscript for important intellectual content: Egon Stenager, Conwell, Andersen, Hawton, Benros, Nordentoft, Elsebeth Stenager.
Statistical analysis: Erlangsen, Andersen, Nordentoft.
Obtained funding: Elsebeth Stenager.
Administrative, technical, or material support: Elsebeth Stenager.
Supervision: Benros, Nordentoft, Elsebeth Stenager.
Conflict of Interest Disclosures: None reported.
Funding/Support: Supported by a grant from the Psychiatric Research Foundation, Region of Southern Denmark, Denmark.
Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.