[Skip to Navigation]
Sign In
Table 1.  Characteristics of the Study Cohorts Within Each Maryland Databasea
Characteristics of the Study Cohorts Within Each Maryland Databasea
Table 2.  Odds Ratios Associated With Variables in Models of Future Opioid Overdose Death and Nonfatal Opioid Overdose Events
Odds Ratios Associated With Variables in Models of Future Opioid Overdose Death and Nonfatal Opioid Overdose Events
Table 3.  C Statistics for Different Models Predicting Fatal and Nonfatal Opioid Overdosesa
C Statistics for Different Models Predicting Fatal and Nonfatal Opioid Overdosesa
Table 4.  Sensitivity, Specificity, and Positive and Negative Predictive Values for Different Probability Score Thresholds When Forecasting Death or Nonfatal Opioid Overdose Eventsa
Sensitivity, Specificity, and Positive and Negative Predictive Values for Different Probability Score Thresholds When Forecasting Death or Nonfatal Opioid Overdose Eventsa
1.
Hedegaard  H, Minino  A, Warner  M. Drug overdose deaths in the United States, 1999–2017. Published 2018. Accessed August 14, 2019. https://www.cdc.gov/nchs/data/databriefs/db329-h.pdf.
2.
National Center for Health Statistics. Provisional Drug Overdose Death Counts. Published 2019. Accessed August 14, 2019. https://www.cdc.gov/nchs/nvss/vsrr/drug-overdose-data.htm
3.
Saloner  B, McGinty  EE, Beletsky  L,  et al.  A public health strategy for the opioid crisis.   Public Health Rep. 2018;133(1_suppl):24S-34S. doi:10.1177/0033354918793627PubMedGoogle ScholarCrossref
4.
Geissert  P, Hallvik  S, Van Otterloo  J,  et al.  High-risk prescribing and opioid overdose: prospects for prescription drug monitoring program-based proactive alerts.   Pain. 2018;159(1):150-156. doi:10.1097/j.pain.0000000000001078PubMedGoogle ScholarCrossref
5.
Liang  Y, Goros  MW, Turner  BJ.  Drug overdose: differing risk models for women and men among opioid users with non-cancer pain.   Pain Med. 2016;17(12):2268-2279. doi:10.1093/pm/pnw071PubMedGoogle ScholarCrossref
6.
Bohnert  ASB, Logan  JE, Ganoczy  D, Dowell  D.  A detailed exploration into the association of prescribed opioid dosage and overdose deaths among patients with chronic pain.   Med Care. 2016;54(5):435-441. doi:10.1097/MLR.0000000000000505PubMedGoogle ScholarCrossref
7.
Glanz  JM, Narwaney  KJ, Mueller  SR,  et al.  Prediction model for two-year risk of opioid overdose among patients prescribed chronic opioid therapy.   J Gen Intern Med. 2018;33(10):1646-1653. doi:10.1007/s11606-017-4288-3PubMedGoogle ScholarCrossref
8.
Zedler  BK, Saunders  WB, Joyce  AR, Vick  CC, Murrelle  EL.  Validation of a screening risk index for serious prescription opioid-induced respiratory depression or overdose in a US commercial health plan claims database.   Pain Med. 2018;19(1):68-78. doi:10.1093/pm/pnx009PubMedGoogle ScholarCrossref
9.
Lo-Ciganic  W-H, Huang  JL, Zhang  HH,  et al.  Evaluation of machine-learning algorithms for predicting opioid overdose risk among Medicare beneficiaries with opioid prescriptions.   JAMA Netw Open. 2019;2(3):e190968. doi:10.1001/jamanetworkopen.2019.0968PubMedGoogle Scholar
10.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   J Clin Epidemiol. 2008;61(4):344-349. doi:10.1016/j.jclinepi.2007.11.008PubMedGoogle ScholarCrossref
11.
Caudarella  A, Dong  H, Milloy  MJ, Kerr  T, Wood  E, Hayashi  K.  Non-fatal overdose as a risk factor for subsequent fatal overdose among people who inject drugs.   Drug Alcohol Depend. 2016;162:51-55. doi:10.1016/j.drugalcdep.2016.02.024PubMedGoogle ScholarCrossref
12.
Chang  H-Y, Krawczyk  N, Schneider  KE,  et al.  A predictive risk model for nonfatal opioid overdose in a statewide population of buprenorphine patients.   Drug Alcohol Depend. 2019;201:127-133. doi:10.1016/j.drugalcdep.2019.04.016PubMedGoogle ScholarCrossref
13.
Krawczyk  N, Eisenberg  M, Schneider  KE,  et al.  Predictors of overdose death among high-risk hospital patients with substance-related encounters: a data linkage cohort study.   Ann Emerg Med. 2020;75(1):1-12. doi:10.1016/j.annemergmed.2019.07.014PubMedGoogle ScholarCrossref
14.
Pencina  MJ, D’Agostino  RB  Sr.  Evaluating discrimination of risk prediction models: the C statistic.   JAMA. 2015;314(10):1063-1064. doi:10.1001/jama.2015.11082PubMedGoogle ScholarCrossref
15.
Maryland Department of Planning. U.S. census bureau population estimates by age, race and gender April 1, 2010 to July 1, 2018. Published 2019. Accessed August 14, 2019. https://planning.maryland.gov/MSDC/Pages/pop_estimate/CensPopEst.aspxx
16.
Weimer  M, Morford  K, Donroe  J.  Treatment of opioid use disorder in the acute hospital setting: a critical review of the literature (2014–2019).   Curr Addict Rep. 2019;(July). doi:10.1007/s40429-019-00267-xGoogle Scholar
17.
Committee on Medication-Assisted Treatment for Opioid Use Disorder.  Medications for Opioid Use Disorder Save Lives. National Academies of Science, Engineering, and Medicine; 2019.
18.
Walley  AY, Bernson  D, Larochelle  MR, Green  TC, Young  L, Land  T.  The contribution of prescribed and illicit opioids to fatal overdoses in Massachusetts, 2013-2015.   Public Health Rep. 2019;134(6):667-674. doi:10.1177/0033354919878429PubMedGoogle ScholarCrossref
19.
Nechuta  SJ, Tyndall  BD, Mukhopadhyay  S, McPheeters  ML.  Sociodemographic factors, prescription history and opioid overdose deaths: a statewide analysis using linked PDMP and mortality data.   Drug Alcohol Depend. 2018;190:62-71. doi:10.1016/j.drugalcdep.2018.05.004PubMedGoogle ScholarCrossref
20.
National Association of State Alcohol and Drug Abuse Directors. NASADAD releases updated STR/SOR funding timeline. Published 2019. Accessed January 6, 2020. https://nasadad.org/2019/04/nasadad-releases-updated-str-sor-funding-timeline/
21.
McCarty  D, Rieckmann  T, Baker  RL, McConnell  KJ.  The perceived impact of 42 CFR part 2 on coordination and integration of care: a qualitative analysis.   Psychiatr Serv. 2017;68(3):245-249. doi:10.1176/appi.ps.201600138PubMedGoogle ScholarCrossref
2 Comments for this article
EXPAND ALL
Predictive Modeling of Opioid Overdose: Lack of Methodological Detail and External Validation Limits Clinical Relevance
Gabrielle Beaudry, BA | Department of Psychiatry, University of Oxford
Saloner et al (1) present a new predictive risk model aimed at identifying persons at risk of opioid overdose (both fatal and nonfatal). Although it provides useful evidence on risk factors of opioid-related deaths, we have concerns whether the tool can be used clinically and some aspects of the methodology.

The existing multivariable prediction model was developed using four large linked datasets. Authors report using bootstrapping to assess internal validity, but the absence of an external validation assessment should caution against its wider use. Prognostic models should be evaluated in a different dataset than that used for derivation, which
ensures that calibration (i.e. the degree to which model predictions align with observed outcomes) is accounted for, and can be, if necessary, adjusted or updated.(2) Poorly calibrated algorithms can generate inaccurate risk predictions which in turn, can result in misinformed clinical decisions. Furthermore, the absence of the full formula precludes others from critically appraising and validating the tool.

The model achieves a seemingly impressive area under the curve for both fatal (0.89) and nonfatal overdose (0.85). However, the low positive predictive value (PPV) – perhaps a more useful real-world performance measure is reported at 1.4%–3.0% in fatal overdoses and 9.1%–17.0% in nonfatal overdoses, which can partly be explained by the low prevalence of opioid overdose. On the basis of this metric, the tool’s high false positive rate might appear excessive, with potential resource implications. If a high risk categorization is regarded as only precautionary rather than definitive, and the consequences of this categorization are not harmful, then a high false positive rate is not a reason in itself to prevent the tool’s use.

The authors state that the utility of their model lies in its capacity to inform resource allocation at a community level (e.g. naloxone programs). Whilst this point is well taken, we think that the prediction model should also be informative at an individual level.(3,4) Such a tool could improve tailoring management for individuals with opiate use disorders by identifying persons at high risk of opioid overdose, and hence offer them additional (non-harmful) interventions. It could also assist clinicians in safer opioid prescribing by pre-screening at-risk individuals. Conversely, the high negative predictive value (NPV) could help in preserving resources by screening out low risk individuals. Without further guidance on whether the tool should be used as a rule-in or as a rule-out mechanism, its practical utility remains unclear.

References

1. Saloner B, Chang H-Y, Krawczyk N, et al. Predictive Modeling of Opioid Overdose Using Linked Statewide Medical and Criminal Justice Data. JAMA Psychiatry. 2020.
2. Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17(1):230.
3. Fazel S, Wolf A, Larsson H, Lichtenstein P, Mallett S, Fanshawe TR. Identification of low risk of violent crime in severe mental illness with a clinical prediction tool (Oxford Mental Illness and Violence tool [OxMIV]): a
derivation and validation study. The Lancet Psychiatry. 2017;4(6):461-468.
4. Fazel S, Wolf A, Larsson H, Mallett S, Fanshawe TR. The prediction of suicide in severe mental illness: devel- opment and validation of a clinical prediction rule (OxMIS). Translational Psychiatry. 2019;9(1):98.

Authors: Gabrielle Beaudry, BA; Denis Yukhnenko, MSc; Seena Fazel, MD
Author affiliations: University of Oxford (all)
CONFLICT OF INTEREST: None Reported
READ MORE
Predictive Modeling of Opioid Overdose: Institutions of Incarceration Must Provide Medication Treatment for OUD
Carolyn Sufrin, MD, PhD | Johns Hopkins University School of Medicine, Dept. of Gynecology and Obstetrics
We appreciate the novel contribution of Saloner et al’s predictive model for opioid overdoses in Maryland (1). We call attention to Table 1 showing that nearly half (47%, 564/1204) of fatal overdoses and more than 7% (635/8430) of non-fatal overdoses occurred in people with criminal justice involvement. Both release from prison and being on probation or parole were significant predictors of overdoses, with release from prison having the strongest association with fatal overdose in the study (Table-2). A CDC analysis showed nearly 4% of all fatal opioid overdoses in the U.S. occurred within 30 days of correctional release (2). Extrapolated to 47,000 opioid overdose deaths each year (3), this represents more than 1700 annual deaths. Saloner el al’s findings corroborate other studies, demonstrating that release from prison or jail is a time of high risk for overdose mortality (4).

These results have urgent implications for overdose prevention and treatment interventions for people returning to their communities after incarceration. If people undergo forced withdrawal in custody, they are likely to relapse upon release, and potentially overdose due to loss of opioid tolerance during incarceration. In contrast, maintenance or initiation of OUD treatment (with access to all FDA-approved medications) in custody is an evidence-based, life-saving strategy (4) For example, the Rhode Island Department of Corrections demonstrated a 2/3 reduction in overdose mortality across the state when they implemented universal treatment for incarcerated people with OUD (5).

The moral and public health imperative to provide medication treatment for people in custody is clear. Yet many obstacles prevent full access to standard-of-care treatment. One is the regulatory framework surrounding methadone, which often leads correctional facilities to awkward and tedious workarounds, or to no provision altogether. Other significant obstacles include lack of consistent funding for treatment programs, irregular state and local legal landscapes, misunderstandings of treatment details, and correctional officials’ and patients’ negative attitudes toward treatment (6).

While it is no small task to implement comprehensive medication treatment in incarcerated settings across a range of geographies, local and state governments, and health service delivery systems, evidence-based strategies show this can be done. The organization we, the under-signed, represent, the National Commission on Correctional Health Care, has been an innovator in this field and was an early partner with the Substance Abuse and Mental Health Services Administration in developing standards for opioid treatment services specifically provided in jails and prisons. The time has come for all institutions of incarceration to provide medication treatment for people with OUD.

Carolyn Sufrin, MD, PhD

Brent Gibson, MD, MPH
National Commission on Correctional Health Care

Kevin Fiscella, MD, MPH
University of Rochester Medical Center, Department of Family Medicine

References
1. Saloner B, et. al. Predictive modeling. . . JAMA Psychiatry. 2020.
2. Mattson CL, et. al. MMWR. 2018;67(34):945-951.
3. Wilson N, et. al. MMWR. 2020;69(11):290-29
4. Sugarman OK, et. al. Interventions for incarcerated adults with opioid use disorder in the United States. PLoS One. 2020;15(1):e0227968.
5. Green TC, et al. Postincarceration fatal overdoses. JAMA Psychiatry. 2018;75(4):405-407.
6. Friedmann PD, et al. Medication-assisted treatment in criminal justice agencies. Subst Abus. 2012;33(1):9-18.
CONFLICT OF INTEREST: I receive funding from the National Institute on Drug Abuse (5K23DA045934-02) and the Society for Family Planning Research Fund (SFPRF11-09).
READ MORE
Original Investigation
June 24, 2020

Predictive Modeling of Opioid Overdose Using Linked Statewide Medical and Criminal Justice Data

Author Affiliations
  • 1Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
  • 2Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
  • 3Center for Population Health IT, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
  • 4Department of Population Health, New York University School of Medicine, New York
  • 5Chesapeake Regional Information System for Our Patients, Columbia, Maryland
  • 6Behavioral Health Administration, Maryland Department of Health, Baltimore
JAMA Psychiatry. 2020;77(11):1155-1162. doi:10.1001/jamapsychiatry.2020.1689
Key Points

Question  What factors most strongly predict opioid overdose in a linked statewide administrative data set?

Findings  In this predictive modeling study of 4 statewide Maryland databases with data from 2.2 million individuals, fatal opioid overdose in the next 12 months could be predicted with an area under the curve as high as 0.89. The factors most strongly associated with the baseline year (by odds ratio) included male sex, use of addiction treatment, at least 1 nonfatal overdose, and release from prison.

Meaning  Public health efforts to prioritize lifesaving interventions should consider the relative risk of overdose across different population groups.

Abstract

Importance  Responding to the opioid crisis requires tools to identify individuals at risk of overdose. Given the expansion of illicit opioid deaths, it is essential to consider risk factors across multiple service systems.

Objective  To develop a predictive risk model to identify opioid overdose using linked clinical and criminal justice data.

Design, Setting, and Participants  A cross-sectional sample was created using 2015 data from 4 Maryland databases: all-payer hospital discharges, the prescription drug monitoring program (PDMP), public-sector specialty behavioral treatment, and criminal justice records for property or drug-associated offenses. Maryland adults aged 18 to 80 years with records in any of 4 databases were included, excluding individuals who died in 2015 or had a non-Maryland zip code. Logistic regression models were estimated separately for risk of fatal and nonfatal opioid overdose in 2016. Model performance was assessed using bootstrapping. Data analysis took place from February 2018 to November 2019.

Exposures  Controlled substance prescription fills and hospital, specialty behavioral health, or criminal justice encounters.

Main Outcomes and Measures  Fatal opioid overdose defined by the state medical examiner and 1 or more nonfatal overdoses treated in Maryland hospitals during 2016.

Results  There were 2 294 707 total individuals in the sample, of whom 42.3% were male (n = 970 019) and 53.0% were younger than 50 years (647 083 [28.2%] aged 18-34 years and 568 160 [24.8%] aged 35-49 years). In 2016, 1204 individuals (0.05%) in the sample experienced fatal opioid overdose and 8430 (0.37%) experienced nonfatal opioid overdose. In adjusted analysis, the factors mostly strongly associated with fatal overdose were male sex (odds ratio [OR], 2.40 [95% CI, 2.08-2.76]), diagnosis of opioid use disorder in a hospital (OR, 2.93 [95% CI, 2.17-3.80]), release from prison in 2015 (OR, 4.23 [95% CI, 2.10-7.11]), and receiving opioid addiction treatment with medication (OR, 2.81 [95% CI, 2.20-3.86]). Similar associations were found for nonfatal overdose. The area under the curve for fatal overdose was 0.82 for a model with hospital variables, 0.86 for a model with both PDMP and hospital variables, and 0.89 for a model that further added behavioral health and criminal justice variables. For nonfatal overdose, the area under the curve using all variables was 0.85.

Conclusions and Relevance  In this analysis, fatal and nonfatal opioid overdose could be accurately predicted with linked administrative databases. Hospital encounter data had higher predictive utility than PDMP data. Model performance was meaningfully improved by adding PDMP records. Predictive models using linked databases can be used to target large-scale public health programs.

Introduction

Drug overdose deaths in the US quadrupled between 1999 and 20171 before reversing slightly in 2018.2 Two-thirds of all overdose deaths are linked to opioids (prescription opioids, heroin, and illicit fentanyl).1 Approaches to decrease overdose include safer prescribing, expanding treatment for people with opioid use disorder (OUD), and harm reduction programs (eg, naloxone distribution).3 Each requires a strategy to identify high-need populations, but policy often operates without a comprehensive understanding of relative risks across the community.

Predictive risk models (PRMs) provide 1 method for identifying those at greatest overdose risk. These models have been developed using prescription drug monitoring programs (PDMPs)4 and individual payers (eg, private insurance5 or the Veterans Administration6). Models typically focus on prescription-associated predictors, such as dosage of prescribed opioids and concurrent benzodiazepine use.5,6 Some incorporate claims-derived indicators, such as diagnosed OUD and nonfatal overdose.7,8 Predictive risk models can accurately predict fatal and nonfatal overdose risks for these specific payers or programs, achieving area under the curve (AUC) statistics ranging from 0.75 to 0.90.4-9

Despite their utility, current models are limited because they often exclude groups such as people without insurance, patients using specialty behavioral health programs that may be carved out of health plans, and people with criminal justice involvement. Overdose risk has substantially shifted away from people exclusively using prescription opioids to people using illicit opioids.1 To effectively identify high-risk populations, PRMs may need to incorporate data on indicators associated with illicit substance use.

Our objective was to develop a PRM for fatal and nonfatal overdose, using a novel, person-level data set that linked Maryland records from 4 statewide data systems. We hypothesized that best-performing models would use risk factors derived from all of the study databases compared with models with fewer variables.

Methods
Data Sources and Study Cohort

The study cohort spanned multiple programs that might be the focus of a statewide risk reduction effort by the Department of Health. Specifically, it included Maryland residents with 1 or more records in 2015 in any of 4 statewide data sets: PDMP prescriptions; statewide hospital inpatient and emergency department visits; public-sector specialty behavioral health care admissions (ie, programs predominantly serving individuals with Medicaid or no insurance); or arrest, postconviction incarceration in state prison, or a parole/probation case associated with a property or drug offense. The Maryland PDMP collects data on controlled substances on schedules II through V that are dispensed in Maryland by pharmacies and other health care services, across all payers, including the Veteran’s Administration. Data were linked using a probabilistic matching algorithm and deidentified by the Maryland state-designated health information exchange that maintains a data-sharing agreement with the Maryland Department of Health. We restricted the sample to individuals between 18 and 80 years and excluded individuals with out-of-state zip codes. We also excluded decedents recorded by the medical examiner in 2015, but we did not have access to noninvestigated deaths. Additional information is available in the eFigure and eTable 1 in the Supplement. We followed the Strengthening of Reporting in Observational Epidemiology (STROBE) reporting guidelines for cross-sectional studies.10 The study was approved by institutional review boards at the Johns Hopkins Bloomberg School of Public Health and the Maryland Department of Health, which provided a waiver of informed consent for study participants because the study was a retrospective analysis that involved no more than minimal risk.

Opioid-Associated Adverse Outcomes

We focused on 2 primary outcomes occurring in 2016. First was fatal opioid overdoses derived from the Maryland medical examiner datafiles. Second was having 1 or more nonfatal opioid overdose identified within the emergency department or inpatient setting within any Maryland hospital. More information on the derivation of these outcomes is in the eMethods and eTable 2 in the Supplement.

Risk Variables

Variables were derived from each study data set based on their conceptual association with overdose risk, interpretability for clinicians, and established statistical significance in other studies.6,11-13 Individuals who did not appear in a database were coded to have 0 values for all variables derived from that database (eg, a patient with no PDMP records was assumed to have never legally received a prescription for a controlled substance in 2015).

Demographic variables included age and sex. The PDMP variables included short-acting opioid analgesics (0, 1-3, ≥4 prescriptions); any long-acting opioid analgesics; buprenorphine indicated for OUD treatment; any benzodiazepine prescription; number of prescribers (0, 1-2, ≥3 prescribers); and total opioid volume, measured as the total quantity of opioids filled by the patient during the year in morphine milligram equivalents (MMEs), which captures the potencies of different opioids as a standard value (none, 1-300 MMEs, or >300 MMEs). Hospital variables included the number of all-cause emergency department visits (0, 1-2, ≥3 visits); any all-cause inpatient admission; and visits associated with a primary diagnosis for OUD, nonopioid substance use disorder, nonfatal overdose (in 2015), and nonoverdose injury. Behavioral health service variables included mental health treatment, as well as substance use disorder treatment defined as either non-OUD substance use treatment, OUD treatment that included medications, or OUD treatment that did not include medications. Criminal justice variables included having any arrest, an arrest for misdemeanor drug charges or felony drug charges, a parole/probation case, and being released from prison in 2015 after a term specific to drug and property crimes. Further details on variable construction are in the eMethods and eTables 3 and 4 in the Supplement.

Statistical Analysis

We trained separate logistic regression models for fatal and nonfatal overdose that sequentially added variables from the 4 databases. Our basic model included only demographic variables. To the basic model, either PDMP-associated variables or hospital-associated variables were added. We next trained a model that included both hospital and PDMP variables. To that model, we added variables from behavioral health. Our final model added criminal justice data. The AUCs were derived from bootstrapping analyses of 300 iterations; bootstrapping is beneficial because it allowed us to construct 95% CIs and is also robust to specification error.6,11,12 For our final model, we examined the sensitivity, specificity, positive predictive value, and negative predictive value of risk scores at different thresholds of risk.14

Data analysis took place from February 2018 to November 2019 using SAS version 9.4 (SAS Institute). Significance was set at a threshold of P < .05.

Results
Descriptive Analysis

The final sample size was 2 294 707 individuals. This was approximately 51.5% of the Maryland population age 18 to 80 years in 2015.15 There were 1839 fatal opioid overdoses in 2016, of which 1204 (65.5%) could be linked to our sample. In addition, 8430 individuals in the sample had 1 or more nonfatal opioid overdose treated in an emergency or inpatient setting in Maryland.

In the sample, 42.3% of individuals were male (n = 970 019), and 28.2% were aged 18 to 35 years (n = 647 083), 24.8% were aged 36 to 50 years (n = 568 160), 29.3% were aged 50 to 64 years (n = 671 204), and 17.8% were aged 65 to 80 years (n = 408 260) (Table 1). Individuals with criminal justice records were more likely than the full sample to be men (81.1% [(n = 20 011], compared with 42.3% across the full sample) and in younger age groups (eg, 63.3% of individuals [n = 15 633] with criminal justice records were aged 18-34 years compared with 28.2% of those in the full sample; conversely, 0.53% of people with criminal justice records [n = 132] were aged 65-80 years, compared with 17.8% of the full sample), while those with hospital records were more likely to be older (19.5% were aged 65-80 years [n = 286 636] vs 17.8% of the full sample).

About 63.9% of the sample (n = 1 466 750) had hospital records in 2015, 32.0% (n = 734 326) had 1 to 2 emergency department visits, and 5.9% (n = 134 177) had 3 or more visits. Inpatient visits were experienced by 12.6% of the sample (n = 289 925). Injury visits were experienced by 17.0% of the sample (n = 389 423), while visits with a diagnosis of OUD were experienced by 1.2% (n = 26 492), visits with a diagnosis of nonopioid substance use disorders by 3.6% (n = 82 528), and visits involving a nonfatal overdose by 0.2% (n = 5366). Hospital visits for opioid and other substance use disorders were highest in the subgroups using behavioral health services (opioid use disorder visit, 9.6% [n = 16 301]; any other substance use disorder visit, 16.9% [n = 28 932]) and involved with criminal justice (opioid use disorder visit, 8.2% [n = 2032]; any other substance use disorder visit, 11.9% [n = 2941]; compared with 1.2% and 3.6%, respectively, for the full sample).

Two-thirds of the sample (1 529 895 [66.7%]) had PDMP records in 2015. About 34.3% of the full sample (n = 786 634) had 1 to 3 opioid prescriptions, and 9.6% (n = 219 228) had 4 or more opioid prescriptions. About 3.0% (n = 69 087) had 1 or more prescriptions for long-acting opioids. About 1.1% of the sample (n = 25 484) had buprenorphine for OUD, and 17.3% (n = 397 195) had any benzodiazepine. Overall, 37.2% (n = 853 918) had 1 to 2 opioid prescribers, and 7.0% (n = 160 088) had 3 or more prescribers. Controlled substance prescription fills were highest in the populations using PDMP and behavioral health services. For example, 26.0% of the PDMP sample (n = 397 195) and 23.9% (n = 40 784) of the behavioral health had prescriptions for benzodiazepines vs 17.3% of the full sample.

Individuals with criminal justice encounters (n = 24 686) and specialty behavioral health treatment (n = 170 752) made up relatively small subgroups. Only 1.1% of the overall sample (n = 24 686) had criminal justice involvement, 0.4% (n = 8894) had an arrest in 2015 for a drug or property offense, and 0.6% (n = 13 147) were under parole or probation for these offenses. About 7.4% of the overall sample (n = 170 752) had any behavioral health service use. Mental health treatment in the behavioral health system was received by 5.5% of the overall sample (n = 126 708), specialty OUD treatment with medication was received by 1.1% (n = 25 780), specialty OUD treatment without medication was received by 0.5% (n = 11 793), and 0.8% (n = 18 806) received specialty substance use disorder with medication. Criminal justice indicators were consistently more than 3 times more prevalent for individuals receiving behavioral health services than in the full sample (eg, any arrest: 1.4% [n = 2320] vs 0.4% [n = 8894], respectively) and, conversely, receipt of specialty behavioral health services was higher among those with criminal justice involvement (eg, any mental health service: 13.9% [n = 3419] vs 5.5% [n = 126 708] in the full sample).

Crude Fatal and Nonfatal Overdose Risk

Overall, 1204 individuals (0.05%) in the sample had a fatal overdose in 2016, and 8430 (0.37%) had 1 or more nonfatal overdoses (Table 1). Fatal and nonfatal overdose rates were lowest in the PDMP sample (respectively 883 [0.06%] and 6271 [0.41%]). The fatal overdose rate was highest in the behavioral health sample (106 [0.43%]), and the nonfatal overdose rate was highest in the criminal justice sample (635 [2.6%]).

Odds of Overdose

Odds ratios for models incorporating risk factors from all databases are shown in Table 2. Among demographic variables, male sex was the factor most strongly associated with both fatal overdose (odds ratio [OR], 2.40 [95% CI, 2.08-2.76]) and nonfatal overdose (OR, 1.41 [95% CI, 1.34-1.47]). Individuals aged 35 to 49 years had lower odds of nonfatal overdose than those aged 18 to 34 years (OR, 0.74 [95% CI, 0.70-0.79]), but this was not true for fatal overdose (OR, 1.11 [95% CI, 0.99-1.28]). The only prescription variables that was associated with fatal overdose were receiving any long-acting opioids (OR, 1.42 [95% CI, 1.11-1.73]), buprenorphine prescriptions for OUD (OR, 2.13 [95% CI, 1.71-2.81]), and benzodiazepine prescriptions (OR, 1.64 [95% CI, 1.42-1.85]). Patterns were generally similar for nonfatal overdose, except that having 4 or more short-acting opioids (vs none) was also associated with nonfatal overdose (OR, 1.44 [95% CI, 1.11-1.98]).

All hospital indicators were significantly associated with both fatal and nonfatal overdoses. Compared with no emergency department visits, 1 or 2 visits (OR, 1.55 [95% CI, 1.32-1.77]) and 3 or more visits (OR, 1.35 [95% CI, 1.09-1.70]) were associated with increased fatal overdose risk. The strongest substance use variable associated with fatal overdose was a hospital visit for OUD (OR, 2.93 [95% CI, 2.17-3.80]) and nonfatal overdose in 2016 (OR, 3.04 [95% CI, 2.26-3.94]). Patterns were similar for nonfatal overdose. Having 3 or more visits to the emergency department was strongly associated with nonfatal overdose (OR, 3.89 [95% CI, 3.47-4.45]).

The factor most strongly associated with fatal overdose was release from prison in 2015 (OR, 4.23 [95% CI, 2.10-7.11]). Parole/probation was significantly associated with fatal overdose risk (OR, 2.00 [95% CI, 1.53-2.71]). Arrests for any drug offenses, compared with property offenses, were not significantly associated with increased fatal overdose risk, but having a drug felony was associated with protection relative to property offenses (OR, 0.37 [95% CI, 0.07-1.37]). Similar associations were found for nonfatal overdose risk, except that there was no protective association with drug felonies. Finally, all variables indicating behavioral health system use were associated with significant increases in fatal and nonfatal overdose risk. Among the variables with highest associations with nonfatal overdose were receipt of specialty OUD treatments with medication (OR, 4.20 [95% CI, 3.67-4.65]) and without medication (OR, 4.21 [95% CI, 3.63-4.80]). Further details are in eTables 5 through 8 in the Supplement.

Model Performance

Comparing model performance for fatal overdose in 2016, the model with the lowest AUC was demographics alone (0.69), followed by demographics and PDMP variables (0.79), and demographics and hospital variables (0.82) (Table 3). The model that included demographics, PDMP, and hospital variables had an AUC of 0.86. Adding behavioral health indicators improved fit (0.89); however, there was no substantial improvement when adding criminal justice variables to this model (0.89).

The AUC statistics followed a similar pattern for nonfatal overdose, but improvements in model fit were more substantial when comparing the model with demographics alone (0.58) with models that included either the PDMP (0.73), hospital data (0.77), or both (0.82). Incorporating behavioral health care utilization slightly increased the AUC (0.85), but there was no further substantial improvement by adding criminal justice variables (0.85).

Models for both fatal and nonfatal overdose that included all variables demonstrated high specificity and negative predictive value (both >99% in all cases), but relatively low sensitivity (from 5.6% to 26.6% in fatal overdoses and from 4.7% to 24.7% in nonfatal overdoses) and positive predictive value (from 1.4% to 3.0% in fatal doses and from 9.1% to 17.0% in nonfatal overdoses; Table 4). This indicates that individuals identified as low risk generally did not experience the outcomes, but also that the models would classify many individuals who did not experience the outcome as high risk. Even when applying the most restrictive threshold of 0.1%, the model had a positive predictive value of 3.0% for fatal overdose (ie, only 3.0% of those screened as high risk would experience fatal overdose) and 17.0% for nonfatal overdose. The sensitivity indicates what percentage of all true positive outcomes would be captured at each threshold. At a threshold of 0.1%, the fatal overdose model identifies 5.6% of all cases, and at the same threshold for nonfatal overdose, 4.7% of all cases are identified. Applying less restrictive thresholds results in slight decreases in specificity, but substantial increases in sensitivity: a 1% threshold correctly captures 26.6% of all fatal overdose cases and 24.7% of all nonfatal overdose cases.

Sensitivity Analyses

We conducted several sensitivity analyses. The AUC statistics substantially decreased when removing (1) measures of opioid dose and (2) variables of high risk that were present for less than 2% of the sample (eg, OUD visits). To examine the implications for a specific payer, we reexamined our main model focused on individuals with opioids reimbursed by Medicaid and found this model could accurately predict both fatal overdose (AUC, 0.85) and nonfatal overdose (AUC, 0.85) in the next year. The main model could successfully predict fatal polydrug overdose (ie, with an opioid and another drug; AUC, 0.89).

Discussion

Using 4 linked administrative databases in Maryland, we created a predictive risk model of fatal and nonfatal opioid-associated overdose risk. In our most comprehensive model, we obtained an AUC of 0.89 for fatal overdose and an AUC of 0.85 for nonfatal overdose. Individuals in the top 1% of the risk distribution accounted for more than one-quarter of all fatal overdoses in the sample. This study builds on prior PRMs by including individuals not necessarily identified in individual payer and prescription records. The PDMP-derived variables improved model fit but were not all statistically significant.

Overall, all-payer hospital data had greater predictive utility than the PDMP. This is likely because hospital records capture a large population with both highly prevalent risk indicators of moderate risk (eg, visits for nonoverdose-associated injuries), as well as acute risk indicators, such as nonfatal overdose. Hospitals have become an increasingly important focus for overdose reduction efforts because of their frequent contact with survivors of overdose and other substance use–associated injuries, such as skin abscesses from injecting drugs.13,16

Specialty behavioral health treatment data modestly improved model fit, but criminal justice records did not. This is likely because individuals who are justice involved made up a small proportion of our overall sample, despite high overdose risk. Within the behavioral health treatment group, the greatest markers of risk were associated with substance use disorder treatment. Although medication treatment is known to reduce overdose risk among people with OUD,17 it was associated with higher risk of overdose in our sample, likely because it is also a proxy for more severe and longer-term disorders.

Despite the relatively high AUC scores, our models had poor positive predictive values. In our highest-performing model, the positive predictive values for fatal overdose when applying a threshold of 0.1% was only 3.0%, and for nonfatal overdose, it was 17.0%. Our model makes broad predictions at a population level, which could be useful when public health agencies are assessing, for example, how to target naloxone programs across communities. However, our models cannot accurately predict levels of risk among people with specific markers, such as people with diagnosed OUD. Our model has inferior sensitivity and specificity to some prior models developed for discrete populations (eg, patients receiving opioid medications),7,9 which may be because of either the greater unmeasured heterogeneity of our study sample or measurement error in key domains. Further, modeling may be improved by considering issues, such as temporal sequence of events (eg, timing of addiction treatment discontinuation or opioid pharmacotherapy relative to death) as have been considered in other large population studies.18,19 Risk prediction may also be improved by examining how known risk factors, such as male sex, interact with other markers of risk.

Even with low positive predictive values, the model has utility for risk stratification. Public health programs are currently in an era of massive scale-up. For example, under the 21st Century Cures Act of 2016, states were called on to spend time-limited grant funds through the State Targeted Response to the Opioid Crisis grants and State Opioid Response Grants for overdose reduction efforts.20 States must determine how to prioritize among groups that could achieve benefit from treatment and harm reduction programs. Going forward, our model can suggest specific populations that can benefit from additional resources, particularly in coordinated responses across systems, such as criminal justice systems and hospitals.

Our study underscores the importance of considering multiple risk factors, particularly the integration of PDMP and hospital data. Integration of PDMP with high-risk flags (eg, recent discharges for nonfatal overdose) has already been implemented in Maryland and is likely to become more widespread through data exchanges. Certain types of information, such as criminal justice involvement and behavioral health treatment receipt, are especially sensitive and may be legally restricted.21 However, adding behavioral health and criminal justice variables to the model did not substantially improve model discrimination, which could indicate that their inclusion is not crucial for predicting overdose risk.

Limitations

While this study has important innovations, including the use of a large, linked patient database, it is subject to several limitations that may either limit generalizability or introduce bias. First, in an observational study using secondary data, risk indicators are not causal. Because our sample represents a group with higher-than-typical risk of overdose, some of the observed associations may not generalize to the statewide population. Second, the database did not include settings that could inform risk prediction, including emergency medical services and outpatient care with private, community physicians. The criminal justice data exclude pretrial detention and individuals who exclusively have been charged with certain offense categories, such as violent offenses. The behavioral health data did not include treatment received in programs that exclusively accept private payment. The data also lack information on access to harm reduction programs, including naloxone distribution. Third, the data are missing information on individual sociodemographics (eg, race/ethnicity, education, and income). Fourth, the study data were collected in 2015 and 2016 in Maryland and may therefore not generalize to the current context or other states. Fifth, although illicit-opioid and prescription-opioid overdoses may have different prevention strategies,18,19 they are often difficult to differentiate with medical examiner records. Our model does not distinguish these 2 types of events.

Conclusions

In this analysis, we have demonstrated that a predictive model for opioid overdose can be created using linked administrative data. These models can optimize program planning and resource allocation decisions. For example, outreach efforts can focus on individuals with multiple markers of risk, such as justice involved people with recent hospital exposure. Pairing effective interventions with objective risk data can steer limited resources toward lifesaving opportunities.

Back to top
Article Information

Accepted for Publication: April 10, 2020.

Corresponding Author: Brendan Saloner, PhD, Department of Health Policy and Management, Johns Hopkins School of Public Health, 624 N Broadway, Room 344, Baltimore, MD 21205 (bsaloner@jhu.edu).

Published Online: June 24, 2020. doi:10.1001/jamapsychiatry.2020.1689

Author Contributions: Dr Saloner had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Saloner, Chang, Weiner.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Saloner, Chang.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Chang, Richards.

Obtained funding: Saloner, Ferris, Baier, Weiner.

Administrative, technical, or material support: Chang, Krawczyk, Ferris, Richards, Schneider, Weiner.

Supervision: Saloner, Baier, Weiner.

Other—data management: Krawczyk.

Conflict of Interest Disclosures: All authors reported grants from the US Department of Justice Bureau of Justice Assistance during the conduct of the study. Drs Chang, Saloner, and Eisenberg also reported grants from the National Institute on Drug Abuse during the conduct of the study. Dr Eisenberg also reported grants from the Agency for Healthcare Research and Quality and the Arnold Foundation outside the submitted work. Dr Saloner also reported grants from the Arnold Foundation outside the submitted work. No other disclosures were reported.

Funding/Support: This project was supported by the Bureau of Justice Assistance (grant 2015-PM-BX-K002). This research was also supported by the National Institute on Drug Abuse (grants 5T32DA007292 [Drs Schneider and Krawczyk] and F31DA047021 [Dr Krawczyk]).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: Points of view or opinions in this document are those of the author and do not necessarily represent the official position or policies of the US Department of Justice, the Maryland Department of Health, or other state agencies.

Additional Contributions: The authors also wish to thank all Maryland state agencies that provided data for this project.

Additional Information: The Bureau of Justice Assistance is a component of the Department of Justice's Office of Justice Programs, which also includes the Bureau of Justice Statistics, the National Institute of Justice, the Office of Juvenile Justice and Delinquency Prevention, the Office for Victims of Crime, and the SMART Office.

References
1.
Hedegaard  H, Minino  A, Warner  M. Drug overdose deaths in the United States, 1999–2017. Published 2018. Accessed August 14, 2019. https://www.cdc.gov/nchs/data/databriefs/db329-h.pdf.
2.
National Center for Health Statistics. Provisional Drug Overdose Death Counts. Published 2019. Accessed August 14, 2019. https://www.cdc.gov/nchs/nvss/vsrr/drug-overdose-data.htm
3.
Saloner  B, McGinty  EE, Beletsky  L,  et al.  A public health strategy for the opioid crisis.   Public Health Rep. 2018;133(1_suppl):24S-34S. doi:10.1177/0033354918793627PubMedGoogle ScholarCrossref
4.
Geissert  P, Hallvik  S, Van Otterloo  J,  et al.  High-risk prescribing and opioid overdose: prospects for prescription drug monitoring program-based proactive alerts.   Pain. 2018;159(1):150-156. doi:10.1097/j.pain.0000000000001078PubMedGoogle ScholarCrossref
5.
Liang  Y, Goros  MW, Turner  BJ.  Drug overdose: differing risk models for women and men among opioid users with non-cancer pain.   Pain Med. 2016;17(12):2268-2279. doi:10.1093/pm/pnw071PubMedGoogle ScholarCrossref
6.
Bohnert  ASB, Logan  JE, Ganoczy  D, Dowell  D.  A detailed exploration into the association of prescribed opioid dosage and overdose deaths among patients with chronic pain.   Med Care. 2016;54(5):435-441. doi:10.1097/MLR.0000000000000505PubMedGoogle ScholarCrossref
7.
Glanz  JM, Narwaney  KJ, Mueller  SR,  et al.  Prediction model for two-year risk of opioid overdose among patients prescribed chronic opioid therapy.   J Gen Intern Med. 2018;33(10):1646-1653. doi:10.1007/s11606-017-4288-3PubMedGoogle ScholarCrossref
8.
Zedler  BK, Saunders  WB, Joyce  AR, Vick  CC, Murrelle  EL.  Validation of a screening risk index for serious prescription opioid-induced respiratory depression or overdose in a US commercial health plan claims database.   Pain Med. 2018;19(1):68-78. doi:10.1093/pm/pnx009PubMedGoogle ScholarCrossref
9.
Lo-Ciganic  W-H, Huang  JL, Zhang  HH,  et al.  Evaluation of machine-learning algorithms for predicting opioid overdose risk among Medicare beneficiaries with opioid prescriptions.   JAMA Netw Open. 2019;2(3):e190968. doi:10.1001/jamanetworkopen.2019.0968PubMedGoogle Scholar
10.
von Elm  E, Altman  DG, Egger  M, Pocock  SJ, Gøtzsche  PC, Vandenbroucke  JP; STROBE Initiative.  The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.   J Clin Epidemiol. 2008;61(4):344-349. doi:10.1016/j.jclinepi.2007.11.008PubMedGoogle ScholarCrossref
11.
Caudarella  A, Dong  H, Milloy  MJ, Kerr  T, Wood  E, Hayashi  K.  Non-fatal overdose as a risk factor for subsequent fatal overdose among people who inject drugs.   Drug Alcohol Depend. 2016;162:51-55. doi:10.1016/j.drugalcdep.2016.02.024PubMedGoogle ScholarCrossref
12.
Chang  H-Y, Krawczyk  N, Schneider  KE,  et al.  A predictive risk model for nonfatal opioid overdose in a statewide population of buprenorphine patients.   Drug Alcohol Depend. 2019;201:127-133. doi:10.1016/j.drugalcdep.2019.04.016PubMedGoogle ScholarCrossref
13.
Krawczyk  N, Eisenberg  M, Schneider  KE,  et al.  Predictors of overdose death among high-risk hospital patients with substance-related encounters: a data linkage cohort study.   Ann Emerg Med. 2020;75(1):1-12. doi:10.1016/j.annemergmed.2019.07.014PubMedGoogle ScholarCrossref
14.
Pencina  MJ, D’Agostino  RB  Sr.  Evaluating discrimination of risk prediction models: the C statistic.   JAMA. 2015;314(10):1063-1064. doi:10.1001/jama.2015.11082PubMedGoogle ScholarCrossref
15.
Maryland Department of Planning. U.S. census bureau population estimates by age, race and gender April 1, 2010 to July 1, 2018. Published 2019. Accessed August 14, 2019. https://planning.maryland.gov/MSDC/Pages/pop_estimate/CensPopEst.aspxx
16.
Weimer  M, Morford  K, Donroe  J.  Treatment of opioid use disorder in the acute hospital setting: a critical review of the literature (2014–2019).   Curr Addict Rep. 2019;(July). doi:10.1007/s40429-019-00267-xGoogle Scholar
17.
Committee on Medication-Assisted Treatment for Opioid Use Disorder.  Medications for Opioid Use Disorder Save Lives. National Academies of Science, Engineering, and Medicine; 2019.
18.
Walley  AY, Bernson  D, Larochelle  MR, Green  TC, Young  L, Land  T.  The contribution of prescribed and illicit opioids to fatal overdoses in Massachusetts, 2013-2015.   Public Health Rep. 2019;134(6):667-674. doi:10.1177/0033354919878429PubMedGoogle ScholarCrossref
19.
Nechuta  SJ, Tyndall  BD, Mukhopadhyay  S, McPheeters  ML.  Sociodemographic factors, prescription history and opioid overdose deaths: a statewide analysis using linked PDMP and mortality data.   Drug Alcohol Depend. 2018;190:62-71. doi:10.1016/j.drugalcdep.2018.05.004PubMedGoogle ScholarCrossref
20.
National Association of State Alcohol and Drug Abuse Directors. NASADAD releases updated STR/SOR funding timeline. Published 2019. Accessed January 6, 2020. https://nasadad.org/2019/04/nasadad-releases-updated-str-sor-funding-timeline/
21.
McCarty  D, Rieckmann  T, Baker  RL, McConnell  KJ.  The perceived impact of 42 CFR part 2 on coordination and integration of care: a qualitative analysis.   Psychiatr Serv. 2017;68(3):245-249. doi:10.1176/appi.ps.201600138PubMedGoogle ScholarCrossref
×