Modeling Mitigation Strategies to Reduce Opioid-Related Morbidity and Mortality in the US | Addiction Medicine | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Model of US Opioid Epidemic (APOLLO)
Model of US Opioid Epidemic (APOLLO)

H indicates prescription heroin use with prior prescription opioid use; HN, heroin use without prior prescription opioid use; MAT, medications for addiction treatment.

Figure 2.  Projected Cumulative Number of Opioid Overdose Deaths Under Status Quo and Intervention Scenarios, 2020-2029
Projected Cumulative Number of Opioid Overdose Deaths Under Status Quo and Intervention Scenarios, 2020-2029
Table 1.  Projected Number of Opioid Overdose Deaths With Uncertainty Ranges by Use Disorder Category Under Status Quo, 2020-2029
Projected Number of Opioid Overdose Deaths With Uncertainty Ranges by Use Disorder Category Under Status Quo, 2020-2029
Table 2.  Projected Outcomes Associated With Interventions to Address US Opioid Epidemic vs Status Quo, 2020-2029
Projected Outcomes Associated With Interventions to Address US Opioid Epidemic vs Status Quo, 2020-2029
Table 3.  Quantitative Comparison of Outcomes for APOLLO and Prior Opioid Modelsa
Quantitative Comparison of Outcomes for APOLLO and Prior Opioid Modelsa
1.
Scholl  L, Seth  P, Kariisa  M, Wilson  N, Baldwin  G.  Drug and opioid-involved overdose deaths—United States, 2013-2017.   MMWR Morb Mortal Wkly Rep. 2018;67(5152):1419-1427. doi:10.15585/mmwr.mm675152e1 PubMedGoogle ScholarCrossref
2.
Phillips  JK, Ford  MA, Bonnie  RJ, eds; National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Health Sciences Policy; Committee on Pain Management and Regulatory Strategies to Address Prescription Opioid Abuse. Pain Management and the Opioid Epidemic: Balancing Societal and Individual Benefits and Risks of Prescription Opioid Use. National Academies Press; 2017:272-273.
3.
Rudd  RA, Seth  P, David  F, Scholl  L.  Increases in drug and opioid-involved overdose deaths—United States, 2010-2015.   MMWR Morb Mortal Wkly Rep. 2016;65(50-51):1445-1452. doi:10.15585/mmwr.mm655051e1 PubMedGoogle ScholarCrossref
4.
Olfson  M, Wall  M, Wang  S, Crystal  S, Blanco  C.  Risks of fatal opioid overdose during the first year following nonfatal overdose.   Drug Alcohol Depend. 2018;190:112-119. doi:10.1016/j.drugalcdep.2018.06.004 PubMedGoogle ScholarCrossref
5.
Jalal  H, Buchanich  JM, Roberts  MS, Balmert  LC, Zhang  K, Burke  DS.  Changing dynamics of the drug overdose epidemic in the United States from 1979 through 2016.   Science. 2018;361(6408):6408. doi:10.1126/science.aau1184 PubMedGoogle ScholarCrossref
6.
Ostling  PS, Davidson  KS, Anyama  BO, Helander  EM, Wyche  MQ, Kaye  AD.  America’s opioid epidemic: a comprehensive review and look into the rising crisis.   Curr Pain Headache Rep. 2018;22(5):32. doi:10.1007/s11916-018-0685-5 PubMedGoogle ScholarCrossref
7.
Pitt  AL, Humphreys  K, Brandeau  ML.  Modeling health benefits and harms of public policy responses to the US opioid epidemic.   Am J Public Health. 2018;108(10):1394-1400. doi:10.2105/AJPH.2018.304590 PubMedGoogle ScholarCrossref
8.
Wakeland  W, Nielsen  A, Geissert  P.  Dynamic model of nonmedical opioid use trajectories and potential policy interventions.   Am J Drug Alcohol Abuse. 2015;41(6):508-518. doi:10.3109/00952990.2015.1043435 PubMedGoogle ScholarCrossref
9.
Chen  Q, Larochelle  MR, Weaver  DT,  et al.  Prevention of prescription opioid misuse and projected overdose deaths in the United States.   JAMA Netw Open. 2019;2(2):e187621. doi:10.1001/jamanetworkopen.2018.7621 PubMedGoogle Scholar
10.
Irvine  MA, Kuo  M, Buxton  JA,  et al.  Modelling the combined impact of interventions in averting deaths during a synthetic-opioid overdose epidemic.   Addiction. 2019;114(9):1602-1613. doi:10.1111/add.14664 PubMedGoogle ScholarCrossref
11.
National Institute on Alcohol Abuse and Alcoholism. Rates of nonmedical prescription opioid use and opioid use disorder double in 10 years. Published June 22, 2016. Accessed December 10, 2018. https://www.niaaa.nih.gov/news-events/news-releases/rates-nonmedical-prescription-opioid-use-and-opioid-use-disorder-double-10
12.
Moran  A, Gu  D, Zhao  D,  et al.  Future cardiovascular disease in china: Markov model and risk factor scenario projections from the coronary heart disease policy model-china.   Circ Cardiovasc Qual Outcomes. 2010;3(3):243-252. doi:10.1161/CIRCOUTCOMES.109.910711PubMedGoogle ScholarCrossref
13.
Cobb  CO, Villanti  AC, Graham  AL,  et al.  Markov modeling to estimate the population impact of emerging tobacco products: a proof-of-concept study.   Tob Regul Sci. 2015;1(2):129-141. doi:10.18001/TRS.1.2.3Google ScholarCrossref
14.
Census  US. Accessed January 23, 2020. https://www.census.gov
15.
Centers for Disease Control and Prevention, National Center for Health Statistics. CDC WONDER. About multiple cause of death, 1999-2017. Accessed January 23, 2020. https://wonder.cdc.gov/mcd-icd10.html
16.
US Dept of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality. Results from the 2017 National Survey on Drug Use and Health: detailed tables. Accessed January 23, 2020. https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHDetailedTabs2017/NSDUHDetailedTabs2017.pdf
17.
Grant  BF, Amsbary  M, Chu  A,  et al  Source and Accuracy Statement: National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III). National Institute on Alcohol Abuse and Alcoholism; 2014.
18.
Dyer  O.  OxyContin maker stops marketing opioids, as report details payments to advocacy groups.   BMJ. 2018;360:k791. doi:10.1136/bmj.k791 PubMedGoogle ScholarCrossref
19.
Drug Enforcement Agency, Diversion Control Division. FR Doc No: 2018-15141. Accessed January 23, 2020. https://www.deadiversion.usdoj.gov/fed_regs/rules/2018/fr0716_3.htm
20.
Dowell  D, Haegerich  TM, Chou  R.  CDC guideline for prescribing opioids for chronic pain—United States, 2016.   JAMA. 2016;315(15):1624-1645. doi:10.1001/jama.2016.1464 PubMedGoogle ScholarCrossref
21.
Midgette G, Davenport S, Caulkins JP, Kilmer B. What America’s users spend on illegal drugs, 2006-2016. RAND. Published 2019. Accessed August 17, 2020. https://www.rand.org/content/dam/rand/pubs/research_reports/RR3100/RR3140/RAND_RR3140.pdf
22.
Alexander  GC, Frattaroli  S, Gielen  AC, eds.  The Opioid Epidemic: From Evidence to Impact. Johns Hopkins Bloomberg School of Public Health; 2017.
23.
Gramlich  J. What the data says about gun deaths in the US. Pew Research Center. Published August 16, 2019. Accessed January 23, 2020. https://www.pewresearch.org/fact-tank/2019/08/16/what-the-data-says-about-gun-deaths-in-the-u-s/
24.
McLellan  AT, Lewis  DC, O’Brien  CP, Kleber  HD.  Drug dependence, a chronic medical illness: implications for treatment, insurance, and outcomes evaluation.   JAMA. 2000;284(13):1689-1695. doi:10.1001/jama.284.13.1689 PubMedGoogle ScholarCrossref
25.
Mackey  K, Veazie  S, Anderson  J, Bourne  D, and Peterson  K. Evidence brief: barriers and facilitators to use of medications for opioid use disorder. Evidence Synthesis Program, Health Services Research and Development Service, Office of Research and Development, Department of Veterans Affairs. VA ESP Project #09-009; 2019.
26.
Friedmann  PD, Alexander  JA, Yey  Y, Nahra  T, Soliman  S, Pollack  HA.  Duration of nonmethadone outpatient treatment: results from a national survey.   Subst Abus. 2006;27(3):47-53. doi:10.1300/J465v27n03_07 PubMedGoogle ScholarCrossref
27.
Andrews  CM, Abraham  AJ, Grogan  CM, Westlake  MA, Pollack  HA, Friedmann  PD.  Impact of Medicaid restrictions on availability of buprenorphine in addiction treatment programs.   Am J Public Health. 2019;109(3):434-436. doi:10.2105/AJPH.2018.304856 PubMedGoogle ScholarCrossref
28.
Morgan  DJ, Dhruva  SS, Wright  SM, Korenstein  D.  2016 Update on medical overuse: a systematic review.   JAMA Intern Med. 2016;176(11):1687-1692. doi:10.1001/jamainternmed.2016.5381 PubMedGoogle ScholarCrossref
29.
Subramani  MS.  Capsule Commentary on Calcaterra et al., Opioid prescribing at hospital discharge contributes to chronic opioid use.   J Gen Intern Med. 2016;31(5):532-532. doi:10.1007/s11606-016-3587-4 PubMedGoogle ScholarCrossref
30.
Kolodny  A, Courtwright  DT, Hwang  CS,  et al.  The prescription opioid and heroin crisis: a public health approach to an epidemic of addiction.   Annu Rev Public Health. 2015;36:559-574. doi:10.1146/annurev-publhealth-031914-122957 PubMedGoogle ScholarCrossref
31.
Compton  WM, Jones  CM, Baldwin  GT.  Relationship between nonmedical prescription-opioid use and heroin use.   N Engl J Med. 2016;374(2):154-163. doi:10.1056/NEJMra1508490 PubMedGoogle ScholarCrossref
32.
Caulkins  JP.  Models pertaining to how drug policy should vary over the course of a drug epidemic.   Adv Health Econ Health Serv Res. 2005;16:397-429. doi:10.1016/S0731-2199(05)16019-1 PubMedGoogle ScholarCrossref
33.
Alpert  A, Powell  D, Pacula  RL.  Supply-side drug policy in the presence of substitutes: evidence from the introduction of abuse-deterrent opioids.   Am Econ J Econ Policy. 2018;10:1-35. doi:10.1257/pol.20170082Google ScholarCrossref
34.
Kilmer  B, Everingham  S, Caulkins  J .  What America’s Users Spend on Illegal Drugs: 2000-2010. Rand Corporation; 2014. doi:10.7249/RR534
35.
Pardo  B, Taylor J,Caulkins   JP, et al. The Future of Fentanyl and Other Synthetic Opioids. RAND Corporation. Published 2019. Accessed January 23, 2020. https://www.rand.org/pubs/research_reports/RR3117.html
36.
Chang  HY, Lyapustina  T, Rutkow  L,  et al.  Impact of prescription drug monitoring programs and pill mill laws on high-risk opioid prescribers: a comparative interrupted time series analysis.   Drug Alcohol Depend. 2016;165:1-8. doi:10.1016/j.drugalcdep.2016.04.033 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Substance Use and Addiction
    November 4, 2020

    Modeling Mitigation Strategies to Reduce Opioid-Related Morbidity and Mortality in the US

    Author Affiliations
    • 1Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
    • 2Center for Drug Safety and Effectiveness, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
    • 3Monument Analytics, Baltimore, Maryland
    • 4The University of Chicago School of Social Service Administration, Chicago, Illinois
    • 5Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
    • 6Department of Medicine, Johns Hopkins Medicine, Baltimore, Maryland
    JAMA Netw Open. 2020;3(11):e2023677. doi:10.1001/jamanetworkopen.2020.23677
    Key Points

    Question  What is the projected burden of the opioid epidemic in fatal overdoses, and interventions such as prescribing reductions, naloxone distribution, and treatment expansion associated with mitigation of the epidemic?

    Findings  In this decision analytical model of the US population aged 12 years or older, under status quo, an estimated 484 429 individuals were projected to die of fatal opioid overdose over 10 years. A combination of reducing opioid prescribing, increasing naloxone distribution, and expanding treatment for opioid use disorder was associated with an estimated 179 151 lives saved when compared with the status quo.

    Meaning  The findings of this study suggest that the number of fatal opioid overdoses in the US is expected to remain high for at least 10 years, but evidence-based interventions may prevent a substantial fraction of these deaths.

    Abstract

    Importance  The US opioid epidemic is complex and dynamic, yet relatively little is known regarding its likely future impact and the potential mitigating impact of interventions to address it.

    Objective  To estimate the future burden of the opioid epidemic and the potential of interventions to address the burden.

    Design, Setting, and Participants  A decision analytic dynamic Markov model was calibrated using 2010-2018 data from the National Survey on Drug Use and Health, Centers for Disease Control and Prevention, National Health and Nutrition Examination Survey, the US Census, and National Epidemiologic Survey on Alcohol and Related Conditions–III. Data on individuals 12 years or older from the US general population or with prescription opioid medical use; prescription opioid nonmedical use; heroin use; prescription, heroin, or combined prescription and heroin opioid use disorder (OUD); 1 of 7 treatment categories; or nonfatal or fatal overdose were examined. The model was designed to project fatal opioid overdoses between 2020 and 2029.

    Exposures  The model projected prescribing reductions (5% annually), naloxone distribution (assumed 5% reduction in case-fatality), and treatment expansion (assumed 35% increase in uptake annually for 4 years and 50% relapse reduction), with each compared vs status quo.

    Main Outcomes and Measures  Projected 10-year overdose deaths and prevalence of OUD.

    Results  Under status quo, 484 429 (95% confidence band, 390 543-576 631) individuals were projected to experience fatal opioid overdose between 2020 and 2029. Projected decreases in deaths were 0.3% with prescribing reductions, 15.4% with naloxone distribution, and 25.3% with treatment expansion; when combined, these interventions were associated with 179 151 fewer overdose deaths (37.0%) over 10 years. Interventions had a smaller association with the prevalence of OUD; for example, the combined intervention was estimated to reduce OUD prevalence by 27.5%, from 2.47 million in 2019 to 1.79 million in 2029. Model projections were most sensitive to assumptions regarding future rates of fatal and nonfatal overdose.

    Conclusions and Relevance  The findings of this study suggest that the opioid epidemic is likely to continue to cause tens of thousands of deaths annually over the next decade. Aggressive deployment of evidence-based interventions may reduce deaths by at least a third but will likely have less impact for the number of people with OUD.

    Introduction

    In 2017, approximately 47 600 individuals in the US died from an opioid overdose,1 and morbidity and mortality from the opioid epidemic continues to accrue. Because of the epidemic’s magnitude and scope, it is important to understand as much as possible about how the crisis may evolve, including the interplay of factors accounting for injuries and deaths. It is also important to understand the potential impact of measures designed to avert future harms. While no epidemiologic model can capture every aspect of the epidemic, formal models allow for systematic analysis of previous research and the explicit and quantitative estimation of the impact of different interventions. Models also can assist in comparing short- with long-term outcomes, examining the impact of interventions in subpopulations of interest, and quantifying the economic, as well as public health, costs, and benefits of different approaches to abate opioid-related harms.2

    To fully estimate the epidemic’s scope and the impact of interventions to address it, it is essential to consider differences in individuals using prescription opioids vs heroin or illicit fentanyl,3 the increased risk of second overdose in people who have experienced an initial overdose,4 and the evolving time-dependent nature of the epidemic.5,6 Despite the contributions of prior models of the epidemic,7-10 most have not incorporated these elements, nor have they accounted for the more than 2.5 million individuals in the US who report lifetime—but not past year—opioid use disorder (OUD).11 In addition, earlier models have tended to regard treatment of OUD as a single entity rather than differentiating among different phases of treatment or recovery to allow for flexible modeling of different subpopulations, such as those receiving more or less intensive care.

    We therefore constructed a dynamic decision analytic Markov model of the opioid epidemic in the US, incorporating these elements, to provide updated estimates of the future magnitude of the epidemic and project the potential association of key interventions with mitigation of the epidemic.

    Methods

    Our model (APOLLO) was designed to capture the associations underpinning the epidemiologic nature of the opioid epidemic. Similar dynamic Markov models have been used to better understand other complex phenomena ranging from risk factor changes for cardiovascular disease12 to the population-level impact of electronic cigarettes and other novel tobacco products.13 APOLLO’s conceptual framework was developed using an iterative process soliciting scientific and clinical input from experts in addiction and pain medicine, public health, health economics, epidemiologic factors, and health policy. The model consists of 32 compartments distinguishing 7 major populations: (1) no opioid use (general population in Figure 1); (2) prescription opioid medical use; (3) prescription opioid nonmedical use; (4) use of heroin, illicit fentanyl, or other illicit opioids; (5) OUD from prescription opioids; (6) heroin use disorder with prior prescription opioid use (HUD-Rx); and (7) heroin use disorder without prior prescription opioid use (HUD-NonRx). Note that, in all compartments, heroin use includes use of illicit fentanyl and other nonprescription opioids. The model also provides for 7 subpopulations with prescription OUD, HUD-Rx, and HUD-NonRx based on degrees of clinical stability and treatment engagement.

    The model, which begins in January 2010 and extends through December 2029 using monthly time-steps, was constructed in Microsoft Excel 2019, version 16.41 (Microsoft Corp) with sensitivity analyses performed using @Risk, version 8 (Palisade Software). The reporting of this study is in accordance with the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) reporting guideline. Our study involved the analysis of data that were recorded such that individuals could not be identified and thus was exempted from institutional review board review (45 CFR 46 [4]).

    Initial Populations

    The initial population for the 32 compartments was estimated using 1 of 4 national databases (US Census,14 Centers for Disease Control and Prevention Wide-Ranging Online Data for Epidemiologic Research,15 National Survey on Drug Use and Health [NSDUH],16 and National Epidemiologic Survey on Alcohol and Related Conditions–III17). For estimates of the active OUD population, we relied on NSDUH data that defines OUD based on specific Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, or Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, diagnostic criteria within a 12-month window. We also used National Epidemiologic Survey on Alcohol and Related Conditions data to estimate lifetime OUD prevalence, assuming an increased risk of relapsing to active OUD. For populations in which 2010 information was missing, we used the next closest year with available data and adjusted for changes in the epidemic using changes in overdose deaths. The eAppendix in the Supplement includes a description of each initial population and source.

    State Transitions

    The model consisted of 109 monthly transitions between the 32 compartments. These included 25 time-dependent transitions reflecting changes in the epidemic, such as increased use of fentanyl and increasing lethality of opioids, decreased prescribing of opioid medications, increased access to medications for addiction treatment (MAT), and population growth in the US. We estimated transition probabilities from data sources including large national databases, peer-reviewed literature, and when necessary, expert opinion. Several key probabilities were calculated using the difference between respondents’ past-month and past-year self-reported opioid use in the NSDUH. Transition probabilities between heroin and prescription OUD compartments were estimated with data on individuals using prescription opioids before initiation of heroin. After transitions were initially defined, we calibrated the model using data from 2010-2018, during which some transitions were incrementally adjusted to better align model outputs with actual data. The eAppendix in the Supplement includes a description of each transition probability and source and eFigure 1 in the Supplement provides an example of select transition probabilities for individuals with prescription OUD.

    Model Calibration

    After programming the model structure, initial populations, and transitions, the model was calibrated against 13 populations from 2010 to 2018, with priority given to 4 populations based on the strength of evidence estimating their size and/or their importance to policymakers: total population, total active OUD, total prescription OUD, and total annual overdose deaths (eTable in the Supplement). We used 3 data sets to calibrate the model and derive information regarding key populations of interest. First, we used US Census data to derive information regarding the population of individuals in the US aged 12 years or older from 2010 to 2018. Second, we used NSDUH data to derive estimates of the numbers of people with prescription opioid nonmedical use, heroin use, and prescription OUD, adjusting for NSDUH’s double-counting of individuals who have both prescription OUD and HUD. Third, we used 2010 to 2017 CDC Wide-Ranging Online Data for Epidemiologic Research to capture estimates of opioid overdose deaths.

    Simulation Strategies

    We simulated 3 interventions set to begin in 2020; in the cases of naloxone distribution and treatment expansion, we simulated these policies by making assumptions about the outcome of such policies. First, we simulated the outcomes associated with reducing opioid prescribing by 5% annually from 2020 to 2029 beyond the status quo, which is equivalent to a total reduction of approximately 40% over 10 years. This intervention reflects policies such as reductions in marketing and promotion,18 stricter Drug Enforcement Agency quotas,19 increased promulgation of clinical guidelines,20 and any number of other local, state, and federal initiatives. Second, we simulated the outcomes associated with policies to expand the distribution and use of naloxone. To do so, we assumed a 5% annual reduction in overdose case fatality over 4 years (2020-2023) and then sustained for 6 years (2024-2029), which is equivalent to a total reduction of approximately 19% over 10 years. Third, we modeled the outcome of treatment expansion among individuals with OUD, evaluating a policy that increased the initiation of medications for MAT by 35% annually from 2020 to 2023 and then sustained these increases through 2029, which is equivalent to a tripling of MAT uptake over 10 years. Our MAT initiation begins with detoxification before an individual is prescribed MAT. We combined this latter intervention with one that decreased treatment relapse out of MAT by 50% for 10 years starting in 2020.

    Outcomes

    We used the model to calculate the annual and cumulative number of overdose deaths from 2020 to 2029 stratified by any opioid, prescription opioids, or heroin and other illicit opioids with and without prior prescription opioid use. We also projected the size of the 7 major populations described above over the same period.

    Sensitivity Analysis

    We performed univariate and multivariate (probabilistic) sensitivity analyses; the univariate analyses tested the sensitivity of model outcomes to a change in each state transition variable. Such sensitivity analyses served 2 main purposes. First, by clarifying the robustness and key components of our model, these analyses allowed us to identify transitions affecting model outcomes and quantify the association of variation in these parameters with our primary outcomes. Second, from a public health perspective, sensitivity analyses help to identify key policy levers and intervention points of greatest value in preventing harmful outcomes.

    Results are presented as tornado diagrams in the eAppendix in the Supplement. For the multivariate probabilistic analysis, we identified key transitions and used a mix of published evidence and expert opinion to create probability distributions for these transitions. The probability distributions took the form of β distributions. Results of the multivariate probabilistic sensitivity analysis are provided as a 95% confidence band, defined as the 2.5th and 97.5th percentiles across 1000 probabilistic simulations.

    We also included sensitivity analyses focused on heroin users, since evidence suggests that NSDUH may significantly underestimate the population of heroin users in the US.21 The results of the sensitivity analyses are described in the eAppendix in the Supplement.

    Results
    Projected Opioid Overdose and Active OUD

    Cumulatively, the model projected 484 429 (95% confidence band, 390 543-576 631) overdose deaths from any opioid from 2020 to 2029, of which 155 628 (95% confidence band, 106 181-208 903) deaths (32.1%; 95% confidence band, 27.2%-36.2%) are from prescription opioid medical use, including both medical and nonmedical use, 199 751 (95% confidence band, 136 253-265 402) deaths (41.2%; 95% confidence band, 34.9%-46.0%) are from heroin use with previous prescription opioid use, and 129 050 (95% confidence band, 89 754-166 926) deaths (26.7%; 95% confidence band, 23.0%-28.9%) are from heroin use without previous prescription opioid use (Table 1).

    The model projected a slight annual increase in the number of overdose deaths from any opioid, estimating 46 735 (95% confidence band, 38 004-55 888) deaths in 2020 increasing to 50 300 (95% confidence band, 40 147-60 372) deaths in 2029. Trends varied across subpopulations. Prescription opioid overdose deaths decreased by approximately 800 deaths from 15 992 (95% confidence band, 10 915-21 034) deaths in 2020 to 15 170 (95% confidence band, 10 309-20 539) deaths in 2029, a 5.1% (95% confidence band, 2.4%-5.6%) decrease. By contrast, heroin overdose deaths among people with previous prescription opioid use (HUD-Rx) increased steadily from 18 149 (95% confidence band, 12 426-23 835) deaths in 2020 to 21 838 (95% confidence band, 14 629-29 416) deaths in 2029, a 20.3% (95% confidence band, 17.7%-23.4%) increase. Similarly, heroin overdose deaths among people who never used prescription opioids (HUD-NonRx) increased from 12 594 (95% confidence band, 8624-16 454) deaths in 2020 to 13 293 (95% confidence band, 9362-17 026) deaths in 2029, a 5.6% (95% confidence band, 3.5%-8.6%) increase.

    Under the status quo, there were approximately 2.47 million individuals with active OUD in December 2019, of whom 2.17 million (87.9%) were not receiving treatment. By December 2029, the model predicted approximately 2.56 (95% confidence band, 2.41-2.75) million individuals with active OUD, of whom 2.25 (95% confidence band, 2.12, 2.42) million (87.9%; 95% confidence band, 87.7%-88.1%) were receiving on treatment.

    Projected Outcomes Associated With Interventions
    Outcomes Associated With Overdose Deaths

    Reducing prescribing rates of opioids had a small association with opioid overdose deaths from 2020 to 2029 (0.3% reduction in all deaths, 3.4% reduction in prescription opioid overdose deaths) compared with status quo (eFigure 2 in the Supplement).

    Increasing naloxone access was projected to be associated with a 15.4% reduction in cumulative overdose deaths, corresponding to 74 510 (95% confidence band, 60 310-87 894) fewer deaths from 2020 to 2029. Expanding MAT had the greatest projected association with cumulative opioid overdose deaths from 2020 to 2029, with a 25.3% (95% confidence band, 22.4%-27.7%) reduction (122 710 deaths; 95% confidence band, 95 451-148 335) compared with status quo, an outcome that reflects both increased MAT uptake (18.9% reduction in overdose deaths; 95% confidence band, 16.3-21.1) and reduced relapse (6.3% reduction in deaths; 95% confidence band, 5.5%-7.1%) (Table 2). MAT was associated with a 38.3% (95% confidence band, 34.4%-41.0%) projected reduction in cumulative HUD-Rx deaths (76 552 deaths; 95% confidence band, 51 827-100 320) compared with status quo. This association was due to relatively higher base rates of MAT uptake among HUD-Rx (5.3% individuals initiating MAT per month) compared with the prescription OUD population (3.0% per month).

    Combining the interventions was associated with a projected reduction in the number of opioid overdose deaths by 37.0% (95% confidence band, 34.3%-38.4%) between 2020 and 2029 compared with status quo, representing 179 151 (140 696-211 323) deaths averted (Figure 2). Reductions were achieved in all overdose categories, although they were larger among individuals who have transitioned from prescription opioids to heroin (46.6%; 95% confidence band, 45.3%-49.1%) than among heroin users who did not begin with prescription opioids (32.6%; 95% confidence band, 32.1%-33.0%) or individuals with prescription opioid use disorder (28.3%; 95% confidence band, 27.7%-28.6%)

    Outcomes Associated With Active Opioid Use Disorder

    Prescribing reductions and naloxone distribution have little association with the number of individuals with active OUD, increasing this estimate from 2.47 million in December 2019 to 2.51 million (95% confidence band, 2.40 million-2.68 million) for prescribing reductions and 2.59 million (2.44 million-2.78 million) for naloxone distribution in 2029 (eFigure 3 in the Supplement). These interventions also did not substantively change the number of individuals receiving treatment, from 431 282 in December 2019 to 452 817 (95% confidence band, 416 187-495 390) for prescribing reductions and to 471 093 (95% confidence band, 430 169-518 402) for naloxone distribution in 2029.

    Increased MAT uptake and reduced relapse were projected to decrease the number of individuals with active OUD from 2.47 million in December 2019 to 1.81 million (95% confidence band, 1.72 million-1.93 million) by December 2029, a 26.7% (95% confidence band, 22.1%-30.2%) decrease. The number of individuals receiving MAT would increase from 431 282 in December 2019 to 1.09 million (95% confidence band, 0.98 million-1.21 million) in December 2029.

    Combining all 3 interventions was associated with a projected decrease in the number of individuals with active OUD from 2.47 million, of whom 2.17 million (87.9%) are not in treatment, in December 2019 to 1.79 million (95% confidence band, 1.71 million-1.89 million), of whom 1.35 million (95% confidence band, 1.30 million-1.43 million) (75.8%; 95% confidence band, 75.6%-76.2%) are not in treatment, in December 2029. This change represents a 27.5% (95% confidence band, 23.4%-30.9%) decrease in the prevalence of OUD and a 37.7% (95% confidence band, 34.1%-40.0%) decrease in the prevalence of untreated OUD compared with status quo. The number of individuals receiving MAT would increase from 431 282 in December 2019 to 1.06 million (95% confidence band, 0.96 million-1.19 million) in December 2029 (146.7% increase; 95% confidence band, 123.4%-175.0%), with changes affected predominately by expanding MAT uptake.

    Sensitivity Analyses

    From the univariate sensitivity analysis, we found the probability of surviving an opioid overdose and the case fatality of prescription overdose to be most influential on cumulative overdose deaths from 2010 to 2029 (eFigure 4 in the Supplement). Rates of transition from medical to nonmedical prescription opioid use, heroin overdose to detoxification, and heroin overdose to death were nearly as influential. Parameters that had small influences on overdose deaths included non–opioid-related mortality rates across all populations.

    Discussion

    Despite the efforts of many stakeholders, morbidity and mortality from the opioid epidemic continue to accrue. We developed a Markov model to quantify the outcomes of prescribing reductions, naloxone distribution, and treatment expansion on fatal opioid overdoses between 2020 and 2029. Using data from standard reference data sets, our model replicated historical trends from 2010 to 2018. Under status quo, the model estimated that 484 429 individuals would experience fatal opioid overdose between 2020 and 2029. Applying 3 broad evidence-based interventions,22 our model projected 179 151 fewer deaths (37.0%) over 10 years, results that are largely driven by increasing MAT uptake and reducing treatment relapse. Our findings are important because they project the continued harm of the opioid epidemic and estimate the potential impact of concerted policy efforts to address it. By way of comparison, this comprehensive opioid strategy is projected to save more lives than an ideal intervention that could prevent all gun homicides in the US over the same period.23

    Our estimates complement other modeling efforts examining the overdose crisis.7-10 Of these, Pitt et al7 and Chen et al9 are most similar to APOLLO in terms of setting and scope; they estimate baseline 10-year cumulative overdose deaths between 513 740 and 704 000, while we estimate 413 963 to 562 252 deaths, albeit over a slightly different time period and with other differences in approach (Table 3). For example, in addition to using time-varying probabilities to more accurately capture changes in phenomena, such as the increased case fatality of overdose due to illicit fentanyl or decreasing prescribing rates, we also account for individuals with a lifetime, but not past year, history of OUD, since these individuals are prone to resumption of opioid use and attendant morbidity and mortality.24

    Our findings suggest that the opioid epidemic will exact a large burden of morbidity and mortality in the US for the coming decade, even in the face of decreasing prescription rates. While increased treatment uptake among individuals with OUD will reduce overdose rates, the public health benefits of increased treatment uptake are magnified with greater treatment continuity. Although the propensity for some level of reuse is a defining characteristic of OUD, well-described barriers also encourage dropout or hinder take-up in OUD treatment. These barriers include addiction; stigma; high out-of-pocket costs; logistical challenges, such as transportation and childcare; prior authorization reimbursement constraints; and poor interpersonal experiences with treatment professionals and staff.25,26 These and other barriers should be addressed through a number of measures, including improved training of service professionals and staff, strengthened linkages to required social services and peer supports, and more comprehensive public and private insurance coverage for OUD treatment.27

    Although there is widespread consensus that opioid oversupply has been a crucial component of the epidemic, our model suggests that reductions in opioid volume will have a relatively small direct association with overdose deaths over the next decade, as the primary component of overdose in APOLLO over this time horizon is the population with OUD—not the population using opioids for medical use. However, reducing prescription opioid volume remains important, as opioids remain widely overused,28 there is a strong and consistent association between their receipt, nonmedical use,29 and other harms,30 and the size of the OUD population, over time, may be influenced by the number of individuals receiving opioids and their transition from medical to nonmedical use.

    Our results also allow for an assessment of different populations impacted by the epidemic, including individuals with prescription opioid use, heroin use, and use disorders arising from these and other opioids, such as illicit fentanyl. While treatment expansion was associated with reductions in overdose deaths among each of these populations, higher rates of lethality in the heroin use populations suggest strong public health potential for getting individuals with heroin use disorder into treatment. This outcome is reflected in the model; results with treatment expansion were associated with a reduction in heroin deaths of 31.5% vs a 12.4% reduction in prescription opioid-related deaths.

    There is some trade-off between reducing prescription opioid volume and heroin use, although additional factors affect baseline heroin overdose rates. In addition, the magnitude of offset between prescription opioid and heroin use can be positively impacted by public policies.31 Optimal policies for individuals who use opioids nonmedically differ across the diverse life trajectories of many drug users. For example, our findings underscore the tension between policies that effectively serve the needs of current opioid users and those designed to deter naive users from becoming opioid dependent.32 Most obviously, policies to prevent opioid initiation, such as strict prescription drug caps or prescription drug monitoring program implementation, may deter nonmedical use of prescription opioids while accelerating transitions from prescription opioids to heroin and illicit fentanyl use among some individuals.

    In addition, our model estimations suggest that the total population of people living with OUD in the US will not markedly change over the next decade. These findings persist despite changes in morbidity and mortality over time and despite the availability of combined interventions addressing many elements of the epidemic.33

    Limitations

    This study had limitations. Despite the value of models such as APOLLO, our findings should be interpreted as model projections of simulated populations, with the inherent limitations of mathematical models. For example, while the parameters underlying this model are based on the best available scientific data, there are no direct data to inform the precise effect that could be or could have been achieved from any one specific intervention, and our simulation of policies, such as naloxone distribution and training, is based on assumptions regarding the impact of such policies. Also, although our model parameters are based on what we believe to be the best available data, both our point estimates and sensitivity analyses underscore gaps in available epidemiologic and survey data and highlight the value of further data collection efforts. While our calibrations closely track NSDUH estimates over the past decade, we know far less about how our model and others track street drug users and others at elevated risk. For example, NSDUH may significantly understate the prevalence and severity of heroin use,34 although our sensitivity analyses suggested the long-term impact of increasing the heroin population was modest, as the steady state equilibrium of the model is defined by other parameters. In addition, much about the trajectory of illicit drug markets remains unknown. For example, the emergence of a global internet-based market for fentanyl may worsen the opioid epidemic.35 Also, we do not attempt to account for heterogeneities across states or localities, the groups in which prescribing reductions are achieved,7,36 or the costs or cost-effectiveness of the interventions deployed.

    Conclusions

    While the opioid crisis has evolved considerably, the findings of this study suggest that the epidemic is likely to continue to exact a large toll during the next decade. Our estimates suggest that aggressive deployment of evidence-supported practices, including expanded use of medications for addiction treatment and improved naloxone distribution, may save many lives. This public health opportunity should be seized to limit the harms associated with perhaps the most serious drug use epidemic in US history.

    Back to top
    Article Information

    Accepted for Publication: August 31, 2020.

    Published: November 4, 2020. doi:10.1001/jamanetworkopen.2020.23677

    Open Access: This is an open access article distributed under the terms of the CC-BY-NC-ND License. © 2020 Ballreich J et al. JAMA Network Open.

    Corresponding Author: G. Caleb Alexander, MD, MS, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe St, W6035, Baltimore, MD 21205 (galexan9@jhmi.edu).

    Author Contributions: Dr Alexander had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Ballreich, Mansour, Pollack, Dowdy, Alexander.

    Acquisition, analysis, or interpretation of data: Ballreich, Mansour, Hu, Chingcuanco, Dowdy, Alexander.

    Drafting of the manuscript: Ballreich, Mansour, Hu, Pollack, Alexander.

    Critical revision of the manuscript for important intellectual content: Ballreich, Mansour, Chingcuanco, Pollack, Dowdy, Alexander.

    Statistical analysis: Ballreich, Mansour, Hu.

    Obtained funding: Alexander.

    Administrative, technical, or material support: Ballreich, Mansour, Hu, Chingcuanco, Pollack, Alexander.

    Supervision: Ballreich, Dowdy, Alexander.

    Conflict of Interest Disclosures: Dr Ballreich reported receiving consulting fees from Monument Analytics during the conduct of the study and is an employee of The Johns Hopkins University outside the submitted work. Mr Mansour and Ms Chingcuanco reported serving as paid employees of Monument Analytics. Ms Hu reported receiving consulting fees from Monument Analytics during the conduct of the study. Dr Pollack reported receiving consulting fees from Monument Analytics during the conduct of the study and is an employee of the University of Chicago outside the submitted work. Dr Dowdy reported receiving consulting fees from Monument Analytics during the conduct of the study and is an employee of The Johns Hopkins University outside the submitted work.

    Funding/Support: This development of the core model was funded in part by plaintiffs in opioid litigation as part of the Multidistrict Litigation (MDL 2804) in the Northern District of Ohio, United States District Court.

    Role of the Funder/Sponsor: Counsel reviewed the manuscript for confidential or privileged information but otherwise had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.

    Additional Contributions: Brendan Saloner, PhD, and Joshua Sharfstein, MD (Johns Hopkins Bloomberg School of Public Health), provided feedback on specific model features and transition probabilities but were not financially remunerated for this feedback.

    References
    1.
    Scholl  L, Seth  P, Kariisa  M, Wilson  N, Baldwin  G.  Drug and opioid-involved overdose deaths—United States, 2013-2017.   MMWR Morb Mortal Wkly Rep. 2018;67(5152):1419-1427. doi:10.15585/mmwr.mm675152e1 PubMedGoogle ScholarCrossref
    2.
    Phillips  JK, Ford  MA, Bonnie  RJ, eds; National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Health Sciences Policy; Committee on Pain Management and Regulatory Strategies to Address Prescription Opioid Abuse. Pain Management and the Opioid Epidemic: Balancing Societal and Individual Benefits and Risks of Prescription Opioid Use. National Academies Press; 2017:272-273.
    3.
    Rudd  RA, Seth  P, David  F, Scholl  L.  Increases in drug and opioid-involved overdose deaths—United States, 2010-2015.   MMWR Morb Mortal Wkly Rep. 2016;65(50-51):1445-1452. doi:10.15585/mmwr.mm655051e1 PubMedGoogle ScholarCrossref
    4.
    Olfson  M, Wall  M, Wang  S, Crystal  S, Blanco  C.  Risks of fatal opioid overdose during the first year following nonfatal overdose.   Drug Alcohol Depend. 2018;190:112-119. doi:10.1016/j.drugalcdep.2018.06.004 PubMedGoogle ScholarCrossref
    5.
    Jalal  H, Buchanich  JM, Roberts  MS, Balmert  LC, Zhang  K, Burke  DS.  Changing dynamics of the drug overdose epidemic in the United States from 1979 through 2016.   Science. 2018;361(6408):6408. doi:10.1126/science.aau1184 PubMedGoogle ScholarCrossref
    6.
    Ostling  PS, Davidson  KS, Anyama  BO, Helander  EM, Wyche  MQ, Kaye  AD.  America’s opioid epidemic: a comprehensive review and look into the rising crisis.   Curr Pain Headache Rep. 2018;22(5):32. doi:10.1007/s11916-018-0685-5 PubMedGoogle ScholarCrossref
    7.
    Pitt  AL, Humphreys  K, Brandeau  ML.  Modeling health benefits and harms of public policy responses to the US opioid epidemic.   Am J Public Health. 2018;108(10):1394-1400. doi:10.2105/AJPH.2018.304590 PubMedGoogle ScholarCrossref
    8.
    Wakeland  W, Nielsen  A, Geissert  P.  Dynamic model of nonmedical opioid use trajectories and potential policy interventions.   Am J Drug Alcohol Abuse. 2015;41(6):508-518. doi:10.3109/00952990.2015.1043435 PubMedGoogle ScholarCrossref
    9.
    Chen  Q, Larochelle  MR, Weaver  DT,  et al.  Prevention of prescription opioid misuse and projected overdose deaths in the United States.   JAMA Netw Open. 2019;2(2):e187621. doi:10.1001/jamanetworkopen.2018.7621 PubMedGoogle Scholar
    10.
    Irvine  MA, Kuo  M, Buxton  JA,  et al.  Modelling the combined impact of interventions in averting deaths during a synthetic-opioid overdose epidemic.   Addiction. 2019;114(9):1602-1613. doi:10.1111/add.14664 PubMedGoogle ScholarCrossref
    11.
    National Institute on Alcohol Abuse and Alcoholism. Rates of nonmedical prescription opioid use and opioid use disorder double in 10 years. Published June 22, 2016. Accessed December 10, 2018. https://www.niaaa.nih.gov/news-events/news-releases/rates-nonmedical-prescription-opioid-use-and-opioid-use-disorder-double-10
    12.
    Moran  A, Gu  D, Zhao  D,  et al.  Future cardiovascular disease in china: Markov model and risk factor scenario projections from the coronary heart disease policy model-china.   Circ Cardiovasc Qual Outcomes. 2010;3(3):243-252. doi:10.1161/CIRCOUTCOMES.109.910711PubMedGoogle ScholarCrossref
    13.
    Cobb  CO, Villanti  AC, Graham  AL,  et al.  Markov modeling to estimate the population impact of emerging tobacco products: a proof-of-concept study.   Tob Regul Sci. 2015;1(2):129-141. doi:10.18001/TRS.1.2.3Google ScholarCrossref
    14.
    Census  US. Accessed January 23, 2020. https://www.census.gov
    15.
    Centers for Disease Control and Prevention, National Center for Health Statistics. CDC WONDER. About multiple cause of death, 1999-2017. Accessed January 23, 2020. https://wonder.cdc.gov/mcd-icd10.html
    16.
    US Dept of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality. Results from the 2017 National Survey on Drug Use and Health: detailed tables. Accessed January 23, 2020. https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHDetailedTabs2017/NSDUHDetailedTabs2017.pdf
    17.
    Grant  BF, Amsbary  M, Chu  A,  et al  Source and Accuracy Statement: National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III). National Institute on Alcohol Abuse and Alcoholism; 2014.
    18.
    Dyer  O.  OxyContin maker stops marketing opioids, as report details payments to advocacy groups.   BMJ. 2018;360:k791. doi:10.1136/bmj.k791 PubMedGoogle ScholarCrossref
    19.
    Drug Enforcement Agency, Diversion Control Division. FR Doc No: 2018-15141. Accessed January 23, 2020. https://www.deadiversion.usdoj.gov/fed_regs/rules/2018/fr0716_3.htm
    20.
    Dowell  D, Haegerich  TM, Chou  R.  CDC guideline for prescribing opioids for chronic pain—United States, 2016.   JAMA. 2016;315(15):1624-1645. doi:10.1001/jama.2016.1464 PubMedGoogle ScholarCrossref
    21.
    Midgette G, Davenport S, Caulkins JP, Kilmer B. What America’s users spend on illegal drugs, 2006-2016. RAND. Published 2019. Accessed August 17, 2020. https://www.rand.org/content/dam/rand/pubs/research_reports/RR3100/RR3140/RAND_RR3140.pdf
    22.
    Alexander  GC, Frattaroli  S, Gielen  AC, eds.  The Opioid Epidemic: From Evidence to Impact. Johns Hopkins Bloomberg School of Public Health; 2017.
    23.
    Gramlich  J. What the data says about gun deaths in the US. Pew Research Center. Published August 16, 2019. Accessed January 23, 2020. https://www.pewresearch.org/fact-tank/2019/08/16/what-the-data-says-about-gun-deaths-in-the-u-s/
    24.
    McLellan  AT, Lewis  DC, O’Brien  CP, Kleber  HD.  Drug dependence, a chronic medical illness: implications for treatment, insurance, and outcomes evaluation.   JAMA. 2000;284(13):1689-1695. doi:10.1001/jama.284.13.1689 PubMedGoogle ScholarCrossref
    25.
    Mackey  K, Veazie  S, Anderson  J, Bourne  D, and Peterson  K. Evidence brief: barriers and facilitators to use of medications for opioid use disorder. Evidence Synthesis Program, Health Services Research and Development Service, Office of Research and Development, Department of Veterans Affairs. VA ESP Project #09-009; 2019.
    26.
    Friedmann  PD, Alexander  JA, Yey  Y, Nahra  T, Soliman  S, Pollack  HA.  Duration of nonmethadone outpatient treatment: results from a national survey.   Subst Abus. 2006;27(3):47-53. doi:10.1300/J465v27n03_07 PubMedGoogle ScholarCrossref
    27.
    Andrews  CM, Abraham  AJ, Grogan  CM, Westlake  MA, Pollack  HA, Friedmann  PD.  Impact of Medicaid restrictions on availability of buprenorphine in addiction treatment programs.   Am J Public Health. 2019;109(3):434-436. doi:10.2105/AJPH.2018.304856 PubMedGoogle ScholarCrossref
    28.
    Morgan  DJ, Dhruva  SS, Wright  SM, Korenstein  D.  2016 Update on medical overuse: a systematic review.   JAMA Intern Med. 2016;176(11):1687-1692. doi:10.1001/jamainternmed.2016.5381 PubMedGoogle ScholarCrossref
    29.
    Subramani  MS.  Capsule Commentary on Calcaterra et al., Opioid prescribing at hospital discharge contributes to chronic opioid use.   J Gen Intern Med. 2016;31(5):532-532. doi:10.1007/s11606-016-3587-4 PubMedGoogle ScholarCrossref
    30.
    Kolodny  A, Courtwright  DT, Hwang  CS,  et al.  The prescription opioid and heroin crisis: a public health approach to an epidemic of addiction.   Annu Rev Public Health. 2015;36:559-574. doi:10.1146/annurev-publhealth-031914-122957 PubMedGoogle ScholarCrossref
    31.
    Compton  WM, Jones  CM, Baldwin  GT.  Relationship between nonmedical prescription-opioid use and heroin use.   N Engl J Med. 2016;374(2):154-163. doi:10.1056/NEJMra1508490 PubMedGoogle ScholarCrossref
    32.
    Caulkins  JP.  Models pertaining to how drug policy should vary over the course of a drug epidemic.   Adv Health Econ Health Serv Res. 2005;16:397-429. doi:10.1016/S0731-2199(05)16019-1 PubMedGoogle ScholarCrossref
    33.
    Alpert  A, Powell  D, Pacula  RL.  Supply-side drug policy in the presence of substitutes: evidence from the introduction of abuse-deterrent opioids.   Am Econ J Econ Policy. 2018;10:1-35. doi:10.1257/pol.20170082Google ScholarCrossref
    34.
    Kilmer  B, Everingham  S, Caulkins  J .  What America’s Users Spend on Illegal Drugs: 2000-2010. Rand Corporation; 2014. doi:10.7249/RR534
    35.
    Pardo  B, Taylor J,Caulkins   JP, et al. The Future of Fentanyl and Other Synthetic Opioids. RAND Corporation. Published 2019. Accessed January 23, 2020. https://www.rand.org/pubs/research_reports/RR3117.html
    36.
    Chang  HY, Lyapustina  T, Rutkow  L,  et al.  Impact of prescription drug monitoring programs and pill mill laws on high-risk opioid prescribers: a comparative interrupted time series analysis.   Drug Alcohol Depend. 2016;165:1-8. doi:10.1016/j.drugalcdep.2016.04.033 PubMedGoogle ScholarCrossref
    ×