[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 34.204.193.85. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    2 Comments for this article
    EXPAND ALL
    Addiction is a Medical Problem?
    Thomas Hilton, PhD | Retired, NIH/NIDA
    I do not doubt the sincerity of the authors in trying to prevent opioid overdose among the elderly, but I find it to be quite a stretch to tacitly suggest that overdose is due to medical mismanagement, and therefore can be prevented by an algorithm's alert. Many of the variables in the equation have been associated with intentional as well as accidental overdose. This fact is acknowledged in the limitations section but inadequately discussed as undermining AI application in clinical practice.

    An AI shotgun cannot match an attending physician's need for a more nuanced understanding of their
    patient than an algorithm can produce even if it has a full magazine of data in the chamber - which is unlikely.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Error in Text, Table 1, Figure 2 and Supplement
    Jenny Lo-Ciganic, PhD | University of Florida, College of Pharmacy
    In the Original Investigation titled “Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose among Medicare Beneficiaries with Opioid Prescriptions”, published March 22, 2019, there were minor errors in the texts, Table 1, Figure 2, and Supplement. We identified these errors when duplicating the analyses for a separate project. When presenting our risk stratification results for the GBM and DNN models, we inadvertently presented results defining high risk differently in the GBM and DNN models, using the 5th percentile in one model and 10th percentile in the other. We have now corrected these errors in the text, Table 1 and Figure 1 by consistently using the 5th percentile of scores. We also provide the results using the 10th percentiles of scores for both GBM and DNN in the Online Supplement. None of the conclusions or interpretations are affected. This article has been corrected.

    References
    1. Lo-Ciganic W, Huang JL, Zhang HH et al. Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions. JAMA Netw Open. 2019;2(3):e190968. doi:10.1001/jamanetworkopen.2019.0968
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Original Investigation
    Substance Use and Addiction
    March 22, 2019

    Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions

    Author Affiliations
    • 1Department of Pharmaceutical Outcomes & Policy, College of Pharmacy, University of Florida, Gainesville
    • 2Department of Mathematics, University of Arizona, Tucson
    • 3Carnegie Mellon University, Heinz College, Pittsburgh, Pennsylvania
    • 4Department of Health Outcomes & Biomedical Informatics, University of Florida, College of Medicine, Gainesville
    • 5Division of Rheumatology, Department of Medicine, and the University of Arizona Arthritis Center, University of Arizona, Tucson
    • 6Department of Health Policy and Management, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania
    • 7Program for Addiction Research, Clinical Care, Knowledge, and Advocacy, Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City
    • 8Informatics, Decision-Enhancement, and Analytic Sciences Center, Veterans Affairs Salt Lake City Health Care System, Salt Lake City, Utah
    • 9Department of Pharmacy, Practice and Science, College of Pharmacy, University of Arizona, Tucson
    • 10Center for Pharmaceutical Policy and Prescribing, Health Policy Institute, University of Pittsburgh, Pittsburgh, Pennsylvania
    • 11Department of Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania
    • 12Center for Health Equity Research Promotion, Veterans Affairs Pittsburgh Healthcare System, Pittsburgh, Pennsylvania
    JAMA Netw Open. 2019;2(3):e190968. doi:10.1001/jamanetworkopen.2019.0968
    Key Points español 中文 (chinese)

    Question  Can machine-learning approaches predict opioid overdose risk among fee-for-service Medicare beneficiaries?

    Findings  In this prognostic study of the administrative claims data of 560 057 Medicare beneficiaries, the deep neural network and gradient boosting machine models outperformed other methods for identifying risk, although positive predictive values were low given the low prevalence of overdose episodes.

    Meaning  Machine-learning algorithms using administrative data appear to be a valuable and feasible tool for more accurate identification of opioid overdose risk.

    Abstract

    Importance  Current approaches to identifying individuals at high risk for opioid overdose target many patients who are not truly at high risk.

    Objective  To develop and validate a machine-learning algorithm to predict opioid overdose risk among Medicare beneficiaries with at least 1 opioid prescription.

    Design, Setting, and Participants  A prognostic study was conducted between September 1, 2017, and December 31, 2018. Participants (n = 560 057) included fee-for-service Medicare beneficiaries without cancer who filled 1 or more opioid prescriptions from January 1, 2011, to December 31, 2015. Beneficiaries were randomly and equally divided into training, testing, and validation samples.

    Exposures  Potential predictors (n = 268), including sociodemographics, health status, patterns of opioid use, and practitioner-level and regional-level factors, were measured in 3-month windows, starting 3 months before initiating opioids until loss of follow-up or the end of observation.

    Main Outcomes and Measures  Opioid overdose episodes from inpatient and emergency department claims were identified. Multivariate logistic regression (MLR), least absolute shrinkage and selection operator–type regression (LASSO), random forest (RF), gradient boosting machine (GBM), and deep neural network (DNN) were applied to predict overdose risk in the subsequent 3 months after initiation of treatment with prescription opioids. Prediction performance was assessed using the C statistic and other metrics (eg, sensitivity, specificity, and number needed to evaluate [NNE] to identify one overdose). The Youden index was used to identify the optimized threshold of predicted score that balanced sensitivity and specificity.

    Results  Beneficiaries in the training (n = 186 686), testing (n = 186 685), and validation (n = 186 686) samples had similar characteristics (mean [SD] age of 68.0 [14.5] years, and approximately 63% were female, 82% were white, 35% had disabilities, 41% were dual eligible, and 0.60% had at least 1 overdose episode). In the validation sample, the DNN (C statistic = 0.91; 95% CI, 0.88-0.93) and GBM (C statistic = 0.90; 95% CI, 0.87-0.94) algorithms outperformed the LASSO (C statistic = 0.84; 95% CI, 0.80-0.89), RF (C statistic = 0.80; 95% CI, 0.75-0.84), and MLR (C statistic = 0.75; 95% CI, 0.69-0.80) methods for predicting opioid overdose. At the optimized sensitivity and specificity, DNN had a sensitivity of 92.3%, specificity of 75.7%, NNE of 542, positive predictive value of 0.18%, and negative predictive value of 99.9%. The DNN classified patients into low-risk (76.2% [142 180] of the cohort), medium-risk (18.6% [34 579] of the cohort), and high-risk (5.2% [9747] of the cohort) subgroups, with only 1 in 10 000 in the low-risk subgroup having an overdose episode. More than 90% of overdose episodes occurred in the high-risk and medium-risk subgroups, although positive predictive values were low, given the rare overdose outcome.

    Conclusions and Relevance  Machine-learning algorithms appear to perform well for risk prediction and stratification of opioid overdose, especially in identifying low-risk subgroups that have minimal risk of overdose.

    ×