Examination of the Accuracy of Existing Overdose Surveillance Systems

This cross-sectional study investigates the accuracy of the Centers for Disease Control and Prevention opioid overdose case definition compared with a state surveillance system.


Introduction
Opioid overdose surveillance is a key public health strategy to inform rapid and targeted responses to the overdose crisis in the US. 1,2States have historically maintained their own surveillance systems, which vary in data collection methods and content and case definitions.The US Centers for Disease Control and Prevention (CDC) has been leading a national effort to standardize opioid overdose syndromic surveillance using emergency department (ED) visit data and a standard opioid overdose case definition from electronic health record (EHR) chief complaint field and opioid overdoserelated diagnostic code data. 3,4However, the relative performance of the CDC opioid overdose case definition vs existing state-based systems is unknown.We assessed the accuracy of the CDC opioid overdose case definition and Rhode Island overdose surveillance system in identifying opioid overdose ED visits by comparing them with a reference based on EHR review.

Methods
This retrospective cross-sectional study was conducted in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.The Lifespan Health System Institutional Review Board approved this study.

Study Sample and Data
We conducted a cross-sectional study of opioid overdose ED visits at 2 large urban EDs in Providence, Rhode Island, from January 1 through May 31, 2021.We reviewed EHR data for opioid overdoses for adult patients identified by the CDC opioid overdose case definition or reported to the RIDOH state surveillance system.Cases were included in the study if they were captured by either surveillance system during the 5-month study period.

CDC Opioid Overdose Case Definition
The CDC opioid overdose case definition uses information from EHR chief complaint fields, opioid overdose-related diagnostic codes (International Classification of Diseases, Ninth Revision [ICD -9]   and International Statistical Classification of Diseases and Related Health Problems, Tenth Revision [ICD-10]), and specific SNOMED CT codes (eFigure in Supplement 1). 3 Hospitals automatically send ED EHR data to RIDOH in near real time, and RIDOH applies the CDC case definition for CDC reporting and inclusion in the CDC Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE). 4This process was under development at the time of this study.For this study, we identified ED visits meeting the CDC opioid overdose case definition through electronic query of the EHR.

Rhode Island State Surveillance System
Since 2014, Rhode Island has required EDs to report suspected opioid overdoses to RIDOH within 48 hours through a secure, web-based platform. 5To identify records for reporting, a daily list of potential opioid overdoses is generated by EHR query for visits with a naloxone order, an opioid-or overdose-related chief complaint, or an opioid overdose-related discharge diagnosis.A quality assurance nurse manually reviews identified records to determine whether the visit was an opioid overdose. 5Subsequently, a quality assurance nurse or unit secretary manually enters deidentified, individual-level data into an online reporting form.For this study, we identified ED visits reported to the state surveillance system during the study period using a database of reported records maintained by the hospital quality and patient safety division.

EHR Review
We conducted a manual EHR review of cases identified by the CDC opioid overdose case definition, reported to the RIDOH surveillance system, or both to determine whether these identified visits

JAMA Network Open | Public Health
Examination of the Accuracy of Existing Overdose Surveillance Systems were true opioid overdoses using a standard case definition.We defined a true opioid overdose as an event in which a patient developed respiratory depression, loss of consciousness, or both immediately after the patient had reportedly ingested an opioid or naloxone was administered with improvement in respiration or consciousness.EHR fields reviewed included emergency medical services run sheets, lab results, orders, and EHR documentation by clinicians, nurses, and social and community health workers.
EHRs were reviewed by 2 data extractors (J.G. and A.G.) trained and supervised by a senior study team member and emergency physician (E.A.S.).Patient sociodemographic data, including race and ethnicity, collected for the study were from EHR data, which is a combination of hospital registration-recorded and patient self-reported data.Race, ethnicity, and sex categories available in the EHR were, respectively, American Indian or Alaskan Native, Asian, Black or African American, Hawaiian or other Pacific Islander, White, not documented, and other; Hispanic or Latinx, not Hispanic or Latinx, not documented, and other; and female, male, and other.For analysis, race and ethnicity categories were grouped into Black, White, and other and Hispanic or Latinx, not Hispanic or Latinx, and unknown, respectively, due to small sample sizes.Most individuals in the other racial category identified as Hispanic or Latino.We assessed age, sex, race, and ethnicity to describe the demographic composition of the study population overall and in study groups.Data extractors entered ED visit data (patient sociodemographic, prehospital, and ED visit characteristics) into a research electronic data capture 6 form and determined whether the ED visit met the standard case definition.A subsample of EHRs (61 of 460 EHRs [13.3%]) were double reviewed (E.A.S.) for accuracy and to determine interrater reliability.

Statistical Analysis
Study data were analyzed using Stata statistical software version 17 (StataCorp) and RStudio statistical software version 4.1.2(RStudio).We descriptively compared agreement between the CDC case definition and state surveillance system.We separately estimated the positive predictive value (PPV) and exact binomial 95% CI 7,8 for each approach compared with the standard case definition (ie, percentage of visits classified as opioid overdoses that were truly opioid overdoses) and descriptively compared 95% CIs.We calculated a κ score to estimate the interrater reliability of each data extractor compared with the emergency medicine physician for our standard case definition.
The level of statistical significance was defined a priori as α = .05,and statistical tests were 2-sided.
Data were analyzed from January through May 2021.1).

JAMA Network Open | Public Health
Examination of the Accuracy of Existing Overdose Surveillance Systems

Discussion
In this cross-sectional study of ED opioid overdose visits, we found that the CDC opioid overdose case definition more often identified true opioid overdoses than the state surveillance system (90.8%vs 75.6%).RIDOH uses state surveillance data to inform rapid and targeted overdose responses. 9,10Our findings suggest that transitioning to the CDC opioid overdose case definition through ESSENCE may be associated with improved surveillance accuracy and timeliness and a decrease in the reporting burden for hospital staff of manual reporting, which is time and labor intensive, subject to human error, and often delayed.Given limited resources for overdose prevention, it may be preferable for overdose surveillance systems to identify true opioid overdoses rather than a mixture of true opioid overdoses and other visit types, so long as it does not significantly undercount overdoses and specific subgroups of patients are not systematically underrepresented.Many departments of health and community harm reduction organizations are working with limited resources; to effectively reduce overdose deaths, resources should be directed where they are needed most, in areas with confirmed high overdose incidence.Further studies are needed to evaluate the sensitivity of this case definition on its own and in comparison with other overdose surveillance approaches to ensure generalizability and accurate account of overdose incidence for allocation of overdose prevention resources.

Limitations
This study has several limitations.As a retrospective EHR review, this study was limited by reporting and recording bias in the EHR which may result in inaccurate classification of ED visits as opioid overdoses and potentially undercounting of overdose events.A major limitation of this study was that our approach may have missed some true opioid overdoses given that it was not feasible to manually review all ED records during the study period.Consequently, we were not able to measure other aspects of the CDC case definition performance (eg, sensitivity or specificity).While the CDC case definition does not erroneously classify many other types of visits as opioid overdoses, the number of true opioid overdose visits that it misses is uncertain.

Table 2 .
Agreement Between Approaches in Opioid Overdose Classification Not assessed given that all records in the study sample were identified by CDC case definition or reported to the state surveillance system. a

Table 3 .
Performance of Classification Approaches vs Standard Case Definition