Development and Validation of a Tool to Identify Patients With Type 2 Diabetes at High Risk of Hypoglycemia-Related Emergency Department or Hospital Use

Key Points

Question  Can electronic medical records be used to reliably categorize risk of future hypoglycemia-related emergency department or hospital use in patients with type 2 diabetes?

Findings  We developed and validated a risk stratification tool that categorized patients’ 12-month risk of hypoglycemia-related utilization using only 6 electronic medical record–based inputs (patient history of hypoglycemia-related utilization, insulin use, sulfonylurea use, emergency department use, chronic kidney disease, and age). Tool performance was validated in 2 fully independent populations.

Meaning  This hypoglycemia risk stratification tool could facilitate efficient targeting of population management interventions to reduce hypoglycemia risk and improve patient safety.

Importance  Hypoglycemia-related emergency department (ED) or hospital use among patients with type 2 diabetes (T2D) is clinically significant and possibly preventable.

Objective  To develop and validate a tool to categorize risk of hypoglycemic-related utilization in patients with T2D.

Design, Setting, and Participants  Using recursive partitioning with a split-sample design, we created a classification tree based on potential predictors of hypoglycemia-related ED or hospital use. The resulting model was transcribed into a tool for practical application and tested in 1 internal and 2 fully independent, external samples. Development and internal testing was conducted in a split sample of 206 435 patients with T2D from Kaiser Permanente Northern California (KPNC), an integrated health care system. The tool was externally tested in 1 335 966 Veterans Health Administration and 14 972 Group Health Cooperative patients with T2D.

Exposures  Based on a literature review, we identified 156 candidate predictor variables (prebaseline exposures) using data collected from electronic medical records.

Main Outcomes and Measures  Hypoglycemia-related ED or hospital use during 12 months of follow-up.

Results  The derivation sample (n = 165 148) had a mean (SD) age of 63.9 (13.0) years and included 78 576 (47.6%) women. The crude annual rate of at least 1 hypoglycemia-related ED or hospital encounter in the KPNC derivation sample was 0.49%. The resulting hypoglycemia risk stratification tool required 6 patient-specific inputs: number of prior episodes of hypoglycemia-related utilization, insulin use, sulfonylurea use, prior year ED use, chronic kidney disease stage, and age. We categorized the predicted 12-month risk of any hypoglycemia-related utilization as high (>5%), intermediate (1%-5%), or low (<1%). In the internal validation sample, 2.0%, 10.7%, and 87.3% were categorized as high, intermediate, and low risk, respectively, with observed 12-month hypoglycemia-related utilization rates of 6.7%, 1.4%, and 0.2%, respectively. There was good discrimination in the internal validation KPNC sample (C statistic = 0.83) and both external validation samples (Veterans Health Administration: C statistic = 0.81; Group Health Cooperative: C statistic = 0.79).

Conclusions and Relevance  This hypoglycemia risk stratification tool categorizes the 12-month risk of hypoglycemia-related utilization in patients with T2D using only 6 inputs. This tool could facilitate targeted population management interventions, potentially reducing hypoglycemia risk and improving patient safety and quality of life.

Advances in diabetes clinical care and medical treatment have reduced the risk of long-term complications and mortality for the more than 25 million Americans who have diabetes.¹ However, iatrogenic hypoglycemia associated with glucose-lowering medication use has become a critical public health and drug safety concern.^2-4Quiz Ref ID Severe hypoglycemia is defined as an event necessitating assistance from another person to actively administer carbohydrates, glucagon, or other resuscitative actions.⁵ Such assistance is often rendered professionally in emergency department (ED) or hospital encounters and is captured as hypoglycemia-related utilization.⁶

Whereas the risk of severe hypoglycemia is known to be elevated in patients with type 1 diabetes, the risk has been historically underappreciated among patients with type 2 diabetes (T2D), which make up most of the diabetes population. Quiz Ref IDHypoglycemia is now one of the most frequent adverse events in patients with T2D and is more common than acute hyperglycemic emergencies (eg, hyperosmolar hyperglycemic state),⁷ particularly among older patients and those with a longer history of diabetes.⁸ One in 4 emergency hospitalizations for adverse drug events is related to hypoglycemia, and these rates are higher in older patients.⁹ Severe hypoglycemia has been associated with falls and automobile accidents,¹⁰ cardiovascular autonomic dysfunction and ventricular arrhythmia,¹¹ dementia,¹² and death.^13,14 Patients report that fear of hypoglycemia can dissuade them from initiating newly prescribed insulin.¹⁵ Hypoglycemia is also strongly predictive of poorer health-related quality of life¹⁶ and more diabetes distress.¹⁷ Hypoglycemia-related utilization is costly; total annual direct medical costs were estimated at approximately $1.8 billion in 2009 in the United States.¹⁸

The risk of hypoglycemia varies widely in patients with T2D.¹⁹ Whereas interventions to prevent hypoglycemia exist,^20-24 there are no validated methods to target these interventions efficiently. Accordingly, we developed and validated a hypoglycemia risk stratification tool to categorize 12-month risk of hypoglycemia-related emergency department (ED) or hospital use among patients with T2D. This study was approved by the institutional review boards of Kaiser Permanente, the Bedford Veterans Health Administration, and Group Health Cooperative; the requirement that informed consent be obtained from study participants was waived.

We used a prospective cohort study design to develop a risk tool to categorize the 12-month risk of hypoglycemia-related ED or hospital use. Selection of prebaseline candidate predictors was based on a literature review of clinical risk factors associated with hypoglycemia and limited to data typically available in electronic medical records (EMRs). We derived and internally validated this tool in a split sample (4:1) of 206 435 adult patients with T2D in an integrated health care delivery system (Kaiser Permanente Northern California [KPNC]) using clinical and demographic data from EMRs. We used recursive partitioning in the derivation sample to create a risk classification tree. The classification tree leaf nodes were further categorized into high-, intermediate-, or low-risk groups on the basis of predicted risk and then transcribed into the hypoglycemia risk stratification tool. After testing the tool in the internal sample, we conducted external validation in 2 completely independent samples of patients with T2D from the Veterans Administration Diabetes Epidemiology Cohort (DEpiC) (VA sample, n = 1 335 966)²⁵ and from Group Health Cooperative (GH sample, n = 14 972).

Using EMR data from KPNC, we identified 233 330 adults (≥21 years as of the baseline date of January 1, 2014) with diabetes with continuous health plan membership for 24 months prebaseline and pharmacy benefits for 12 months prebaseline. We excluded 24 719 patients with unknown diabetes type and 3615 with probable type 1 diabetes according to an algorithm (based on age of onset <30 years and use of insulin alone²⁶). The remaining 206 435 eligible patients with T2D were randomly split into an 80% derivation sample (n = 165 148) for tool development and a 20% internal validation sample (n = 41 287). Similar eligibility criteria were applied in the creation of the 2 external validation samples.

Our outcome was the occurrence of any hypoglycemia-related ED or hospital use during 12 months postbaseline. This was defined by having any ED visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained according to a validated definition²⁷ (any of the following International Classification of Diseases, Ninth Revision (ICD-9), codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes). Secondary discharge diagnoses for hypoglycemia were not used because they are often attributable to events that occurred during the ED or hospital encounter (eg, inpatient insulin management, sepsis, acute renal failure²⁸).

On the basis of a literature review of clinical risk factors associated with hypoglycemia in T2D, we selected 156 (122 categorical and 34 continuous) candidate clinical, demographic, and behavioral predictor variables for model development (eTable 1 in the Supplement). To increase the utility of our prediction model in other health care settings (transportability, usability²⁹), we excluded variables that were expensive or impractical to collect, had ambiguous meanings, or were not typically available in an EMR. Medication exposures were based on pharmacy dispensings during 6 months prebaseline; laboratory values were based on the last test results within 2 years prebaseline; and prior medical events (eg, history of hypoglycemia-related utilization using the same outcome definition) were based on all available prebaseline records (maximum, 18 years of medical history; mean [SD], 16.5 [3.4] years).

We used standard methodology for model development including a split sample and internal and external validation.^30-33 We first regressed the outcome (any hypoglycemia-related utilization) on each of the 156 candidate predictors using univariate logistic regression models to generate odds ratios. We selected candidate variables that had a resulting P < .10. We then used recursive partitioning (using SAS JMP, version 12³⁴) on the selected candidates to construct a binary classification tree to predict the occurrence of at least 1 hypoglycemia-related utilization episode 12 months postbaseline. Recursive partitioning is widely used to generate clinical decision support tools.^32,35,36 This method uses a machine-learning, nonlinear, and nonparametric approach to split (“partition”) events into pairs of subgroups based on continuous or categorical predictors, and has the unique advantage that it identifies complex nested interactions, unlike linear modeling methods. Recursive partitioning also optimizes cut points rather than relying on prespecification. Thus, the resultant classification tree identifies predictors that may be important for 1 segment of the population but not others, as well as identifying critical thresholds in continuous or ordinal predictors.

We pruned branches from the classification tree in an attempt to optimize predictive accuracy (performance), model simplicity, practicality of implementation, and intuitive clinical interpretation.^31,33 Overly complex models (overfitting), while potentially offering somewhat greater precision, may be less practical, increase the decision and classification costs (expense and time of compiling the predictors), and introduce propagated error associated with predictors measured with uncertainty.

Model accuracy was assessed in the internal validation sample using standard metrics.^{30-32(pp255-310)37-39}Discrimination, the ability of a model to accurately distinguish between subjects who do vs do not develop the outcome, is based on the area under the receiver-operator curve (C statistic), with greater than 0.7 classified as good discrimination. We also visually assessed calibration (the extent to which the predicted risks over- or underestimate the observed risks) using calibration plots. Given that our goal was not to quantify the numeric probability of a hypoglycemia episode for a given patient but rather to stratify our population into categories of risk, we focused on model discrimination over calibration.^33,40

We further evaluated components of generalizability^33,39 (ie, reproducibility in patients not used for the derivation of the model) and transportability (practical application in different settings) in the 2 external primary care T2D populations (VA and GH samples).^32,33,41 Because these samples included a different disease severity and case mix of patients with T2D from distinct geographical locations with different methods for identifying patients with diabetes, the validation exercises also tested the spectrum, geographic, and methodological transportability of the model.^32,33

We developed this risk stratification tool using all of the patients’ available medical histories at KPNC (up to 18 years); however, long enrollment may be uncommon in other health care settings. Thus, we conducted further analyses to evaluate whether our tool was sensitive to restrictions in available length of enrollment (prebaseline period transportability³³). We also evaluated temporal sensitivity (historical transportability³³) of the tool by applying it to KPNC data in the subsequent year (using a baseline date of January 1, 2015). This tool was optimized for patients with T2D, and because not all health care settings can reliably determine diabetes type from their EMR, we also evaluated tool performance when patients with type 1 diabetes were included (ie, sensitivity to misclassification). Finally, as a measure of ecological validity of this tool, we evaluated the association between the predicted level of risk of hypoglycemia-related utilization and actual, self-reported severe hypoglycemia events based on a 2005 survey⁴² of 15 231 patients with T2D.

Quiz Ref IDThe final classification tree was based on 6 patient-specific variables: total number of prior episodes of hypoglycemia-related ED or hospital utilization (0, 1-2, ≥3 times), number of ED encounters for any reason in the prior 12 months (<2, ≥2 times), insulin use (yes/no), sulfonylurea use (yes/no), presence of severe or end-stage kidney disease (dialysis or chronic kidney disease stage 4 or 5 determined by estimated glomerular filtration rate of ≤29 mL/min/1.73 m² calculated by the Chronic Kidney Disease Epidemiology Collaboration creatinine equation)⁴³ (yes/no), and age younger than 77 years (yes/no) (Figure 1). This classification tree resulted in 10 mutually exclusive leaf nodes, each yielding an estimated annual risk of hypoglycemia-related utilization, which were categorized as high (>5%), intermediate (1%-5%), or low (<1%). In the KPNC internal validation sample, 2.0% were categorized as high risk, 10.7% as intermediate risk, and 87.3% as low risk.

We then transcribed the classification model into a simple, checklist style, hypoglycemia risk stratification tool by mapping the combinations of risk factors to high, intermediate, or low risk of having any hypoglycemia-related utilization in the following 12 months (Figure 2). This tool instructs the user to identify only 1 of 6 mutually exclusive options, where the first 5 are each defined by a unique combination of predictor variables, and the sixth option is indicated only after ruling out all other options (eTable 2 in the Supplement provides the source code).

We compared the distribution of the 6 predictor variables in our derivation and validation samples (Table 1). There were no significant differences in the distribution of the 6 predictors between the KPNC derivation vs validation samples, but there were significant differences across external validation samples. The proportion of men and women was similar in the KPNC and GH samples, while the VA sample was predominantly men. The mean age was similar across sites, although the proportion older than 77 years was greater in the VA (24.6%), followed by KPNC (17.8%), and GH samples (13.2%). The VA sample had the highest proportion of patients with severe or end-stage kidney disease (3.7%), and 3 or more prior hypoglycemic events (0.4%) vs 2.0% and 0.1%, respectively, in GH. Insulin use was lower in the KPNC samples (20.3%) compared with the GH (30.9%) and VA samples (30.2%). Sulfonylurea use was higher in the KPNC samples (34.9%) compared with the VA (25.0%) and GH (22.5%) samples. The observed annual rate of hypoglycemia-related utilization was lower in GH patients (0.30%) compared with the KPNC derivation sample (0.49%) and the VA (0.51%).

Internal validation of the classification tree model indicated high discrimination (C statistic = 0.83) and good calibration (no significant differences between predicted and observed risk: Pearson χ² goodness-of-fit P = .31) (Table 2). The odds ratios of hypoglycemia-related utilization among those categorized as high relative to low risk were large in each sample: KPNC internal validation sample (34.6; 95% CI, 24.2-49.3), VA (23.3; 95% CI, 21.9-24.7), and GH (20.7; 95% CI, 8.6-45.0; P < .001).

The tool also performed well in terms of discrimination in the external validation samples (VA C statistic = 0.81; GH C statistic = 0.79). Visual inspection of the calibration plots showed a reasonable match between the predicted and observed risk of hypoglycemia-related utilization within the 10 leaf nodes (Figure 3). However, the tool somewhat overestimated risk among the leaf nodes in the intermediate- and higher-risk categories in the external validation samples.

In the sensitivity analyses for length of available medical history, good discrimination was confirmed despite shorter medical history (C statistic = 0.82, 0.83, 0.84 for ≤2, ≤5, and ≤10 years, respectively). We then evaluated temporal sensitivity (historical transportability³³). There was good discrimination (C statistic = 0.83) even after KPNC experienced a significant 21% increase in the rate of hypoglycemia-related utilization (0.48% vs 0.59% for 2014 and 2015, respectively). We also evaluated sensitivity to misclassification of diabetes type by including all diabetes patients in our sample and found good discrimination (C statistic = 0.84).

In the assessment of ecological validity, there was a strong association between predicted risk of hypoglycemia-related utilization and self-reported severe hypoglycemia (ie, hypoglycemia necessitating assistance in the past 12 months). Patients categorized as high risk by the tool were 5 times more likely (49.7% vs 9.2%; P < .001) to self-report a severe hypoglycemic episode relative to those categorized as low risk.

Health care systems currently lack an evidence-based method for efficiently and systematically identifying patients with T2D at risk of hypoglycemia-related ED or hospital use. We developed and validated a pragmatic hypoglycemia risk stratification tool that uses 6 factors to categorize the 12-month risk of hypoglycemia-related utilization. This tool uses EMR data only and requires no patient contact; it offers an efficient, low-cost approach for identifying patients for targeted interventions to reduce their risk of hypoglycemia.^21,44-48 Because of the harms and costs associated with hypoglycemia, high-risk patients are candidates for an elevated level of scrutiny. Identifying medication overtreatment^49-51 and including hypoglycemia rates as a health plan–level quality measure have been recommended to drive accountability and quality improvement.^52-54

This tool is intended to offer a practical method to risk stratify patients for population management. For example, intensive interventions aimed at reducing hypoglycemia risk could be targeted at the minority of patients with T2D in the high-risk category (2% of patients with diabetes at KPNC). Quiz Ref IDThese interventions could include deintensifying or simplifying medication regimens, addressing impaired hypoglycemic awareness, prescribing glucagon kits or continuous glucose monitors, making referrals to clinical pharmacists or nurse care managers, providing additional diabetes education, and regularly asking about hypoglycemia events occurring outside the medical setting. Clinician discussions could address potential contributors to hypoglycemia, including behavioral (eg, meal skipping⁵⁵), psychosocial (eg, food insufficiency^56,57), or socioeconomic (eg, deprivation⁵⁸) factors. Similarly, a lower-cost, less intensive intervention could be designed for patients in the intermediate-risk category (11% at KPNC). The intervention could include system-level structural modifications such as risk-based glycemic targets, automated clinical alert flags in the EMR, and automated messaging to patients with elevated risk. Moreover, the tool could be modified to identify specific subsets of risk groups such those with 3 or more hypoglycemia-related ED or hospital encounters whom our model identifies as having the highest risk.

While it is unknown to what extent clinicians are aware of a patient’s hypoglycemia risk, there is evidence that clinicians and patients with diabetes do not communicate about hypoglycemia events that occur outside clinical settings.^59,60 Almost all (roughly 95%) severe hypoglycemia events may go clinically unrecognized because they did not result in ED or hospital use.⁶¹ In an internal review of EMRs of KPNC patients with T2D, hypoglycemia was absent from the problem lists in 85% of patients categorized by our tool as being at high risk, underscoring the potential for this tool to increase clinicians’ awareness of the risk of hypoglycemia in their patients with T2D.

Some limitations should be noted. The final classification tree was one of many possible options, chosen on the basis of performance (eg, C statistic), parsimony, and pragmatism (eg, we excluded predictors that are typically unavailable, impractical, or costly to assess in usual care settings). In developing the model, we excluded secondary discharge diagnoses for hypoglycemia because these events may occur during the ED or hospital encounter (eg, inpatient insulin management, sepsis, acute renal failure²⁸), rather than being a cause of the encounter. On the other hand, hypoglycemia could be a secondary diagnosis if the primary or principal diagnosis is trauma due to an automobile accident or a serious fall caused by hypoglycemia. Although we did not include those events in our model development, we estimate that this would include less than 2% of ED encounters (data not shown). When validating this model, we emphasized discrimination rather than calibration given that our goal was to risk stratify patients into broad categories rather than predicting the continuous level of risk.⁴⁰ Thus, while the tool successfully stratifies the population into 3 levels of risk, it should not be used to estimate the probability of hypoglycemic-related utilization for an individual patient. Discrimination performance of the tool was validated internally and externally in 2 large, independent populations, suggesting generalizability. Geographic, methodologic, and spectrum transportability were also demonstrated because these external validation populations were from different geographical locations, used differing methods of caring for and identifying patients with T2D, and had different distributions of disease severity and case mix. While the observed rates of hypoglycemia-related utilization were lower in GH (0.3%) and somewhat higher in VA (0.51%) compared with KPNC sample (0.49%), the tool had good discrimination in all 3 samples. Some of the inconsistent findings are attributable to sparse data in the GH validation sample; only 45 events were observed during follow-up. It is also possible that the 2 external sample populations may experience more prevention efforts than at KPNC, explaining the somewhat lower than expected risk in higher-risk patients in the validation samples.

Quiz Ref IDThe tool was designed to predict hypoglycemia-related utilization and thus did not take into account severe hypoglycemia occurring outside the health care system (roughly 95% of hypoglycemic events necessitating third-party assistance are treated by persons other than medical professionals, eg, friends or family^61,62). However, we demonstrated a strong and significant association between the tool’s stratification of the risk of hypoglycemia-related utilization and a patient’s self-reported severe hypoglycemia events. Therefore, we believe that preventive interventions targeting patients identified as high risk by this tool may reduce the rate of severe hypoglycemic events that do not result in utilization or clinical recognition. Finally, the tool also proved robust in internal validations when patients with type 1 diabetes were included and when the availability of longitudinal EMR data was restricted.

This risk stratification tool was developed and validated in 3 vertically integrated health care delivery systems (KPNC, GH, and the VA). The tool logic may be programmed into a system’s EMR to allow for automated risk stratification based on the EMR data. Practical barriers may complicate implementing this tool for population management in horizontally integrated or nonintegrated health care delivery systems, for example, where pharmacy claims may not be readily available.

This tool offers a practical, EMR-based method to stratify patients with T2D by their 12-month risk of hypoglycemia-related ED or hospital utilization. This tool could be integrated with targeted preventive interventions to facilitate population management, which ultimately could reduce future hypoglycemia risk and improve patient safety. The 2 criteria indicating high risk are easily memorized (ie, ≥3 previous episodes of hypoglycemia-related utilization, or 1 or 2 episodes if treated with insulin). The criteria for intermediate risk are more nuanced and therefore may be less likely to provoke clinical action in primary care without prompting. Health care systems could adopt a 2-level intervention, with intensive (more expensive) interventions reserved for high-risk patients and less intensive (lower cost) interventions for the intermediate-risk patients. Implementation of this tool could conceivably increase clinician awareness of patients’ hypoglycemia risk. Clinical researchers may also find this tool helpful in identifying patients at high risk for hypoglycemic episodes for either purposeful inclusion or exclusion in clinical trials of novel therapies and diagnostic tests. Quality improvement and impact studies²⁹ are needed to evaluate whether and how implementation of this hypoglycemia risk stratification tool may influence clinician behavior, patient decision making, drug safety, and hypoglycemia incidence. Future research is needed to develop patient-centered and cost-effective interventions to reduce hypoglycemia risk in those identified as being at high risk for hypoglycemia.

Corresponding Author: Andrew J. Karter, PhD, Division of Research, Kaiser Permanente Northern California, 2000 Broadway, Oakland, CA 94612 (andy.j.karter@kp.org).

Accepted for Publication: June 20, 2017.

Published Online: August 21, 2017. doi:10.1001/jamainternmed.2017.3844

Author Contributions: Dr Karter had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Karter, Warton, Lipska, Moffet, Huang, Miller.

Acquisition, analysis, or interpretation of data: Karter, Warton, Lipska, Ralston, Moffet, Jackson, Miller.

Drafting of the manuscript: Karter, Warton, Jackson.

Critical revision of the manuscript for important intellectual content: Karter, Warton, Lipska, Ralston, Moffet, Huang, Miller.

Statistical analysis: Karter, Warton, Miller.

Obtained funding: Karter, Moffet.

Administrative, technical, or material support: Karter, Ralston, Moffet, Jackson, Miller.

Supervision: Karter, Ralston, Huang.

Conflict of Interest Disclosures: The National Institutes of Health supplied additional funding for our hypoglycemia-related research (NIDDK R01DK103721, R01DK081796). Drs Karter and Huang are also supported by the NIDDK Centers for Diabetes Translational Research (P30 DK092924 and P30 DK092949, respectively). Dr Huang was supported by K24 DK105340. Dr Lipska receives support from the Centers for Medicare & Medicaid Services to develop and maintain publicly reported quality measures, and from the Yale Claude D. Pepper Older Americans Independence Center (P30AG021342) and the National Institute on Aging through the Paul Beeson Career Development Award (K23AG048359). No other disclosures are reported.

Funding/Support: This project was funded by the US Food and Drug Administration (FDA BAA-13 00119).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.