[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 34.226.208.185. Please contact the publisher to request reinstatement.
[Skip to Content Landing]
Figure 1.
Classification Tree for Hypoglycemia-Related Emergency Department (ED) or Hospital Use
Classification Tree for Hypoglycemia-Related Emergency Department (ED) or Hospital Use

Hypoglycemic-related utilization was defined by having any ED visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes.27 The classification tree was developed using the 808 out of 165 148 T2D adults (derivation sample) from Kaiser Permanente who had such utilization (4.9 events per 1000 person-years) in 2014. The classification is based on 6 predictor variables from the electronic medical record and resulted in 10 mutually exclusive leaf nodes. The criterion for each node is displayed with the corresponding number of individuals (n) who met that criterion. The 12-month observed rate of any hypoglycemia-related ED or hospital use is displayed in each leaf node and categorized as high (>5% risk), intermediate (1%-5% risk), or low risk (<1% risk). CKD indicates chronic kidney disease.

Figure 2.
Hypoglycemia Risk Stratification Tool
Hypoglycemia Risk Stratification Tool

aHypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes.27

bHypoglycemic-related utilization was defined by having any emergency department (ED) visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia.

Figure 3.
Calibration Plots Comparing the Expected vs Observed 12-Month Rate of Having Any Hypoglycemia-Related Utilizationa for the Interval Derivation Sample From Kaiser Permanente Northern California (KPNC) (n = 165 148), the KPNC Internal Validation Sample (n = 41 287), the External Validation Sample From Group Health (GH) (n = 14 972), and the External Validation Sample From the Veterans Administration (VA) (n = 1 335 966)
Calibration Plots Comparing the Expected vs Observed 12-Month Rate of Having Any Hypoglycemia-Related Utilizationa for the Interval Derivation Sample From Kaiser Permanente Northern California (KPNC) (n = 165 148), the KPNC Internal Validation Sample (n = 41 287), the External Validation Sample From Group Health (GH) (n = 14 972), and the External Validation Sample From the Veterans Administration (VA) (n = 1 335 966)

aHypoglycemic-related utilization was defined by having any emergency department visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes.27

Table 1.  
Demographic Characteristics, Baseline Predictor Variables, and Outcome Rates in the Internal Derivation and Validation Samples (Kaiser Permanente Northern California [KPNC]) and the 2 External Validation Samples (Group Health Cooperative [GH] and Veterans Administration [VA])
Demographic Characteristics, Baseline Predictor Variables, and Outcome Rates in the Internal Derivation and Validation Samples (Kaiser Permanente Northern California [KPNC]) and the 2 External Validation Samples (Group Health Cooperative [GH] and Veterans Administration [VA])
Table 2.  
Calibration of Expected and Observed Rates of Hypoglycemia-Related Utilizationa Across the 3 Risk Strata in the Validation Samples
Calibration of Expected and Observed Rates of Hypoglycemia-Related Utilizationa Across the 3 Risk Strata in the Validation Samples
1.
Gregg  EW, Li  Y, Wang  J,  et al.  Changes in diabetes-related complications in the United States, 1990-2010.  N Engl J Med. 2014;370(16):1514-1523.PubMedGoogle ScholarCrossref
2.
Pogach  L, Aron  D.  Balancing hypoglycemia and glycemic control: a public health approach for insulin safety.  JAMA. 2010;303(20):2076-2077.PubMedGoogle ScholarCrossref
3.
Lee  SJ.  So much insulin, so much hypoglycemia.  JAMA Intern Med. 2014;174(5):686-688.PubMedGoogle ScholarCrossref
4.
Lipska  KJ.  Improving safety of diabetes mellitus management.  JAMA Intern Med. 2014;174(10):1612-1613.PubMedGoogle ScholarCrossref
5.
Workgroup on Hypoglycemia, American Diabetes Association.  Defining and reporting hypoglycemia in diabetes: a report from the American Diabetes Association Workgroup on Hypoglycemia.  Diabetes Care. 2005;28(5):1245-1249.PubMedGoogle ScholarCrossref
6.
Majumdar  SR, Hemmelgarn  BR, Lin  M, McBrien  K, Manns  BJ, Tonelli  M.  Hypoglycemia associated with hospitalization and adverse events in older people: population-based cohort study.  Diabetes Care. 2013;36(11):3585-3590.PubMedGoogle ScholarCrossref
7.
Lipska  KJ, Ross  JS, Wang  Y,  et al.  National trends in US hospital admissions for hyperglycemia and hypoglycemia among Medicare beneficiaries, 1999 to 2011.  JAMA Intern Med. 2014;174(7):1116-1124.PubMedGoogle ScholarCrossref
8.
Huang  ES, Laiteerapong  N, Liu  JY, John  PM, Moffet  HH, Karter  AJ.  Rates of complications and mortality in older patients with diabetes mellitus: the diabetes and aging study.  JAMA Intern Med. 2014;174(2):251-258.PubMedGoogle ScholarCrossref
9.
Budnitz  DS, Lovegrove  MC, Shehab  N, Richards  CL.  Emergency hospitalizations for adverse drug events in older Americans.  N Engl J Med. 2011;365(21):2002-2012.PubMedGoogle ScholarCrossref
10.
Signorovitch  JE, Macaulay  D, Diener  M,  et al.  Hypoglycaemia and accident risk in people with type 2 diabetes mellitus treated with non-insulin antidiabetes drugs.  Diabetes Obes Metab. 2013;15(4):335-341.PubMedGoogle ScholarCrossref
11.
Stahn  A, Pistrosch  F, Ganz  X,  et al.  Relationship between hypoglycemic episodes and ventricular arrhythmias in patients with type 2 diabetes and cardiovascular diseases: silent hypoglycemias and silent arrhythmias.  Diabetes Care. 2014;37(2):516-520.PubMedGoogle ScholarCrossref
12.
Whitmer  RA, Karter  AJ, Yaffe  K, Quesenberry  CP  Jr, Selby  JV.  Hypoglycemic episodes and risk of dementia in older patients with type 2 diabetes mellitus.  JAMA. 2009;301(15):1565-1572.PubMedGoogle ScholarCrossref
13.
McCoy  RG, Van Houten  HK, Ziegenfuss  JY, Shah  ND, Wermers  RA, Smith  SA.  Increased mortality of patients with diabetes reporting severe hypoglycemia.  Diabetes Care. 2012;35(9):1897-1901.PubMedGoogle ScholarCrossref
14.
Zoungas  S, Patel  A, Chalmers  J,  et al; ADVANCE Collaborative Group.  Severe hypoglycemia and risks of vascular events and death.  N Engl J Med. 2010;363(15):1410-1418.PubMedGoogle ScholarCrossref
15.
Karter  AJ, Subramanian  U, Saha  C,  et al.  Barriers to insulin initiation: the translating research into action for diabetes insulin starts project.  Diabetes Care. 2010;33(4):733-735.PubMedGoogle ScholarCrossref
16.
Laiteerapong  N, Karter  AJ, Liu  JY,  et al.  Correlates of quality of life in older adults with diabetes: the diabetes & aging study.  Diabetes Care. 2011;34(8):1749-1753.PubMedGoogle ScholarCrossref
17.
Nicolucci  A, Pintaudi  B, Rossi  MC,  et al.  The social burden of hypoglycemia in the elderly.  Acta Diabetol. 2015;52(4):677-685.PubMedGoogle ScholarCrossref
18.
Peusens  G, De Jonghe  K, De Rood  I, ,  et al.  Phytoplasmas in pome fruit trees: update of their presence and their vectors in Belgium.  Commun Agric Appl Biol Sci. 2015;80(2):143-148.PubMedGoogle Scholar
19.
Amiel  SA, Dixon  T, Mann  R, Jameson  K.  Hypoglycaemia in type 2 diabetes.  Diabet Med. 2008;25(3):245-254.PubMedGoogle ScholarCrossref
20.
Meulstee  M, Whittemore  R, Watts  SA.  Development of an educational program on prevention of hypoglycemic events among elderly veterans with type 2 diabetes.  Diabetes Educ. 2015;41(6):690-697.PubMedGoogle ScholarCrossref
21.
Rondags  SM, de Wit  M, Snoek  FJ.  HypoAware: development and pilot study of a brief and partly web-based psychoeducational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia.  Diabet Med. 2016;33(2):184-191.PubMedGoogle ScholarCrossref
22.
Elliott  J, Rankin  D, Jacques  RM,  et al; NIHR DAFNE Research Study Group.  A cluster randomized controlled non-inferiority trial of 5-day dose adjustment for normal eating (DAFNE) training delivered over 1 week versus 5-day DAFNE training delivered over 5 weeks: the DAFNE 5 × 1-day trial.  Diabet Med. 2015;32(3):391-398.PubMedGoogle ScholarCrossref
23.
Seaquist  ER, Anderson  J, Childs  B,  et al; American Diabetes Association; Endocrine Society.  Hypoglycemia and diabetes: a report of a workgroup of the American Diabetes Association and the Endocrine Society.  J Clin Endocrinol Metab. 2013;98(5):1845-1859.PubMedGoogle ScholarCrossref
24.
Heinemann  L, Devries  JH.  Evidence for continuous glucose monitoring: sufficient for reimbursement?  Diabet Med. 2014;31(2):122-125.PubMedGoogle ScholarCrossref
25.
Miller  DR, Pogach  L.  Longitudinal approaches to evaluate health care quality and outcomes: the Veterans Health Administration diabetes epidemiology cohorts.  J Diabetes Sci Technol. 2008;2(1):24-32.PubMedGoogle ScholarCrossref
26.
Huang  ES, Liu  JY, Moffet  HH, John  PM, Karter  AJ.  Glycemic control, complications, and death in older diabetic patients: the diabetes and aging study.  Diabetes Care. 2011;34(6):1329-1336.PubMedGoogle ScholarCrossref
27.
Ginde  AA, Blanc  PG, Lieberman  RM, Camargo  CA  Jr.  Validation of ICD-9-CM coding algorithm for improved identification of hypoglycemia visits.  BMC Endocr Disord. 2008;8:4.PubMedGoogle ScholarCrossref
28.
Miller  SI, Wallace  RJ  Jr, Musher  DM, Septimus  EJ, Kohl  S, Baughn  RE.  Hypoglycemia as a manifestation of sepsis.  Am J Med. 1980;68(5):649-654.PubMedGoogle ScholarCrossref
29.
Moons  KG, Altman  DG, Vergouwe  Y, Royston  P.  Prognosis and prognostic research: application and impact of prognostic models in clinical practice.  BMJ. 2009;338:b606.PubMedGoogle ScholarCrossref
30.
Royston  P, Moons  KG, Altman  DG, Vergouwe  Y.  Prognosis and prognostic research: developing a prognostic model.  BMJ. 2009;338:b604.PubMedGoogle ScholarCrossref
31.
Altman  DG, Vergouwe  Y, Royston  P, Moons  KG.  Prognosis and prognostic research: validating a prognostic model.  BMJ. 2009;338:b605.PubMedGoogle ScholarCrossref
32.
Steyerberg  EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009.
33.
Justice  AC, Covinsky  KE, Berlin  JA.  Assessing the generalizability of prognostic information.  Ann Intern Med. 1999;130(6):515-524.PubMedGoogle ScholarCrossref
34.
Breiman  L, Friedman  JH, Olshen  RA, Stone  CJ.  Classification and Regression Trees. Monterey, CA: Wadsworth & Brooks/Cole Advanced Books & Software; 1984.
35.
Fonarow  GC, Adams  KF  Jr, Abraham  WT, Yancy  CW, Boscardin  WJ; ADHERE Scientific Advisory Committee, Study Group, and Investigators.  Risk stratification for in-hospital mortality in acutely decompensated heart failure: classification and regression tree analysis.  JAMA. 2005;293(5):572-580.PubMedGoogle ScholarCrossref
36.
Lewis  RJ. An Introduction to Classification and Regression Tree (CART) Analysis. 2000. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.4103&rep=rep1&type=pdf. Accessed July 6, 2017.
37.
Hosmer  DW, Lemeshow  S.  Applied Logistic Regression. New York, NY: John Wiley & Sons; 1989.
38.
Chambless  LE, Cummiskey  CP, Cui  G.  Several methods to assess improvement in risk prediction models: extension to survival analysis.  Stat Med. 2011;30(1):22-38.PubMedGoogle ScholarCrossref
39.
Moons  KG, de Groot  JA, Bouwmeester  W,  et al.  Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.  PLoS Med. 2014;11(10):e1001744.PubMedGoogle ScholarCrossref
40.
Meurer  WJ, Tolles  J.  Logistic regression diagnostics: understanding how well a model predicts outcomes.  JAMA. 2017;317(10):1068-1069.PubMedGoogle ScholarCrossref
41.
Walter  LC, Brand  RJ, Counsell  SR,  et al.  Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization.  JAMA. 2001;285(23):2987-2994.PubMedGoogle ScholarCrossref
42.
Moffet  HH, Adler  N, Schillinger  D,  et al.  Cohort profile: the Diabetes Study of Northern California (DISTANCE)—objectives and design of a survey follow-up study of social health disparities in a managed care population.  Int J Epidemiol. 2009;38(1):38-47.PubMedGoogle ScholarCrossref
43.
Levey  AS, Bosch  JP, Lewis  JB, Greene  T, Rogers  N, Roth  D; Modification of Diet in Renal Disease Study Group.  A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation.  Ann Intern Med. 1999;130(6):461-470.PubMedGoogle ScholarCrossref
44.
Senior  PA, Bellin  MD, Alejandro  R,  et al; Clinical Islet Transplantation Consortium.  Consistency of quantitative scores of hypoglycemia severity and glycemic lability and comparison with continuous glucose monitoring system measures in long-standing type 1 diabetes.  Diabetes Technol Ther. 2015;17(4):235-242.PubMedGoogle ScholarCrossref
45.
Olsen  SE, Bjørgaas  MR, Åsvold  BO,  et al.  Impaired awareness of hypoglycemia in adults with type 1 diabetes is not associated with autonomic dysfunction or peripheral neuropathy.  Diabetes Care. 2016;39(3):426-433.PubMedGoogle ScholarCrossref
46.
Rondags  SM, de Wit  M, van Tulder  MW, Diamant  M, Snoek  FJ.  HypoAware—a brief and partly web-based psycho-educational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia: design of a cost-effectiveness randomised controlled trial.  BMC Endocr Disord. 2015;15:43.PubMedGoogle ScholarCrossref
47.
Shepard  JA, Vajda  K, Nyer  M, Clarke  W, Gonder-Frederick  L.  Understanding the construct of fear of hypoglycemia in pediatric type 1 diabetes.  J Pediatr Psychol. 2014;39(10):1115-1125.PubMedGoogle ScholarCrossref
48.
Rondags  SM, de Wit  M, Twisk  JW, Snoek  FJ.  Effectiveness of HypoAware, a brief partly web-based psychoeducational intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycemia: a cluster randomized controlled trial.  Diabetes Care. 2016;39(12):2190-2196.PubMedGoogle ScholarCrossref
49.
Pogach  L, Aron  D.  The other side of quality improvement in diabetes for seniors: a proposal for an overtreatment glycemic measure.  Arch Intern Med. 2012;172(19):1510-1512.PubMedGoogle ScholarCrossref
50.
Tseng  CL, Soroka  O, Maney  M, Aron  DC, Pogach  LM.  Assessing potential glycemic overtreatment in persons at hypoglycemic risk.  JAMA Intern Med. 2014;174(2):259-268.PubMedGoogle ScholarCrossref
51.
McCoy  RG, Lipska  KJ, Yao  X, Ross  JS, Montori  VM, Shah  ND.  Intensive treatment and severe hypoglycemia among adults with type 2 diabetes.  JAMA Intern Med. 2016;176(7):969-978.PubMedGoogle ScholarCrossref
52.
Lipska  KJ, Kosiborod  M.  Hypoglycemia and adverse outcomes: marker or mediator?  Rev Cardiovasc Med. 2011;12(3):132-135.PubMedGoogle Scholar
53.
Rodriguez-Gutierrez  R, Lipska  KJ, McCoy  RG, Ospina  NS, Ting  HH, Montori  VM; Hypoglycemia as a Quality Measure in Diabetes Study Group.  Hypoglycemia as an indicator of good diabetes care.  BMJ. 2016;352:i1084.PubMedGoogle ScholarCrossref
54.
Rodriguez-Gutierrez  R, Ospina  NS, McCoy  RG, Lipska  KJ, Shah  ND, Montori  VM; Hypoglycemia as a Quality Measure in Diabetes Study Group.  Inclusion of hypoglycemia in clinical practice guidelines and performance measures in the care of patients with diabetes.  JAMA Intern Med. 2016;176(11):1714-1716.PubMedGoogle ScholarCrossref
55.
Bonds  DE, Miller  ME, Dudl  J,  et al.  Severe hypoglycemia symptoms, antecedent behaviors, immediate consequences and association with glycemia medication usage: secondary analysis of the ACCORD clinical trial data.  BMC Endocr Disord. 2012;12:5.PubMedGoogle ScholarCrossref
56.
Seligman  HK, Bolger  AF, Guzman  D, López  A, Bibbins-Domingo  K.  Exhaustion of food budgets at month’s end and hospital admissions for hypoglycemia.  Health Aff (Millwood). 2014;33(1):116-123.PubMedGoogle ScholarCrossref
57.
Basu  S, Berkowitz  SA, Seligman  H.  The monthly cycle of hypoglycemia: an observational claims-based study of emergency room visits, hospital admissions, and costs in a commercially insured population.  Med Care. 2017;55(7):639-645.PubMedGoogle ScholarCrossref
58.
Berkowitz  SA, Karter  AJ, Lyles  CR,  et al.  Low socioeconomic status is associated with increased risk for hypoglycemia in diabetes patients: the Diabetes Study of Northern California (DISTANCE).  J Health Care Poor Underserved. 2014;25(2):478-490.PubMedGoogle ScholarCrossref
59.
Frier  BM, Jensen  MM, Chubb  BD.  Hypoglycaemia in adults with insulin-treated diabetes in the UK: self-reported frequency and effects.  Diabet Med. 2016;33(8):1125-1132.PubMedGoogle ScholarCrossref
60.
Östenson  CG, Geelhoed-Duijvestijn  P, Lahtela  J, Weitgasser  R, Markert Jensen  M, Pedersen-Bjergaard  U.  Self-reported non-severe hypoglycaemic events in Europe.  Diabet Med. 2014;31(1):92-101.PubMedGoogle ScholarCrossref
61.
Sarkar  U, Karter  AJ, Liu  JY, Moffet  HH, Adler  NE, Schillinger  D.  Hypoglycemia is more common among type 2 diabetes patients with limited health literacy: the Diabetes Study of Northern California (DISTANCE).  J Gen Intern Med. 2010;25(9):962-968.PubMedGoogle ScholarCrossref
62.
Lipska  KJ, Warton  EM, Huang  ES,  et al.  HbA1c and risk of severe hypoglycemia in type 2 diabetes: the Diabetes and Aging Study.  Diabetes Care. 2013;36(11):3535-3542.PubMedGoogle ScholarCrossref
Original Investigation
October 2017

Development and Validation of a Tool to Identify Patients With Type 2 Diabetes at High Risk of Hypoglycemia-Related Emergency Department or Hospital Use

Author Affiliations
  • 1Division of Research, Kaiser Permanente Northern California, Oakland
  • 2Department of General Internal Medicine, University of California, San Francisco
  • 3Department of Epidemiology, University of Washington, Seattle
  • 4Department of Health Services, University of Washington, Seattle
  • 5Section of Endocrinology, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
  • 6Kaiser Permanente Washington Health Research Institute, Seattle
  • 7Section of General Internal Medicine, Department of Medicine, University of Chicago, Chicago, Illinois
  • 8Center for Healthcare Organization and Implementation Research, Edith Nourse Rogers Memorial Veterans Hospital, Bedford, Massachusetts
JAMA Intern Med. 2017;177(10):1461-1470. doi:10.1001/jamainternmed.2017.3844
Key Points

Question  Can electronic medical records be used to reliably categorize risk of future hypoglycemia-related emergency department or hospital use in patients with type 2 diabetes?

Findings  We developed and validated a risk stratification tool that categorized patients’ 12-month risk of hypoglycemia-related utilization using only 6 electronic medical record–based inputs (patient history of hypoglycemia-related utilization, insulin use, sulfonylurea use, emergency department use, chronic kidney disease, and age). Tool performance was validated in 2 fully independent populations.

Meaning  This hypoglycemia risk stratification tool could facilitate efficient targeting of population management interventions to reduce hypoglycemia risk and improve patient safety.

Abstract

Importance  Hypoglycemia-related emergency department (ED) or hospital use among patients with type 2 diabetes (T2D) is clinically significant and possibly preventable.

Objective  To develop and validate a tool to categorize risk of hypoglycemic-related utilization in patients with T2D.

Design, Setting, and Participants  Using recursive partitioning with a split-sample design, we created a classification tree based on potential predictors of hypoglycemia-related ED or hospital use. The resulting model was transcribed into a tool for practical application and tested in 1 internal and 2 fully independent, external samples. Development and internal testing was conducted in a split sample of 206 435 patients with T2D from Kaiser Permanente Northern California (KPNC), an integrated health care system. The tool was externally tested in 1 335 966 Veterans Health Administration and 14 972 Group Health Cooperative patients with T2D.

Exposures  Based on a literature review, we identified 156 candidate predictor variables (prebaseline exposures) using data collected from electronic medical records.

Main Outcomes and Measures  Hypoglycemia-related ED or hospital use during 12 months of follow-up.

Results  The derivation sample (n = 165 148) had a mean (SD) age of 63.9 (13.0) years and included 78 576 (47.6%) women. The crude annual rate of at least 1 hypoglycemia-related ED or hospital encounter in the KPNC derivation sample was 0.49%. The resulting hypoglycemia risk stratification tool required 6 patient-specific inputs: number of prior episodes of hypoglycemia-related utilization, insulin use, sulfonylurea use, prior year ED use, chronic kidney disease stage, and age. We categorized the predicted 12-month risk of any hypoglycemia-related utilization as high (>5%), intermediate (1%-5%), or low (<1%). In the internal validation sample, 2.0%, 10.7%, and 87.3% were categorized as high, intermediate, and low risk, respectively, with observed 12-month hypoglycemia-related utilization rates of 6.7%, 1.4%, and 0.2%, respectively. There was good discrimination in the internal validation KPNC sample (C statistic = 0.83) and both external validation samples (Veterans Health Administration: C statistic = 0.81; Group Health Cooperative: C statistic = 0.79).

Conclusions and Relevance  This hypoglycemia risk stratification tool categorizes the 12-month risk of hypoglycemia-related utilization in patients with T2D using only 6 inputs. This tool could facilitate targeted population management interventions, potentially reducing hypoglycemia risk and improving patient safety and quality of life.

Introduction

Advances in diabetes clinical care and medical treatment have reduced the risk of long-term complications and mortality for the more than 25 million Americans who have diabetes.1 However, iatrogenic hypoglycemia associated with glucose-lowering medication use has become a critical public health and drug safety concern.2-4Quiz Ref ID Severe hypoglycemia is defined as an event necessitating assistance from another person to actively administer carbohydrates, glucagon, or other resuscitative actions.5 Such assistance is often rendered professionally in emergency department (ED) or hospital encounters and is captured as hypoglycemia-related utilization.6

Whereas the risk of severe hypoglycemia is known to be elevated in patients with type 1 diabetes, the risk has been historically underappreciated among patients with type 2 diabetes (T2D), which make up most of the diabetes population. Quiz Ref IDHypoglycemia is now one of the most frequent adverse events in patients with T2D and is more common than acute hyperglycemic emergencies (eg, hyperosmolar hyperglycemic state),7 particularly among older patients and those with a longer history of diabetes.8 One in 4 emergency hospitalizations for adverse drug events is related to hypoglycemia, and these rates are higher in older patients.9 Severe hypoglycemia has been associated with falls and automobile accidents,10 cardiovascular autonomic dysfunction and ventricular arrhythmia,11 dementia,12 and death.13,14 Patients report that fear of hypoglycemia can dissuade them from initiating newly prescribed insulin.15 Hypoglycemia is also strongly predictive of poorer health-related quality of life16 and more diabetes distress.17 Hypoglycemia-related utilization is costly; total annual direct medical costs were estimated at approximately $1.8 billion in 2009 in the United States.18

The risk of hypoglycemia varies widely in patients with T2D.19 Whereas interventions to prevent hypoglycemia exist,20-24 there are no validated methods to target these interventions efficiently. Accordingly, we developed and validated a hypoglycemia risk stratification tool to categorize 12-month risk of hypoglycemia-related emergency department (ED) or hospital use among patients with T2D. This study was approved by the institutional review boards of Kaiser Permanente, the Bedford Veterans Health Administration, and Group Health Cooperative; the requirement that informed consent be obtained from study participants was waived.

Methods
Study Design

We used a prospective cohort study design to develop a risk tool to categorize the 12-month risk of hypoglycemia-related ED or hospital use. Selection of prebaseline candidate predictors was based on a literature review of clinical risk factors associated with hypoglycemia and limited to data typically available in electronic medical records (EMRs). We derived and internally validated this tool in a split sample (4:1) of 206 435 adult patients with T2D in an integrated health care delivery system (Kaiser Permanente Northern California [KPNC]) using clinical and demographic data from EMRs. We used recursive partitioning in the derivation sample to create a risk classification tree. The classification tree leaf nodes were further categorized into high-, intermediate-, or low-risk groups on the basis of predicted risk and then transcribed into the hypoglycemia risk stratification tool. After testing the tool in the internal sample, we conducted external validation in 2 completely independent samples of patients with T2D from the Veterans Administration Diabetes Epidemiology Cohort (DEpiC) (VA sample, n = 1 335 966)25 and from Group Health Cooperative (GH sample, n = 14 972).

Study Population

Using EMR data from KPNC, we identified 233 330 adults (≥21 years as of the baseline date of January 1, 2014) with diabetes with continuous health plan membership for 24 months prebaseline and pharmacy benefits for 12 months prebaseline. We excluded 24 719 patients with unknown diabetes type and 3615 with probable type 1 diabetes according to an algorithm (based on age of onset <30 years and use of insulin alone26). The remaining 206 435 eligible patients with T2D were randomly split into an 80% derivation sample (n = 165 148) for tool development and a 20% internal validation sample (n = 41 287). Similar eligibility criteria were applied in the creation of the 2 external validation samples.

Outcome

Our outcome was the occurrence of any hypoglycemia-related ED or hospital use during 12 months postbaseline. This was defined by having any ED visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained according to a validated definition27 (any of the following International Classification of Diseases, Ninth Revision (ICD-9), codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes). Secondary discharge diagnoses for hypoglycemia were not used because they are often attributable to events that occurred during the ED or hospital encounter (eg, inpatient insulin management, sepsis, acute renal failure28).

Exposures

On the basis of a literature review of clinical risk factors associated with hypoglycemia in T2D, we selected 156 (122 categorical and 34 continuous) candidate clinical, demographic, and behavioral predictor variables for model development (eTable 1 in the Supplement). To increase the utility of our prediction model in other health care settings (transportability, usability29), we excluded variables that were expensive or impractical to collect, had ambiguous meanings, or were not typically available in an EMR. Medication exposures were based on pharmacy dispensings during 6 months prebaseline; laboratory values were based on the last test results within 2 years prebaseline; and prior medical events (eg, history of hypoglycemia-related utilization using the same outcome definition) were based on all available prebaseline records (maximum, 18 years of medical history; mean [SD], 16.5 [3.4] years).

Statistical Analysis

We used standard methodology for model development including a split sample and internal and external validation.30-33 We first regressed the outcome (any hypoglycemia-related utilization) on each of the 156 candidate predictors using univariate logistic regression models to generate odds ratios. We selected candidate variables that had a resulting P < .10. We then used recursive partitioning (using SAS JMP, version 1234) on the selected candidates to construct a binary classification tree to predict the occurrence of at least 1 hypoglycemia-related utilization episode 12 months postbaseline. Recursive partitioning is widely used to generate clinical decision support tools.32,35,36 This method uses a machine-learning, nonlinear, and nonparametric approach to split (“partition”) events into pairs of subgroups based on continuous or categorical predictors, and has the unique advantage that it identifies complex nested interactions, unlike linear modeling methods. Recursive partitioning also optimizes cut points rather than relying on prespecification. Thus, the resultant classification tree identifies predictors that may be important for 1 segment of the population but not others, as well as identifying critical thresholds in continuous or ordinal predictors.

We pruned branches from the classification tree in an attempt to optimize predictive accuracy (performance), model simplicity, practicality of implementation, and intuitive clinical interpretation.31,33 Overly complex models (overfitting), while potentially offering somewhat greater precision, may be less practical, increase the decision and classification costs (expense and time of compiling the predictors), and introduce propagated error associated with predictors measured with uncertainty.

Validation Studies

Model accuracy was assessed in the internal validation sample using standard metrics.30-32(pp255-310)37-39Discrimination, the ability of a model to accurately distinguish between subjects who do vs do not develop the outcome, is based on the area under the receiver-operator curve (C statistic), with greater than 0.7 classified as good discrimination. We also visually assessed calibration (the extent to which the predicted risks over- or underestimate the observed risks) using calibration plots. Given that our goal was not to quantify the numeric probability of a hypoglycemia episode for a given patient but rather to stratify our population into categories of risk, we focused on model discrimination over calibration.33,40

We further evaluated components of generalizability33,39 (ie, reproducibility in patients not used for the derivation of the model) and transportability (practical application in different settings) in the 2 external primary care T2D populations (VA and GH samples).32,33,41 Because these samples included a different disease severity and case mix of patients with T2D from distinct geographical locations with different methods for identifying patients with diabetes, the validation exercises also tested the spectrum, geographic, and methodological transportability of the model.32,33

Sensitivity Analyses

We developed this risk stratification tool using all of the patients’ available medical histories at KPNC (up to 18 years); however, long enrollment may be uncommon in other health care settings. Thus, we conducted further analyses to evaluate whether our tool was sensitive to restrictions in available length of enrollment (prebaseline period transportability33). We also evaluated temporal sensitivity (historical transportability33) of the tool by applying it to KPNC data in the subsequent year (using a baseline date of January 1, 2015). This tool was optimized for patients with T2D, and because not all health care settings can reliably determine diabetes type from their EMR, we also evaluated tool performance when patients with type 1 diabetes were included (ie, sensitivity to misclassification). Finally, as a measure of ecological validity of this tool, we evaluated the association between the predicted level of risk of hypoglycemia-related utilization and actual, self-reported severe hypoglycemia events based on a 2005 survey42 of 15 231 patients with T2D.

Results
Model Selection

Quiz Ref IDThe final classification tree was based on 6 patient-specific variables: total number of prior episodes of hypoglycemia-related ED or hospital utilization (0, 1-2, ≥3 times), number of ED encounters for any reason in the prior 12 months (<2, ≥2 times), insulin use (yes/no), sulfonylurea use (yes/no), presence of severe or end-stage kidney disease (dialysis or chronic kidney disease stage 4 or 5 determined by estimated glomerular filtration rate of ≤29 mL/min/1.73 m2 calculated by the Chronic Kidney Disease Epidemiology Collaboration creatinine equation)43 (yes/no), and age younger than 77 years (yes/no) (Figure 1). This classification tree resulted in 10 mutually exclusive leaf nodes, each yielding an estimated annual risk of hypoglycemia-related utilization, which were categorized as high (>5%), intermediate (1%-5%), or low (<1%). In the KPNC internal validation sample, 2.0% were categorized as high risk, 10.7% as intermediate risk, and 87.3% as low risk.

We then transcribed the classification model into a simple, checklist style, hypoglycemia risk stratification tool by mapping the combinations of risk factors to high, intermediate, or low risk of having any hypoglycemia-related utilization in the following 12 months (Figure 2). This tool instructs the user to identify only 1 of 6 mutually exclusive options, where the first 5 are each defined by a unique combination of predictor variables, and the sixth option is indicated only after ruling out all other options (eTable 2 in the Supplement provides the source code).

Patient Characteristics

We compared the distribution of the 6 predictor variables in our derivation and validation samples (Table 1). There were no significant differences in the distribution of the 6 predictors between the KPNC derivation vs validation samples, but there were significant differences across external validation samples. The proportion of men and women was similar in the KPNC and GH samples, while the VA sample was predominantly men. The mean age was similar across sites, although the proportion older than 77 years was greater in the VA (24.6%), followed by KPNC (17.8%), and GH samples (13.2%). The VA sample had the highest proportion of patients with severe or end-stage kidney disease (3.7%), and 3 or more prior hypoglycemic events (0.4%) vs 2.0% and 0.1%, respectively, in GH. Insulin use was lower in the KPNC samples (20.3%) compared with the GH (30.9%) and VA samples (30.2%). Sulfonylurea use was higher in the KPNC samples (34.9%) compared with the VA (25.0%) and GH (22.5%) samples. The observed annual rate of hypoglycemia-related utilization was lower in GH patients (0.30%) compared with the KPNC derivation sample (0.49%) and the VA (0.51%).

Model Validation

Internal validation of the classification tree model indicated high discrimination (C statistic = 0.83) and good calibration (no significant differences between predicted and observed risk: Pearson χ2 goodness-of-fit P = .31) (Table 2). The odds ratios of hypoglycemia-related utilization among those categorized as high relative to low risk were large in each sample: KPNC internal validation sample (34.6; 95% CI, 24.2-49.3), VA (23.3; 95% CI, 21.9-24.7), and GH (20.7; 95% CI, 8.6-45.0; P < .001).

The tool also performed well in terms of discrimination in the external validation samples (VA C statistic = 0.81; GH C statistic = 0.79). Visual inspection of the calibration plots showed a reasonable match between the predicted and observed risk of hypoglycemia-related utilization within the 10 leaf nodes (Figure 3). However, the tool somewhat overestimated risk among the leaf nodes in the intermediate- and higher-risk categories in the external validation samples.

Sensitivity Analyses

In the sensitivity analyses for length of available medical history, good discrimination was confirmed despite shorter medical history (C statistic = 0.82, 0.83, 0.84 for ≤2, ≤5, and ≤10 years, respectively). We then evaluated temporal sensitivity (historical transportability33). There was good discrimination (C statistic = 0.83) even after KPNC experienced a significant 21% increase in the rate of hypoglycemia-related utilization (0.48% vs 0.59% for 2014 and 2015, respectively). We also evaluated sensitivity to misclassification of diabetes type by including all diabetes patients in our sample and found good discrimination (C statistic = 0.84).

In the assessment of ecological validity, there was a strong association between predicted risk of hypoglycemia-related utilization and self-reported severe hypoglycemia (ie, hypoglycemia necessitating assistance in the past 12 months). Patients categorized as high risk by the tool were 5 times more likely (49.7% vs 9.2%; P < .001) to self-report a severe hypoglycemic episode relative to those categorized as low risk.

Discussion

Health care systems currently lack an evidence-based method for efficiently and systematically identifying patients with T2D at risk of hypoglycemia-related ED or hospital use. We developed and validated a pragmatic hypoglycemia risk stratification tool that uses 6 factors to categorize the 12-month risk of hypoglycemia-related utilization. This tool uses EMR data only and requires no patient contact; it offers an efficient, low-cost approach for identifying patients for targeted interventions to reduce their risk of hypoglycemia.21,44-48 Because of the harms and costs associated with hypoglycemia, high-risk patients are candidates for an elevated level of scrutiny. Identifying medication overtreatment49-51 and including hypoglycemia rates as a health plan–level quality measure have been recommended to drive accountability and quality improvement.52-54

This tool is intended to offer a practical method to risk stratify patients for population management. For example, intensive interventions aimed at reducing hypoglycemia risk could be targeted at the minority of patients with T2D in the high-risk category (2% of patients with diabetes at KPNC). Quiz Ref IDThese interventions could include deintensifying or simplifying medication regimens, addressing impaired hypoglycemic awareness, prescribing glucagon kits or continuous glucose monitors, making referrals to clinical pharmacists or nurse care managers, providing additional diabetes education, and regularly asking about hypoglycemia events occurring outside the medical setting. Clinician discussions could address potential contributors to hypoglycemia, including behavioral (eg, meal skipping55), psychosocial (eg, food insufficiency56,57), or socioeconomic (eg, deprivation58) factors. Similarly, a lower-cost, less intensive intervention could be designed for patients in the intermediate-risk category (11% at KPNC). The intervention could include system-level structural modifications such as risk-based glycemic targets, automated clinical alert flags in the EMR, and automated messaging to patients with elevated risk. Moreover, the tool could be modified to identify specific subsets of risk groups such those with 3 or more hypoglycemia-related ED or hospital encounters whom our model identifies as having the highest risk.

While it is unknown to what extent clinicians are aware of a patient’s hypoglycemia risk, there is evidence that clinicians and patients with diabetes do not communicate about hypoglycemia events that occur outside clinical settings.59,60 Almost all (roughly 95%) severe hypoglycemia events may go clinically unrecognized because they did not result in ED or hospital use.61 In an internal review of EMRs of KPNC patients with T2D, hypoglycemia was absent from the problem lists in 85% of patients categorized by our tool as being at high risk, underscoring the potential for this tool to increase clinicians’ awareness of the risk of hypoglycemia in their patients with T2D.

Limitations

Some limitations should be noted. The final classification tree was one of many possible options, chosen on the basis of performance (eg, C statistic), parsimony, and pragmatism (eg, we excluded predictors that are typically unavailable, impractical, or costly to assess in usual care settings). In developing the model, we excluded secondary discharge diagnoses for hypoglycemia because these events may occur during the ED or hospital encounter (eg, inpatient insulin management, sepsis, acute renal failure28), rather than being a cause of the encounter. On the other hand, hypoglycemia could be a secondary diagnosis if the primary or principal diagnosis is trauma due to an automobile accident or a serious fall caused by hypoglycemia. Although we did not include those events in our model development, we estimate that this would include less than 2% of ED encounters (data not shown). When validating this model, we emphasized discrimination rather than calibration given that our goal was to risk stratify patients into broad categories rather than predicting the continuous level of risk.40 Thus, while the tool successfully stratifies the population into 3 levels of risk, it should not be used to estimate the probability of hypoglycemic-related utilization for an individual patient. Discrimination performance of the tool was validated internally and externally in 2 large, independent populations, suggesting generalizability. Geographic, methodologic, and spectrum transportability were also demonstrated because these external validation populations were from different geographical locations, used differing methods of caring for and identifying patients with T2D, and had different distributions of disease severity and case mix. While the observed rates of hypoglycemia-related utilization were lower in GH (0.3%) and somewhat higher in VA (0.51%) compared with KPNC sample (0.49%), the tool had good discrimination in all 3 samples. Some of the inconsistent findings are attributable to sparse data in the GH validation sample; only 45 events were observed during follow-up. It is also possible that the 2 external sample populations may experience more prevention efforts than at KPNC, explaining the somewhat lower than expected risk in higher-risk patients in the validation samples.

Quiz Ref IDThe tool was designed to predict hypoglycemia-related utilization and thus did not take into account severe hypoglycemia occurring outside the health care system (roughly 95% of hypoglycemic events necessitating third-party assistance are treated by persons other than medical professionals, eg, friends or family61,62). However, we demonstrated a strong and significant association between the tool’s stratification of the risk of hypoglycemia-related utilization and a patient’s self-reported severe hypoglycemia events. Therefore, we believe that preventive interventions targeting patients identified as high risk by this tool may reduce the rate of severe hypoglycemic events that do not result in utilization or clinical recognition. Finally, the tool also proved robust in internal validations when patients with type 1 diabetes were included and when the availability of longitudinal EMR data was restricted.

This risk stratification tool was developed and validated in 3 vertically integrated health care delivery systems (KPNC, GH, and the VA). The tool logic may be programmed into a system’s EMR to allow for automated risk stratification based on the EMR data. Practical barriers may complicate implementing this tool for population management in horizontally integrated or nonintegrated health care delivery systems, for example, where pharmacy claims may not be readily available.

Conclusions

This tool offers a practical, EMR-based method to stratify patients with T2D by their 12-month risk of hypoglycemia-related ED or hospital utilization. This tool could be integrated with targeted preventive interventions to facilitate population management, which ultimately could reduce future hypoglycemia risk and improve patient safety. The 2 criteria indicating high risk are easily memorized (ie, ≥3 previous episodes of hypoglycemia-related utilization, or 1 or 2 episodes if treated with insulin). The criteria for intermediate risk are more nuanced and therefore may be less likely to provoke clinical action in primary care without prompting. Health care systems could adopt a 2-level intervention, with intensive (more expensive) interventions reserved for high-risk patients and less intensive (lower cost) interventions for the intermediate-risk patients. Implementation of this tool could conceivably increase clinician awareness of patients’ hypoglycemia risk. Clinical researchers may also find this tool helpful in identifying patients at high risk for hypoglycemic episodes for either purposeful inclusion or exclusion in clinical trials of novel therapies and diagnostic tests. Quality improvement and impact studies29 are needed to evaluate whether and how implementation of this hypoglycemia risk stratification tool may influence clinician behavior, patient decision making, drug safety, and hypoglycemia incidence. Future research is needed to develop patient-centered and cost-effective interventions to reduce hypoglycemia risk in those identified as being at high risk for hypoglycemia.

Back to top
Article Information

Corresponding Author: Andrew J. Karter, PhD, Division of Research, Kaiser Permanente Northern California, 2000 Broadway, Oakland, CA 94612 (andy.j.karter@kp.org).

Accepted for Publication: June 20, 2017.

Published Online: August 21, 2017. doi:10.1001/jamainternmed.2017.3844

Author Contributions: Dr Karter had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Karter, Warton, Lipska, Moffet, Huang, Miller.

Acquisition, analysis, or interpretation of data: Karter, Warton, Lipska, Ralston, Moffet, Jackson, Miller.

Drafting of the manuscript: Karter, Warton, Jackson.

Critical revision of the manuscript for important intellectual content: Karter, Warton, Lipska, Ralston, Moffet, Huang, Miller.

Statistical analysis: Karter, Warton, Miller.

Obtained funding: Karter, Moffet.

Administrative, technical, or material support: Karter, Ralston, Moffet, Jackson, Miller.

Supervision: Karter, Ralston, Huang.

Conflict of Interest Disclosures: The National Institutes of Health supplied additional funding for our hypoglycemia-related research (NIDDK R01DK103721, R01DK081796). Drs Karter and Huang are also supported by the NIDDK Centers for Diabetes Translational Research (P30 DK092924 and P30 DK092949, respectively). Dr Huang was supported by K24 DK105340. Dr Lipska receives support from the Centers for Medicare & Medicaid Services to develop and maintain publicly reported quality measures, and from the Yale Claude D. Pepper Older Americans Independence Center (P30AG021342) and the National Institute on Aging through the Paul Beeson Career Development Award (K23AG048359). No other disclosures are reported.

Funding/Support: This project was funded by the US Food and Drug Administration (FDA BAA-13 00119).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Gregg  EW, Li  Y, Wang  J,  et al.  Changes in diabetes-related complications in the United States, 1990-2010.  N Engl J Med. 2014;370(16):1514-1523.PubMedGoogle ScholarCrossref
2.
Pogach  L, Aron  D.  Balancing hypoglycemia and glycemic control: a public health approach for insulin safety.  JAMA. 2010;303(20):2076-2077.PubMedGoogle ScholarCrossref
3.
Lee  SJ.  So much insulin, so much hypoglycemia.  JAMA Intern Med. 2014;174(5):686-688.PubMedGoogle ScholarCrossref
4.
Lipska  KJ.  Improving safety of diabetes mellitus management.  JAMA Intern Med. 2014;174(10):1612-1613.PubMedGoogle ScholarCrossref
5.
Workgroup on Hypoglycemia, American Diabetes Association.  Defining and reporting hypoglycemia in diabetes: a report from the American Diabetes Association Workgroup on Hypoglycemia.  Diabetes Care. 2005;28(5):1245-1249.PubMedGoogle ScholarCrossref
6.
Majumdar  SR, Hemmelgarn  BR, Lin  M, McBrien  K, Manns  BJ, Tonelli  M.  Hypoglycemia associated with hospitalization and adverse events in older people: population-based cohort study.  Diabetes Care. 2013;36(11):3585-3590.PubMedGoogle ScholarCrossref
7.
Lipska  KJ, Ross  JS, Wang  Y,  et al.  National trends in US hospital admissions for hyperglycemia and hypoglycemia among Medicare beneficiaries, 1999 to 2011.  JAMA Intern Med. 2014;174(7):1116-1124.PubMedGoogle ScholarCrossref
8.
Huang  ES, Laiteerapong  N, Liu  JY, John  PM, Moffet  HH, Karter  AJ.  Rates of complications and mortality in older patients with diabetes mellitus: the diabetes and aging study.  JAMA Intern Med. 2014;174(2):251-258.PubMedGoogle ScholarCrossref
9.
Budnitz  DS, Lovegrove  MC, Shehab  N, Richards  CL.  Emergency hospitalizations for adverse drug events in older Americans.  N Engl J Med. 2011;365(21):2002-2012.PubMedGoogle ScholarCrossref
10.
Signorovitch  JE, Macaulay  D, Diener  M,  et al.  Hypoglycaemia and accident risk in people with type 2 diabetes mellitus treated with non-insulin antidiabetes drugs.  Diabetes Obes Metab. 2013;15(4):335-341.PubMedGoogle ScholarCrossref
11.
Stahn  A, Pistrosch  F, Ganz  X,  et al.  Relationship between hypoglycemic episodes and ventricular arrhythmias in patients with type 2 diabetes and cardiovascular diseases: silent hypoglycemias and silent arrhythmias.  Diabetes Care. 2014;37(2):516-520.PubMedGoogle ScholarCrossref
12.
Whitmer  RA, Karter  AJ, Yaffe  K, Quesenberry  CP  Jr, Selby  JV.  Hypoglycemic episodes and risk of dementia in older patients with type 2 diabetes mellitus.  JAMA. 2009;301(15):1565-1572.PubMedGoogle ScholarCrossref
13.
McCoy  RG, Van Houten  HK, Ziegenfuss  JY, Shah  ND, Wermers  RA, Smith  SA.  Increased mortality of patients with diabetes reporting severe hypoglycemia.  Diabetes Care. 2012;35(9):1897-1901.PubMedGoogle ScholarCrossref
14.
Zoungas  S, Patel  A, Chalmers  J,  et al; ADVANCE Collaborative Group.  Severe hypoglycemia and risks of vascular events and death.  N Engl J Med. 2010;363(15):1410-1418.PubMedGoogle ScholarCrossref
15.
Karter  AJ, Subramanian  U, Saha  C,  et al.  Barriers to insulin initiation: the translating research into action for diabetes insulin starts project.  Diabetes Care. 2010;33(4):733-735.PubMedGoogle ScholarCrossref
16.
Laiteerapong  N, Karter  AJ, Liu  JY,  et al.  Correlates of quality of life in older adults with diabetes: the diabetes & aging study.  Diabetes Care. 2011;34(8):1749-1753.PubMedGoogle ScholarCrossref
17.
Nicolucci  A, Pintaudi  B, Rossi  MC,  et al.  The social burden of hypoglycemia in the elderly.  Acta Diabetol. 2015;52(4):677-685.PubMedGoogle ScholarCrossref
18.
Peusens  G, De Jonghe  K, De Rood  I, ,  et al.  Phytoplasmas in pome fruit trees: update of their presence and their vectors in Belgium.  Commun Agric Appl Biol Sci. 2015;80(2):143-148.PubMedGoogle Scholar
19.
Amiel  SA, Dixon  T, Mann  R, Jameson  K.  Hypoglycaemia in type 2 diabetes.  Diabet Med. 2008;25(3):245-254.PubMedGoogle ScholarCrossref
20.
Meulstee  M, Whittemore  R, Watts  SA.  Development of an educational program on prevention of hypoglycemic events among elderly veterans with type 2 diabetes.  Diabetes Educ. 2015;41(6):690-697.PubMedGoogle ScholarCrossref
21.
Rondags  SM, de Wit  M, Snoek  FJ.  HypoAware: development and pilot study of a brief and partly web-based psychoeducational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia.  Diabet Med. 2016;33(2):184-191.PubMedGoogle ScholarCrossref
22.
Elliott  J, Rankin  D, Jacques  RM,  et al; NIHR DAFNE Research Study Group.  A cluster randomized controlled non-inferiority trial of 5-day dose adjustment for normal eating (DAFNE) training delivered over 1 week versus 5-day DAFNE training delivered over 5 weeks: the DAFNE 5 × 1-day trial.  Diabet Med. 2015;32(3):391-398.PubMedGoogle ScholarCrossref
23.
Seaquist  ER, Anderson  J, Childs  B,  et al; American Diabetes Association; Endocrine Society.  Hypoglycemia and diabetes: a report of a workgroup of the American Diabetes Association and the Endocrine Society.  J Clin Endocrinol Metab. 2013;98(5):1845-1859.PubMedGoogle ScholarCrossref
24.
Heinemann  L, Devries  JH.  Evidence for continuous glucose monitoring: sufficient for reimbursement?  Diabet Med. 2014;31(2):122-125.PubMedGoogle ScholarCrossref
25.
Miller  DR, Pogach  L.  Longitudinal approaches to evaluate health care quality and outcomes: the Veterans Health Administration diabetes epidemiology cohorts.  J Diabetes Sci Technol. 2008;2(1):24-32.PubMedGoogle ScholarCrossref
26.
Huang  ES, Liu  JY, Moffet  HH, John  PM, Karter  AJ.  Glycemic control, complications, and death in older diabetic patients: the diabetes and aging study.  Diabetes Care. 2011;34(6):1329-1336.PubMedGoogle ScholarCrossref
27.
Ginde  AA, Blanc  PG, Lieberman  RM, Camargo  CA  Jr.  Validation of ICD-9-CM coding algorithm for improved identification of hypoglycemia visits.  BMC Endocr Disord. 2008;8:4.PubMedGoogle ScholarCrossref
28.
Miller  SI, Wallace  RJ  Jr, Musher  DM, Septimus  EJ, Kohl  S, Baughn  RE.  Hypoglycemia as a manifestation of sepsis.  Am J Med. 1980;68(5):649-654.PubMedGoogle ScholarCrossref
29.
Moons  KG, Altman  DG, Vergouwe  Y, Royston  P.  Prognosis and prognostic research: application and impact of prognostic models in clinical practice.  BMJ. 2009;338:b606.PubMedGoogle ScholarCrossref
30.
Royston  P, Moons  KG, Altman  DG, Vergouwe  Y.  Prognosis and prognostic research: developing a prognostic model.  BMJ. 2009;338:b604.PubMedGoogle ScholarCrossref
31.
Altman  DG, Vergouwe  Y, Royston  P, Moons  KG.  Prognosis and prognostic research: validating a prognostic model.  BMJ. 2009;338:b605.PubMedGoogle ScholarCrossref
32.
Steyerberg  EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009.
33.
Justice  AC, Covinsky  KE, Berlin  JA.  Assessing the generalizability of prognostic information.  Ann Intern Med. 1999;130(6):515-524.PubMedGoogle ScholarCrossref
34.
Breiman  L, Friedman  JH, Olshen  RA, Stone  CJ.  Classification and Regression Trees. Monterey, CA: Wadsworth & Brooks/Cole Advanced Books & Software; 1984.
35.
Fonarow  GC, Adams  KF  Jr, Abraham  WT, Yancy  CW, Boscardin  WJ; ADHERE Scientific Advisory Committee, Study Group, and Investigators.  Risk stratification for in-hospital mortality in acutely decompensated heart failure: classification and regression tree analysis.  JAMA. 2005;293(5):572-580.PubMedGoogle ScholarCrossref
36.
Lewis  RJ. An Introduction to Classification and Regression Tree (CART) Analysis. 2000. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.4103&rep=rep1&type=pdf. Accessed July 6, 2017.
37.
Hosmer  DW, Lemeshow  S.  Applied Logistic Regression. New York, NY: John Wiley & Sons; 1989.
38.
Chambless  LE, Cummiskey  CP, Cui  G.  Several methods to assess improvement in risk prediction models: extension to survival analysis.  Stat Med. 2011;30(1):22-38.PubMedGoogle ScholarCrossref
39.
Moons  KG, de Groot  JA, Bouwmeester  W,  et al.  Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.  PLoS Med. 2014;11(10):e1001744.PubMedGoogle ScholarCrossref
40.
Meurer  WJ, Tolles  J.  Logistic regression diagnostics: understanding how well a model predicts outcomes.  JAMA. 2017;317(10):1068-1069.PubMedGoogle ScholarCrossref
41.
Walter  LC, Brand  RJ, Counsell  SR,  et al.  Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization.  JAMA. 2001;285(23):2987-2994.PubMedGoogle ScholarCrossref
42.
Moffet  HH, Adler  N, Schillinger  D,  et al.  Cohort profile: the Diabetes Study of Northern California (DISTANCE)—objectives and design of a survey follow-up study of social health disparities in a managed care population.  Int J Epidemiol. 2009;38(1):38-47.PubMedGoogle ScholarCrossref
43.
Levey  AS, Bosch  JP, Lewis  JB, Greene  T, Rogers  N, Roth  D; Modification of Diet in Renal Disease Study Group.  A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation.  Ann Intern Med. 1999;130(6):461-470.PubMedGoogle ScholarCrossref
44.
Senior  PA, Bellin  MD, Alejandro  R,  et al; Clinical Islet Transplantation Consortium.  Consistency of quantitative scores of hypoglycemia severity and glycemic lability and comparison with continuous glucose monitoring system measures in long-standing type 1 diabetes.  Diabetes Technol Ther. 2015;17(4):235-242.PubMedGoogle ScholarCrossref
45.
Olsen  SE, Bjørgaas  MR, Åsvold  BO,  et al.  Impaired awareness of hypoglycemia in adults with type 1 diabetes is not associated with autonomic dysfunction or peripheral neuropathy.  Diabetes Care. 2016;39(3):426-433.PubMedGoogle ScholarCrossref
46.
Rondags  SM, de Wit  M, van Tulder  MW, Diamant  M, Snoek  FJ.  HypoAware—a brief and partly web-based psycho-educational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia: design of a cost-effectiveness randomised controlled trial.  BMC Endocr Disord. 2015;15:43.PubMedGoogle ScholarCrossref
47.
Shepard  JA, Vajda  K, Nyer  M, Clarke  W, Gonder-Frederick  L.  Understanding the construct of fear of hypoglycemia in pediatric type 1 diabetes.  J Pediatr Psychol. 2014;39(10):1115-1125.PubMedGoogle ScholarCrossref
48.
Rondags  SM, de Wit  M, Twisk  JW, Snoek  FJ.  Effectiveness of HypoAware, a brief partly web-based psychoeducational intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycemia: a cluster randomized controlled trial.  Diabetes Care. 2016;39(12):2190-2196.PubMedGoogle ScholarCrossref
49.
Pogach  L, Aron  D.  The other side of quality improvement in diabetes for seniors: a proposal for an overtreatment glycemic measure.  Arch Intern Med. 2012;172(19):1510-1512.PubMedGoogle ScholarCrossref
50.
Tseng  CL, Soroka  O, Maney  M, Aron  DC, Pogach  LM.  Assessing potential glycemic overtreatment in persons at hypoglycemic risk.  JAMA Intern Med. 2014;174(2):259-268.PubMedGoogle ScholarCrossref
51.
McCoy  RG, Lipska  KJ, Yao  X, Ross  JS, Montori  VM, Shah  ND.  Intensive treatment and severe hypoglycemia among adults with type 2 diabetes.  JAMA Intern Med. 2016;176(7):969-978.PubMedGoogle ScholarCrossref
52.
Lipska  KJ, Kosiborod  M.  Hypoglycemia and adverse outcomes: marker or mediator?  Rev Cardiovasc Med. 2011;12(3):132-135.PubMedGoogle Scholar
53.
Rodriguez-Gutierrez  R, Lipska  KJ, McCoy  RG, Ospina  NS, Ting  HH, Montori  VM; Hypoglycemia as a Quality Measure in Diabetes Study Group.  Hypoglycemia as an indicator of good diabetes care.  BMJ. 2016;352:i1084.PubMedGoogle ScholarCrossref
54.
Rodriguez-Gutierrez  R, Ospina  NS, McCoy  RG, Lipska  KJ, Shah  ND, Montori  VM; Hypoglycemia as a Quality Measure in Diabetes Study Group.  Inclusion of hypoglycemia in clinical practice guidelines and performance measures in the care of patients with diabetes.  JAMA Intern Med. 2016;176(11):1714-1716.PubMedGoogle ScholarCrossref
55.
Bonds  DE, Miller  ME, Dudl  J,  et al.  Severe hypoglycemia symptoms, antecedent behaviors, immediate consequences and association with glycemia medication usage: secondary analysis of the ACCORD clinical trial data.  BMC Endocr Disord. 2012;12:5.PubMedGoogle ScholarCrossref
56.
Seligman  HK, Bolger  AF, Guzman  D, López  A, Bibbins-Domingo  K.  Exhaustion of food budgets at month’s end and hospital admissions for hypoglycemia.  Health Aff (Millwood). 2014;33(1):116-123.PubMedGoogle ScholarCrossref
57.
Basu  S, Berkowitz  SA, Seligman  H.  The monthly cycle of hypoglycemia: an observational claims-based study of emergency room visits, hospital admissions, and costs in a commercially insured population.  Med Care. 2017;55(7):639-645.PubMedGoogle ScholarCrossref
58.
Berkowitz  SA, Karter  AJ, Lyles  CR,  et al.  Low socioeconomic status is associated with increased risk for hypoglycemia in diabetes patients: the Diabetes Study of Northern California (DISTANCE).  J Health Care Poor Underserved. 2014;25(2):478-490.PubMedGoogle ScholarCrossref
59.
Frier  BM, Jensen  MM, Chubb  BD.  Hypoglycaemia in adults with insulin-treated diabetes in the UK: self-reported frequency and effects.  Diabet Med. 2016;33(8):1125-1132.PubMedGoogle ScholarCrossref
60.
Östenson  CG, Geelhoed-Duijvestijn  P, Lahtela  J, Weitgasser  R, Markert Jensen  M, Pedersen-Bjergaard  U.  Self-reported non-severe hypoglycaemic events in Europe.  Diabet Med. 2014;31(1):92-101.PubMedGoogle ScholarCrossref
61.
Sarkar  U, Karter  AJ, Liu  JY, Moffet  HH, Adler  NE, Schillinger  D.  Hypoglycemia is more common among type 2 diabetes patients with limited health literacy: the Diabetes Study of Northern California (DISTANCE).  J Gen Intern Med. 2010;25(9):962-968.PubMedGoogle ScholarCrossref
62.
Lipska  KJ, Warton  EM, Huang  ES,  et al.  HbA1c and risk of severe hypoglycemia in type 2 diabetes: the Diabetes and Aging Study.  Diabetes Care. 2013;36(11):3535-3542.PubMedGoogle ScholarCrossref
×