[Skip to Navigation]
Sign In
Figure 1.  Flowcharts of the Derivation and External Validation Cohorts
Flowcharts of the Derivation and External Validation Cohorts

AKDN indicates Alberta Kidney Disease Network; AKI, acute kidney injury; ESKD, end-stage kidney disease; KRT, kidney replacement therapy.

Figure 2.  Observed vs Predicted Risk of Acute Kidney Injury Requiring Kidney Replacement Therapy in Model 2 by Clinical Risk Categories in the Derivation and External Validation Cohorts
Observed vs Predicted Risk of Acute Kidney Injury Requiring Kidney Replacement Therapy in Model 2 by Clinical Risk Categories in the Derivation and External Validation Cohorts

AKI indicates acute kidney injury.

Table 1.  Baseline Characteristics of the Derivation and External Validation Cohorts
Baseline Characteristics of the Derivation and External Validation Cohorts
Table 2.  Odds Ratios for Predictive Variables Included in Each Model Developed in the Derivation Cohorta
Odds Ratios for Predictive Variables Included in Each Model Developed in the Derivation Cohorta
Table 3.  Predictive Values for Model 2 and the Risk Index in the Derivation and External Validation Cohorts
Predictive Values for Model 2 and the Risk Index in the Derivation and External Validation Cohorts
1.
Bellomo  R, Kellum  JA, Ronco  C.  Acute kidney injury.   Lancet. 2012;380(9843):756-766. doi:10.1016/S0140-6736(11)61454-2 PubMedGoogle ScholarCrossref
2.
Brienza  N, Giglio  MT, Marucci  M.  Preventing acute kidney injury after noncardiac surgery.   Curr Opin Crit Care. 2010;16(4):353-358. doi:10.1097/MCC.0b013e32833a9ef5 PubMedGoogle ScholarCrossref
3.
Chertow  GM, Burdick  E, Honour  M, Bonventre  JV, Bates  DW.  Acute kidney injury, mortality, length of stay, and costs in hospitalized patients.   J Am Soc Nephrol. 2005;16(11):3365-3370. doi:10.1681/ASN.2004090740 PubMedGoogle ScholarCrossref
4.
Bellomo  R.  Acute renal failure.   Semin Respir Crit Care Med. 2011;32(5):639-650. doi:10.1055/s-0031-1287872 PubMedGoogle ScholarCrossref
5.
Siddiqui  NF, Coca  SG, Devereaux  PJ,  et al.  Secular trends in acute dialysis after elective major surgery—1995 to 2009.   CMAJ. 2012;184(11):1237-1245. doi:10.1503/cmaj.110895 PubMedGoogle ScholarCrossref
6.
Hsu  RK, McCulloch  CE, Dudley  RA, Lo  LJ, Hsu  C.  Temporal changes in incidence of dialysis-requiring AKI.   J Am Soc Nephrol. 2013;24(1):37-42. doi:10.1681/ASN.2012080800 PubMedGoogle ScholarCrossref
7.
Collister  D, Pannu  N, Ye  F,  et al; Alberta Kidney Disease Network.  Health care costs associated with AKI.   Clin J Am Soc Nephrol. 2017;12(11):1733-1743. doi:10.2215/CJN.00950117 PubMedGoogle ScholarCrossref
8.
James  MT, Bhatt  M, Pannu  N, Tonelli  M.  Long-term outcomes of acute kidney injury and strategies for improved care.   Nat Rev Nephrol. 2020;16(4):193-205. doi:10.1038/s41581-019-0247-z PubMedGoogle ScholarCrossref
9.
Bell  S, Prowle  J.  Postoperative AKI-prevention is better than cure?   J Am Soc Nephrol. 2019;30(1):4-6. doi:10.1681/ASN.2018111127 PubMedGoogle ScholarCrossref
10.
Golden  D, Corbett  J, Forni  LG.  Peri-operative renal dysfunction: prevention and management.   Anaesthesia. 2016;71(suppl 1):51-57. doi:10.1111/anae.13313 PubMedGoogle ScholarCrossref
11.
Wilson  T, Quan  S, Cheema  K,  et al.  Risk prediction models for acute kidney injury following major noncardiac surgery: systematic review.   Nephrol Dial Transplant. 2015;31(2):231-240. doi:10.1093/ndt/gfv415 PubMedGoogle Scholar
12.
Park  S, Cho  H, Park  S,  et al.  Simple postoperative AKI risk (SPARK) classification before noncardiac surgery: a prediction index development study with external validation.   J Am Soc Nephrol. 2019;30(1):170-181. doi:10.1681/ASN.2018070757 PubMedGoogle ScholarCrossref
13.
Bihorac  A, Yavas  S, Subbiah  S,  et al.  Long-term risk of mortality and acute kidney injury during hospitalization after major surgery.   Ann Surg. 2009;249(5):851-858. doi:10.1097/SLA.0b013e3181a40a0bGoogle ScholarCrossref
14.
Hoste  EA, Bagshaw  SM, Bellomo  R,  et al.  Epidemiology of acute kidney injury in critically ill patients: the multinational AKI-EPI study.   Intensive Care Med. 2015;41(8):1411-1423. doi:10.1007/s00134-015-3934-7 PubMedGoogle ScholarCrossref
15.
Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group.  KDIGO Clinical Practice Guideline for Acute Kidney Injury.   Kidney Int Suppl. 2012;2(1):1-138. doi:10.1038/kisup.2012.6Google ScholarCrossref
16.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD).   Ann Intern Med. 2015;162(10):735-736. doi:10.7326/L15-5093-2 PubMedGoogle ScholarCrossref
17.
Hemmelgarn  BR, Clement  F, Manns  BJ,  et al.  Overview of the Alberta Kidney Disease Network.   BMC Nephrol. 2009;10:30. doi:10.1186/1471-2369-10-30 PubMedGoogle ScholarCrossref
18.
Quan  H, Sundararajan  V, Halfon  P,  et al.  Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.   Med Care. 2005;43(11):1130-1139. doi:10.1097/01.mlr.0000182534.19832.83 PubMedGoogle ScholarCrossref
19.
Quan  H, Khan  N, Hemmelgarn  BR,  et al; Hypertension Outcome and Surveillance Team of the Canadian Hypertension Education Programs.  Validation of a case definition to define hypertension using administrative data.   Hypertension. 2009;54(6):1423-1428. doi:10.1161/HYPERTENSIONAHA.109.139279 PubMedGoogle ScholarCrossref
20.
Vlasschaert  ME, Bejaimal  SA, Hackam  DG,  et al.  Validity of administrative database coding for kidney disease: a systematic review.   Am J Kidney Dis. 2011;57(1):29-43. doi:10.1053/j.ajkd.2010.08.031 PubMedGoogle ScholarCrossref
21.
De Coster  C, Li  B, Quan  H.  Comparison and validity of procedures coded with ICD-9-CM and ICD-10-CA/CCI.   Med Care. 2008;46(6):627-634. doi:10.1097/MLR.0b013e3181649439 PubMedGoogle ScholarCrossref
22.
Hux  JE, Ivis  F, Flintoft  V, Bica  A.  Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm.   Diabetes Care. 2002;25(3):512-516. doi:10.2337/diacare.25.3.512 PubMedGoogle ScholarCrossref
23.
Juurlink  D, Preyra C, Croxford  R, et al  .  Canadian Institute for Health Information Discharge Abstract Database: A Validation Study. ICES; 2006.
24.
Waikar  SS, Wald  R, Chertow  GM,  et al.  Validity of International Classification of Diseases, Ninth Revision, Clinical Modification codes for acute renal failure.   J Am Soc Nephrol. 2006;17(6):1688-1694. doi:10.1681/ASN.2006010073 PubMedGoogle ScholarCrossref
25.
Biteker  M, Dayan  A, Tekkeşin  AI,  et al.  Incidence, risk factors, and outcomes of perioperative acute kidney injury in noncardiac and nonvascular surgery.   Am J Surg. 2014;207(1):53-59. doi:10.1016/j.amjsurg.2013.04.006 PubMedGoogle ScholarCrossref
26.
Borthwick  E, Ferguson  A.  Perioperative acute kidney injury: risk factors, recognition, management, and outcomes.   BMJ. 2010;341:c3365. doi:10.1136/bmj.c3365 PubMedGoogle ScholarCrossref
27.
Kheterpal  S, Tremper  KK, Heung  M,  et al.  Development and validation of an acute kidney injury risk index for patients undergoing general surgery: results from a national data set.   Anesthesiology. 2009;110(3):505-515. doi:10.1097/ALN.0b013e3181979440 PubMedGoogle ScholarCrossref
28.
Levey  AS, Stevens  LA, Schmid  CH,  et al; CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration).  A new equation to estimate glomerular filtration rate.   Ann Intern Med. 2009;150(9):604-612. doi:10.7326/0003-4819-150-9-200905050-00006 PubMedGoogle ScholarCrossref
29.
Lamb  EJ, MacKenzie  F, Stevens  PE.  How should proteinuria be detected and measured?   Ann Clin Biochem. 2009;46(Pt 3):205-217. doi:10.1258/acb.2009.009007 PubMedGoogle Scholar
30.
Harrell  FE Jr.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer; 2001. doi:10.1007/978-1-4757-3462-1
31.
Steyerberg  EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating.: Springer; 2009. doi:10.1007/978-0-387-77244-8
32.
Pencina  MJ, D’Agostino  RB  Sr, D’Agostino  RB  Jr, Vasan  RS.  Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.   Stat Med. 2008;27(2):157-172. doi:10.1002/sim.2929 PubMedGoogle ScholarCrossref
33.
Pencina  KM, Pencina  MJ, D’Agostino  RB  Sr.  What to expect from net reclassification improvement with three categories.   Stat Med. 2014;33(28):4975-4987. doi:10.1002/sim.6286 PubMedGoogle ScholarCrossref
34.
Leening  MJG, Vedder  MM, Witteman  JCM, Pencina  MJ, Steyerberg  EW.  Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide.   Ann Intern Med. 2014;160(2):122-131. doi:10.7326/M13-1522 PubMedGoogle ScholarCrossref
35.
Pepe  MS, Janes  H, Li  CI.  Net risk reclassification p values: valid or misleading?   J Natl Cancer Inst. 2014;106(4):dju041. doi:10.1093/jnci/dju041 PubMedGoogle Scholar
36.
Saito  T, Rehmsmeier  M.  The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets.   PLoS One. 2015;10(3):e0118432. doi:10.1371/journal.pone.0118432 PubMedGoogle Scholar
37.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: the Framingham Study risk score functions.   Stat Med. 2004;23(10):1631-1660. doi:10.1002/sim.1742 PubMedGoogle ScholarCrossref
38.
Janssen  KJM, Moons  KGM, Kalkman  CJ, Grobbee  DE, Vergouwe  Y.  Updating methods improved the performance of a clinical prediction model in new patients.   J Clin Epidemiol. 2008;61(1):76-86. doi:10.1016/j.jclinepi.2007.04.018 PubMedGoogle ScholarCrossref
39.
Grams  ME, Astor  BC, Bash  LD, Matsushita  K, Wang  Y, Coresh  J.  Albuminuria and estimated glomerular filtration rate independently associate with acute kidney injury.   J Am Soc Nephrol. 2010;21(10):1757-1764. doi:10.1681/ASN.2010010128 PubMedGoogle ScholarCrossref
40.
Coca  SG, Yalavarthy  R, Concato  J, Parikh  CR.  Biomarkers for the diagnosis and risk stratification of acute kidney injury: a systematic review.   Kidney Int. 2008;73(9):1008-1016. doi:10.1038/sj.ki.5002729 PubMedGoogle ScholarCrossref
41.
Hsu  RK, Hsu  CY.  Proteinuria and reduced glomerular filtration rate as risk factors for acute kidney injury.   Curr Opin Nephrol Hypertens. 2011;20(3):211-217. doi:10.1097/MNH.0b013e3283454f8d PubMedGoogle ScholarCrossref
42.
Wessler  BS, Ruthazer  R, Udelson  JE,  et al.  Regional validation and recalibration of clinical predictive models for patients with acute heart failure.   J Am Heart Assoc. 2017;6(11):e006121. doi:10.1161/JAHA.117.006121 PubMedGoogle Scholar
43.
Davis  SE, Lasko  TA, Chen  G, Siew  ED, Matheny  ME.  Calibration drift in regression and machine learning models for acute kidney injury.   J Am Med Inform Assoc. 2017;24(6):1052-1061. doi:10.1093/jamia/ocx030 PubMedGoogle ScholarCrossref
44.
Halbesma  N, Jansen  DF, Heymans  MW, Stolk  RP, de Jong  PE, Gansevoort  RT; PREVEND Study Group.  Development and validation of a general population renal risk score.   Clin J Am Soc Nephrol. 2011;6(7):1731-1738. doi:10.2215/CJN.08590910 PubMedGoogle ScholarCrossref
45.
Teixeira  C, Rosa  R, Rodrigues  N,  et al.  Acute kidney injury after major abdominal surgery: a retrospective cohort analysis.   Crit Care Res Pract. 2014;2014:132175. doi:10.1155/2014/132175 PubMedGoogle Scholar
46.
Kheterpal  S, Tremper  KK, Englesbe  MJ,  et al.  Predictors of postoperative acute renal failure after noncardiac surgery in patients with previously normal renal function.   Anesthesiology. 2007;107(6):892-902. doi:10.1097/01.anes.0000290588.29668.38 PubMedGoogle ScholarCrossref
47.
Bagshaw  SM, Wald  R, Barton  J,  et al.  Clinical factors associated with initiation of renal replacement therapy in critically ill patients with acute kidney injury-a prospective multicenter observational study.   J Crit Care. 2012;27(3):268-275. doi:10.1016/j.jcrc.2011.06.003 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Views 3,121
    Citations 0
    Original Investigation
    Nephrology
    August 23, 2021

    Derivation and External Validation of a Risk Index for Predicting Acute Kidney Injury Requiring Kidney Replacement Therapy After Noncardiac Surgery

    Author Affiliations
    • 1Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
    • 2Department of Pathology and Laboratory Medicine, University of Calgary, Calgary, Alberta, Canada
    • 3Alberta Precision Laboratories, Calgary, Alberta, Canada
    • 4Libin Cardiovascular Institute, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
    • 5Department of Medicine, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
    • 6O’Brien Institute for Public Health, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
    • 7ICES, Toronto, Ontario, Canada
    • 8Department of Medicine, Western University, London, Ontario, Canada
    • 9Department of Epidemiology & Biostatistics, Western University, London, Ontario, Canada
    • 10Department of Medicine, University of Alberta, Edmonton, Alberta, Canada
    JAMA Netw Open. 2021;4(8):e2121901. doi:10.1001/jamanetworkopen.2021.21901
    Key Points

    Question  Can acute kidney injury requiring kidney replacement therapy after major noncardiac surgery be accurately predicted from readily available preoperative data?

    Findings  In this prognostic study of a risk prediction model developed with 92 114 adults and externally validated with 709 086 adults, in external validation, a model including age, sex, surgery type, estimated glomerular filtration rate, hemoglobin concentration, albuminuria, and history of myocardial infarction or liver disease had sensitivity of 21.2%, specificity of 99.9%, positive predictive value of 38.1%, and negative predictive value of 99.7% at a predicted risk threshold of 10% or greater.

    Meaning  The findings suggest that this risk model can predict acute kidney injury requiring kidney replacement therapy after noncardiac surgery using routine preoperative data and may be feasible for implementation in perioperative care.

    Abstract

    Importance  Severe acute kidney injury (AKI) is a serious postoperative complication. A tool for predicting the risk of AKI requiring kidney replacement therapy (KRT) after major noncardiac surgery might assist with patient counseling and targeted use of measures to reduce this risk.

    Objective  To derive and validate a predictive model for AKI requiring KRT after major noncardiac surgery.

    Design, Setting, and Participants  In this prognostic study, 5 risk prediction models were derived and internally validated in a population-based cohort of adults without preexisting kidney failure who underwent noncardiac surgery in Alberta, Canada, between January 1, 2004, and December 31, 2013. The best performing model and corresponding risk index were externally validated in a population-based cohort of adults without preexisting kidney failure who underwent noncardiac surgery in Ontario, Canada, between January 1, 2007, and December 31, 2017. Data analysis was conducted from September 1, 2019, to May 31, 2021.

    Exposures  Demographic characteristics, surgery type, laboratory measures, and comorbidities before surgery.

    Main Outcomes and Measures  Acute kidney injury requiring KRT within 14 days after surgery. Discrimination was assessed using the C statistic; calibration was assessed using calibration intercept and slope. Logistic recalibration was used to optimize model calibration in the external validation cohort.

    Results  The derivation cohort included 92 114 patients (52.2% female; mean [SD] age, 62.3 [18.0] years), and the external validation cohort included 709 086 patients (50.8% female; mean [SD] age, 61.0 [16.0] years). A total of 529 patients (0.6%) developed postoperative AKI requiring KRT in the derivation cohort, and 2956 (0.4%) developed postoperative AKI requiring KRT in the external validation cohort. The following factors were consistently associated with the risk of AKI requiring KRT: younger age (40-69 years: odds ratio [OR], 2.07 [95% CI, 1.69-2.53]; <40 years: OR, 3.73 [95% CI, 2.61-5.33]), male sex (OR, 1.55; 95% CI, 1.28-1.87), surgery type (colorectal: OR, 4.86 [95% CI, 3.28-7.18]; liver or pancreatic: OR, 6.46 [95% CI, 3.85-10.83]; other abdominal: OR, 2.19 [95% CI, 1.66-2.89]; abdominal aortic aneurysm repair: OR, 19.34 [95% CI, 14.31-26.14]; other vascular: OR, 7.30 [95% CI, 5.48-9.73]; thoracic: OR, 3.41 [95% CI, 2.07-5.59]), lower estimated glomerular filtration rate (OR, 0.97; 95% CI, 0.97-0.97 per 1 mL/min/1.73 m2 increase), lower hemoglobin concentration (OR, 0.99; 95% CI, 0.98-0.99 per 0.1 g/dL increase), albuminuria (mild: OR, 1.88 [95% CI, 1.52-2.33]; heavy: OR, 3.74 [95% CI, 2.98-4.69]), history of myocardial infarction (OR, 1.63; 95% CI, 1.32-2.03), and liver disease (mild: OR, 2.32 [95% CI, 1.66-3.24]; moderate or severe: OR, 4.96 [95% CI, 3.58-6.85]). In external validation, a final model including these variables showed excellent discrimination (C statistic, 0.95; 95% CI, 0.95-0.96), with sensitivity of 21.2%, specificity of 99.9%, positive predictive value of 38.1%, and negative predictive value of 99.7% at a predicted risk threshold of 10% or greater.

    Conclusions and Relevance  The findings suggest that this risk model can predict AKI requiring KRT after noncardiac surgery using routine preoperative data. The model may be feasible for implementation in clinical perioperative risk stratification for severe AKI.

    Introduction

    Acute kidney injury (AKI) is common after major surgery and is associated with adverse outcomes, including prolonged hospitalization, chronic kidney disease, and death.1-3 Postoperative AKI accounts for more than one-third of cases of AKI in the hospital setting,4 and the incidence of AKI requiring kidney replacement therapy (KRT) after major elective surgery increased almost 4-fold between 1995 and 2009.5,6 When AKI is severe enough to require KRT, it is resource intensive and is associated with adverse patient experiences and outcomes.7,8 Quantifying the risk of AKI requiring KRT is an important part of the surgical informed consent process and may impact perioperative decision-making.9,10

    Several validated prediction models for AKI after cardiac surgery are available; however, few such tools exist for AKI after noncardiac surgical procedures,11,12 even though the risk may be substantial for some patients.5,6,13,14 Clinical practice guidelines15 have included the recommendation that patients be stratified for risk of AKI. However, a lack of accurate risk-stratification approaches for clinically significant AKI after noncardiac surgery is a barrier to further research and clinical uptake of approaches to mitigate the risk of AKI. Accurate identification of patients at increased risk for AKI may allow for increased monitoring and supportive strategies. Therefore, AKI guidelines from Kidney Disease: Improving Global Outcomes have included a recommendation for research in the development and validation of AKI risk prediction tools for use in contexts other than cardiac surgery.15

    We used population-based cohorts to develop and externally validate risk prediction models and an integer-based risk index for AKI requiring KRT after major noncardiac surgery. We framed the risk prediction tools for risk stratification in preoperative clinical settings and focused on risk factors readily identifiable before surgery.

    Methods

    This prognostic study was approved by the Conjoint Health Research Ethics Board at the University of Calgary, Calgary, Alberta, Canada, with a waiver of informed consent because retrospective deidentified data were used. We adhered to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline for reporting multivariable prediction model development and validation.16

    Derivation and Validation Cohorts

    We obtained the model derivation cohort from the Alberta Kidney Disease Network database and Alberta Health administrative data using approaches described elsewhere.17 We used International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) from physician claims in Alberta and International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) and Canadian Classification of Health Intervention procedure codes from hospital discharge abstracts to characterize patient comorbidities and procedures.18-21 We included all patients aged 18 to 95 years who underwent surgical repair of abdominal aortic aneurysm or had surgical procedures performed as treatment for colorectal, liver, or pancreatic disease and those who underwent other abdominal, retroperitoneal, thoracic, vascular, or musculoskeletal surgical procedures in Alberta, Canada, between January 1, 2004, and December 31, 2013. We excluded patients without a preoperative serum creatinine measurement within 30 days before surgery and patients with end-stage kidney disease (defined by receipt of dialysis or kidney transplant or a baseline estimated glomerular filtration rate [eGFR] <10 mL/min/1.73 m2) before surgery. Diagnostic and procedure codes for cohort development and characterization are shown in eTables 1 and 2 in the Supplement.

    The external validation cohort was derived using data sets from Ontario, Canada, from January 1, 2007, through December 31, 2017, that were analyzed at ICES (formerly Institute for Clinical Evaluative Science) in London, Ontario. ICES houses laboratory information and provincial administrative data for patients in Ontario, Canada. Data analysis was conducted from September 1, 2019, to May 31, 2021. Cohort inclusion criteria were identical to those of the derivation cohort, with similar ICD-9-CM and ICD-10 coding approaches used to characterize patient comorbidities and procedures.22,23

    Outcome Measure and Candidate Predictor Variables

    We defined AKI requiring KRT using ICD-10 diagnosis codes for AKI and Canadian Classification of Health procedure codes for KRT (eTable 1 in the Supplement) within 14 days of surgery by following a validated approach that has been shown to have high sensitivity and specificity for AKI requiring KRT.20,21,24 We identified candidate predictor variables by means of literature review and their routine availability in clinical care.17,25-27 We included demographic data, laboratory measures, comorbidities, surgical procedures,5 and status of the surgery as emergent or urgent vs elective. We used the closest measurement within 30 days and included the patient’s date of surgery to determine the baseline serum creatinine concentration and eGFR based on the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation.28 We used hemoglobin measurements from before surgery closest to the patient’s surgery date. Albuminuria was based on measurements closest to the surgery date using urine dipstick or the albumin to creatinine ratio, categorized as normal (negative; <30 mg/g; to convert albumin to creatine ratio to mg/mmol, multiply by 0.113), mild (trace; ≥1 or 30-300 mg/g), or heavy (≥2 or ≥300 mg/g) within 1 year before surgery.29 We identified comorbidities up to 3 years before surgery from hospital discharge abstracts and physician claims using validated coding algorithms.18,19

    Statistical Analysis

    We evaluated the linearity of associations between the outcome and continuous variables (age, eGFR, and hemoglobin concentration) using locally weighted scatterplot smoothing and restricted cubic spline analysis. We confirmed linear associations for eGFR and hemoglobin concentration with AKI requiring KRT. Locally weighted scatterplot smoothing analysis showed a quadratic association of the outcome with age, which we therefore categorized as younger than 40 years, 40 to 69 years, and 70 years or older in the models. To address unmeasured albuminuria status, we performed a single imputation, separately within each cohort, with an ordinal logistic regression model with all variables included in the full model to impute missing values. Missing hemoglobin measurements for 22 patients (<0.001%) were imputed to the mean hemoglobin concentration in the cohort (127 g/L; to convert to g/dL, divide by 10.0).

    We developed 5 multivariable logistic regression models based on 3 different strategies. First, we fit a full model including all candidate predictors. We performed bootstrap resampling for variable selection using 1000 bootstrap samples of the full cohort and fit a model using the variables that were statistically significant at a 2-sided 1% level of significance in at least 80% of bootstrap samples. We developed other models with increasing simplicity by further removing variables that might be less consistently available in a preoperative setting. We determined the discrimination of each model in the derivation cohort by calculating the apparent C statistic.

    The full model (model 1) included 30 variables. After bootstrap resampling, 15 variables were selected in at least 80% of samples and included in model 2. Model 3 included variables from model 2 but excluded albuminuria. Model 4 consisted of age, sex, surgery type, eGFR, and hemoglobin concentration (10 variables); only age, sex, and surgery type were included in model 5.

    We internally validated each model using bootstrap resampling. We took 100 bootstrap samples and fit the respective model, calculated the C statistic, and determined bootstrap and test C statistics. We subtracted the difference between the bootstrap statistic and test C statistic from the apparent C statistic to determine the optimism-corrected C statistic for each model.30

    We evaluated calibration based on the calibration intercept and slope from the 100 bootstrap samples using logistic regression of the predicted risk against the outcome. The average of the 100 regression coefficients was taken as the calibration slope for the respective model. Perfect calibration is indicated by a calibration intercept of 0 and calibration slope of 1.30,31

    We further compared models using the bayesian information criteria, area under the precision recall curve, categorical net reclassification improvement, and integrated discrimination improvement. The net reclassification improvement assesses the improvement in stratification of cases and controls into high and low predicted risk categories with the addition of new variables.32,33 We prespecified risk categories of less than 1%, 1% to less than 5%, 5% to less than 10%, 10% to less than 20%, and 20% or greater based on clinician input. The integrated discrimination improvement is the difference between mean predicted risks of patients with events and nonevents between the new and old models.32,34,35 We further compared models using reclassification tables with stratification of events and nonevents and the same prespecified predicted risk categories.36

    We transformed the best performing model based on accuracy and parsimony into an integer-based risk index using methods from the Framingham Heart Study.37 This process assigns an integer number of points to each variable, and the total points for each patient in the cohort are used to estimate the corresponding risk.

    We performed external validation of the best performing model and corresponding risk index by applying them to the external validation cohort from ICES Ontario to evaluate their predictive performance in a geographically distinct, independent cohort. We assessed discrimination with the C statistic and calibration with calibration intercept and slope. We used logistic recalibration to optimize model calibration in the external validation cohort.38 Analyses were performed using Stata, version 16 (StataCorp LLC).

    Results
    Cohort Descriptions

    The derivation cohort included 92 114 patients (47.8% male and 52.2% female; mean [SD] age, 62.3 [18.0] years); 529 (0.6%) developed AKI requiring KRT within 14 days of noncardiac surgery (Figure 1). Patients who developed AKI requiring KRT were more often male; had undergone repair of abdominal aortic aneurysm or vascular, colorectal, liver, or pancreatic surgery; and had lower baseline eGFR and hemoglobin concentration (eTable 3 in the Supplement).

    The external validation cohort included 709 086 patients (49.2% male and 50.8% female; mean [SD] age, 61.0 [16.0] years); 2956 (0.4%) developed AKI requiring KRT within 14 days of noncardiac surgery. The mean age was similar to that of the derivation cohort; however, the percentages of patients younger than 40 years and 70 years or older were smaller in the external validation cohort (<40 years: 10.6% vs 13.3%; ≥70 years: 32.2% vs 38.9%) (Table 1). In the external validation cohort, a larger percentage of patients had moderate or severe liver disease, a history of abdominal and musculoskeletal surgery, and a higher mean baseline eGFR. A smaller percentage of patients had a history of myocardial infarction or abdominal aortic aneurysm and vascular, colorectal, liver, or pancreatic surgery.

    Model Characteristics

    With bootstrap resampling (model 2), the following factors were associated with the risk of AKI requiring KRT: younger age (40-69 years: odds ratio [OR], 2.07 [95% CI, 1.69-2.53]; <40 years: OR, 3.73 [95% CI, 2.61-5.33]), male sex (OR, 1.55; 95% CI, 1.28-1.87), surgery type (colorectal: OR, 4.86 [95% CI, 3.28-7.18]; liver or pancreatic: OR, 6.46 [95% CI, 3.85-10.83]; other abdominal: OR, 2.19 [95% CI, 1.66-2.89]; abdominal aortic aneurysm repair: OR, 19.34 [95% CI, 14.31-26.14]; other vascular: OR, 7.30 [95% CI, 5.48-9.73]; thoracic: OR, 3.41 [95% CI, 2.07-5.59]), lower eGFR (OR, 0.97; 95% CI, 0.97-0.97 per 1 mL/min/1.73 m2 increase), lower hemoglobin concentration (OR, 0.99; 95% CI, 0.98-0.99 per 0.1 g/dL increase), albuminuria (mild: OR, 1.88 [95% CI, 1.52-2.33]; heavy: OR, 3.74 [95% CI, 2.98-4.69]), history of myocardial infarction (OR, 1.63; 95% CI, 1.32-2.03), and liver disease (mild: OR, 2.32 [95% CI, 1.66-3.24]; moderate or severe: OR, 4.96 [95% CI, 3.58-6.85]). The predictive variables and odds ratios for the full model (model 1) and reduced models (models 3-5) are shown in Table 2.

    Internal Validation

    Discrimination, measured by the C statistic, ranged from 0.80 (95% CI, 0.78-0.82) for model 5 to 0.89 (95% CI, 0.87-0.90) for model 1 and 0.89 (95% CI, 0.88-0.91) for model 2 (eTable 4 in the Supplement). Model discrimination was similar for models 1 and 2, with the exclusion of most of the comorbidities and elective vs emergent or urgent surgery status; however, the C statistic was lower for model 3 (0.87; 95% CI, 0.860.89) and model 4 (0.87; 95% CI, 0.85-0.88), with the removal of albuminuria and additional comorbidities, respectively. The lowest bayesian information criterion was observed for model 2 (5278), indicating the best global model fit after penalization for model complexity. The area under the precision recall curve was similar for models 1 (0.080) and 2 (0.079) and decreased for models 3 (0.069), 4 (0.060), and 5 (0.026). The area under the precision recall curve for the risk index (0.076) was lower than that for model 2. Calibration slope was close to ideal for all models, ranging from 0.98 (95% CI, 0.92-1.06) for model 5 to 0.99 (95% CI, 0.94-1.03) for model 2 in internal validation. Figure 2A shows calibration for model 2 in the derivation cohort for most of the predicted risks, although higher predicted risk vs observed risk occurred at the highest predicted risks (>20%).

    Model comparisons based on categorical net reclassification improvement and integrated discrimination improvement are shown in eTable 5 in the Supplement. The inclusion of additional comorbidities in model 1 resulted in few patients with and without the outcome being reclassified into higher and lower risk categories, respectively, compared with model 2, and little improvement observed in the integrated discrimination improvement. Larger changes in net reclassification and discrimination improvement were seen with inclusion of albuminuria in model 2 compared with model 3, from which it was excluded, with an increase of 20 patients (3.8%) with the outcome correctly reclassified into a higher risk category and an increase of 167 patients (0.2%) without the outcome correctly reclassified into a lower risk category. Greater changes in the net reclassification improvement and integrated discrimination improvement were found for model 2 compared with models 4 and 5. Reclassification tables for models 1, 3, 4, and 5 compared with model 2 are shown in eTables 6 to 9 in the Supplement.

    An integer-based risk index developed from the best performing model (model 2) (eTable 10 in the Supplement) generated points ranging from 0 to 30, with predicted risks corresponding to each point score ranging from 0.01% (0 points) to 96.7% (30 points) (eTable 11 in the Supplement). The discrimination of the risk index in the derivation cohort was similar to that in model 2, with a C statistic of 0.89 (95% CI, 0.87-0.90). The calibration slope for the risk index was 0.97 (95% CI, 0.92-1.03) in the internal validation cohort. Observed and predicted probabilities of AKI requiring KRT for each point value in the risk index are shown in the eFigure in the Supplement.

    External Validation

    Model 2 showed excellent discrimination in the external validation cohort, with a C statistic of 0.95 (95% CI, 0.95-0.96). On initial application, the calibration slope was 1.28 (95% CI, 1.26-1.30), and the calibration intercept was 1.12 (95% CI, 1.03-1.21). After logistic recalibration of the model coefficients and intercept, the recalibrated model 2 had a calibration slope of 1.00 (95% CI, 0.98-1.02) and calibration intercept of 0.00 (95% CI, –0.07 to 0.07). The area under the precision recall curve for recalibrated model 2 was 0.233, and the recalibrated risk index was 0.218. Figure 2B shows calibration for model 2 in the external validation cohort over a range of predicted risk categories. Regression coefficients for models 2 and 3 and for recalibrated models 2 and 3 are shown in eTable 12 in the Supplement.

    The risk index also showed excellent discrimination in the external validation cohort (C statistic, 0.95; 95% CI, 0.95-0.96). The original risk index had a calibration slope of 1.22 (95% CI, 1.20-1.24), and calibration intercept of 0.80 (95% CI, 0.72-0.88) for the external validation cohort. With recalibration, the calibration slope improved to 0.96 (95% CI, 0.95-0.98), and the calibration intercept improved to –0.23 (95% CI, –0.30 to –0.16). Observed and predicted probabilities of AKI requiring KRT for each point value of the risk index in the external validation cohort are shown in the eFigure in the Supplement.

    A sensitivity of 21.2%, specificity of 99.9%, positive predictive value of 38.1%, and negative predictive value of 99.7% were observed at a predicted risk threshold of 10% or greater (Table 3). An electronic risk calculator for the original and recalibrated models 2 and 3 is available in the eAppendix in the Supplement.

    Discussion

    We developed and internally validated risk prediction models for predicting AKI requiring KRT after major, noncardiac surgery in a large, population-based cohort. We externally validated the best performing model and corresponding risk index in an independent, geographically distinct, population-based cohort, showing excellent discrimination and accuracy of predictions after recalibration. Our models use patient demographics, laboratory data, and comorbidities, which are all readily available variables in preoperative care. This risk index can be implemented before surgery to identify patients at increased risk for AKI requiring KRT after noncardiac surgery. Preoperative risk stratification for severe AKI could be valuable for patient education, potential modification of treatment decisions, and resource allocation because AKI requiring KRT is a serious adverse outcome that is important to patients and is resource intensive for the health care system.

    Our final model and risk index showed excellent discrimination and calibration in internal validation.30,31 Precision (positive predictive value) and recall (sensitivity) were relatively low, whereas negative predictive values and specificity of the final model were high, as expected, given the low incidence of the outcome. Reduced models excluding albuminuria resulted in some loss of discrimination and poorer stratification into low-, moderate-, and high-risk groups, supporting the prognostic importance of albuminuria,39-41 a relatively simple laboratory measure that can be routinely incorporated into assessment of risk for AKI. A model without albuminuria performed reasonably well and could be used when albuminuria measurements are not available. The greatest decrease in model performance was observed with a model based on demographic variables and surgery type alone, which highlights the value of multivariable approaches incorporating laboratory measures of eGFR and hemoglobin concentration for AKI risk prediction. External validation of our final model and corresponding risk index showed strong discrimination and excellent calibration after the model was recalibrated to reflect differences in baseline risk and associations between predictors and outcome in the external validation cohort. These findings highlight the importance of recalibrating risk models for application in new populations, which may be particularly important for the outcome of AKI requiring KRT because thresholds for initiating KRT may vary among regions and over time.42,43

    The variables included in our final model and risk index have been previously associated with AKI, including lower eGFR, greater albuminuria, lower hemoglobin concentration, history of vascular surgery, and presence of heart and liver disease.27,40,44,45 We found that increasing age was associated with a lower risk of AKI requiring KRT, which differed from findings of other studies that examined the association with AKI that were not limited to patients who received KRT.11,25,46 A reason for these results may be that older patients are less likely to be receive KRT than younger patients or older patients at risk for AKI may be preselected for nonsurgical treatment options.

    The development of these models is valuable because little research has been conducted on predicting AKI after noncardiac surgery.11 Most published models have been limited to smaller, single-center cohorts of patients undergoing liver transplantation or resection only. Kheterpal et al27 developed a risk index for predicting AKI after general surgery using a large national clinical research data set. However, their study predicted the risk of AKI defined as a postoperative increase in creatinine concentration of more than 2 mg/dL (177 μmol/L) or the need for dialysis within 30 days of surgery.15 More recently, Park et al12 developed and validated the Simple Postoperative AKI Risk (SPARK) index, which can be used for predicting any postoperative AKI or critical AKI, defined as Kidney Disease: Improving Global Outcomes stage 2 or 3 AKI, post-AKI death, or dialysis within 90 days of noncardiac surgery.12 Both models differ from our model because they include milder stages of AKI in the outcome as opposed to our model. Our model was developed to predict AKI requiring KRT, which has a greater effect on patient health and health resource utilization.

    This study may have implications for patients, care providers, and decision makers. Obtaining individualized information on the risk of AKI requiring KRT may help inform decisions about pursuing a course of treatment or help better prepare patients for an adverse postoperative event. Care providers could use this tool to identify patients at high risk of AKI, which would assist with patient counseling, targeting use of preventative measures to reduce the risk of AKI, and prompting more careful postoperative monitoring of kidney function in patients at high risk. Being able to identify patients at high risk for requiring KRT before surgery may allow for better planning of resource allocation, including a need for preoperative consultation with nephrologists or planning surgical procedures in centers with access to KRT. Additional research is needed to test the clinical impact of the risk prediction tool for perioperative decision-making, including studies to characterize risk thresholds for surgical procedures for which patient preferences or clinicians’ recommendations to undergo or avoid surgery might change, or to identify clinically motivated risk thresholds at which to test interventions for AKI prevention in particular patient populations.

    Limitations

    This study has limitations. Although comorbidities were identified from administrative data using validated approaches, these data were not specifically collected for AKI risk prediction, which may have limited the candidate variables available for study; important risk factors for AKI, including disease severity, exposure to periprocedure nephrotoxins, and details on surgical approaches, may not have been included. A proportion of the cohort did not have a preoperative albuminuria measurement, which may have influenced the performance of the models. However, we used an imputation method to address the missing data and found improvement in model performance with inclusion of albuminuria, which provided reassuring findings about the validity of imputing albuminuria in the models. Kidney replacement therapy for patients with AKI may differ based on patient preferences and institutional practices. Thus, the risk models may not be generalizable for use in all locations and patient populations.47 The model and risk index may have overestimated absolute risk at high levels of predicted risk (ie, risk index ≥20 points; predicted risk >20%). However, this would be unlikely to influence decisions based on the model if a risk threshold of 10% were used to stratify patients into moderate- vs high-risk status in clinical practice. For example, with a risk threshold of 10% or greater selected to stratify patients at high risk for AKI requiring KRT, the model had a sensitivity of 21.2%, specificity of 99.9%, positive predictive value of 38.1%, and negative predictive value of 99.7%, suggesting that the model may be clinically useful for identifying patients at high risk.

    Conclusions

    In this prognostic study, we developed and externally validated a logistic regression model and a risk index for predicting AKI requiring KRT after major, noncardiac surgical procedures that showed excellent performance. This prognostic tool uses readily available preoperative data that may facilitate its implementation for use and assessment in clinical perioperative risk stratification.

    Back to top
    Article Information

    Accepted for Publication: June 15, 2021.

    Published: August 23, 2021. doi:10.1001/jamanetworkopen.2021.21901

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Wilson TA et al. JAMA Network Open.

    Corresponding Author: Matthew T. James, MD, PhD, Department of Medicine, Cumming School of Medicine, University of Calgary, 3280 Hospital Drive NW Calgary, Alberta, Canada T2N 4Z6 (mjames@ucalgary.ca).

    Author Contributions: Drs Garg and James had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Wilson, Quinn, Zarnke, Iskander, Hemmelgarn, Pannu, James.

    Acquisition, analysis, or interpretation of data: Wilson, de Koning, Quinn, Zarnke, McArthur, Roshanov, Garg, Pannu, James.

    Drafting of the manuscript: Wilson, Roshanov, Hemmelgarn, James.

    Critical revision of the manuscript for important intellectual content: Wilson, de Koning, Quinn, Zarnke, McArthur, Iskander, Roshanov, Garg, Pannu, James.

    Statistical analysis: Wilson, de Koning, Zarnke, McArthur, Roshanov, James.

    Obtained funding: Hemmelgarn, James.

    Administrative, technical, or material support: Wilson, Zarnke, Iskander, James.

    Supervision: Wilson, Quinn, Zarnke, Iskander, Hemmelgarn, Pannu, James.

    Conflict of Interest Disclosures: Dr Quinn reported having a patent for Dialysis Measurement and Reporting data system. Dr James reported receiving grants from Amgen Canada outside the submitted work. No other disclosures were reported.

    Funding/Support: The study was funded by a Canadian Institutes for Health Research (CIHR) Foundation award (Dr James) and by the ICES Western site. ICES is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care. Core funding for ICES Western is provided by the Academic Medical Organization of Southwestern Ontario, the Schulich School of Medicine and Dentistry, Western University, the Lawson Health Research Institute, and 14 clinical departments. Mr Wilson was supported by the Roy and Vi Baay Chair for Kidney Research at the University of Calgary. Dr Garg was supported by a Clinician Investigator Award from the CIHR and the Dr. Adam Linton Chair in Kidney Health Analytics. Dr James was supported by a CIHR New Investigator and Foundation Awards.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Disclaimer: This study is based in part on data provided by Alberta Health and Alberta Health Services. The interpretation and conclusions contained herein are those of the researchers and do not represent the views of the Government of Alberta or Alberta Health Services. The Government of Alberta, Alberta Health, and Alberta Health Services do not express any opinion in relation to this study. The opinions, results, and conclusions reported in this paper are those of the authors and are independent from these funding sources. Some of this work was conducted with members of the provincial ICES Kidney, Dialysis, and Transplantation Research Program, which receives programmatic grant funding from the CIHR. No endorsement by ICES or the Ontario Ministry of Health and Long-Term Care is intended or should be inferred. Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information. However, the analyses, conclusions, opinions, and statements expressed herein are those of the authors and not necessarily those of the Canadian Institute for Health Information.

    References
    1.
    Bellomo  R, Kellum  JA, Ronco  C.  Acute kidney injury.   Lancet. 2012;380(9843):756-766. doi:10.1016/S0140-6736(11)61454-2 PubMedGoogle ScholarCrossref
    2.
    Brienza  N, Giglio  MT, Marucci  M.  Preventing acute kidney injury after noncardiac surgery.   Curr Opin Crit Care. 2010;16(4):353-358. doi:10.1097/MCC.0b013e32833a9ef5 PubMedGoogle ScholarCrossref
    3.
    Chertow  GM, Burdick  E, Honour  M, Bonventre  JV, Bates  DW.  Acute kidney injury, mortality, length of stay, and costs in hospitalized patients.   J Am Soc Nephrol. 2005;16(11):3365-3370. doi:10.1681/ASN.2004090740 PubMedGoogle ScholarCrossref
    4.
    Bellomo  R.  Acute renal failure.   Semin Respir Crit Care Med. 2011;32(5):639-650. doi:10.1055/s-0031-1287872 PubMedGoogle ScholarCrossref
    5.
    Siddiqui  NF, Coca  SG, Devereaux  PJ,  et al.  Secular trends in acute dialysis after elective major surgery—1995 to 2009.   CMAJ. 2012;184(11):1237-1245. doi:10.1503/cmaj.110895 PubMedGoogle ScholarCrossref
    6.
    Hsu  RK, McCulloch  CE, Dudley  RA, Lo  LJ, Hsu  C.  Temporal changes in incidence of dialysis-requiring AKI.   J Am Soc Nephrol. 2013;24(1):37-42. doi:10.1681/ASN.2012080800 PubMedGoogle ScholarCrossref
    7.
    Collister  D, Pannu  N, Ye  F,  et al; Alberta Kidney Disease Network.  Health care costs associated with AKI.   Clin J Am Soc Nephrol. 2017;12(11):1733-1743. doi:10.2215/CJN.00950117 PubMedGoogle ScholarCrossref
    8.
    James  MT, Bhatt  M, Pannu  N, Tonelli  M.  Long-term outcomes of acute kidney injury and strategies for improved care.   Nat Rev Nephrol. 2020;16(4):193-205. doi:10.1038/s41581-019-0247-z PubMedGoogle ScholarCrossref
    9.
    Bell  S, Prowle  J.  Postoperative AKI-prevention is better than cure?   J Am Soc Nephrol. 2019;30(1):4-6. doi:10.1681/ASN.2018111127 PubMedGoogle ScholarCrossref
    10.
    Golden  D, Corbett  J, Forni  LG.  Peri-operative renal dysfunction: prevention and management.   Anaesthesia. 2016;71(suppl 1):51-57. doi:10.1111/anae.13313 PubMedGoogle ScholarCrossref
    11.
    Wilson  T, Quan  S, Cheema  K,  et al.  Risk prediction models for acute kidney injury following major noncardiac surgery: systematic review.   Nephrol Dial Transplant. 2015;31(2):231-240. doi:10.1093/ndt/gfv415 PubMedGoogle Scholar
    12.
    Park  S, Cho  H, Park  S,  et al.  Simple postoperative AKI risk (SPARK) classification before noncardiac surgery: a prediction index development study with external validation.   J Am Soc Nephrol. 2019;30(1):170-181. doi:10.1681/ASN.2018070757 PubMedGoogle ScholarCrossref
    13.
    Bihorac  A, Yavas  S, Subbiah  S,  et al.  Long-term risk of mortality and acute kidney injury during hospitalization after major surgery.   Ann Surg. 2009;249(5):851-858. doi:10.1097/SLA.0b013e3181a40a0bGoogle ScholarCrossref
    14.
    Hoste  EA, Bagshaw  SM, Bellomo  R,  et al.  Epidemiology of acute kidney injury in critically ill patients: the multinational AKI-EPI study.   Intensive Care Med. 2015;41(8):1411-1423. doi:10.1007/s00134-015-3934-7 PubMedGoogle ScholarCrossref
    15.
    Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group.  KDIGO Clinical Practice Guideline for Acute Kidney Injury.   Kidney Int Suppl. 2012;2(1):1-138. doi:10.1038/kisup.2012.6Google ScholarCrossref
    16.
    Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD).   Ann Intern Med. 2015;162(10):735-736. doi:10.7326/L15-5093-2 PubMedGoogle ScholarCrossref
    17.
    Hemmelgarn  BR, Clement  F, Manns  BJ,  et al.  Overview of the Alberta Kidney Disease Network.   BMC Nephrol. 2009;10:30. doi:10.1186/1471-2369-10-30 PubMedGoogle ScholarCrossref
    18.
    Quan  H, Sundararajan  V, Halfon  P,  et al.  Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.   Med Care. 2005;43(11):1130-1139. doi:10.1097/01.mlr.0000182534.19832.83 PubMedGoogle ScholarCrossref
    19.
    Quan  H, Khan  N, Hemmelgarn  BR,  et al; Hypertension Outcome and Surveillance Team of the Canadian Hypertension Education Programs.  Validation of a case definition to define hypertension using administrative data.   Hypertension. 2009;54(6):1423-1428. doi:10.1161/HYPERTENSIONAHA.109.139279 PubMedGoogle ScholarCrossref
    20.
    Vlasschaert  ME, Bejaimal  SA, Hackam  DG,  et al.  Validity of administrative database coding for kidney disease: a systematic review.   Am J Kidney Dis. 2011;57(1):29-43. doi:10.1053/j.ajkd.2010.08.031 PubMedGoogle ScholarCrossref
    21.
    De Coster  C, Li  B, Quan  H.  Comparison and validity of procedures coded with ICD-9-CM and ICD-10-CA/CCI.   Med Care. 2008;46(6):627-634. doi:10.1097/MLR.0b013e3181649439 PubMedGoogle ScholarCrossref
    22.
    Hux  JE, Ivis  F, Flintoft  V, Bica  A.  Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm.   Diabetes Care. 2002;25(3):512-516. doi:10.2337/diacare.25.3.512 PubMedGoogle ScholarCrossref
    23.
    Juurlink  D, Preyra C, Croxford  R, et al  .  Canadian Institute for Health Information Discharge Abstract Database: A Validation Study. ICES; 2006.
    24.
    Waikar  SS, Wald  R, Chertow  GM,  et al.  Validity of International Classification of Diseases, Ninth Revision, Clinical Modification codes for acute renal failure.   J Am Soc Nephrol. 2006;17(6):1688-1694. doi:10.1681/ASN.2006010073 PubMedGoogle ScholarCrossref
    25.
    Biteker  M, Dayan  A, Tekkeşin  AI,  et al.  Incidence, risk factors, and outcomes of perioperative acute kidney injury in noncardiac and nonvascular surgery.   Am J Surg. 2014;207(1):53-59. doi:10.1016/j.amjsurg.2013.04.006 PubMedGoogle ScholarCrossref
    26.
    Borthwick  E, Ferguson  A.  Perioperative acute kidney injury: risk factors, recognition, management, and outcomes.   BMJ. 2010;341:c3365. doi:10.1136/bmj.c3365 PubMedGoogle ScholarCrossref
    27.
    Kheterpal  S, Tremper  KK, Heung  M,  et al.  Development and validation of an acute kidney injury risk index for patients undergoing general surgery: results from a national data set.   Anesthesiology. 2009;110(3):505-515. doi:10.1097/ALN.0b013e3181979440 PubMedGoogle ScholarCrossref
    28.
    Levey  AS, Stevens  LA, Schmid  CH,  et al; CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration).  A new equation to estimate glomerular filtration rate.   Ann Intern Med. 2009;150(9):604-612. doi:10.7326/0003-4819-150-9-200905050-00006 PubMedGoogle ScholarCrossref
    29.
    Lamb  EJ, MacKenzie  F, Stevens  PE.  How should proteinuria be detected and measured?   Ann Clin Biochem. 2009;46(Pt 3):205-217. doi:10.1258/acb.2009.009007 PubMedGoogle Scholar
    30.
    Harrell  FE Jr.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer; 2001. doi:10.1007/978-1-4757-3462-1
    31.
    Steyerberg  EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating.: Springer; 2009. doi:10.1007/978-0-387-77244-8
    32.
    Pencina  MJ, D’Agostino  RB  Sr, D’Agostino  RB  Jr, Vasan  RS.  Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.   Stat Med. 2008;27(2):157-172. doi:10.1002/sim.2929 PubMedGoogle ScholarCrossref
    33.
    Pencina  KM, Pencina  MJ, D’Agostino  RB  Sr.  What to expect from net reclassification improvement with three categories.   Stat Med. 2014;33(28):4975-4987. doi:10.1002/sim.6286 PubMedGoogle ScholarCrossref
    34.
    Leening  MJG, Vedder  MM, Witteman  JCM, Pencina  MJ, Steyerberg  EW.  Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide.   Ann Intern Med. 2014;160(2):122-131. doi:10.7326/M13-1522 PubMedGoogle ScholarCrossref
    35.
    Pepe  MS, Janes  H, Li  CI.  Net risk reclassification p values: valid or misleading?   J Natl Cancer Inst. 2014;106(4):dju041. doi:10.1093/jnci/dju041 PubMedGoogle Scholar
    36.
    Saito  T, Rehmsmeier  M.  The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets.   PLoS One. 2015;10(3):e0118432. doi:10.1371/journal.pone.0118432 PubMedGoogle Scholar
    37.
    Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: the Framingham Study risk score functions.   Stat Med. 2004;23(10):1631-1660. doi:10.1002/sim.1742 PubMedGoogle ScholarCrossref
    38.
    Janssen  KJM, Moons  KGM, Kalkman  CJ, Grobbee  DE, Vergouwe  Y.  Updating methods improved the performance of a clinical prediction model in new patients.   J Clin Epidemiol. 2008;61(1):76-86. doi:10.1016/j.jclinepi.2007.04.018 PubMedGoogle ScholarCrossref
    39.
    Grams  ME, Astor  BC, Bash  LD, Matsushita  K, Wang  Y, Coresh  J.  Albuminuria and estimated glomerular filtration rate independently associate with acute kidney injury.   J Am Soc Nephrol. 2010;21(10):1757-1764. doi:10.1681/ASN.2010010128 PubMedGoogle ScholarCrossref
    40.
    Coca  SG, Yalavarthy  R, Concato  J, Parikh  CR.  Biomarkers for the diagnosis and risk stratification of acute kidney injury: a systematic review.   Kidney Int. 2008;73(9):1008-1016. doi:10.1038/sj.ki.5002729 PubMedGoogle ScholarCrossref
    41.
    Hsu  RK, Hsu  CY.  Proteinuria and reduced glomerular filtration rate as risk factors for acute kidney injury.   Curr Opin Nephrol Hypertens. 2011;20(3):211-217. doi:10.1097/MNH.0b013e3283454f8d PubMedGoogle ScholarCrossref
    42.
    Wessler  BS, Ruthazer  R, Udelson  JE,  et al.  Regional validation and recalibration of clinical predictive models for patients with acute heart failure.   J Am Heart Assoc. 2017;6(11):e006121. doi:10.1161/JAHA.117.006121 PubMedGoogle Scholar
    43.
    Davis  SE, Lasko  TA, Chen  G, Siew  ED, Matheny  ME.  Calibration drift in regression and machine learning models for acute kidney injury.   J Am Med Inform Assoc. 2017;24(6):1052-1061. doi:10.1093/jamia/ocx030 PubMedGoogle ScholarCrossref
    44.
    Halbesma  N, Jansen  DF, Heymans  MW, Stolk  RP, de Jong  PE, Gansevoort  RT; PREVEND Study Group.  Development and validation of a general population renal risk score.   Clin J Am Soc Nephrol. 2011;6(7):1731-1738. doi:10.2215/CJN.08590910 PubMedGoogle ScholarCrossref
    45.
    Teixeira  C, Rosa  R, Rodrigues  N,  et al.  Acute kidney injury after major abdominal surgery: a retrospective cohort analysis.   Crit Care Res Pract. 2014;2014:132175. doi:10.1155/2014/132175 PubMedGoogle Scholar
    46.
    Kheterpal  S, Tremper  KK, Englesbe  MJ,  et al.  Predictors of postoperative acute renal failure after noncardiac surgery in patients with previously normal renal function.   Anesthesiology. 2007;107(6):892-902. doi:10.1097/01.anes.0000290588.29668.38 PubMedGoogle ScholarCrossref
    47.
    Bagshaw  SM, Wald  R, Barton  J,  et al.  Clinical factors associated with initiation of renal replacement therapy in critically ill patients with acute kidney injury-a prospective multicenter observational study.   J Crit Care. 2012;27(3):268-275. doi:10.1016/j.jcrc.2011.06.003 PubMedGoogle ScholarCrossref
    ×