[Skip to Content]
[Skip to Content Landing]
Figure 1.
Formation of the Derivation, Internal Validation, and External Validation Cohorts Hospitalized With Acute Kidney Injury
Formation of the Derivation, Internal Validation, and External Validation Cohorts Hospitalized With Acute Kidney Injury

aPatients without prehospitalization serum creatinine measurements were excluded from the study (49 patients of 2946 [1.6%] had advanced chronic kidney events from the Alberta source population).

bTo ascertain whether patients had advanced chronic kidney disease, the first estimated glomerular filtration rate (eGFR) of less than 30 mL/min/1.73 m2 was obtained a median of 75 days (interquartile range [IQR], 40-160 days) after discharge and the last a median of 267 days (IQR, 125-321 days) after discharge in the derivation and internal validation cohorts. There were 29 patients (0.2%) in the derivation and internal validation cohorts with only 1 eGFR less than 30 mL/min/1.73 m2 and did not have a subsequent measurement during follow-up. They were not considered to have developed advanced chronic kidney disease. Patients were followed up through March 2015.

Figure 2.
Six-Variable Risk Index for Advanced Chronic Kidney Disease Following Hospitalization With Acute Kidney Injury
Six-Variable Risk Index for Advanced Chronic Kidney Disease Following Hospitalization With Acute Kidney Injury

Acute kidney injury stage 1 is defined by serum creatinine (SCr) increase of 0.3 mg/dL or more or 1.5 to 1.9 times the baseline within index hospitalization; stage 2, serum creatinine increase of between 2.0 and 2.9 times baseline within index hospitalization; stage 3, serum creatinine increase of 3.0 mg/g or more times baseline or 4.0 mg/dL or more within index hospitalization. Normal albuminuria is indicated by an albumin:creatinine ratio (ACR) of 30 or less or a dipstick negative urinalysis protein; mild, ACR of 30 mg/g to 300 mg/g or dipstick positive urinalysis protein trace or 1+; heavy, ACR of more than 300 mg/g or dipstick positive urinalysis protein of 2+ or more. To convert urine ACR to mg/mmol, multiply by 0.113.

Points assigned to values of each variable can be summed to obtain a patient’s total risk score, which can be used to determine his/her corresponding predicted risk of developing advanced chronic kidney disease (Supplement 2). A smartphone app is available on Calculate by QxMD for iOS, Android, and Windows (free install at https://qxmd.com/getcalculate).

Figure 3.
Predicted vs Observed Probability of Advanced Chronic Kidney Disease by the 6-Variable Risk index in the Internal Validation and External Validation Cohorts
Predicted vs Observed Probability of Advanced Chronic Kidney Disease by the 6-Variable Risk index in the Internal Validation and External Validation Cohorts

Internal validation cohort (n = 4956) and external validation cohort (n = 2761). Error bars indicate 95% confidence intervals. The risk scores observed in both cohorts ranged from a minimum value of 0 to maximum value of 24.

Table 1.  
Baseline Characteristics of the Derivation, Internal Validation, and External Validation Cohorts
Baseline Characteristics of the Derivation, Internal Validation, and External Validation Cohorts
Table 2.  
Advanced Chronic Kidney Disease Predictors and Performance of Models in the Derivation Cohorta
Advanced Chronic Kidney Disease Predictors and Performance of Models in the Derivation Cohorta
Table 3.  
Predictive Performance of Models for Advanced Chronic Kidney Disease in Internal and External Validation Cohortsa
Predictive Performance of Models for Advanced Chronic Kidney Disease in Internal and External Validation Cohortsa
Supplement 1.

eTable 1. Diagnostic and procedure codes used to identify comorbidities and procedures in the derivation, internal validation (AKDN) and external validation (ICES) cohorts

eTable 2. Predictors of advanced CKD and their frequency in the derivation, internal validation (AKDN) and external validation (ICES) cohorts

eTable 3. Frequency, and observed risk of risk categories for 5 prediction models for advanced CKD in the derivation (AKDN) cohort

eTable 4. Distribution of model predicted risk of advanced CKD (%) estimated in the derivation, internal validation (AKDN), and external validation (ICES) cohorts

eTable 5. Discrimination and calibration of a six-variable model for advanced CKD in the internal validation (AKDN) cohort, stratified by timing of baseline Scr measurement and level of baseline eGFR

eTable 6. Predictive performance of models for advanced CKD in the patients excluded from original (AKDN) cohorts due to lack of pre-hospitalization Scr measurements between 7-365 days prior to hospital admission

eTable 7. Varying thresholds of predicted risk based on the six-variable risk-index, proportion of patients who would be risk stratified for community CKD follow-up, and corresponding sensitivity, specificity, positive and negative predictive values for progression to advanced CKD during follow-up after a hospitalization with AKI

eFigure 1. Calibration of the six-variable (Model 1) and reduced models (Models 2-5) in the internal validation (AKDN) cohort using locally weighted least squares regression smoother plots

eFigure 2. Calibration of the six-variable (Model 1) and reduced models (Models 2-5) in the external validation (ICES) cohort using locally weighted least squares regression smoother plots

1.
Coca  SG, Singanamala  S, Parikh  CR.  Chronic kidney disease after acute kidney injury: a systematic review and meta-analysis.  Kidney Int. 2012;81(5):442-448.PubMedGoogle ScholarCrossref
2.
Sawhney  S, Marks  A, Fluck  N,  et al.  Post-discharge kidney function is associated with subsequent ten-year renal progression risk among survivors of acute kidney injury.  Kidney Int. 2017;92(2):440-452.PubMedGoogle ScholarCrossref
3.
Ishani  A, Nelson  D, Clothier  B,  et al.  The magnitude of acute serum creatinine increase after cardiac surgery and the risk of chronic kidney disease, progression of kidney disease, and death.  Arch Intern Med. 2011;171(3):226-233.PubMedGoogle ScholarCrossref
4.
Kidney International Supplements. KDIGO clinical practice guideline for acute kidney injury. http://www.kisupplements.org/issue/S2157-1716(12)X7200-9. Published March 2012. Accessed October 5, 2017.
5.
Siew  ED, Peterson  JF, Eden  SK,  et al.  Outpatient nephrology referral rates after acute kidney injury.  J Am Soc Nephrol. 2012;23(2):305-312.PubMedGoogle ScholarCrossref
6.
Harel  Z, Wald  R, Bargman  JM,  et al.  Nephrologist follow-up improves all-cause mortality of severe acute kidney injury survivors.  Kidney Int. 2013;83(5):901-908.PubMedGoogle ScholarCrossref
7.
Goldstein  SL, Jaber  BL, Faubel  S, Chawla  LS; Acute Kidney Injury Advisory Group of American Society of Nephrology.  AKI transition of care: a potential opportunity to detect and prevent CKD.  Clin J Am Soc Nephrol. 2013;8(3):476-483.PubMedGoogle ScholarCrossref
8.
James  M, Bouchard  J, Ho  J,  et al.  Canadian Society of Nephrology commentary on the 2012 KDIGO clinical practice guideline for acute kidney injury.  Am J Kidney Dis. 2013;61(5):673-685.PubMedGoogle ScholarCrossref
9.
Coleman  EA, Berenson  RA.  Lost in transition: challenges and opportunities for improving the quality of transitional care.  Ann Intern Med. 2004;141(7):533-536.PubMedGoogle ScholarCrossref
10.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD).  Ann Intern Med. 2015;162(10):735-736.PubMedGoogle ScholarCrossref
11.
Hemmelgarn  BR, Clement  F, Manns  BJ,  et al.  Overview of the Alberta Kidney Disease Network.  BMC Nephrol. 2009;10:30.PubMedGoogle ScholarCrossref
12.
Quan  H, Sundararajan  V, Halfon  P,  et al.  Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.  Med Care. 2005;43(11):1130-1139.PubMedGoogle ScholarCrossref
13.
Quan  H, Khan  N, Hemmelgarn  BR,  et al; Hypertension Outcome and Surveillance Team of the Canadian Hypertension Education Programs.  Validation of a case definition to define hypertension using administrative data.  Hypertension. 2009;54(6):1423-1428.PubMedGoogle ScholarCrossref
14.
Hux  JE, Ivis  F, Flintoft  V, Bica  A.  Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm.  Diabetes Care. 2002;25(3):512-516.PubMedGoogle ScholarCrossref
15.
Jain  AK, Cuerden  MS, McLeod  I,  et al.  Reporting of the estimated glomerular filtration rate was associated with increased use of angiotensin-converting enzyme inhibitors and angiotensin-II receptor blockers in CKD.  Kidney Int. 2012;81(12):1248-1253.PubMedGoogle ScholarCrossref
16.
Juurlink  D, Preyra  C, Croxford  R,  et al.  Canadian Institute for Health Information Discharge Abstract Database: A Validation Study. Toronto, ON: Institute for Clinical Evaluative Sciences; 2006.
17.
Siew  ED, Ikizler  TA, Matheny  ME,  et al.  Estimating baseline kidney function in hospitalized patients with impaired kidney function.  Clin J Am Soc Nephrol. 2012;7(5):712-719.PubMedGoogle ScholarCrossref
18.
Clement  FM, James  MT, Chin  R,  et al; Alberta Kidney Disease Network.  Validation of a case definition to define chronic dialysis using outpatient administrative data.  BMC Med Res Methodol. 2011;11:25.PubMedGoogle ScholarCrossref
19.
Levey  AS, Stevens  LA, Schmid  CH,  et al; CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration).  A new equation to estimate glomerular filtration rate.  Ann Intern Med. 2009;150(9):604-612.PubMedGoogle ScholarCrossref
20.
Go  AS, Chertow  GM, Fan  D, McCulloch  CE, Hsu  CY.  Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization.  N Engl J Med. 2004;351(13):1296-1305.PubMedGoogle ScholarCrossref
21.
Hemmelgarn  BR, Manns  BJ, Lloyd  A,  et al; Alberta Kidney Disease Network.  Relation between kidney function, proteinuria, and adverse outcomes.  JAMA. 2010;303(5):423-429.PubMedGoogle ScholarCrossref
22.
Andrassy  KM.  Comments on “KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease.”  Kidney Int. 2013;84(3):622-623.PubMedGoogle ScholarCrossref
23.
Levin  A, Hemmelgarn  B, Culleton  B,  et al; Canadian Society of Nephrology.  Guidelines for the management of chronic kidney disease.  CMAJ. 2008;179(11):1154-1162.PubMedGoogle ScholarCrossref
24.
National Institute for Health and Care Excellence,National Clinical Guideline Centre. Chronic kidney disease in adults in primary and secondary care: clinical guideline 182. Methods, evidence and recommendations. https://www.nice.org.uk/guidance/cg182. July 2014. Accessed April 17, 2017.
25.
Stevens  LA, Greene  T, Levey  AS.  Surrogate end points for clinical trials of kidney disease progression.  Clin J Am Soc Nephrol. 2006;1(4):874-884.PubMedGoogle ScholarCrossref
26.
Lamb  EJ, MacKenzie  F, Stevens  PE.  How should proteinuria be detected and measured?  Ann Clin Biochem. 2009;46(pt 3):205-217.PubMedGoogle ScholarCrossref
27.
Waikar  SS, Wald  R, Chertow  GM,  et al.  Validity of International Classification of Diseases, Ninth Revision, clinical modification codes for acute renal failure.  J Am Soc Nephrol. 2006;17(6):1688-1694.PubMedGoogle ScholarCrossref
28.
Kellum  JA.  How can we define recovery after acute kidney injury? considerations from epidemiology and clinical trial design.  Nephron Clin Pract. 2014;127(1-4):81-88.PubMedGoogle ScholarCrossref
29.
Wald  R, Quinn  RR, Luo  J,  et al; University of Toronto Acute Kidney Injury Research Group.  Chronic dialysis and death among survivors of acute kidney injury requiring dialysis.  JAMA. 2009;302(11):1179-1185.PubMedGoogle ScholarCrossref
30.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387.PubMedGoogle ScholarCrossref
31.
Steyerberg  EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009.
32.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: the Framingham Study risk score functions.  Stat Med. 2004;23(10):1631-1660.PubMedGoogle ScholarCrossref
33.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
34.
Austin  PC, Steyerberg  EW.  Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers.  Stat Med. 2014;33(3):517-535.PubMedGoogle ScholarCrossref
35.
Pencina  MJ, D’Agostino  RB  Sr, D’Agostino  RB  Jr, Vasan  RS.  Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.  Stat Med. 2008;27(2):157-172.PubMedGoogle ScholarCrossref
36.
Cook  NR, Ridker  PM.  Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures.  Ann Intern Med. 2009;150(11):795-802.PubMedGoogle ScholarCrossref
37.
Hosmer  DW, Lemeshow  S.  Applied Logistic Regression. 2nd ed. New York, NY: John Wiley & Sons; 2000.Crossref
38.
Tangri  N, Stevens  LA, Griffith  J,  et al.  A predictive model for progression of chronic kidney disease to kidney failure.  JAMA. 2011;305(15):1553-1559.PubMedGoogle ScholarCrossref
39.
Haggerty  JL, Reid  RJ, Freeman  GK, Starfield  BH, Adair  CE, McKendry  R.  Continuity of care: a multidisciplinary review.  BMJ. 2003;327(7425):1219-1221.PubMedGoogle ScholarCrossref
40.
Rennke  S, Nguyen  OK, Shoeb  MH, Magan  Y, Wachter  RM, Ranji  SR.  Hospital-initiated transitional care interventions as a patient safety strategy: a systematic review.  Ann Intern Med. 2013;158(5 pt 2):433-440.PubMedGoogle ScholarCrossref
Original Investigation
November 14, 2017

Derivation and External Validation of Prediction Models for Advanced Chronic Kidney Disease Following Acute Kidney Injury

Author Affiliations
  • 1Department of Medicine, Cumming School of Medicine, University of Calgary, Alberta, Canada
  • 2Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Alberta, Canada
  • 3O’Brien Institute for Public Health, Cumming School of Medicine, University of Calgary, Alberta, Canada
  • 4Libin Cardiovascular Institute of Alberta, Cumming School of Medicine, University of Calgary, Alberta, Canada
  • 5Department of Medicine, University of Alberta, Edmonton, Canada
  • 6Institute for Clinical Evaluative Sciences, Ontario, Canada
  • 7Institute of Health Policy, Management and Evaluation, University of Toronto, Ontario, Canada
  • 8Department of Medicine, University of Toronto, Ontario, Canada
  • 9Department of Medicine, Western University, London, Ontario, Canada
JAMA. 2017;318(18):1787-1797. doi:10.1001/jama.2017.16326
Key Points

Question  Can the risk of progression to advanced chronic kidney disease be accurately predicted when patients who had acute kidney injury during their hospital stay are discharged from the hospital?

Findings  A multivariable model was developed with 9973 participants and was externally validated with 2761 participants. In the external validation cohort, a model that included age, sex, acute kidney injury stage, prehospitalization serum creatinine level, albuminuria, and discharge serum creatinine achieved a C statistic of 0.81 for predicting advanced chronic kidney disease after hospital discharge.

Meaning  This model was able to predict advanced chronic kidney disease following hospitalization with acute kidney injury but requires evaluation of its utility in a clinical setting.

Abstract

Importance  Some patients will develop chronic kidney disease after a hospitalization with acute kidney injury; however, no risk-prediction tools have been developed to identify high-risk patients requiring follow-up.

Objective  To derive and validate predictive models for progression of acute kidney injury to advanced chronic kidney disease.

Design, Setting, and Participants  Data from 2 population-based cohorts of patients with a prehospitalization estimated glomerular filtration rate (eGFR) of more than 45 mL/min/1.73 m2 and who had survived hospitalization with acute kidney injury (defined by a serum creatinine increase during hospitalization > 0.3 mg/dL or > 50% of their prehospitalization baseline), were used to derive and validate multivariable prediction models. The risk models were derived from 9973 patients hospitalized in Alberta, Canada (April 2004-March 2014, with follow-up to March 2015). The risk models were externally validated with data from a cohort of 2761 patients hospitalized in Ontario, Canada (June 2004-March 2012, with follow-up to March 2013).

Exposures  Demographic, laboratory, and comorbidity variables measured prior to discharge.

Main Outcomes and Measures  Advanced chronic kidney disease was defined by a sustained reduction in eGFR less than 30 mL/min/1.73 m2 for at least 3 months during the year after discharge. All participants were followed up for up to 1 year.

Results  The participants (mean [SD] age, 66 [15] years in the derivation and internal validation cohorts and 69 [11] years in the external validation cohort; 40%-43% women per cohort) had a mean (SD) baseline serum creatinine level of 1.0 (0.2) mg/dL and more than 20% had stage 2 or 3 acute kidney injury. Advanced chronic kidney disease developed in 408 (2.7%) of 9973 patients in the derivation cohort and 62 (2.2%) of 2761 patients in the external validation cohort. In the derivation cohort, 6 variables were independently associated with the outcome: older age, female sex, higher baseline serum creatinine value, albuminuria, greater severity of acute kidney injury, and higher serum creatinine value at discharge. In the external validation cohort, a multivariable model including these 6 variables had a C statistic of 0.81 (95% CI, 0.75-0.86) and improved discrimination and reclassification compared with reduced models that included age, sex, and discharge serum creatinine value alone (integrated discrimination improvement, 2.6%; 95% CI, 1.1%-4.0%; categorical net reclassification index, 13.5%; 95% CI, 1.9%-25.1%) or included age, sex, and acute kidney injury stage alone (integrated discrimination improvement, 8.0%; 95% CI, 5.1%-11.0%; categorical net reclassification index, 79.9%; 95% CI, 60.9%-98.9%).

Conclusions and Relevance  A multivariable model using routine laboratory data was able to predict advanced chronic kidney disease following hospitalization with acute kidney injury. The utility of this model in clinical care requires further research.

Introduction

Acute kidney injury is common among hospitalized patients and associated with poor long-term outcomes.1,2 Several studies between 2008 to 2017 have demonstrated that although acute kidney injury is usually reversible, some patients may experience incomplete recovery of kidney function, while others subsequently develop accelerated loss of kidney function, resulting in an increased risk of chronic kidney disease.1-3 Current clinical practice guidelines recommend that patients be followed-up within 3 months to assess whether they have developed chronic kidney disease and when identified undergo treatment according to current guidelines.4

Despite these recommendations, many patients with acute kidney injury do not receive a follow-up assessment, nor do they receive appropriate care when kidney function has not recovered,5,6 resulting in lost opportunities to intervene and potentially improve long-term outcomes.7 However, because not all patients who survive acute kidney injury progress to chronic kidney disease,1-3,7 follow-up of all patients hospitalized with acute kidney injury could lead to unnecessary use of clinical resources. Identifying and screening patients at high risk of developing advanced chronic kidney disease could improve outcomes for patients following acute kidney injury.8,9

We used population-based laboratory and administrative data from 2 provinces in Canada to derive and validate risk prediction models for advanced chronic kidney disease following hospitalization with acute kidney injury. Our objective was to develop a practical risk stratification approach that could be used to identify patients at high risk of chronic kidney disease after they are discharged.

Methods

We followed the TRIPOD10 (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) statement for reporting multivariable prediction model development and validation. The health research ethics boards of the Universities of Calgary, Alberta, and Sunnybrook Health Sciences Centre, Ontario, Canada, approved the study and granted waiver of participant consent.

Derivation and Validation Cohorts

We formed a derivation cohort using the Alberta Kidney Disease Network (AKDN) database,11 a population-based repository that includes serum creatinine measurements from more than 1.8 million adults obtained from acute care and outpatient settings for all 3.5 million residents of the province, from April 1, 2004, through March 31, 2014. The database includes patient-level data from the provincial administrative health sources of Alberta Health. Patient comorbidities and procedures are characterized using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and the Canadian Classification of Procedures (CCP) and the ICD-10 diagnosis and the Canadian Classification of Heath Intervention (CCI) procedure codes.12-14 We randomly selected two-thirds of the AKDN cohort to serve as a model derivation cohort and used the remaining third as an internal validation cohort.

We formed the external validation cohort from Southwestern Ontario using data from the Institute of Clinical Evaluative Sciences Kidney, Dialysis and Transplantation (ICES KDT) Program. These data include laboratory information from more than 74 000 patients from 2 regional laboratory providers (Cerner and Gamma-Dynacare) in Southwestern Ontario (catchment population 2.5 million) from June 1, 2004, through March 31, 2012.15 The laboratory data sets were linked with comprehensive provincial administrative data using unique encoded identifiers and analyzed at ICES. The ICD-9-CM-CPP and ICD-10-CCI coding approaches were used to characterize patient comorbidities and procedures.16

Study Population

We identified all adult residents 18 years of age or older who were hospitalized during the study enrollment period (Figure 1). Eligible participants required at least 1 outpatient estimated glomerular filtration rate (eGFR) value between 7 and 365 days prior to hospitalization to establish baseline kidney function (outpatient serum creatinine tests in this period generally reflect stable values).17 We restricted the cohorts to those hospitalized with acute kidney injury (based on an increase in serum creatinine during hospitalization >0.3 mg/dL or >50% of their prehospitalization baseline),4 a baseline eGFR higher than 45 mL/min/1.73 m2, and at least 2 outpatient serum creatinine measurements separated by a minimum of 3 months with the first between 30 days and 1 year after the hospital discharge date. This required that patients survived more than a month after hospital charge and had sufficient follow-up information to ascertain development of advanced chronic kidney disease, and reflected a pragmatic approach that recognizes that organizing chronic kidney disease care is not needed when patients do not survive long enough to develop advanced chronic kidney disease. We excluded people treated with chronic dialysis or who had had a kidney transplant prior to or during the index hospitalization, identified by a validated approach using physician billing claims for a continuous period of 90 days of dialysis in provincial administrative databases or registration for chronic dialysis or kidney transplant with the Canadian Organ Replacement Registry.18

Study Outcome

The outcome of the study was progression to advanced chronic kidney disease, defined as a sustained outpatient eGFR value of less than 30 mL/min/1.73 m2 (estimated based on age, sex, and serum creatinine levels using the Chronic Kidney Disease Epidemiology Collaboration equation19) identified between 30 days to 1 year after hospital discharge or initiation of chronic dialysis up to 1 year after hospital discharge. This stage of chronic kidney disease is associated with a high risk of adverse outcomes20,21 and is accompanied by guideline recommendations and performance indicators for referral to a nephrologist.22-25 We required a minimum of 2 outpatient measurements of eGFR lower than 30 mL/min/1.73 m2 separated by at least 3 months, including an average of all outpatient eGFR measurements within this period of less than 30 mL/min/1.73 m2 to satisfy the definition of this outcome to align with current consensus criteria for chronic kidney disease.22

Candidate Predictor Variables

We identified candidate predictor variables for potential inclusion in our models from a review of the literature and identified potential predictor variables available prior to and during the index hospitalization with acute kidney injury.1-4 Baseline kidney function was determined from the most recent outpatient serum creatinine measurement made 7 to 365 days prior to the index hospitalization.17 Albuminuria was characterized by urine albumin:creatinine ratio (ACR) or dipstick using random spot urine measurements during the index admission or for up to 6 months prior to admission. We used urine ACR measurements preferentially, with urine dipstick measurements added for patients without ACR measurements in the derivation cohort. We defined albuminuria categories as normal (ACR, <30 mg/g or urine-dipstick negative), mild (ACR, 30-300 mg/g or urine dipstick trace or 1+), or heavy (ACR, >300 mg/g or urine dipstick positive ≥2+) and characterized patients with multiple tests using the median of multiple measurements, recognizing that albuminuria can be transient.26 Absence of a result for albuminuria was included as a separate category. We categorized the severity of acute kidney injury according to 3 categories of the Kidney Disease, Improving Global Outcomes (KDIGO) staging system based on the highest serum creatinine value identified during hospitalization or requirement for dialysis during the index hospitalization.4,27 We identified the extent of kidney function recovery at the time of hospital discharge based on the last inpatient serum creatinine value measured before hospital discharge.2,28 Comorbidities and procedures were identified from hospital discharge records and physician claims using validated ICD-9-CM and ICD-10 coding algorithms12-14,18,29 (eTable 1 in Supplement 1).

Model Derivation

Candidate variables from the derivation cohort were included as potential covariates in multivariable logistic regression models. Stepwise backward variable selection with a significance level of .05 for variable retention was used in 1000 bootstrapped samples of the same size as the derivation sample, to develop a parsimonious predictor model while minimizing overfitting.30,31 Variables selected in at least 70% of samples were included in the full multivariable model. We then fit a series of reduced models by sequentially removing variables and compared the full multivariable model with the simplified models. Reduced models were structured to include variables with greater ease of ascertainment at hospital discharge and to include removing variables requiring prehospitalization laboratory data. To further support clinical use of the model, we created a simple integer risk index using the method described by Sullivan et al.32 The performance of the risk index was determined based on modeling the logit of the total risk index for each participant.

Prediction Model Performance

The regression coefficients from the logistic regression models in the derivation sample were fixed. The fitted models were applied to both the internal and external validation cohorts. The overall goodness-of-fit of the models was compared based on Akaike Information Criteria. Model discrimination was compared based on the C statistic and integrated discrimination improvement (IDI). Model calibration was assessed by the calibration intercept, by the calibration slope, and graphically by locally weighted scatterplot smoothing (LOESS) plots of observed vs predicted probabilities of the outcome.33,34 The ability of models to reclassify patients into high- or low-risk categories was compared using the net reclassification index (NRI).35,36 Risk thresholds for the outcome were defined as less than 1.0%, 1.0% to less than 5.0%, 5.0% to less than 10.0%, 10.0% to less than 20.0%, and 20.0% or higher. Reclassification was also compared using the continuous NRI, which assesses reclassification across a continuous range of risk thresholds. A P value of .05 was considered statistically significant using 2-sided testing. Statistical analyses were performed using SAS version 9.4 (SAS Institute Inc) and R (https://r-project.org).

Results
Characteristics of the Cohorts

A total of 14 958 participants were included in the derivation and internal validation cohorts (9973 in the derivation cohort and 4985 in the internal validation cohort) and 2761 in the external validation cohort (Figure 1). The mean age of participants ranged from 66 years through 69 years, the mean baseline serum creatinine value was 1.0 mg/dL, obtained a median of 66 days (interquartile range [IQR], 24-161 days) prior to admission in the derivation and internal validation cohorts and 44 days (IQR, 17-124 days) in the external validation cohort, more than 20% had stage 2 or 3 acute kidney injury and the serum creatinine value was 1.3 mg/dL or higher at the time of hospital discharge in one-third of patients (Table 1; to convert creatinine from mg/dL to μmol/L, multiply by 88.4). Albuminuria status was available for 57% of the derivation and internal validation cohorts and 36% of the external validation cohort. Compared with the derivation and internal validation cohorts, patients in the external validation cohort were slightly older and were more likely to have received cardiovascular procedures or mechanical ventilation. The median number of outpatient serum creatinine measurements per patient during follow-up was 11 (IQR, 6-22) in the derivation and internal validation cohorts and 12 (IQR, 7-21) in the external validation cohort. There were 29 patients (0.2%) in the derivation and internal validation cohorts with only 1 eGFR less than 30 mL/min/1.73 m2 and did not have a subsequent measurement during follow-up. They were not considered to have developed advanced chronic kidney disease.

Four hundred eight patients (2.7%) in the derivation and internal validation cohorts developed the outcome of having a sustained eGFR value of less than 30 mL/min/1.73m2 after hospital discharge. Seven hundred seventy-five patients (5.2%) in the derivation and internal validation cohorts had an eGFR value of less than 30 mL/min/1.73 m2 at discharge, of whom 172 (22.2%) later developed advanced chronic kidney disease, which comprised 42.2% of the total number of advanced chronic kidney disease events. A similar proportion of patients experienced the outcome in the external validation cohort (62 events, 2.2%).

Prediction of Advanced Chronic Kidney Disease in the Derivation Cohort

Six variables were associated with a higher risk of progression to advanced chronic kidney disease in bootstrapped samples of the derivation cohort: older age, female sex, higher baseline serum creatinine values, albuminuria, greater acute kidney injury severity, and higher discharge serum creatinine values (Table 2; eTable 2 in Supplement 1). The 6-variable model (model 1) had the highest C statistic (0.86; 95% CI, 0.84-0.89) and lowest Akaike Information Criterion (1967.0). Based on the predicted risk using model 1, 5228 patients (52.4%) in the derivation cohort had less than 1%; 3512 (35.2%), 1% to less than 5%; 626 (6.2%), 5% to less than 10%; 360 (3.6%), 10% to less than 20%; and 247 (2.5%), 20% or more risk of developing chronic kidney disease (eTable 3 in Supplement 1). Discrimination (based on the C statistic and integrated discrimination improvement [IDI]) was lower in reduced models without inclusion of albuminuria, discharge serum creatinine values, and baseline serum creatinine values. The lowest C statistic was observed for model 5, which included age, sex, and acute kidney injury stage only (C statistic, 0.71; 95% CI, 0.67-0.74).

Prediction Model Performance in the Internal Validation Cohort

In the internal validation cohort, the 6-variable model remained well calibrated (eFigure 1 in Supplement 1) and also had a higher C statistic (0.87; 95% CI, 0.84-0.90) than did the models with fewer variables (Table 3; eTable 4 in Supplement 1). The IDI, continuous NRI, and categorical NRI favored the 6-variable model compared with the reduced-variable models. For example, significantly improved classification into risk categories was observed with the 6-variable model 1 compared with model 4—age, sex, and discharge serum creatinine values (categorical NRI, 16.6%; 95% CI, 7.7%-25.6%; P value <.001) and model 5—age, sex, and acute kidney injury stage (categorical NRI, 99.3%; 95% CI, 87.6%-110.9%; P value <.001). Model calibration and discrimination were similar for patients with baseline serum creatinine measurements taken within 3 months or taken more than 3 months prior to admission and with baseline eGFR greater than vs less than 60 mL/min/1.73 m2 (eTable 5 in Supplement 1), as well as for patients excluded from the derivation and validation cohorts due to absence of outpatient serum creatinine measurements prior to index hospitalization (eTable 6 in Supplement 1).

The integer-based risk index developed from the 6-variable model (Figure 2) showed similar discrimination in the internal validation cohort (C statistic, 0.87; 95% CI, 0.83-0.91) to that of the logistic regression model and also remained well calibrated (calibration intercept, −0.08; P value = .72; calibration slope, 1.02; P value = .98). Observed vs predicted risks from the integer-based risk index across the range of values of the risk index are shown in Figure 3A.

Prediction Model Performance in the External Validation Cohort

The 6-variable model remained well calibrated in the external validation cohort (eFigure 2 in Supplement 1) and had a C statistic of 0.81 (95% CI, 0.75-0.86). Discrimination and reclassification remained improved using the 6-variable model compared with simpler models—model 4, age, sex, and discharge serum creatinine or model 5, age, sex, and acute kidney injury stage (Table 3). The observed vs predicted risks applying the integer-based risk index in the external validation cohort is shown in Figure 3B.

Clinical Utility of the Prediction Model

The potential clinical utility of applying the 6-variable risk index to plan follow-up care, based on a range of predicted risk thresholds for advanced chronic kidney disease, is shown in eTable 7 in Supplement 1. The percentage of patients above the risk threshold for follow-up ranged from 48% to 2.6%, while positive predictive values ranged from 5.2% to 30.5% for corresponding risk thresholds of 1% or higher and 20% or higher, respectively. Negative predictive values were 98% or higher for each of the risk thresholds.

Discussion

A 6-variable model and corresponding risk index (including age, sex, baseline serum creatinine value, albuminuria, acute kidney injury severity, and discharge serum creatinine value) showed the best performance for predicting whether patients who had acute kidney injury during their hospital stay would later develop advanced chronic kidney disease in internal and external validation cohorts, with better performance than reduced models based on age, sex, and discharge serum creatinine alone or based on age, sex, or acute kidney injury severity alone. The predictor variables used in this model and risk index may be readily ascertained at the time of hospital discharge making it feasible for implementation within discharge planning processes to identify patients at high risk of developing advanced chronic kidney disease following hospitalization with acute kidney injury. Such risk stratification could help guide prognostic assessment and follow-up during transition to outpatient medical care, and inform resource allocation since advanced chronic kidney disease carries a high risk of complications, is accompanied by recommendations for assessment by nephrology specialists, and leads to high resource requirements to provide care.7,8,22,24 The predictor variables contained in these risk models have been previously associated with chronic kidney disease or end-stage kidney disease following acute kidney injury.1-4 This work has integrated these variables into multivariable risk models and a practical risk index, so that these variables can be readily used to obtain individualized estimates of advanced chronic kidney disease for patients with acute kidney injury encountered in clinical practice.

Model validation is an important aspect of developing clinical prediction models. There was no deterioration in model calibration upon evaluation in the internal validation cohort, indicating that the models were likely not overfit. An even more important aspect is external validation in a different population of patients before widespread model adoption and use. These models performed well in the geographically distinct Ontario cohort, despite some expected loss of discriminative performance (model 1 C statistic, 0.81; 95% CI, 0.75-0.86). Models are typically considered useful for clinical decision making when the C statistic is higher than 0.70 and strong when the C statistic exceeds 0.80, suggesting this model could support clinical decision making.37 Furthermore, the net reclassification improvement of the full model compared with simpler models demonstrates that using this model could improve accuracy of decision making for follow-up compared with that based on fewer variables alone.

Strengths of this study include that the model was developed in a large, population-representative cohort of adults hospitalized with acute kidney injury. The predictor variables in these risk models consist of readily available patient demographics and laboratory tests, which could allow them to be implemented into clinical practice using automated approaches within electronic medical record systems (an electronic risk calculator is available in Supplement 2). The models were validated in a distinct, external cohort.

Limitations

This study has several limitations. First, candidate variables were identified from secondary analysis of data, which does not include all potential risk factors for chronic kidney disease such as blood pressure, urine sediment, cause of acute kidney injury, and details of dialysis treatments. However, similar to this study, another risk prediction model for chronic kidney disease also found that clinical variables provided little improvement in model performance beyond kidney-related laboratory variables.38 Second, if low-risk patients were systematically excluded from these cohorts due to lack of follow-up creatinine testing, then estimates from the resulting models could overestimate risk of advanced chronic kidney disease. However, the predicted risks of excluded patients were similar to those of included patients, suggesting that such bias is unlikely. Third, some participants did not have albuminuria measured, which may influence the performance of the models. However, models without albuminuria also performed well, suggesting that risk prediction for chronic kidney disease following acute kidney injury could still be achieved without albuminuria measurements. Fourth, models were derived and validated in cohorts from Canada, and generalizability to patients from other regions was not examined.

This study may have implications for patients, clinicians, and policy makers. Studies from both Canada and the United States have demonstrated low rates of follow-up for chronic kidney disease care and poor clinical outcomes in individuals with chronic kidney disease after acute kidney injury.5-8 This may be because clinicians in the community are not aware of acute kidney injury episodes that occur during hospitalization, lack awareness of the prognostic implications of these events, or encounter barriers to timely access to chronic kidney disease care in the community. This highlights the common clinical challenge of providing continuity of care between the hospital and community.7,9,39 These risk prediction models provide an accurate but simple strategy that could be used to stratify patients into clinically meaningful risk groups at the time of hospital discharge and guide further management in the community.40

Conclusions

A multivariable model using routine laboratory data was able to predict advanced chronic kidney disease following hospitalization with acute kidney injury. The utility of this model in clinical care requires further research.

Back to top
Article Information

Corresponding Author: Matthew T. James, MD, PhD, Foothills Medical Center, 3280 Hospital Dr NW, Calgary, AB T2N 4Z6, Canada (mjames@ucalgary.ca).

Accepted for Publication: October 4, 2017.

Author Contributions: Dr James had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: James, Pannu, Hemmelgarn, Tan, Manns, Wald, Quinn, Ravani.

Acquisition, analysis, or interpretation of data: James, Pannu, Hemmelgarn, Austin, Tan, McArthur, Tonelli, Wald, Quinn, Ravani, Garg.

Drafting of the manuscript: James, Tan, Ravani.

Critical revision of the manuscript for important intellectual content: Pannu, Hemmelgarn, Austin, Tan, McArthur, Manns, Tonelli, Wald, Quinn, Ravani, Garg.

Statistical analysis: James, Pannu, Austin, Tan, McArthur, Ravani.

Obtained funding: James, Hemmelgarn, Wald.

Administrative, technical, or material support: James, Hemmelgarn, Manns, Tonelli, Garg.

Supervision: James, Hemmelgarn, Manns, Quinn.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr James reported receiving support from a Canadian Institutes of Health Research (CIHR) New Investigator and CIHR Foundation award and grant support from Amgen Canada. Dr Hemmelgarn reported receiving support from the Roy and Vi Bay chair in Kidney Research at the University of Calgary. Dr Austin reported receiving support from a Career Investigator award from the Heart and Stroke Foundation of Canada. Dr Manns reported receiving support from the Svare Professorship in Health Economics at the University of Calgary. Dr Tonelli reported receiving support from a CIHR Foundation award. Dr Quinn reported having a patent for a dialysis measurement analysis reporting system. Dr Garg was supported by the Dr Adam Linton Chair in Kidney Health Analytics at Western University and a CIHR Clinician Investigator Award. No other disclosures are reported.

Funding/Support: The study was funded by an Open Operating grant from the Canadian Institutes of Health Research (CIHR). This study was supported by the Institute for Clinical Evaluative Sciences (ICES) Western site. ICES is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). Core funding for ICES Western is provided by the Academic Medical Organization of Southwestern Ontario, the Schulich School of Medicine and Dentistry, Western University, the Lawson Health Research Institute, and 14 clinical departments.

Role of the Funder/Sponsor: None of sponsors had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: This study is based in part on data provided by Alberta Health and Alberta Health Services. The interpretation and conclusions contained herein are those of the researchers and do not necessarily represent the views of the government of Alberta or Alberta Health Services. Neither the government of Alberta nor the Alberta Health or Alberta Health Services express any opinion in relation to this study. The opinions, results, and conclusions reported herein are those of the authors and are independent from these funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred. Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information (CIHI). However, the analyses, conclusions, opinions, and statements expressed herein are those of the authors, and not necessarily those of CIHI. Some of this work was conducted with members of the provincial ICES Kidney, Dialysis, and Transplantation Research Program, which receives programmatic grant funding from CIHR.

References
1.
Coca  SG, Singanamala  S, Parikh  CR.  Chronic kidney disease after acute kidney injury: a systematic review and meta-analysis.  Kidney Int. 2012;81(5):442-448.PubMedGoogle ScholarCrossref
2.
Sawhney  S, Marks  A, Fluck  N,  et al.  Post-discharge kidney function is associated with subsequent ten-year renal progression risk among survivors of acute kidney injury.  Kidney Int. 2017;92(2):440-452.PubMedGoogle ScholarCrossref
3.
Ishani  A, Nelson  D, Clothier  B,  et al.  The magnitude of acute serum creatinine increase after cardiac surgery and the risk of chronic kidney disease, progression of kidney disease, and death.  Arch Intern Med. 2011;171(3):226-233.PubMedGoogle ScholarCrossref
4.
Kidney International Supplements. KDIGO clinical practice guideline for acute kidney injury. http://www.kisupplements.org/issue/S2157-1716(12)X7200-9. Published March 2012. Accessed October 5, 2017.
5.
Siew  ED, Peterson  JF, Eden  SK,  et al.  Outpatient nephrology referral rates after acute kidney injury.  J Am Soc Nephrol. 2012;23(2):305-312.PubMedGoogle ScholarCrossref
6.
Harel  Z, Wald  R, Bargman  JM,  et al.  Nephrologist follow-up improves all-cause mortality of severe acute kidney injury survivors.  Kidney Int. 2013;83(5):901-908.PubMedGoogle ScholarCrossref
7.
Goldstein  SL, Jaber  BL, Faubel  S, Chawla  LS; Acute Kidney Injury Advisory Group of American Society of Nephrology.  AKI transition of care: a potential opportunity to detect and prevent CKD.  Clin J Am Soc Nephrol. 2013;8(3):476-483.PubMedGoogle ScholarCrossref
8.
James  M, Bouchard  J, Ho  J,  et al.  Canadian Society of Nephrology commentary on the 2012 KDIGO clinical practice guideline for acute kidney injury.  Am J Kidney Dis. 2013;61(5):673-685.PubMedGoogle ScholarCrossref
9.
Coleman  EA, Berenson  RA.  Lost in transition: challenges and opportunities for improving the quality of transitional care.  Ann Intern Med. 2004;141(7):533-536.PubMedGoogle ScholarCrossref
10.
Collins  GS, Reitsma  JB, Altman  DG, Moons  KG.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD).  Ann Intern Med. 2015;162(10):735-736.PubMedGoogle ScholarCrossref
11.
Hemmelgarn  BR, Clement  F, Manns  BJ,  et al.  Overview of the Alberta Kidney Disease Network.  BMC Nephrol. 2009;10:30.PubMedGoogle ScholarCrossref
12.
Quan  H, Sundararajan  V, Halfon  P,  et al.  Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.  Med Care. 2005;43(11):1130-1139.PubMedGoogle ScholarCrossref
13.
Quan  H, Khan  N, Hemmelgarn  BR,  et al; Hypertension Outcome and Surveillance Team of the Canadian Hypertension Education Programs.  Validation of a case definition to define hypertension using administrative data.  Hypertension. 2009;54(6):1423-1428.PubMedGoogle ScholarCrossref
14.
Hux  JE, Ivis  F, Flintoft  V, Bica  A.  Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm.  Diabetes Care. 2002;25(3):512-516.PubMedGoogle ScholarCrossref
15.
Jain  AK, Cuerden  MS, McLeod  I,  et al.  Reporting of the estimated glomerular filtration rate was associated with increased use of angiotensin-converting enzyme inhibitors and angiotensin-II receptor blockers in CKD.  Kidney Int. 2012;81(12):1248-1253.PubMedGoogle ScholarCrossref
16.
Juurlink  D, Preyra  C, Croxford  R,  et al.  Canadian Institute for Health Information Discharge Abstract Database: A Validation Study. Toronto, ON: Institute for Clinical Evaluative Sciences; 2006.
17.
Siew  ED, Ikizler  TA, Matheny  ME,  et al.  Estimating baseline kidney function in hospitalized patients with impaired kidney function.  Clin J Am Soc Nephrol. 2012;7(5):712-719.PubMedGoogle ScholarCrossref
18.
Clement  FM, James  MT, Chin  R,  et al; Alberta Kidney Disease Network.  Validation of a case definition to define chronic dialysis using outpatient administrative data.  BMC Med Res Methodol. 2011;11:25.PubMedGoogle ScholarCrossref
19.
Levey  AS, Stevens  LA, Schmid  CH,  et al; CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration).  A new equation to estimate glomerular filtration rate.  Ann Intern Med. 2009;150(9):604-612.PubMedGoogle ScholarCrossref
20.
Go  AS, Chertow  GM, Fan  D, McCulloch  CE, Hsu  CY.  Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization.  N Engl J Med. 2004;351(13):1296-1305.PubMedGoogle ScholarCrossref
21.
Hemmelgarn  BR, Manns  BJ, Lloyd  A,  et al; Alberta Kidney Disease Network.  Relation between kidney function, proteinuria, and adverse outcomes.  JAMA. 2010;303(5):423-429.PubMedGoogle ScholarCrossref
22.
Andrassy  KM.  Comments on “KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease.”  Kidney Int. 2013;84(3):622-623.PubMedGoogle ScholarCrossref
23.
Levin  A, Hemmelgarn  B, Culleton  B,  et al; Canadian Society of Nephrology.  Guidelines for the management of chronic kidney disease.  CMAJ. 2008;179(11):1154-1162.PubMedGoogle ScholarCrossref
24.
National Institute for Health and Care Excellence,National Clinical Guideline Centre. Chronic kidney disease in adults in primary and secondary care: clinical guideline 182. Methods, evidence and recommendations. https://www.nice.org.uk/guidance/cg182. July 2014. Accessed April 17, 2017.
25.
Stevens  LA, Greene  T, Levey  AS.  Surrogate end points for clinical trials of kidney disease progression.  Clin J Am Soc Nephrol. 2006;1(4):874-884.PubMedGoogle ScholarCrossref
26.
Lamb  EJ, MacKenzie  F, Stevens  PE.  How should proteinuria be detected and measured?  Ann Clin Biochem. 2009;46(pt 3):205-217.PubMedGoogle ScholarCrossref
27.
Waikar  SS, Wald  R, Chertow  GM,  et al.  Validity of International Classification of Diseases, Ninth Revision, clinical modification codes for acute renal failure.  J Am Soc Nephrol. 2006;17(6):1688-1694.PubMedGoogle ScholarCrossref
28.
Kellum  JA.  How can we define recovery after acute kidney injury? considerations from epidemiology and clinical trial design.  Nephron Clin Pract. 2014;127(1-4):81-88.PubMedGoogle ScholarCrossref
29.
Wald  R, Quinn  RR, Luo  J,  et al; University of Toronto Acute Kidney Injury Research Group.  Chronic dialysis and death among survivors of acute kidney injury requiring dialysis.  JAMA. 2009;302(11):1179-1185.PubMedGoogle ScholarCrossref
30.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387.PubMedGoogle ScholarCrossref
31.
Steyerberg  EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009.
32.
Sullivan  LM, Massaro  JM, D’Agostino  RB  Sr.  Presentation of multivariate data for clinical use: the Framingham Study risk score functions.  Stat Med. 2004;23(10):1631-1660.PubMedGoogle ScholarCrossref
33.
Steyerberg  EW, Vickers  AJ, Cook  NR,  et al.  Assessing the performance of prediction models: a framework for traditional and novel measures.  Epidemiology. 2010;21(1):128-138.PubMedGoogle ScholarCrossref
34.
Austin  PC, Steyerberg  EW.  Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers.  Stat Med. 2014;33(3):517-535.PubMedGoogle ScholarCrossref
35.
Pencina  MJ, D’Agostino  RB  Sr, D’Agostino  RB  Jr, Vasan  RS.  Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.  Stat Med. 2008;27(2):157-172.PubMedGoogle ScholarCrossref
36.
Cook  NR, Ridker  PM.  Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures.  Ann Intern Med. 2009;150(11):795-802.PubMedGoogle ScholarCrossref
37.
Hosmer  DW, Lemeshow  S.  Applied Logistic Regression. 2nd ed. New York, NY: John Wiley & Sons; 2000.Crossref
38.
Tangri  N, Stevens  LA, Griffith  J,  et al.  A predictive model for progression of chronic kidney disease to kidney failure.  JAMA. 2011;305(15):1553-1559.PubMedGoogle ScholarCrossref
39.
Haggerty  JL, Reid  RJ, Freeman  GK, Starfield  BH, Adair  CE, McKendry  R.  Continuity of care: a multidisciplinary review.  BMJ. 2003;327(7425):1219-1221.PubMedGoogle ScholarCrossref
40.
Rennke  S, Nguyen  OK, Shoeb  MH, Magan  Y, Wachter  RM, Ranji  SR.  Hospital-initiated transitional care interventions as a patient safety strategy: a systematic review.  Ann Intern Med. 2013;158(5 pt 2):433-440.PubMedGoogle ScholarCrossref
×