Development and Validation of a Model for Predicting the Risk of Acute Kidney Injury Associated With Contrast Volume Levels During Percutaneous Coronary Intervention | Acute Kidney Injury | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Scatterplot of Patients’ Preprocedural Acute Kidney Injury (AKI) Risks and Contrast Volumes
Scatterplot of Patients’ Preprocedural Acute Kidney Injury (AKI) Risks and Contrast Volumes

The scatterplot was colored with observed risks of AKI estimated by each patients’ 250 nearest neighbors. To convert creatinine to micromoles per liter, multiply by 88.4.

Figure 2.  Calibration Plots of the Model Predicting Acute Kidney Injury via Splines in the Test Set
Calibration Plots of the Model Predicting Acute Kidney Injury via Splines in the Test Set

Risk of absolute increase in creatinine levels was calculated in the observed vs predicted risks via cubic spline smoothing. To convert creatinine to micromoles per liter, multiply by 88.4.

Figure 3.  Prediction of Risks of Acute Kidney Injury (AKI) as a Function of Contrast Volume
Prediction of Risks of Acute Kidney Injury (AKI) as a Function of Contrast Volume

Risks of absolute increase in creatinine was calculated for patients with preprocedural AKI risk of 5%, 45%, and 80%. Odds ratios (ORs) and absolute risk differences (RDs) of AKI risks are given for increases in contrast volume from 100 to 300 mL, from 400 to 600 mL, and from 700 to 900 mL. The colored bands indicate 95% CIs. To convert creatinine to micromoles per liter, multiply by 88.4.

Table 1.  Study Population Characteristics
Study Population Characteristics
Table 2.  Model Performance for Predicting Risk of Acute Kidney Injurya
Model Performance for Predicting Risk of Acute Kidney Injurya
1.
Tsai  TT, Patel  UD, Chang  TI,  et al.  Validated contemporary risk model of acute kidney injury in patients undergoing percutaneous coronary interventions: insights from the National Cardiovascular Data Registry Cath-PCI Registry.  J Am Heart Assoc. 2014;3(6):e001380. doi:10.1161/JAHA.114.001380PubMedGoogle Scholar
2.
Amin  AP, Bach  RG, Caruso  ML, Kennedy  KF, Spertus  JA.  Association of variation in contrast volume with acute kidney injury in patients undergoing percutaneous coronary intervention.  JAMA Cardiol. 2017;2(9):1007-1012. doi:10.1001/jamacardio.2017.2156PubMedGoogle ScholarCrossref
3.
Cigarroa  RG, Lange  RA, Williams  RH, Hillis  LD.  Dosing of contrast material to prevent contrast nephropathy in patients with renal disease.  Am J Med. 1989;86(6 Pt 1):649-652. doi:10.1016/0002-9343(89)90437-3PubMedGoogle ScholarCrossref
4.
Freeman  RV, O’Donnell  M, Share  D,  et al; Blue Cross-Blue Shield of Michigan Cardiovascular Consortium (BMC2).  Nephropathy requiring dialysis after percutaneous coronary intervention and the critical role of an adjusted contrast dose.  Am J Cardiol. 2002;90(10):1068-1073. doi:10.1016/S0002-9149(02)02771-6PubMedGoogle ScholarCrossref
5.
Brown  JR, Robb  JF, Block  CA,  et al.  Does safe dosing of iodinated contrast prevent contrast-induced acute kidney injury?  Circ Cardiovasc Interv. 2010;3(4):346-350. doi:10.1161/CIRCINTERVENTIONS.109.910638PubMedGoogle ScholarCrossref
6.
Tan  N, Liu  Y, Zhou  YL,  et al.  Contrast medium volume to creatinine clearance ratio: a predictor of contrast-induced nephropathy in the first 72 hours following percutaneous coronary intervention.  Catheter Cardiovasc Interv. 2012;79(1):70-75. doi:10.1002/ccd.23048PubMedGoogle ScholarCrossref
7.
Brown  JR, DeVries  JT, Piper  WD,  et al; Northern New England Cardiovascular Disease Study Group.  Serious renal dysfunction after percutaneous coronary interventions can be predicted.  Am Heart J. 2008;155(2):260-266. doi:10.1016/j.ahj.2007.10.007PubMedGoogle ScholarCrossref
8.
Huang  C, Murugiah  K, Mahajan  S,  et al.  Enhancing the prediction of acute kidney injury risk after percutaneous coronary intervention using machine learning techniques: a retrospective cohort study.  PLoS Med. 2018;15(11):e1002703. doi:10.1371/journal.pmed.1002703PubMedGoogle Scholar
9.
Weintraub  WS, McKay  CR, Riner  RN,  et al; American College of Cardiology Database Committee.  The American College of Cardiology National Database: progress and challenges.  J Am Coll Cardiol. 1997;29(2):459-465.PubMedGoogle ScholarCrossref
10.
Brindis  RG, Fitzgerald  S, Anderson  HV, Shaw  RE, Weintraub  WS, Williams  JF.  The American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR): building a national clinical data repository.  J Am Coll Cardiol. 2001;37(8):2240-2245. doi:10.1016/S0735-1097(01)01372-9PubMedGoogle ScholarCrossref
11.
Hobson  C, Ruchi  R, Bihorac  A.  Perioperative acute kidney injury: risk factors and predictive strategies.  Crit Care Clin. 2017;33(2):379-396. doi:10.1016/j.ccc.2016.12.008PubMedGoogle ScholarCrossref
12.
Yuan  M, Lin  Y.  Model selection and estimation in regression with grouped variables.  J R Stat Soc Series B Stat Methodol. 2006;68(1):49-67. doi:10.1111/j.1467-9868.2005.00532.xGoogle ScholarCrossref
13.
Machado  MN, Nakazone  MA, Maia  LN.  Acute kidney injury based on KDIGO (Kidney Disease Improving Global Outcomes) criteria in patients with elevated baseline serum creatinine undergoing cardiac surgery.  Rev Bras Cir Cardiovasc. 2014;29(3):299-307. doi:10.5935/1678-9741.20140049PubMedGoogle Scholar
14.
Gottlieb  SS, Abraham  W, Butler  J,  et al.  The prognostic importance of different definitions of worsening renal function in congestive heart failure.  J Card Fail. 2002;8(3):136-141. doi:10.1054/jcaf.2002.125289PubMedGoogle ScholarCrossref
15.
Smith  GL, Vaccarino  V, Kosiborod  M,  et al.  Worsening renal function: what is a clinically meaningful change in creatinine during hospitalization with heart failure?  J Card Fail. 2003;9(1):13-25. doi:10.1054/jcaf.2003.3PubMedGoogle ScholarCrossref
16.
Tsai  TT, Patel  UD, Chang  TI,  et al.  Contemporary incidence, predictors, and outcomes of acute kidney injury in patients undergoing percutaneous coronary interventions: insights from the NCDR Cath-PCI registry.  JACC Cardiovasc Interv. 2014;7(1):1-9. doi:10.1016/j.jcin.2013.06.016PubMedGoogle ScholarCrossref
17.
Hastie  T, Tibshirani  R.  Generalized additive models.  Stat Sci. 1986;1(3):297-310. doi:10.1214/ss/1177013604Google ScholarCrossref
18.
R Core Team. The R project for statistical computing. http://www.R-project.org/. Accessed March 26, 2018.
19.
Wood  SN.  Generalized Additive Models: An Introduction with R. 2nd ed. Boca Raton, FL: CRC Press; 2017. doi:10.1201/9781315370279
20.
Tan  N, Liu  Y, Chen  JY,  et al.  Use of the contrast volume or grams of iodine-to-creatinine clearance ratio to predict mortality after percutaneous coronary intervention.  Am Heart J. 2013;165(4):600-608. doi:10.1016/j.ahj.2012.12.017PubMedGoogle ScholarCrossref
21.
Liu  Y, Liu  YH, Chen  JY,  et al.  Safe contrast volumes for preventing contrast-induced nephropathy in elderly patients with relatively normal renal function during percutaneous coronary intervention.  Medicine (Baltimore). 2015;94(12):e615. doi:10.1097/MD.0000000000000615PubMedGoogle Scholar
22.
Laskey  WK, Jenkins  C, Selzer  F,  et al; NHLBI Dynamic Registry Investigators.  Volume-to-creatinine clearance ratio: a pharmacokinetically based risk factor for prediction of early creatinine increase after percutaneous coronary intervention.  J Am Coll Cardiol. 2007;50(7):584-590. doi:10.1016/j.jacc.2007.03.058PubMedGoogle ScholarCrossref
23.
Mehta  RL, Kellum  JA, Shah  SV,  et al; Acute Kidney Injury Network.  Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury.  Crit Care. 2007;11(2):R31-R31. doi:10.1186/cc5713PubMedGoogle ScholarCrossref
24.
Kellum  JA, Bellomo  R, Ronco  C, Mehta  R, Clark  W, Levin  NW.  The 3rd International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI).  Int J Artif Organs. 2005;28(5):441-444. doi:10.1177/039139880502800503PubMedGoogle ScholarCrossref
25.
Gruberg  L, Mintz  GS, Mehran  R,  et al.  The prognostic implications of further renal function deterioration within 48 h of interventional coronary procedures in patients with pre-existent chronic renal insufficiency.  J Am Coll Cardiol. 2000;36(5):1542-1548. doi:10.1016/S0735-1097(00)00917-7PubMedGoogle ScholarCrossref
26.
Marenzi  G, Lauri  G, Assanelli  E,  et al.  Contrast-induced nephropathy in patients undergoing primary angioplasty for acute myocardial infarction.  J Am Coll Cardiol. 2004;44(9):1780-1785. doi:10.1016/j.jacc.2004.07.043PubMedGoogle ScholarCrossref
27.
Bartholomew  BA, Harjai  KJ, Dukkipati  S,  et al.  Impact of nephropathy after percutaneous coronary intervention and a method for risk stratification.  Am J Cardiol. 2004;93(12):1515-1519. doi:10.1016/j.amjcard.2004.03.008PubMedGoogle ScholarCrossref
28.
Slocum  NK, Grossman  PM, Moscucci  M,  et al.  The changing definition of contrast-induced nephropathy and its clinical implications: insights from the Blue Cross Blue Shield of Michigan Cardiovascular Consortium (BMC2).  Am Heart J. 2012;163(5):829-834. doi:10.1016/j.ahj.2012.02.011PubMedGoogle ScholarCrossref
29.
Duncan  IG.  Healthcare Risk Adjustment and Predictive Modeling. New Hartford, CT: ACTEX Learning; 2011.
30.
Janssen  KJ, Moons  KG, Kalkman  CJ, Grobbee  DE, Vergouwe  Y.  Updating methods improved the performance of a clinical prediction model in new patients.  J Clin Epidemiol. 2008;61(1):76-86. doi:10.1016/j.jclinepi.2007.04.018PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Cardiology
    November 22, 2019

    Development and Validation of a Model for Predicting the Risk of Acute Kidney Injury Associated With Contrast Volume Levels During Percutaneous Coronary Intervention

    Author Affiliations
    • 1Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut
    • 2Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
    • 3Program of Applied Translational Research, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
    • 4Division of Cardiology, University of Colorado Anschutz Medical Campus, Aurora
    • 5Saint Luke’s Mid America Heart Institute, Department of Cardiology, University of Missouri, Kansas City
    • 6Department of Computer Science and Engineering, Texas A&M University, College Station
    • 7Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut
    JAMA Netw Open. 2019;2(11):e1916021. doi:10.1001/jamanetworkopen.2019.16021
    Key Points español 中文 (chinese)

    Question  What is the association of contrast volume used during percutaneous coronary intervention with the risk of acute kidney injury?

    Findings  In this prognostic study of nearly 3 million adults undergoing percutaneous coronary intervention, a predictive model for the association of contrast volume with the risk of acute kidney injury was developed and validated. The association was nonlinear along the spectrum of contrast volume and varied according to patient baseline risk.

    Meaning  This model could support personalized approaches that consider the complex association between contrast volume and the risk of acute kidney injury when determining contrast dosing during interventional procedures.

    Abstract

    Importance  Determining the association of contrast volume during percutaneous coronary intervention (PCI) with the risk of acute kidney injury (AKI) is important for optimizing PCI safety.

    Objective  To quantify how the risk of AKI is associated with contrast volume, accounting for the possibility of nonlinearity and heterogeneity among different baseline risks.

    Design, Setting, and Participants  This prognostic study used data from the American College of Cardiology National Cardiovascular Data Registry CathPCI Registry for 1694 US hospitals. Derivation analysis included 2 076 694 individuals who underwent PCI from July 1, 2011, to June 30, 2015. Validation analysis included 961 863 individuals who underwent PCI from July 1, 2015, to March 31, 2017. Data analysis took place from July 2018 to May 2019.

    Exposure  Contrast volume during PCI.

    Main Outcomes and Measures  Acute kidney injury was defined using 3 thresholds for preprocedure to postprocedure creatinine level increase (ie, ≥0.3 mg/dL, ≥0.5 mg/dL, and ≥1.0 mg/dL). A model quantifying the association of contrast volume with AKI was developed, and the existence of nonlinearity and heterogeneity were examined by likelihood ratio tests. The model was derived in the training set (a random 50% of the derivation cohort), and performance was evaluated in the test set (the remaining 50% of the derivation cohort) and an independent validation set by area under the receiver operating characteristic curve (AUC) and calibration slope of observed vs predicted risks.

    Results  The 2 076 694 patients in the derivation set had a mean (SD) age of 65.1 (12.1) years, and 662 525 (31.9%) were women; 133 306 (6.4%) had creatinine level increases of at least 0.3 mg/dL, 66 626 (3.2%) had creatinine level increases of at least 0.5 mg/dL, and 28 378 (1.4%) had creatinine level increases of at least 1.0 mg/dL. In the validation set of 961 843 patients (mean [SD] age, 65.7 [12.1] years; 305 577 [31.8%] women), these rates were 62 913 (6.5%), 34 229 (3.6%), and 15 555 (1.6%), respectively. The association of contrast volume and AKI risk was nonlinear (χ226 = 1436.2; P < .001) and varied by preprocedural risk (χ220 = 105.6; P < .001). In the test set, the model yielded an AUC of 0.777 (95% CI, 0.775-0.779) for predicting risk of a creatinine level increase of at least 0.3 mg/dL, 0.839 (95% CI, 0.837-0.841) for predicting risk of a creatinine level increase of at least 0.5 mg/dL, and 0.870 (95% CI, 0.867-0.873) for predicting risk of a creatinine level increase of at least 1.0 mg/dL; it achieved a calibration slope of 0.998 (95% CI, 0.989-1.007), 0.999 (95% CI, 0.989-1.008), and 0.986 (95% CI, 0.973-0.998), respectively, for the AKI severity levels. The model had similar performance in the validation set (creatinine level increase of ≥0.3 mg/dL: AUC, 0.794; 95% CI, 0.792-0.795; calibration slope, 1.039; 95% CI, 1.030-1.047; creatinine level increase of ≥0.5 mg/dL: AUC, 0.845; 95% CI, 0.843-0.848; calibration slope, 1.063; 95% CI, 1.054-1.074; creatinine level increase of ≥1.0 mg/dL: AUC, 0.872; 95% CI, 0.869-0.875; calibration slope, 1.103; 95% CI, 1.089-1.117).

    Conclusions and Relevance  The association of contrast volume with AKI risk is complex, varies by baseline risk, and can be predicted by a model. Future research to evaluate the effect of the model on AKI is needed.

    Introduction

    Acute kidney injury (AKI) is a common and serious complication of percutaneous coronary intervention (PCI) that is associated with an increased risk of death and higher costs of care. To inform personalized risk-mitigation interventions for patients undergoing PCI, researchers and clinicians have sought accurate risk estimations using the patients’ baseline characteristics.1 Reducing contrast volume during PCI is considered the most actionable and effective intervention.2 Various studies support this approach, with lower rates of AKI among individuals who receive lower contrast volumes.3-5

    While limiting contrast volume is an acknowledged method of reducing AKI risk, clinicians need to balance reducing contrast volume with providing adequate contrast to permit coronary evaluation and treatment.6 Thus, there is a need to better understand how AKI risk varies continuously with increasing contrast volume. Furthermore, among patients with different baseline risks, the association between contrast volume and AKI risk may differ and may not be linear across the spectrum of contrast volume. Identification of such heterogeneity and nonlinearity in risk could potentially improve clinical decision-making for individual patients. However, few studies have quantified the change in AKI risk based on differences in contrast volume, and fewer have examined potential heterogeneity in the association of contrast volume with AKI as a function of patient baseline risk of AKI. Moreover, studies on AKI risk estimation have exclusively considered a binary AKI definition, focusing on a single severity level. Providing risk estimates for multiple severities of AKI, especially for more serious AKIs that are associated with higher rates of morbidity and mortality,7 could facilitate better informed decisions on intervention.

    Accordingly, we developed a prediction model for AKI to determine the association of contrast volume with AKI risk in the National Cardiovascular Data Registry (NCDR) CathPCI Registry. Building on results of a previously developed machine learning model for estimating patients’ preprocedural risk of AKI,8 we employed a generalized additive model (GAM) to accommodate the potential nonlinear associations between contrast volume and AKI risk and the heterogeneity of this association among different risk groups.

    Methods
    Data Source and Study Population

    The NCDR CathPCI Registry has been described elsewhere.9,10 This study included PCIs performed from July 1, 2011, to June 30, 2015 (2 710 866 procedures) (eFigure 1 in the Supplement). We applied the same exclusion criteria used in developing the NCDR AKI model.1 We excluded PCIs that were not the first during a single hospitalization (n = 72 065), that had same-day discharge (n = 161 271), that were missing serum creatinine levels before or after the procedure (n = 324 438), and that were performed on patients who were already on dialysis (n = 60 523). Additionally, we excluded PCIs that were missing contrast volume measurements (n = 7209). We further excluded patients with preprocedural creatinine levels of 0.3 mg/dL or less (n = 2247) or higher than 4.0 mg/dL (n = 6419) (to convert creatinine levels to micromoles per liter, multiply by 88.4); the decision-making process in the care of these patients is different from that used for other patients,11 and including them in the analyses could mask the association between AKI and contrast volume for other patients. The final analytic cohort included the remaining 2 076 694 PCI procedures. To ensure the generalizability of the derived model for prospective use, we validated our results in a cohort that included 961 843 PCI procedures from July 1, 2015, to March 31, 2017, by implementing the same inclusion and exclusion criteria. The Yale University institutional review board approved the study and waived the patient informed consent requirement for the use of retrospective data from the NCDR registry. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.

    Study Outcome

    The primary outcome was AKI, defined by the difference between preprocedure and postprocedure creatinine levels. We considered a spectrum of AKI severity, defined by the following 3 thresholds for absolute creatinine level increase: at least 0.3 mg/dL, at least 0.5 mg/dL, and at least 1.0 mg/dL. This definition was different from the staging criteria used by the Acute Kidney Injury Network (AKIN)12 and the Kidney Disease Improving Global Outcomes.13 Stage 2 and stage 3 definitions from AKIN or Kidney Disease Improving Global Outcomes resulted in an event rate less than 1% in the study cohort, which was not sufficient to produce a useful risk-stratification model. The thresholds used in this study were gradations within stage 1 AKI. Previous studies have shown that modest increases in creatinine levels within stage 1 AKI are independently associated with short-term and long-term adverse clinical outcomes and increased length of hospitalization.14-16

    Risk Factors

    We used preprocedural AKI risk and contrast volume as risk factors to develop the model for guiding contrast dosing. The preprocedural risk was stage 1 AKI, defined according to AKIN as the risk of a creatinine increase of at least 0.3 mg/dL or 150% of the preprocedural level. This risk estimate was used to characterize patients’ baseline risk and was calculated from a previously developed prediction model.8 This prediction model was derived from an earlier cohort from the NCDR CathPCI Registry (with procedures from June 1, 2009, to June 30, 2011) using gradient descent boosting and 13 variables. The detailed definitions and coding of these variables can be found in eTable 1 in the Supplement. Missing values of these baseline variables were imputed by the mode for categorical variables and by the median for continuous variables.

    Statistical Analysis

    The distribution of contrast volume used for patients with different preprocedural risks was summarized as the number and percentage of patients using varying contrast volumes. The pattern of contrast volume usage among individual patients was also shown in 2-dimensional scatterplots. The scatterplots were colored by observed AKI risks estimated by the event rate among each patient’s 250 nearest neighbors, which were determined by the Mahalanobis distance in preprocedural risk and contrast volume.

    To derive and validate the model, we randomly divided the final analytic cohort into a training set (50% of the cohort) and a test set (the remaining 50%). The model was built using the training set, and the performance of the model was evaluated on both the test set and an independent temporal validation set.

    We used a GAM to model and predict the association of contrast volume with AKI risk for different patients. To enable modeling for the 3 severity levels of AKI, we converted the creatinine changes into 4 mutually exclusive categories (ie, <0.3 mg/dL; 0.3 to <0.5 mg/dL, 0.5 to <1.0 mg/dL, and ≥1.0 mg/dL) as the dependent variable and used the logit of the preprocedural risk and the contrast volume as the predictors, employing a multinomial logit link function. The GAM was capable of learning complex associations via a sum of nonlinear functions and was therefore useful in uncovering nonlinear associations and interactions.17 For this study, the nonlinear functions were the tensor products of cubic regression splines of the 2 predictors. Cubic regression splines and tensor products were used to model potential nonlinearity interactions of the predictors. To assess the existence of nonlinearity and heterogeneity in the association of contrast volume with AKI, we further derived 3 alternative prediction models, as follows: (1) linear terms of the 2 predictors, (2) linear predictors plus an interaction term, and (3) nonlinear predictors with no interaction term. The fitness of these models to the data was compared by likelihood ratio tests. These models were also compared visually by 2-dimensional contour plots of the predictions against the observed risks. We also examined whether the association of contrast volume with AKI differed for different severity levels of AKI by comparing the prediction performance of the model in the training set with an alternative model using ordinal logit link function with a proportional odds assumption.

    Model discrimination was evaluated by the area under the receiver operating characteristic curve (AUC), and calibration was evaluated by the calibration intercept and slope of the observed rates vs the deciles of the predicted risks. A smooth calibration plot showing model calibration for a wider range of risks was also provided using cubic spline smoothers. Prediction accuracy was assessed by the Brier score, defined as the mean squared difference between the observed outcome and predicted risks. Brier scores quantify the error made by predictions from observations. We also calculated the predictive range of the model by the difference of the observed rates in the first and tenth deciles of predicted risks. We reported the performance of the model for predicting each of the 3 AKI severities.

    To demonstrate the utility of the prediction model to guide contrast dosing, we plotted the predicted AKI risks with 95% CIs as a function of contrast volume for a range of preprocedural risks. We further calculated the odds ratio (OR) and absolute risk difference (RD) of AKI by an increase of contrast volume of 200 mL along the spectrum of contrast volume. To safeguard against overextrapolation from what the data can inform, predictions were calculated for values of preprocedural risks and contrast volumes where there were at least 10 patients in the training set within the neighborhood of these values; the neighborhood was defined as a 5% difference in preprocedural risk and a 50 mL difference in contrast volume.

    Model performance was further evaluated in the validation cohort, and all performance metrics in the main analyses were reported. All analyses were developed in R version 3.4.0 (the R Foundation for Statistical Computing).18 The GAM was performed using the mgcv version 1.8.24 R package.19 For all 2-tailed χ2 tests, P < .05 was considered statistically significant.

    Results

    Table 1 describes event rates, characteristics, and contrast volumes for patients included in the derivation and validation sets. In the derivation set of 2 076 694 patients (mean [SD] age, 65.1 [12.1] years; 662 525 [31.9%] women), 133 306 patients (6.4%) had a creatinine level increase of at least 0.3 mg/dL, 66 626 (3.2%) had a creatinine level increase of at least 0.5 mg/dL, and 28 378 (1.4%) had a creatinine level increase of at least 1.0 mg/dL. In the validation set of 961 843 patients (mean [SD] age, 65.7 [12.1] years; 305 577 [31.8%] women), these rates were 62 913 (6.5%), 34 229 (3.6%), and 15 555 (1.6%), respectively. Compared with patients in the derivation set, patients in the validation set had higher median (interquartile range) preprocedural risk of stage 1 AKI (5.4% [3.4%-10.0%] vs 5.0% [3.2%-9.2%]) and received lower mean (SD) contrast volume (182.4 [82.3] mL vs 190.6 [86.4] mL).

    Similar patterns of contrast volume were received by patients in different preprocedural risk groups (eTable 2 in the Supplement). For each preprocedural risk group, about half the patients received 100 to 200 mL of contrast, with the percentage gradually decreasing when contrast volume decreased to less than 100 mL or increased to greater than 200 mL. Figure 1 shows the contrast volume received by individual patients ordered by their preprocedural risk estimates and colored by the observed AKI rates. The AKI rate was higher for patients with higher preprocedural risk and generally increased with contrast volume.

    Comparing the models using linear predictors vs nonlinear predictors, a test for the existence of nonlinearity was statistically significant (χ226 = 1436.2; P < .001). Tests for the existence of interaction were statistically significant for comparing models using linear predictors (χ23 = 9.1; P = .03) and comparing models using nonlinear predictors (χ220 = 105.6; P < .001). Comparing the observed risks with predictions from different candidate models for predicting the risk of a creatinine level increase of at least 0.3 mg/dL also revealed a better fit of data to the model employing nonlinearity and interaction terms (eFigure 2 in the Supplement). Finally, the model that used an ordinal link function had worse calibration than the model using a multinomial link function (eTable 3 in the Supplement), showing the inadequacy of using the proportional odds specification to model the association of contrast volume with AKI for different severity levels. Evaluated on the test set, the best model achieved an AUC of 0.777 (95% CI, 0.775-0.779) for predicting risk of a creatinine level increase of at least 0.3 mg/dL, 0.839 (95% CI, 0.837-0.841) for predicting risk of a creatinine level increase of at least 0.5 mg/dL, and 0.870 (95% CI, 0.867-0.873) for predicting risk of a creatinine level increase of at least 1.0 mg/dL (Table 2). It achieved a calibration slope of 0.998 (95% CI, 0.989-1.007), 0.999 (95% CI, 0.989-1.008), and 0.986 (95% CI, 0.973-0.998), respectively, for the AKI severity levels. This model also had Brier scores of 0.0539 (95% CI, 0.0538-0.0540), 0.0276 (95% CI, 0.0275-0.0276), and 0.0123 (95% CI, 0.0123-0.0124), respectively, demonstrating low error and high accuracy. Good calibration between the observed and predicted risks for all 3 AKI severities are shown in eFigure 3 in the Supplement. Smooth calibration plots also show good calibration for patients (Figure 2), except for a slight overestimation toward those with higher risk. When applied to the validation set, the model achieved a similar AUC of 0.794 (95% CI, 0.792-0.795), 0.845 (95% CI, 0.843-0.848), and 0.872 (95% CI, 0.869-0.875) for the 3 AKI severities, with satisfactory calibration slopes (creatinine level increase of ≥0.3 mg/dL: 1.039; 95% CI, 1.030-1.047; creatinine level increase of ≥0.5 mg/dL: 1.063; 95% CI, 1.054-1.074; creatinine level increase of ≥1.0 mg/dL: 1.103; 95% CI, 1.089-1.117) (Table 2; eFigure 4 and eFigure 5 in the Supplement).

    Figure 3 plots the estimated risks of creatinine level increases of at least 0.3 mg/dL, 0.5 mg/dL, and 1.0 mg/dL as a function of contrast volume for preprocedural risks of 5%, 45%, and 80%. The learned model (eFigure 6 in the Supplement) revealed that larger differences in risk were associated with contrast volume for patients with preprocedural risk between 25% to 75% compared with other patients. We thus chose these 3 preprocedural risks to quantify the difference in how AKI risk was associated with increasing contrast volume. For each preprocedural risk, the OR and RD of AKI were given for contrast volume increases from 100 to 300 mL, from 400 to 600 mL, or from 700 to 900 mL. First, for a given preprocedural risk, there were differences along the spectrum of contrast in the OR of AKI by 200 mL increase of contrast volume. For example, for the risk of a creatinine level increase of at least 0.3 mg/dL, the OR for a patient with 5% preprocedural risk with a contrast volume increase of 100 to 300 mL was 1.36 (95% CI, 1.32-1.43) and became 1.64 (95% CI, 1.52-1.77) for a contrast volume increase of 400 to 600 mL (Figure 3A; eTable 4 in the Supplement). Second, there were differences in ORs between preprocedural risk groups by the same contrast increase. For example, for an increase of contrast volume of 100 to 300 mL, patients with a 45% preprocedural risk had the highest OR and RD compared with patients with a 5% or 80% preprocedural risk (eg, risk of ≥0.5 mg/dL increase in creatinine levels: 45% procedural risk, OR, 1.56; 95% CI, 1.48-1.65; RD, 9.7%; 95% CI, 8.5%-10.8%; 5% preprocedural risk, OR, 1.41; 95% CI, 1.33-1.49; RD, 0.4%; 95% CI, 0.3%-0.4%; 80% preprocedural risk, OR, 1.26; 95% CI, 1.06-1.50; RD, 5.3%; 95% CI, 1.3%-9.2%) (Figure 3B; eTable 4 in the Supplement). Third, there were differences between severity levels of AKI in the association of contrast volume with AKI. For example, for patients with a 5% preprocedural risk, the ORs and RDs for an increase of at least 0.5 mg/dL or 1.0 mg/dL had a larger difference from a 100 to 300 mL increase in contrast volume and from a 400 to 600 mL increase in contrast volume than the OR and RD for the risk of a creatinine level increase of at least 0.3 mg/dL (risk of ≥0.3 mg/dL increase in creatinine levels with 100-300 mL increase in contrast volume, OR, 1.36; 95% CI, 1.32-1.42; RD, 1.1%; 95% CI, 1.0%-1.2%; with 400-600 mL increase in contrast volume, OR, 1.64; 95% CI, 1.52-1.77; RD, 3.1%; 95% CI, 2.5%-3.7%; risk of ≥0.5 mg/dL increase in creatinine levels with 100-300 mL increase in contrast volume, OR, 1.41; 95% CI, 1.33-1.49; RD, 0.4%; 95% CI, 0.3%-0.4%; with 400-600 mL increase in contrast volume, OR, 1.97; 95% CI, 1.75-2.21; RD, 1.7%; 95% CI, 1.3%-2.2%; risk of ≥1.0 mg/dL increase in creatinine levels with 100-300 mL increase in contrast volume, OR, 1.51; 95% CI, 1.36-1.67; RD, 0.1%; 95% CI, 0.1%-0.2%; with 400-600 mL increase in contrast volume, OR, 2.27; 95% CI, 1.91-2.70; RD, 0.7%; 95% CI, 0.5%-1.0%) (Figure 3; eTable 4 in the Supplement).

    Discussion

    We developed and validated a model for quantifying the association of contrast volume with the risk of 3 AKI severities for patients undergoing PCI. We found that the risk increase varied with the volume of contrast and the patient’s baseline risk. Given our results and prior studies showing little variation in the current use of contrast by patient risk,2 the model has the potential to support personalized estimates of contrast dosing to reduce AKI risk.

    Previous work on guiding contrast dosing focused on deriving a safe maximum level of contrast volume.4,20,21 However, different safe levels were derived from cohorts with different inclusion criteria, posing a challenge for their use in a general population of patients undergoing PCI.6,21,22 Moreover, information on AKI risks associated with incremental contrast volume below and above the safe level is valuable for decision-making.6 Furthermore, these studies have assumed a linear association between AKI risk and contrast volume, ignoring possible heterogeneity in this association among different patients. In contrast, our model was built on more than 1 million patients across a broad spectrum of baseline risk, making it applicable for a general population of patients undergoing PCI. More importantly, the advanced analytics used in developing this model allow insight into a more complex association between contrast volume and AKI risk, beyond that acknowledged by the literature. The heterogeneity found in the association between AKI and contrast volume among patients could enable personalized decisions on contrast dosing and could even help to inform the decision to have PCI.

    Physicians base the diagnosis of AKI on an elevation in serum creatinine concentration, but there are different definitions of the degree of elevation required for a diagnosis of AKI. Common definitions use an absolute increase above 0.3 mg/dL, 0.5 mg/dL, or 1.0 mg/dL, a relative increase above 25%, or a composite of the absolute and relative increase.23-27 These definitions would identify different patients with an AKI diagnosis; thus, prediction models built on 1 definition cannot be applied to other definitions.28 To develop a prediction model for guiding contrast dosing that is applicable irrespective of the AKI definition under consideration, we chose to use 3 thresholds of absolute increase in creatinine level as the study outcome, including the 0.3 mg/dL threshold as part of the AKIN definition, the 0.5 mg/dL threshold commonly used in previous studies, and a highest threshold of 1.0 mg/dL. Providing predictions for different severities of AKI would enable decision-making according to the severity level for each patient as determined by the clinician. We did not include the relative changes in our outcome definition because they depend on baseline creatinine level and may introduce discontinuity in the association between outcome and the 2 predictors, resulting in difficulty in modeling. Even though we did not use relative creatinine, it can be calculated for decision-making by using baseline creatinine levels. The derived model simultaneously predicted risks for the spectrum of AKI severity, providing a more nuanced understanding of the association of contrast volume with AKI to guide informed decisions on contrast dosing.

    The approach implemented in this study could be applied to other clinical outcomes and modifiable procedural risk factors to investigate the association of procedural variables with the estimate of outcome risk, providing guidance for decisions surrounding risk-mitigation strategies. A large body of work on clinical prediction modeling is based on models that exclusively use baseline information. Such a model is useful for both benchmarking quality assessment and planning treatment.29 With the additional integration of procedural variables, there is opportunity to identify new and helpful information for personalized strategies from the data.

    Our model quantifying the association between AKI risk and contrast volume for individual patients is a step toward personalized decision-making about the net benefit of PCI and personalized treatment planning for PCI to reduce AKI risk. There are several next steps for implementing this model. First, the current model used registry data, in which some of the data elements required manual abstraction. To integrate the model as decision support at the point of care, there is a need to automate variable extraction, most likely from electronic health records that include clinical notes. This need is increasingly being addressed by advancements in the computational capacities of electronic health records and advanced analytics as active research topics. Second, the effective implementation of the model through integration with clinical workflow to encourage model utility in decision-making also needs to be established. Last, the predictions provided by the model should be considered alongside information on the complexity of the case and the risk profile of the patient. These factors should be presented in an interactive interface for clinicians to make informed decisions.

    Limitations

    This study has several limitations. First, the model was developed using observational data on PCI patients who had exposure to contrast. However, the similar patterns in contrast use observed for different baseline risks ensured that the bias owing to contrast use preference conditioned on baseline risk was limited. Second, although the model was temporally validated, it may need to be continually updated to reflect changes in patient characteristics and associations in data.30 Third, the model cannot provide predictions for patients with very high preprocedural risks and very high contrast volumes owing to lack of data in the cohort for these combinations. However, such high volume is rarely considered for these patients. In the event that high volume is needed, predictions from the model should not be used for decision support because so few patients with high risk in our database had that exposure. Fourth, we have no information on when the postprocedural creatinine levels were assessed, and patients discharged the same day of the procedure were excluded. These omissions may have failed to recognize AKIs in which creatinine levels rose after discharge. However, these patients are often at the lowest risk for AKI and do not have postprocedural creatinine levels assessed. The model remains applicable for general PCI patients, especially for those with an elevated risk of AKI who could benefit most from the model. Fifth, we do not have access to outcomes after discharge for these patients to evaluate how the identified creatinine level increases were associated with those outcomes. Further study should investigate whether the model could reduce the risk of clinical adverse outcomes. Sixth, this study did not assess types of contrast agents in association with AKI risk, which may be a topic for further investigation.

    Conclusions

    We developed a model to estimate the association of contrast volume with AKI risk for patients undergoing PCI. We found this association to be nonlinear and heterogeneous among patients with different baseline risks. The model provided information that could guide personalized contrast dosing for individual patients to reduce their risks of AKI. Further investigation is needed to apply and validate the model at the point of care and evaluate its impact on decision-making and outcomes.

    Back to top
    Article Information

    Accepted for Publication: October 2, 2019.

    Published: November 22, 2019. doi:10.1001/jamanetworkopen.2019.16021

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Huang C et al. JAMA Network Open.

    Corresponding Author: Harlan M. Krumholz, MD, SM, Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, One Church Street, Ste 200, New Haven, CT 06510 (harlan.krumholz@yale.edu).

    Author Contributions: Dr Huang had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Drs Mortazavi and Krumholz are co–senior authors.

    Concept and design: Huang, Li, Mahajan, Testani, Rumsfeld, Spertus, Mortazavi, Krumholz.

    Acquisition, analysis, or interpretation of data: Huang, Li, Mahajan, Wilson, Mena, Masoudi, Mortazavi, Krumholz.

    Drafting of the manuscript: Huang, Li.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Huang, Li, Mortazavi.

    Obtained funding: Krumholz.

    Administrative, technical, or material support: Masoudi.

    Supervision: Li, Wilson, Mena, Krumholz.

    Conflict of Interest Disclosures: Dr Testani reported receiving personal fees from Reprieve Medical outside the submitted work. Dr Wilson reported receiving grants R1DK11311 and P30DK079310 from the National Institutes of Health during the conduct of the study. Dr Mena reported serving as a consultant for and receiving personal fees from Medtronic, the Cook Group, Cardinal Health, and Boston Scientific outside the submitted work. Dr Masoudi reported serving as chief scientific advisor for the National Cardiovascular Data Registry of the American College of Cardiology outside the submitted work. Dr Spertus reported owning equity in Health Outcomes Sciences outside the submitted work. Dr Mortazavi reported a patent to US20180315507A1 pending; while not related to acute kidney injury, the patent did look at infection and respiratory failure after percutaneous coronary intervention and therefore could be considered broadly relevant to the work in the article. Dr Krumholz reported having a contract with the Centers for Medicare & Medicaid Services to support quality measurement programs; receiving grants from Medtronic, the US Food and Drug Administration, and Johnson and Johnson; receiving a research agreement from the Shenzhen Center for Health Information; collaborating with the National Center for Cardiovascular Diseases in Beijing; receiving payment from the Arnold and Porter law firm, the Ben C. Martin law firm, and the Siegfried and Jensen law firm; chairing a cardiac scientific advisory board for UnitedHealth; participating in the IBM Watson Health Life Sciences Board; being a member of the advisory boards for Element Science, Facebook, and Aetna; having founded HugoHealth, a personal health information platform; and having cofounded Refactor Health, an enterprise healthcare artificial intelligence–augmented data management company. No other disclosures were reported.

    Disclaimer: The research data were provided by the American College of Cardiology National Cardiovascular Data Registry. The views expressed in this article represent those of the authors and do not necessarily represent the official views of the National Cardiovascular Data Registry or its associated professional societies, identified at cvquality.acc.org/NCDR-Home.

    References
    1.
    Tsai  TT, Patel  UD, Chang  TI,  et al.  Validated contemporary risk model of acute kidney injury in patients undergoing percutaneous coronary interventions: insights from the National Cardiovascular Data Registry Cath-PCI Registry.  J Am Heart Assoc. 2014;3(6):e001380. doi:10.1161/JAHA.114.001380PubMedGoogle Scholar
    2.
    Amin  AP, Bach  RG, Caruso  ML, Kennedy  KF, Spertus  JA.  Association of variation in contrast volume with acute kidney injury in patients undergoing percutaneous coronary intervention.  JAMA Cardiol. 2017;2(9):1007-1012. doi:10.1001/jamacardio.2017.2156PubMedGoogle ScholarCrossref
    3.
    Cigarroa  RG, Lange  RA, Williams  RH, Hillis  LD.  Dosing of contrast material to prevent contrast nephropathy in patients with renal disease.  Am J Med. 1989;86(6 Pt 1):649-652. doi:10.1016/0002-9343(89)90437-3PubMedGoogle ScholarCrossref
    4.
    Freeman  RV, O’Donnell  M, Share  D,  et al; Blue Cross-Blue Shield of Michigan Cardiovascular Consortium (BMC2).  Nephropathy requiring dialysis after percutaneous coronary intervention and the critical role of an adjusted contrast dose.  Am J Cardiol. 2002;90(10):1068-1073. doi:10.1016/S0002-9149(02)02771-6PubMedGoogle ScholarCrossref
    5.
    Brown  JR, Robb  JF, Block  CA,  et al.  Does safe dosing of iodinated contrast prevent contrast-induced acute kidney injury?  Circ Cardiovasc Interv. 2010;3(4):346-350. doi:10.1161/CIRCINTERVENTIONS.109.910638PubMedGoogle ScholarCrossref
    6.
    Tan  N, Liu  Y, Zhou  YL,  et al.  Contrast medium volume to creatinine clearance ratio: a predictor of contrast-induced nephropathy in the first 72 hours following percutaneous coronary intervention.  Catheter Cardiovasc Interv. 2012;79(1):70-75. doi:10.1002/ccd.23048PubMedGoogle ScholarCrossref
    7.
    Brown  JR, DeVries  JT, Piper  WD,  et al; Northern New England Cardiovascular Disease Study Group.  Serious renal dysfunction after percutaneous coronary interventions can be predicted.  Am Heart J. 2008;155(2):260-266. doi:10.1016/j.ahj.2007.10.007PubMedGoogle ScholarCrossref
    8.
    Huang  C, Murugiah  K, Mahajan  S,  et al.  Enhancing the prediction of acute kidney injury risk after percutaneous coronary intervention using machine learning techniques: a retrospective cohort study.  PLoS Med. 2018;15(11):e1002703. doi:10.1371/journal.pmed.1002703PubMedGoogle Scholar
    9.
    Weintraub  WS, McKay  CR, Riner  RN,  et al; American College of Cardiology Database Committee.  The American College of Cardiology National Database: progress and challenges.  J Am Coll Cardiol. 1997;29(2):459-465.PubMedGoogle ScholarCrossref
    10.
    Brindis  RG, Fitzgerald  S, Anderson  HV, Shaw  RE, Weintraub  WS, Williams  JF.  The American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR): building a national clinical data repository.  J Am Coll Cardiol. 2001;37(8):2240-2245. doi:10.1016/S0735-1097(01)01372-9PubMedGoogle ScholarCrossref
    11.
    Hobson  C, Ruchi  R, Bihorac  A.  Perioperative acute kidney injury: risk factors and predictive strategies.  Crit Care Clin. 2017;33(2):379-396. doi:10.1016/j.ccc.2016.12.008PubMedGoogle ScholarCrossref
    12.
    Yuan  M, Lin  Y.  Model selection and estimation in regression with grouped variables.  J R Stat Soc Series B Stat Methodol. 2006;68(1):49-67. doi:10.1111/j.1467-9868.2005.00532.xGoogle ScholarCrossref
    13.
    Machado  MN, Nakazone  MA, Maia  LN.  Acute kidney injury based on KDIGO (Kidney Disease Improving Global Outcomes) criteria in patients with elevated baseline serum creatinine undergoing cardiac surgery.  Rev Bras Cir Cardiovasc. 2014;29(3):299-307. doi:10.5935/1678-9741.20140049PubMedGoogle Scholar
    14.
    Gottlieb  SS, Abraham  W, Butler  J,  et al.  The prognostic importance of different definitions of worsening renal function in congestive heart failure.  J Card Fail. 2002;8(3):136-141. doi:10.1054/jcaf.2002.125289PubMedGoogle ScholarCrossref
    15.
    Smith  GL, Vaccarino  V, Kosiborod  M,  et al.  Worsening renal function: what is a clinically meaningful change in creatinine during hospitalization with heart failure?  J Card Fail. 2003;9(1):13-25. doi:10.1054/jcaf.2003.3PubMedGoogle ScholarCrossref
    16.
    Tsai  TT, Patel  UD, Chang  TI,  et al.  Contemporary incidence, predictors, and outcomes of acute kidney injury in patients undergoing percutaneous coronary interventions: insights from the NCDR Cath-PCI registry.  JACC Cardiovasc Interv. 2014;7(1):1-9. doi:10.1016/j.jcin.2013.06.016PubMedGoogle ScholarCrossref
    17.
    Hastie  T, Tibshirani  R.  Generalized additive models.  Stat Sci. 1986;1(3):297-310. doi:10.1214/ss/1177013604Google ScholarCrossref
    18.
    R Core Team. The R project for statistical computing. http://www.R-project.org/. Accessed March 26, 2018.
    19.
    Wood  SN.  Generalized Additive Models: An Introduction with R. 2nd ed. Boca Raton, FL: CRC Press; 2017. doi:10.1201/9781315370279
    20.
    Tan  N, Liu  Y, Chen  JY,  et al.  Use of the contrast volume or grams of iodine-to-creatinine clearance ratio to predict mortality after percutaneous coronary intervention.  Am Heart J. 2013;165(4):600-608. doi:10.1016/j.ahj.2012.12.017PubMedGoogle ScholarCrossref
    21.
    Liu  Y, Liu  YH, Chen  JY,  et al.  Safe contrast volumes for preventing contrast-induced nephropathy in elderly patients with relatively normal renal function during percutaneous coronary intervention.  Medicine (Baltimore). 2015;94(12):e615. doi:10.1097/MD.0000000000000615PubMedGoogle Scholar
    22.
    Laskey  WK, Jenkins  C, Selzer  F,  et al; NHLBI Dynamic Registry Investigators.  Volume-to-creatinine clearance ratio: a pharmacokinetically based risk factor for prediction of early creatinine increase after percutaneous coronary intervention.  J Am Coll Cardiol. 2007;50(7):584-590. doi:10.1016/j.jacc.2007.03.058PubMedGoogle ScholarCrossref
    23.
    Mehta  RL, Kellum  JA, Shah  SV,  et al; Acute Kidney Injury Network.  Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury.  Crit Care. 2007;11(2):R31-R31. doi:10.1186/cc5713PubMedGoogle ScholarCrossref
    24.
    Kellum  JA, Bellomo  R, Ronco  C, Mehta  R, Clark  W, Levin  NW.  The 3rd International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI).  Int J Artif Organs. 2005;28(5):441-444. doi:10.1177/039139880502800503PubMedGoogle ScholarCrossref
    25.
    Gruberg  L, Mintz  GS, Mehran  R,  et al.  The prognostic implications of further renal function deterioration within 48 h of interventional coronary procedures in patients with pre-existent chronic renal insufficiency.  J Am Coll Cardiol. 2000;36(5):1542-1548. doi:10.1016/S0735-1097(00)00917-7PubMedGoogle ScholarCrossref
    26.
    Marenzi  G, Lauri  G, Assanelli  E,  et al.  Contrast-induced nephropathy in patients undergoing primary angioplasty for acute myocardial infarction.  J Am Coll Cardiol. 2004;44(9):1780-1785. doi:10.1016/j.jacc.2004.07.043PubMedGoogle ScholarCrossref
    27.
    Bartholomew  BA, Harjai  KJ, Dukkipati  S,  et al.  Impact of nephropathy after percutaneous coronary intervention and a method for risk stratification.  Am J Cardiol. 2004;93(12):1515-1519. doi:10.1016/j.amjcard.2004.03.008PubMedGoogle ScholarCrossref
    28.
    Slocum  NK, Grossman  PM, Moscucci  M,  et al.  The changing definition of contrast-induced nephropathy and its clinical implications: insights from the Blue Cross Blue Shield of Michigan Cardiovascular Consortium (BMC2).  Am Heart J. 2012;163(5):829-834. doi:10.1016/j.ahj.2012.02.011PubMedGoogle ScholarCrossref
    29.
    Duncan  IG.  Healthcare Risk Adjustment and Predictive Modeling. New Hartford, CT: ACTEX Learning; 2011.
    30.
    Janssen  KJ, Moons  KG, Kalkman  CJ, Grobbee  DE, Vergouwe  Y.  Updating methods improved the performance of a clinical prediction model in new patients.  J Clin Epidemiol. 2008;61(1):76-86. doi:10.1016/j.jclinepi.2007.04.018PubMedGoogle ScholarCrossref
    ×