Validation and Comparison of a Novel Screening Guideline for Kidney Disease: KEEPing SCORED

Chronic kidney disease (CKD) is one of the world's major public health problems. Nearly 1 in 9 adults (20 million people) in the United States have CKD, and it is estimated that another 20 million are at increased risk.¹ Given the asymptomatic nature of kidney disease, affected individuals and health care providers may be unaware of the condition in patients. Identifying individuals with early kidney disease would be a useful first step in preventing progression to end-stage renal disease as well as reducing morbidity and mortality from cardiovascular disease (CVD).

We recently published an instrument (SCreening for Occult REnal Disease [SCORED]) to systematically identify individuals with a high likelihood of prevalent CKD.² Derived from the National Health and Nutrition Examination Survey (NHANES) 1999-2002 and (partially) validated in the Atherosclerosis Risk in Communities (ARIC) study, SCORED identified 9 demographic and medical variables that could be assigned integer values and then entered into a scoring algorithm. The scoring algorithm was intentionally simplified to be accessible to lay persons and health care providers.

To our knowledge, SCORED is the only algorithm derived from scientific modeling, rather than expert scientific opinion, targeted for general population screening. Alternatively, the National Kidney Foundation has encouraged screening strategies targeted at high-risk groups. Through the Kidney Early Evaluation Program (KEEP), the National Kidney Foundation recommends using the following characteristics to identify individuals with a high likelihood of kidney disease: “if a person is 18 years or older and has one or more of the following: diabetes; high blood pressure; or a family history of diabetes, high blood pressure or kidney disease.”^3,4

It is unclear how the SCORED algorithm compares with the KEEP guidelines. Such comparisons would be useful for clinicians and others. Since the publication of SCORED, we have performed an additional validation study using new, independent samples: (1) NHANES 2003-2004 and (2) a combined cohort of the ARIC study and Cardiovascular Heart Study (ARIC/CHS). We report the findings herein.

The ARIC study enrolled 15 792 participants aged 45 to 64 years between 1987 and 1989, and CHS recruited 5201 subjects 65 years and older between 1989 and 1990. Both are community studies, and detailed descriptions have been published previously.^2,5,6 Some data disparities and limitations to be noted are summarized in a footnote of the Table.

The SCORED model is a multivariable mathematical function that gives an estimated probability of having CKD as follows:

Probability(CKD) = 1/[1 + exp(−β" × x)],

where

β" × x = −5.4 + 1.55 × I(Age 50-59 years) + 2.31 × I(Age 60-69 years) + 3.23 × I(Age ≥70 years) + 0.29 × I(Female) + 0.93 × I(Anemia) + 0.45 × I(Hypertension) + 0.44 × I(Diabetes) + 0.59 × I(History of CVD) + 0.45 × I(History of Heart Failure) + 0.74 × I(Peripheral Vascular Disease) + 0.83 × I(Proteinuria).

The total integer score is a weighted sum, ranging from 0 to 12, and is calculated as follows:

Total score = 2 × I(Age 50-59 years) + 3 × I(Age 60-69 years) + 4 × I(Age ≥70 years) + 1 × I(Female) + 1 × I(Anemia) + 1 × I(Hypertension) +1 × I(Diabetes) + 1 × I(History of CVD) + 1 × I(History of Heart Failure) + 1 × I(Peripheral Vascular Disease) + 1 × I(Proteinuria),

where I(A) is an indicator taking 1 for condition A and 0 for otherwise. If the total score is greater than or equal to 4, a confirmatory blood examination (eg, serum creatinine and/or urinalysis) by a physician is strongly recommended. A user-friendly questionnaire is also provided in the original article.²

Creatinine level was calibrated in all studies as recommended by the National Kidney Disease Education Program and the National Center for Health Statistics.^7-9 Kidney function was quantified by glomerular filtration rate from the 4-variable Modification of Diet in Renal Disease study equation, and CKD was defined as a glomerular filtration rate lower than 60 mL/min/1.73m², which corresponds to stage 3 or higher kidney disease.^3,10

The SCORED model was evaluated in the validation datasets using standard measures: percentage of people identified to be at high risk by the given rule, sensitivity, specificity, positive predictive value, negative predictive value, and area under the receiver operating characteristic curve (AUC).

Because the ARIC study and CHS did not measure urinary protein level for proteinuria, we created 3 different approaches to handling missing or incomplete information and assessed sensitivity of the estimates of the selected measures. First, we started the analysis with ARIC/CHS datasets without proteinuria, which reflects the situation that no one reported that she or he had proteinuria. Second, we substituted urinary protein level measured at the year 5 follow-up visit for CHS subjects if available or imputed the measurement via a missing data analysis method otherwise.¹¹ Third, we defined the highest 10 percentiles of uric acid level as proteinuria (to compare, 9.5% of subjects had proteinuria in NHANES). (The rationale for using high uric acid level as a surrogate marker for proteinuria, at least for sensitivity checking, is that it is used in the diagnosis and treatment of renal disorders, including renal failure or injury and is associated with proteinuria.¹²)

We compared SCORED vs KEEP using the same set of diagnostic measures. It is important to note that none of the datasets we examined contained information on “family history of kidney disease,” a criterion needed for KEEP. As a result, the SCORED model could not examine the effect of this potentially important factor. The absence of this data was also a problem for the direct comparison between KEEP vs SCORED. To improve comparability, we repeated our analysis in 2 scenarios: (1) ignoring this criterion and (2) after applying the fill-in method based on missing data imputation and replacement by surrogate variables. These statistical strategies, although imperfect, would offer a sound method of sensitivity analysis in practice.

For the NHANES dataset, sensitivity, specificity, positive predictive value, and negative predictive value of SCORED were estimated to be 95%, 65%, 20%, and 99%, respectively, with an AUC of 0.88. The application to ARIC/CHS provided sensitivity of 88% to 90%, specificity of 50% to 52%, positive predictive value of 13% to 14%, and negative predictive value of 98%, with an AUC of 0.78 to 0.80 (Table). In contrast, KEEP tended to place larger proportions of subjects into the high-risk category than SCORED (67%-77% vs 40%-53%), and the sensitivity for detecting CKD was 86% to 92%, with a specificity of 24% to 35%, positive predictive value of 3% to 12%, negative predictive value of 95% to 98%, and an AUC of 0.65 to 0.77. Results were robust under the different assumptions and approaches we adopted. Our comparisons give some indication about the expected performance of the 2 screening programs in real-world settings.

We report the diagnostic characteristics of the SCORED model and its performance compared with the KEEP guidelines. Ninety-five percent of individuals with CKD were identified by the SCORED algorithm as being at high risk (cutoff of ≥4); among those who did not have CKD, 65% scored less than 4 in NHANES. Those values were somewhat reduced in the ARIC/CHS samples, possibly owing to differences in population characteristics, variable definitions, and available data.

The SCORED model exhibited improved performances in all the criteria we examined. For KEEP as well as SCORED, NHANES showed better results than ARIC/CHS. Based on the higher AUC of the SCORED model compared with the KEEP guidelines, the improved performance characteristics are likely because of the use of additional variables and different weights for age groups. Indeed, questions about underlying CVD, common in the CKD population as demonstrated by the KEEP data,¹³ may improve their screening efforts.

There are also some limitations. Some variables that are a part of SCORED or KEEP were unavailable or incomplete in the databases we used, including family history of kidney disease and proteinuria. We addressed this issue with different statistical strategies. The absence of these variables in well-known studies may reflect low awareness of these important risk factors. We encourage the collection of such information in future epidemiological studies including the next iteration of NHANES. Furthermore, there are known negative consequences of screening. Overdiagnosis, misdiagnosis, anxiety, cost, and creating a false sense of security are some adverse effects of screening and render screening in some diseases controversial.^14-18 We propose that SCORED would minimize the potential adverse outcomes associated with screening by identifying approximately 25% fewer screeners as being at high risk compared with KEEP. The greater specificity of SCORED may reduce overdiagnosis and enhance resource use.

Screening is a public health strategy for identifying an unrecognized disease in asymptomatic populations, in which subjects are asked questions or offered a test, to identify those individuals who are more likely to be helped than harmed by further tests or treatment to reduce the risk of a disease or its complications. Diseases suitable for screening are those with serious consequences, those in which treatment is more effective at an earlier stage, and conditions with a long preclinical phase.¹⁶ Chronic kidney disease fulfills these criteria; however, it is not known whether screening will in fact result in improved outcomes and reduced progression to end-stage renal disease.¹⁷ The benefits of screening for CKD and its effectiveness and cost-effectiveness are yet to be determined.¹⁹

In conclusion, SCORED and KEEP are simple and inexpensive to use and carry some, albeit, minor, consequences of a false-positive result. Because of its general population sampling, we believe that SCORED has greater applicability and could also serve as an educational tool to raise CKD awareness. There is ample evidence that CKD is underdiagnosed and undertreated and that the burden of CKD is increasing worldwide.²⁰ Not many people are aware that undiagnosed CKD can lead to serious consequences, including kidney failure, dialysis, and even death, and is associated with CVD. We propose that increased screening would identify greater numbers of individuals who may potentially benefit from strategies known to slow the progression of renal disease, a hypothesis that warrants testing in prospective clinical trials. We should “act on the best available evidence—as opposed to waiting for the best possible evidence.”^21(p111)

Correspondence: Dr Bang, Division of Biostatistics and Epidemiology, Department of Public Health, Weill Medical College of Cornell University, 411 E 69th St, New York, NY 10021 (heb2013@med.cornell.edu).

Author Contributions: Dr Bang had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Bang, Mazumdar, Shoham, and Kshirsagar. Acquisition of data: Bang. Analysis and interpretation of data: Bang, Mazumdar, Kern, Shoham, August, and Kshirsagar. Drafting of the manuscript: Bang, Shoham, August, and Kshirsagar. Critical revision of the manuscript for important intellectual content: Bang, Mazumdar, Kern, and Kshirsagar. Statistical analysis: Bang, Mazumdar, and Shoham. Obtained funding: Mazumdar. Administrative, technical, and material support: Kern and Kshirsagar. Study supervision: August and Kshirsagar.

Financial Disclosure: None reported.

Funding/Support: The ARIC and CHS studies are conducted and supported by the National Heart, Lung, and Blood Institute (NHLBI) in collaboration with ARIC and CHS investigators. Drs Bang and Mazumdar were partially supported by the Tolly Vinik Trust through the Institute of Clinical Research of Weill Medical College of Cornell University. Dr Mazumdar was additionally supported by grant R25CA105012 (Collaborative Program in Nutrition and Cancer Prevention). Dr Shoham is a postdoctoral trainee supported by NHLBI institutional training grant 5T-32HL007055-29.

Disclaimer: This manuscript was prepared using a limited-access dataset obtained from the NHLBI and does not necessarily reflect the opinions or views of the ARIC/CHS or NHLBI.

Additional Contributions: Kylie Braynt, MS, provided CHS data setup and Sean Coady, MA, at NHLBI assisted in data issues and clarifications. We thank the staff and participants of the NHANES, ARIC study, and CHS for their important contributions and valuable information for health research.