ED indicates emergency department; UTI, urinary tract infection.
eAppendix. Additional Details Regarding Development of the Predictive Models
eTable 1. Accuracy of Symptoms in the Diagnosis of Urinary Tract Infection (UTI) From the Present Study Compared to Previous Studies
eTable 2. Overview of the Variables Included (and Their Odds Ratios) in Each of the Five Multivariable Models Developed in the Training Database
eTable 3. Multilevel Likelihood Ratios of Combination of Findings in the Training Database
eFigure. Predicted Pre-Test Probability of UTI for Individual Children According to Their Clinical Characteristics in the Training Dataset (Clinical Model)
Customize your JAMA Network experience by selecting one or more topics from the list below.
Shaikh N, Hoberman A, Hum SW, et al. Development and Validation of a Calculator for Estimating the Probability of Urinary Tract Infection in Young Febrile Children. JAMA Pediatr. Published online April 16, 2018. doi:10.1001/jamapediatrics.2018.0217
In febrile children younger than 2 years, what combination of clinical and laboratory findings best predicts the risk of urinary tract infection?
This nested case-control study of 2070 children aged 2 to 23 months with a documented temperature of 38°C or higher tested the accuracy of UTICalc, an algorithm that uses clinical and laboratory findings to estimate the probability of urinary tract infection. Compared with the American Academy of Pediatrics algorithm, UTICalc reduced testing by 8.1% and decreased the number of urinary tract infections that were missed.
The UTICalc calculator can be used to guide testing and treatment in children with suspected urinary tract infection.
Accurately estimating the probability of urinary tract infection (UTI) in febrile preverbal children is necessary to appropriately target testing and treatment.
To develop and test a calculator (UTICalc) that can first estimate the probability of UTI based on clinical variables and then update that probability based on laboratory results.
Design, Setting, and Participants
Review of electronic medical records of febrile children aged 2 to 23 months who were brought to the emergency department of Children’s Hospital of Pittsburgh, Pittsburgh, Pennsylvania. An independent training database comprising 1686 patients brought to the emergency department between January 1, 2007, and April 30, 2013, and a validation database of 384 patients were created. Five multivariable logistic regression models for predicting risk of UTI were trained and tested. The clinical model included only clinical variables; the remaining models incorporated laboratory results. Data analysis was performed between June 18, 2013, and January 12, 2018.
Documented temperature of 38°C or higher in children aged 2 months to less than 2 years.
Main Outcomes and Measures
With the use of culture-confirmed UTI as the main outcome, cutoffs for high and low UTI risk were identified for each model. The resultant models were incorporated into a calculation tool, UTICalc, which was used to evaluate medical records.
A total of 2070 children were included in the study. The training database comprised 1686 children, of whom 1216 (72.1%) were female and 1167 (69.2%) white. The validation database comprised 384 children, of whom 291 (75.8%) were female and 200 (52.1%) white. Compared with the American Academy of Pediatrics algorithm, the clinical model in UTICalc reduced testing by 8.1% (95% CI, 4.2%-12.0%) and decreased the number of UTIs that were missed from 3 cases to none. Compared with empirically treating all children with a leukocyte esterase test result of 1+ or higher, the dipstick model in UTICalc would have reduced the number of treatment delays by 10.6% (95% CI, 0.9%-20.4%).
Conclusions and Relevance
UTICalc estimates the probability of UTI by evaluating the risk factors present in the individual child. As a result, testing and treatment can be tailored, thereby improving outcomes for children with UTI.
Approximately 7% of children younger than 2 years who present to an emergency department with fever have a urinary tract infection (UTI).1,2 However, testing for UTI in young children, whether by catheterization or via the 2-step process (testing bag specimens first and limiting catheterization to children with positive results),3,4 is challenging. Accordingly, clinicians obtain samples only when they judge the probability of UTI to be sufficiently high. Estimating the probability of UTI by using each child’s unique set of presenting signs and symptoms can be challenging given the relatively large number of variables that modify the risk.5 Although algorithms have been developed to assist clinicians in identifying children who may benefit from further diagnostic testing,5-9 available evidence suggests that clinicians generally do not adhere to them.9-12
If screening tests (urine dipstick or urinalysis) are ordered, the clinician must reestimate the probability of UTI based on the results obtained and decide whether empirical antimicrobial treatment is warranted before urine culture results are available. Interpreting screening test results is often not straightforward (eg, for a child with trace amounts of leukocyte esterase).
To assist clinicians in identifying children likely to benefit from testing and empirical treatment with antimicrobial drugs, we designed UTICalc, a calculator that first estimates the probability of UTI based on clinical variables (pretest probability) and then, if laboratory testing is performed, updates the probability estimate based on the results (posttest probability). We present data on the development and validation of this tool.
We studied a consecutive population of febrile children younger than 2 years who were evaluated for UTI at the emergency department of Children’s Hospital of Pittsburgh between January 1, 2007, and April 30, 2013, in whom a urine specimen was obtained by bladder catheterization. We retrospectively reviewed medical records of all children with culture-confirmed UTI (n = 570) and of randomly selected children without UTI (n = 1312), conducting a nested case-control study with a case to control ratio of approximately 1:2 (Figure). We chose this design because the prevalence of UTI in young children is relatively low and because estimates of diagnostic accuracy obtained from nested case-control studies closely approximate the values in the general population.13 The University of Pittsburgh Institutional Review Board approved this study and waived informed consent. Data analysis was performed between June 18, 2013, and January 12, 2018.
In this database, hereinafter referred to as the training database, we developed 5 multivariable logistic regression models (details of variable selection are available in the eAppendix in the Supplement) to estimate the risk of UTI. The clinical model included 5 dichotomous clinical risk factors (aged <12 months, temperature ≥39°C, nonblack race, female or uncircumcised male, and no other fever source). The remaining 4 models included laboratory tests and are referred to as laboratory models. The “dipstick model” included variables from the clinical model plus leukocyte esterase and nitrite values. The “dipstick + Gram stain model” included variables from the clinical and dipstick models plus the results of a Gram-stained urine smear. The “hemocytometer model” included variables from our clinical and dipstick models plus urine white blood cell (WBC) count (WBC/μL). The “enhanced urinalysis model” included variables from the clinical and hemocytometer models plus Gram stain results. We also developed a urinalysis model that included variables from the clinical and dipstick models plus bacteria per high-power field (HPF) (leukocytes per HPF did not add significantly to the model) in urinalysis results in 248 children; however, because the area under the curve (AUC) of this model was the same as the AUC of the dipstick model, this model was not incorporated into UTICalc. Because of the nested case-control study design, we corrected the constant term (β0) in the final logistic regression models using the prevalence of UTI (6.1%) in our source population.14
Pyuria (defined as WBC count of ≥5/HPF or ≥10/μL, or the presence of any leukocyte esterase) and growth of a uropathogen at a concentration of at least 50 000 colony-forming units per milliliter15 were both required for the diagnosis of UTI.7
For each model, we calculated the AUC and accuracy at various cutoffs to assign children into high-risk and low-risk categories. To determine cutoffs for each model, we reasoned that most clinicians would require a minimum sensitivity of 95%. Using this criterion, we arrived at a probability cutoff of 2% for the clinical model and 5% for the laboratory models. The 2% cutoff corresponds to the point at or above which children were determined to have a high pretest probability of UTI, thus requiring urine testing. The 5% cutoff corresponds to the point at or above which children were determined to have a relatively high posttest probability of UTI, thus requiring antimicrobial therapy. These cutoffs seemed consistent with our clinical judgment and with published studies.5,7,12
We tested the accuracy of each model at the identified cutoff in an independent database. To create the database, we reviewed medical records of children aged 2 months to less than 2 years who presented to the emergency department at Children’s Hospital of Pittsburgh between July 7, 2015, and December 30, 2016, with a documented temperature of 38°C or higher (Figure). A research assistant periodically (subject to research assistant availability) reviewed medical records of children evaluated in the past 72 hours.
We assessed outcomes of using the clinical model in UTICalc in a hypothetical cohort of 1000 children being evaluated for a UTI. As a comparison, we evaluated outcomes of using the algorithm proposed in the UTI guideline of the American Academy of Pediatrics.7 Similarly, we compared outcomes of using the laboratory models in UTICalc with clinical practice. For the latter, we used the cutoff most commonly used in practice to dichotomize test results. Statistical analyses were performed using SAS, version 9.4 (SAS Institute Inc) and Stata, version 14 (StataCorp).
Of the 1686 children aged 2 to 23 months in the training database, 1229 (72.9%) were aged 2 to 11 months, 1216 (72.1%) were female, and 1167 (69.2%) were white. The validation database comprised 384 children aged 2 to 23 months, of whom 231 (60.2%) were aged 2 to 11 months, 291 (75.8%) were female, and 200 (52.1%) were white. Table 1 describes the clinical characteristics of children included in the training and validation databases. The prevalence of UTI was higher in the training database (542 of 1686 [32.1%] vs 30 of 384 [7.8%]), reflecting our 1:2 case-control sampling strategy. The training database also differed from the validation databases with regard to age, circumcision status, sex, race, and source of fever. These differences were not surprising because the training database was enriched with children with UTI (who are known to differ from children without UTI in the above characteristics).
eTable 1 in the Supplement shows the results of the univariate analysis, and eTable 2 in the Supplement shows the variables included in each of the 5 final multivariate models. The final laboratory models include clinical variables because dropping these variables resulted in lower accuracy (eg, AUC of the dipstick model was 97% with and 96% without clinical variables in the training database). The calculator (UTICalc), which calculates the probability of UTI based on the models developed, can be found at https://uticalc.pitt.edu/.
Table 2 compares the accuracy of the models in the training and validation databases. In the training database, the clinical model had lower accuracy than the laboratory models, reflecting the nonspecific signs and symptoms of UTI in preverbal children (clinical model AUC, 0.80 [95% CI, 0.77-0.82] vs 0.97 [95% CI, 0.96-0.98] to 0.98 [95% CI, 0.98-0.99] for the laboratory models). In general, models that included a Gram-stained smear (dipstick + Gram stain model and the enhanced urinalysis model) performed better than models that did not include this test. The difference in accuracy of the models in the validation and the training database was slight, suggesting that overfitting is not a concern.
Table 3 summarizes the clinical implications of using UTICalc in a population of 1000 children younger than 2 years presenting with fever. Compared with the algorithm endorsed by the American Academy of Pediatrics, using the clinical model in UTICalc would have reduced the need for urine sampling by 8.1% (95% CI, 4.2%-12.0%), at the same time decreasing the number of cases of UTI that were missed from 3 cases to none. Compared with empirically treating all children who had a leukocyte esterase result of 1+ or higher, the dipstick component of UTICalc would have reduced the number of children whose treatment was delayed by 10.6% (95% CI, 0.9%-20.4%) without substantially higher rates of antimicrobial use.
Compared with the dipstick test alone, use of a Gram-stained smear or a hemocytometer reduced the number of children with delayed treatment from 10 children to 5 with the use of a dipstick test and Gram-stained smear or to 3 with the use of a hemocytometer. A similar trend was observed in the UTICalc models (Table 3).
The eFigure in the Supplement summarizes the pretest probability of UTI for children with all possible combinations of risk factors, enabling clinicians to better understand the contribution of these risk factors when estimating the probability of UTI; eTable 3 in the Supplement presents multilevel likelihood ratios of these combinations. Table 4 shows how the posttest probability of UTI varies according to the results of the leukocyte esterase and nitrite tests and according to the pretest probability of UTI.
We have developed a UTI calculator that estimates the pretest and posttest probability of UTI at the bedside according to clinical and laboratory characteristics of the child being assessed. Our study is unique in that the database we used to develop the models included approximately 10 times as many children with UTI as previous similar studies.6 Moreover, external validation of the models in an independent database seems to confirm their validity. Use of UTICalc could reduce unnecessary testing and delays in the treatment of children with UTI.
UTICalc can be used by clinicians evaluating children aged 2 to less than 24 months with fever and a suspected UTI to decide whether urine sampling is warranted. After evaluating the child, the clinician inputs data on 5 variables: age, race, sex/circumcision status, maximum temperature, and absence of another source for fever. The calculator outputs the probability of UTI for a child with those characteristics and assigns a risk category (low or high).
Use of the clinical model in UTICalc would detect 95% to 100% of UTIs in febrile children younger than 2 years. The algorithm proposed by the American Academy of Pediatrics would also detect most UTIs at its lower cutoff, but following that algorithm would lead to approximately 81 more children undergoing urine sampling per 1000 children evaluated. Although the 2% cutoff may seem low to some clinicians,12 higher cutoffs would have resulted in substantially lower sensitivity. A cutoff of 2% does not mean that 50 urine samples must be collected to detect 1 UTI, which would be unreasonable. As seen in Table 3, using a cutoff of 2% would subject approximately 10 children to urine sample collection for every UTI detected (ie, number needed to test, 9.8).
If testing is performed, the clinician enters the results into UTICalc. UTICalc is helpful because interpreting screening test results is often not straightforward. The clinician often must interpret the significance of borderline (eg, trace or 1+ leukocyte esterase test result with no nitrites) or continuous (eg, WBC count of 15/μL) results while considering the pretest probability of UTI in the individual child. UTICalc automatically selects the correct model and estimates the posttest probability of UTI based on clinical information and the type of test results entered. As in the first step, in addition to providing the probability of UTI, the calculator will assign a risk category. Our results suggest that clinicians could minimize the number of UTIs for which treatment is delayed by ordering a Gram-stained smear of the urine.
Although duration of fever and history of UTI were significantly associated with UTI on univariate analysis, the clinical significance of dropping these variables from the baseline clinical model was minor; the accuracy of models with and without these variables in the validation database was similar (eAppendix in the Supplement).
As seen in the eFigure in the Supplement, the risk of UTI in circumcised male children was less than 2% with 1 exception (nonblack infants younger than 12 months with a temperature of ≥39°C and no other source of fever); in female or uncircumcised male patients, the highest risk group was nonblack infants younger than 12 months with no other source of fever. These findings are consistent with previous reports.5,6,16,17
To simplify interpretation of dipstick results, investigators have proposed various methods of classifying the result as positive or negative (eg, ≥1+ leukocyte esterase as positive). As illustrated in Table 4, the probability of UTI varies substantially in children according to leukocyte esterase level, suggesting that dichotomizing the results of this test may be simplistic. Furthermore, it appears that considering the pretest probability of UTI influences interpretation of the results of the dipstick test. This influence is also apparent in Table 3; compared with using dichotomized versions of the tests, use of UTICalc, which incorporates the full (nondichotomized) information provided by laboratory tests and information about the child’s clinical findings, often substantially reduced the number of children in whom treatment of UTI was delayed, the number of children who received unnecessary prescriptions for antimicrobial agents, or both.
The high sensitivity of the clinical model ensures that very few children with UTI are left untested and minimizes unnecessary testing; the relatively high specificity of the models that include laboratory tests ensures that antimicrobial treatment is directed to the children most likely to have a UTI. Nevertheless, more accurate point-of-care tests are needed to improve the care of children with UTI. Currently available screening tests for UTI have relatively low sensitivities, and their high negative predictive values to a large extent reflect the low prevalence of UTI in young preverbal children. In a hypothetical population of 1000 febrile children screened for UTI by using the dipstick test, 3 of 70 children (4%) with UTI would receive delayed treatment and 70 of 101 children (69%) treated for UTI would have a true UTI (Table 3). Accordingly, until more sensitive and specific tests are developed, it is prudent to obtain urine cultures from young febrile children being evaluated for UTI.
We envision that UTICalc, should it prove cost-effective, could eventually be incorporated as a decision-support tool in electronic health records and could, to a large extent, be prepopulated before the clinician assesses the child. Of the 5 variables in the clinical model, all except absence of other source of fever can easily be obtained by a triage nurse (in person or by phone). Because many children have upper respiratory tract symptoms, this variable will also often be known.
Our study has several limitations. The database used to train the models was a retrospective sample of children tested for UTI. Accordingly, UTICalc is likely to perform best when applied to a population of children with a reasonable pretest probability of UTI; clinicians should refrain from using it when UTI is not suspected on clinical grounds. A second limitation relates to the practice pattern at the Children’s Hospital of Pittsburgh; in lieu of a dipstick at bedside, clinicians often order a urinalysis. We assumed that the diagnostic accuracy of the leukocyte esterase and nitrite tests would be equivalent whether performed at the bedside and interpreted visually or performed as part of a urinalysis in the hospital laboratory and interrelated using a colorimeter. The sensitivity and specificity estimates we obtained for these tests were similar to previously reported values (eTable 1 in the Supplement).18 Another limitation is that our training and validation samples were from a single institution. Finally, a true test of the utility of UTICalc would be a quasi-experimental or randomized study in primary care clinics and emergency departments.
Accurate diagnosis of UTI is important to reduce the delay in diagnosis and to avoid unnecessary treatment with antimicrobial drugs.19-21 The approach advocated here tailors testing and treatment to the risk factors present in the child being assessed, thus offering the potential to improve outcomes for children with UTI.
Accepted for Publication: January 22, 2018.
Corresponding Author: Nader Shaikh, MD, MPH, Division of General Academic Pediatrics, Children’s Hospital of Pittsburgh of UPMC, One Children’s Hospital Drive, 4401 Penn Ave, Pittsburgh, PA 15224 (firstname.lastname@example.org).
Published Online: April 16, 2018. doi:10.1001/jamapediatrics.2018.0217
Author Contributions: Dr Shaikh had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Shaikh, Shope.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Shaikh, Alberty, Shope.
Critical revision of the manuscript for important intellectual content: Shaikh, Hoberman, Hum, Muniz, Kurs-Lasky, Landsittel, Shope.
Statistical analysis: Shaikh, Muniz, Kurs-Lasky, Landsittel, Shope.
Administrative, technical, or material support: Shaikh, Hum.
Study supervision: Shaikh.
Conflict of Interest Disclosures: None reported.