Author Affiliations: RAND Corporation, Arlington, Virginia (Drs Anhang Price and Kellermann and Ms Harris); Children's National Medical Center, Washington, DC (Dr Fagbuyi); and Inova Fairfax Hospital, Falls Church, Virginia (Drs Hanfling and Place). Dr Taylor is a medical informaticist consultant in Nashville, Tennessee.
Importance Self-triage using web-based decision support could be a useful way to encourage appropriate care-seeking behavior and reduce health system surge in epidemics. However, the feasibility and safety of this strategy have not previously been evaluated.
Objective To assess the usability and safety of Strategy for Off-site Rapid Triage (SORT) for Kids, a web-based decision support tool designed to translate clinical guidance developed by the Centers for Disease Control and Prevention to help parents and adult caregivers determine if a child with influenza-like illness requires immediate care in an emergency department (ED).
Design Prospective pilot validation study conducted between February 8 and April 30, 2012. Staff who abstracted medical records and made follow-up calls were blinded to the SORT algorithm's assessment of the child's level of risk.
Setting Two pediatric emergency departments in the National Capital Region.
Participants Convenience sample of 294 parents and adult caregivers who were at least 18 years of age; able to read and speak English; and the parent or legal guardian of a child 18 years or younger presenting to 1 of 2 EDs with signs and symptoms meeting Centers for Disease Control and Prevention criteria for influenza-like illness.
Intervention Completion of the SORT for Kids survey.
Main Outcome Measures Caregiver ratings of the website's usability and the sensitivity of the underlying algorithm for identifying children who required immediate ED management of influenza-like illness, defined as receipt of 1 or more of 5 essential clinical services.
Results Ninety percent of participants reported that the website was “very easy” to understand and use. Ratings did not differ by respondent race, ethnicity, or educational attainment. Of the 15 patients whose initial ED visit met explicit criteria for clinical necessity, the Centers for Disease Control and Prevention algorithm classified 14 as high risk, resulting in an overall sensitivity of 93.3% (exact 95% CI, 68.1%-99.8%). Specificity of the algorithm was poor.
Conclusions and Relevance This pilot study suggests that web-based decision support to help parents and adult caregivers self-triage children with influenza-like illness is feasible. However, prospective refinement of the clinical algorithm is needed to improve its specificity without compromising patient safety.
Emergency department (ED) crowding undermines the clinical quality, safety, and timeliness of care of both adults and children,1- 4 particularly when demand for ED care is high, such as during an influenza epidemic. Both the Institute of Medicine and Emergency Medical Services for Children program have highlighted the importance of researching best practices to reduce ED crowding.5,6
One promising strategy is to develop automated algorithms that allow patients to self-triage themselves to determine their need for ED care. This approach has been recommended as a potentially useful way to blunt health system surge.7 Although the number of web-based tools and smartphone applications offered to the public to support consumer decisions about health care is growing at a rapid rate,8 to our knowledge, no published studies have rigorously assessed how well they identify patients who need immediate care while safely reassuring those who do not.
In 2009, a group of volunteer experts from the fields of public health, emergency medicine, and infectious disease collaborated with the Institute of Medicine, the Centers for Disease Control and Prevention (CDC), and the US Department of Health and Human Services Office of Assistant Secretary for Preparedness and Response to develop the Strategy for Off-site Rapid Triage (SORT), a streamlined approach intended to triage large numbers of people affected by the 2009 novel influenza A(H1N1) pandemic.9,10 Ultimately, 2 age-specific versions of SORT were adopted by the CDC, one designed for symptomatic adults and the other for children. At the height of the pandemic, the adult algorithm was offered directly to the public through 2 interactive websites: flu.gov and a free, private-sector site.10 The creators of SORT had hoped to offer an interactive web-based version of the pediatric algorithm as well, but the CDC and the American Academy of Pediatrics (AAP), which had jointly devised the pediatric algorithm, declined to endorse the effort without prospective evidence of its safety.
To lay the groundwork for a prospective nationwide trial to validate a consumer website based on the pediatric algorithm, we conducted a pilot cohort study during the 2012 influenza season. Our goals were to elicit parental feedback regarding the SORT for Kids website's usability and obtain a preliminary assessment of its sensitivity and specificity relative to a clinical gold standard and subsequent telephone follow-up.
The website we devised for our pilot study closely adheres to the guidance contained in the CDC/AAP clinical algorithm produced during the 2009 novel influenza A(H1N1) pandemic to help health care professionals and call center personnel efficiently triage children with influenza-like illness (ILI) to various levels of care (Figure 1). The algorithm had 4 age-specific pathways, (1) infants (aged 0 to <12 weeks); (2) toddlers (aged 12 weeks to <2 years); (3) younger children (aged 2 years to <5 years); and (4) older children (aged 5 years to ≤18 years), reflecting differences in how influenza can present at various ages. Based on the child's age and usual health status, the algorithm assigns a child with ILI into 1 of 3 risk groups: (1) high-risk cases who should receive immediate care in an ED; (2) intermediate-risk cases who warrant expedited evaluation by the child's pediatrician or another primary care provider, and (3) low-risk children who should be able to safely recover at home, provided their condition does not worsen.
Figure 1. The 2009-2010 Centers for Disease Control and Prevention and American Academy of Pediatrics Influenza Season Triage Algorithm for Children (≤18 years) With Influenza-Like Illness.
To make the CDC guidance accessible and understandable to parents and adult caregivers of widely varying age, race, socioeconomic status, and health literacy,11 we translated the CDC/AAP pediatric algorithm into plain English and converted the various decision points into a highly interactive, web-based questionnaire. We dubbed the prototype website SORT for Kids. To increase the likelihood that individuals with limited health literacy could navigate the website, we worked with the same team of health literacy experts who previously collaborated with us to produce the adult SORT websites during the 2009 novel influenza A(H1N1) pandemic.10
The study was conducted between February 8 and April 30, 2012, at 2 EDs in the National Capital Region: Children's National Medical Center, a large, urban, academic tertiary care pediatric medical center, and Inova Fairfax Hospital, a suburban tertiary care medical center that treats children and adults.
At each of the participating hospitals, ED triage nurses recruited convenience samples of parents and adult caregivers bringing an ill child for evaluation of flulike symptoms. To be eligible to participate, a study subject had to be at least 18 years of age; able to read and speak English; and be the parent or legal guardian of a child 18 years or younger presenting to the ED with signs and symptoms meeting CDC criteria for ILI: temperature of 37.8°C or more and cough and/or sore throat.12
The study protocol was approved by the institutional review boards at both hospitals and the Human Subjects Protection Committee of RAND Corporation. Following informed consent, willing adult participants were asked to complete the SORT for Kids web-based survey using a dedicated study computer placed in each ED. The survey covered 3 content areas: (1) a series of questions crafted to closely follow the CDC/AAP algorithm for triaging children with ILI; (2) 2 questions regarding the usability of the website; and (3) user sociodemographics (ie, age, education, and race/ethnicity) (Figure 2). To ensure that using the website did not influence the parent or health care provider's subsequent decision making, the website gave no feedback regarding the child's risk status and made no recommendations.
Figure 2. Screenshot of Strategy for Off-site Rapid Triage for Kids web interface.
One or more days following each ED visit, hospital research staff queried the child's electronic health record to abstract data on any clinical services provided during the index visit. In keeping with recommended strategies for enhancing accuracy and consistency of medical record abstraction,13 precise definitions of each clinical service were established; explicit protocols were developed for identifying the criteria in each hospital's record; standardized electronic abstraction forms were used; hospital staff received training on a set of practice records prior to initiation of the study; and staff who abstracted records were blinded to the SORT algorithm's assessment of the child's level of risk.
Beginning 7 to 9 days after the visit, hospital research staff made multiple attempts to reach study participants by telephone to determine if a second ED visit for ILI had been made since study enrollment and, if applicable, when the second visit occurred. As with the medical record abstraction, staff members making follow-up calls were blinded to the SORT algorithm's assessment of the child's level of risk.
Primary study outcomes were (1) caregiver ratings of the survey's usability and (2) the sensitivity of the SORT for Kids tool for identifying children with ILI who require immediate ED evaluation. The website's usability was determined from participant responses to 2 questions: “How easy was it to understand the questions?” and “How easy was it to answer the questions?” Response options ranged on a 5-point scale from “very easy” to “very hard.”
The sensitivity of the SORT for Kids clinical algorithm for identifying needed ED care was based on an explicit gold standard: documented evidence that the child received 1 or more of the 5 ED-specific interventions. Prior studies have established that use of ED resources is an accurate indicator of illness severity in children and have defined categories of pediatric ED resource use that are indicative of routine nursing and medical care (ie, no ED visit needed), ED-level intervention (ie, diagnostic studies and therapeutic procedures performed and ED visit needed), and hospitalization.14,15 Clinical experts on the study team (D.F. and A.L.K.), aided by investigators affiliated with the Pediatric Emergency Care Applied Research Network,16 identified the following instances of resource use as confirming that ED-level intervention for ILI was necessary: (1) administration of supplemental oxygen to a patient with an oxygen saturation less than 93%; (2) administration of antibiotics during the ED visit or via discharge prescription to a patient with a radiology-confirmed diagnosis of pneumonia; (3) delivery of an intravenous fluid bolus of 20 mL/kg or pressors; (4) performance of a diagnostic lumbar puncture; and/or (5) hospital admission for influenza-related reasons, such as pneumonia or viral myocarditis. Receipt of 1 or more of these services was considered indicative of a clinically necessary ED visit.
Secondary outcomes included the sensitivity of the SORT for Kids clinical algorithm for identifying children in need of a second ED evaluation for ILI within the following 7 days, as reported during follow-up calls with parents or adult caregivers, and the algorithm's specificity for identifying children who did not require any of the 5 ED-specific interventions, including emergent hospitalization.
Sensitivity of the SORT for Kids tool was calculated with exact 95% confidence intervals. We used χ2 tests to examine the bivariate associations between participant race/ethnicity and education and survey usability responses. All analyses were conducted in SAS version 9.2 (SAS Institute Inc).
Two hundred ninety-four parents and adult caregivers completed the web survey. Average time to complete the survey was 4.2 minutes. Participants were diverse with regard to education, race, and ethnicity (Table 1). Although a wide age range of ill children was represented, only 3.1% were younger than 12 weeks.
Electronic health records were abstracted for 286 encounters (97.3%). Of these, 7.3% received antibiotics for confirmed pneumonia, 3.2% were admitted at least overnight to the hospital, 1.8% received intravenous fluids for dehydration, and 1.7% received supplemental oxygen for hypoxemia. No child received a lumbar puncture. Overall, 5.2% of ED visits (n = 15) were deemed clinically necessary, according to explicit criteria.
Telephone follow-up was obtained from 56.1% of participants (n = 165). In 73% of these cases, the parents or adult caregiver reported that the child was feeling “a lot better” since the ED visit and 19.4% reported that the child was feeling “somewhat better.” Eight of these parents or caregivers (4.8%) brought their children back to an ED for flulike symptoms within 7 days of the index visit. None of these visits involved children whose original ED care was classified as “necessary” based on explicit clinical criteria. According to the reports of parents and caregivers, none of these 8 children received any of the necessary ED services listed earlier during the second visit.
Ninety percent of respondents reported that it was “very easy” to understand the website questions, and an additional 8.0% said they were “somewhat easy” to understand. Nine in 10 participants (91.4%) described the website as “very easy” to use. There were no significant differences in usability responses between the study sites or by race, ethnicity, or education.
Based on responses provided by parents and adult caregivers, the SORT for Kids algorithm classified 10.2% of patients as low risk, 2.4% as intermediate risk, and 87.4% as high risk.
Of the 15 patients whose initial ED visit met explicit criteria for clinical necessity, SORT for Kids classified 14 as high risk, for an overall sensitivity of 93.3% (exact 95% CI, 68.1%-99.8%). The 1 false-negative result was for a 4-year-old whose parent described him as having a cough but no fever. This child was classified by SORT for Kids as low risk but subsequently received intravenous fluids for dehydration and was discharged with a prescription for antibiotics for chest radiography–confirmed pneumonia.
The SORT algorithm classified 28 of the 271 patients whose initial ED visit was not medically necessary as low risk and 7 as intermediate risk, for an overall specificity of 12.9% (95% CI, 9.2%-17.5%). The main reasons the algorithm classified so many of these children as high risk were survey reports that the child had not urinated in the last 8 hours, was “fussy or cranky,” was “much sleepier or more tired than usual,” or was confused (Table 2).
The SORT algorithm classified all 8 patients whose parents or caregivers reported a second ED visit for ILI as high risk, for an overall sensitivity of 100% (95% CI, 69.2%-100.0%). Of the 157 patients whose parents or caregivers reported no return ED visit, SORT for Kids identified 13 as low risk and 4 as intermediate risk, for an overall specificity of 10.8% (95% CI, 6.4%-16.8%).
Internet access is now widely available, even to traditionally underserved racial and ethnic minority groups and economically disadvantaged populations.17 This affords opportunities for web-based tools to inform decision making regarding when and where to seek health care. We performed this pilot study to determine if a prototype SORT for Kids could be easily understood and used by parents and adult caregivers across a range of educational, racial, and ethnic backgrounds. We also wanted to conduct a preliminary assessment of the performance of the CDC clinical algorithm on which it is based.
Among the modest number of subjects recruited to participate in our pilot study, the algorithm correctly classified 93% of pediatric patients with ILI who made necessary ED visits and all children who made a second ED visit for ILI within the subsequent week. Prior research has found that approximately 5% of patients initially classified as low acuity by triage nurses at the ED triage desk are subsequently determined to require immediate hospitalization.18
Safety was the chief concern of the CDC/AAP working group that developed the clinical algorithm; therefore, the algorithm was designed to avoid misclassification of high-risk cases. As a consequence, SORT for Kids deemed the vast majority of children with ILI in our study as high risk, sacrificing specificity for the sake of sensitivity. An unintended consequence of such a cautious approach might be to significantly overtriage mildly and moderately ill children to hospital EDs, worsening ED crowding in the process and quite possibly contributing to a range of adverse events.19 This is not what the algorithm's authors had in mind. Because the goal of self-triage is to reduce health system surge, not increase it, a much larger prospective study, examining a wider range of clinical questions, will be needed to refine the algorithm to achieve a higher level of specificity without compromising patient safety.
Our study has important limitations. First, study participants were recruited from ED waiting rooms rather than at home, possibly resulting in a study population that was sicker on average than the universe of children with ILI in a nonpandemic year. Second, because this was a modestly funded pilot study, recruitment of subjects was limited to 2 institutions, when convenient to triage nurses, and our team lacked the financial resources to offer Spanish and other language translations of the website. Although there was substantial diversity in study participants' educational, racial, and ethnic backgrounds, our results may not be generalizable to the United States at large. Third, although we made numerous attempts to follow up study participants via telephone, the overall response rate was low. Should telephone follow-up be needed in future studies of this sort, additional strategies beyond a modest incentive may be needed to increase response rates. Fourth, interrater reliability between researchers conducting the medical record abstraction was not measured. Finally, to make our gold standard as objective and rigorous as possible, we based our definition of “medical necessity” on documented provision of 1 or more of 5 emergency care services determined a priori by expert consensus to be indicative of a necessary ED visit. Although resource use is a well-established indicator of disease severity, some might feel that this definition does not sufficiently consider pediatricians' referrals to the ED, a child's clinical presentation, or the level of concern of adult caregivers. The CDC pediatric ILI algorithm was originally developed to address ED surge during a global influenza pandemic that disproportionately affected children. In that context, it is not unreasonable to seek limiting ED visits to those that are clinically necessary.
Our findings present a cautionary tale regarding the potential effects of self-triage tools. Although a diverse set of consumers found the SORT for Kids website easy to use, the underlying algorithm's specificity was poor. Had it been made available to the public in its current form, it might have led more, rather than fewer, parents to bring their children to an ED, thereby worsening, rather than ameliorating, ED crowding. It is certainly possible, but not provable, that call centers and other clinicians who used the CDC algorithm during the 2009 novel influenza A(H1N1) pandemic inadvertently contributed to higher levels of ED use.
It will never be possible, in an evolving pandemic, to prospectively validate novel triage tools. By the time such data are collected and analyzed, the crisis will have passed. But for seasonal influenza and other epidemic illnesses, prospective validation of self-triage tools is needed to properly balance the risks of undertriage and overtriage.
Correspondence: Arthur L. Kellermann, MD, MPH, RAND Corp, Health, 1200 S Hayes St, Arlington, VA 22202-5050 (firstname.lastname@example.org).
Accepted for Publication: November 30, 2012.
Published Online: December 12, 2012. doi:10.1001/jamapediatrics.2013.1573
Author Contributions:Study concept and design: Fagbuyi and Kellermann. Acquisition of data: Fagbuyi, Harris, Hanfling, Place, and Taylor. Analysis and interpretation of data: Anhang Price, Fagbuyi, Hanfling, and Kellermann. Drafting of the manuscript: Anhang Price, Fagbuyi, Harris, and Kellermann. Critical revision of the manuscript for important intellectual content: Fagbuyi, Hanfling, Place, Taylor, and Kellermann. Statistical analysis: Anhang Price and Fagbuyi. Obtained funding: Fagbuyi, Harris, Hanfling, Taylor, and Kellermann. Administrative, technical, and material support: Anhang Price, Fagbuyi, Harris, Hanfling, Place, Taylor, and Kellermann. Study supervision: Fagbuyi, Harris, Hanfling, Place, and Kellermann.
Conflict of Interest Disclosures: None reported.
Funding/Support: Support for this pilot study was provided by a grant from the de Beaumont Foundation. In-kind support for the project was provided by Walgreens in the form of gift cards for participants.
Additional Contributions: We are grateful to the leadership of Inova Health System and Children's National Medical Center for allowing us to conduct this project in their EDs during the 2011-2012 flu season. We deeply appreciate the hard work and good cheer of the nursing and medical staffs in the EDs of Children's National Medical Center and Inova Fairfax Hospital, as well as the efforts of the project champions at each site: Patricia Todd, RN, and Alyssa Carlin (Inova Fairfax) and James M. Chamberlain, MD, Kate Shreve, and Bobbe Thomas (Children's National Medical Center). Finally, we thank Jeremy Root for drafting staff training materials and Alastair Rough for his work developing the web survey.
Anhang Price R, Fagbuyi D, Harris R, Hanfling D, Place F, Taylor TB, Kellermann AL. Feasibility of Web-Based Self-Triage by Parents of Children With Influenza-Like IllnessA Cautionary Tale. JAMA Pediatr. 2013;167(2):112-118. doi:10.1001/jamapediatrics.2013.1573