Customize your JAMA Network experience by selecting one or more topics from the list below.
Copyright 2003 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2003
Advice nurse call centers are used to ensure access to medical advice, thereby potentially reducing the costs of health services.
To determine if medical advice from advice nurses and on-call physicians delays significant medical treatment in a general pediatrics population.
Randomized controlled trial.
A university general pediatrics faculty practice.
Parents or guardians calling for after-hours advice regarding their children.
After-hours medical advice calls were randomized at the time of the call to an advice nurse or an on-call pediatrician.
Main Outcome Measures
The proportion of callers who sought medical care not advised by the advice nurse or on-call pediatrician and the proportion who received unadvised significant care.
There were 1182 advice calls: 566 in the pediatrician group and 616 in the advice nurse group. There were no significant differences in the types of telephone triage advice in the physician and advice nurse groups. There was no significant difference in the proportion of callers who sought unadvised care (108 [19.9%] in the physician group vs 110 [19.0%] in the advice nurse group) or in the proportion of callers who received unadvised significant care (23 [4.2%] in the physician group vs 25 [4.3%] in the advice nurse group).
The proportions of callers who sought unadvised medical care and who received unadvised significant care were not significantly different in the advice nurse and pediatrician groups. This suggests that advice nurses do not delay significant medical treatment when compared with pediatricians.
PHYSICIANS HAVE been providing medical advice by telephone for more than a century.1 In 1977, the National Health Survey reported 693 calls to physician practices per 1000 patients per year.2 Rapid changes in the health care marketplace, however, have led to an increased reliance on telephone interactions as a means to control patient demand for medical care, sometimes referred to as telephone triage.3-5
Telephone triage takes place in various settings, including physician practices, health care maintenance organizations, hospitals and clinics, school and university health programs, and managed care organizations.4 These entities can either develop an in-house system or purchase this service from a commercial vendor or another provider.5 By the end of 1996, telephone triage services were available to 35 million people, and were expected to reach 100 million people by the end of 2001.6
Little research has been performed to evaluate the quality of telephone triage. The studies in pediatric populations have suggested the potential for significant nonmonetary benefits, including high patient satisfaction,7-10 high physician satisfaction,7,11,12 improved quality of information gathering,13,14 and improved access by patients to medical advice.15 Telephone interaction has evolved from unstructured protocols, which rely heavily on the subjective judgment of the call taker, to physician-developed clinical algorithms. Most large call centers employ specially trained full-time call takers, usually nurses, who often use computerized algorithms in conjunction with their clinical judgment to gather information and give medical advice.
Two previous studies16,17 of large call centers using registered nurses and computerized algorithms demonstrated that 80% to 90% of after-hour referrals for urgent or emergency care were appropriate. However, these studies did not determine whether patients who had been referred to delayed care or self-care received significant medical treatment that had been delayed. This study evaluates whether telephone triage advice delays significant medical treatment.
The study population was drawn from a university general pediatrics faculty practice, and included parents who called for after-hours telephone advice during the study period. There are approximately 6000 children in this practice. Of the children, 70% are white, 14% are Asian, and 8% are African American, with the remainder being Latino, Native American, and other. More than 98% have health insurance, 75% are managed care patients, and fewer than 2% are enrolled in Medicaid or Medicare. Based on a survey from a concurrent study, 89.1% of the parents have some college education and 41.8% have attended graduate school.
The inclusion criteria were all after-hours calls for the on-call pediatrician regarding patients in the university general pediatrics faculty practice. The exclusion criteria were as follows: (1) calls not for medical advice (eg, for a prescription refill), (2) calls for emergency medical conditions (eg, someone is unconscious and not breathing or someone is in respiratory distress), and (3) calls in which persons who answered were unable to consent (they were non–English speaking, they were not the legal guardian, or they were minors). When an emergency medical condition was present, the caller was instructed to call 911.
This investigation was approved by the institutional review board, which deemed that telephone consent at the time of the advice call was appropriate. All families in the general pediatrics faculty practice were notified of the study by a special mailing that described the study, including the rationale and methods, and subsequently by an article in the quarterly General Pediatrics Newsletter mailed to all families before the initiation of the study. As was the practice before the study, all telephone calls after office hours, including those for medical advice, were forwarded to the medical center page operators. At the time of the call, the page operator attempted to obtain verbal telephone consent from all callers for patients who did not meet the exclusion criteria. For callers who did not consent, the operator recorded the age of the patient and the reason for the call. Completeness of page operator enrollment attempts and data collection was determined by comparing page operator data forms with daily message prints of all pages to the on-call pediatric faculty.
We estimated a rate of unadvised significant medical care events of 4%, based on a study18 that used a similar measure in a pediatric population and a pilot study (T.J.L. and L.J.B., unpublished data, 1998) we had conducted. We anticipated that the study physicians would refer patients to higher levels of care and would most likely have a lower rate of these events. We wanted to have a sufficient sample size to demonstrate that the advice nurses would have a rate of 8%. Assuming an α of .05 (1-tailed) and a β of .20, we calculated a sample size of 484 patients in each group using Stata (Stata Corp, College Station, Tex).
A single interim analysis was performed, which demonstrated that the study physicians and advice nurses referred patients to the same level of care, and that the rates of unadvised significant care were similar. Discontinuation criteria included the occurrence of death or permanent disability resulting from medical care that had not been advised by the advice nurses. All emergency department (ED) visits and hospitalizations were monitored continuously for discontinuation criteria, which did not occur.
If the caller consented, the page operator selected a numbered opaque envelope, which enclosed a 7.6 × 12.7-cm (3 × 5-in) card with the word "physician" or "nurse." Cards were ordered by the study coordinator (J.G.) using a random number table in which the group was determined by whether the last digit was odd or even. Based on this result, the telephone operator either paged the on-call pediatrician to speak with the parent or forwarded the call directly to the advice nurse at the telephone triage service. All callers randomized to the advice nurse were told they would have the option of speaking to the on-call pediatrician after speaking to the nurse, if they so desired. This is the standard policy of the advice nurse service.
When callers were randomized to the on-call pediatrician, the on-call pediatric faculty member was paged via an alphanumeric pager. The message included the name of the child, the caller's telephone number, the name of the child's pediatrician, and the reason for the call. The on-call pediatricians are the 16 university general pediatrics practice faculty (including H.W.) who function as a group practice. Although they spend time performing research and teaching, most members of the general pediatrics faculty spend at least 50% of their time in clinical care. Medical care is provided at a multispecialty medical office building on the medical campus and at 5 satellite clinics. A member of the pediatric faculty is on call on a rotating basis at all times for the group. Before the study, the usual practice was for the on-call pediatrician to handle all after-hours calls for medical advice.
When callers were assigned to the advice nurse, the page operator forwarded calls to a dedicated telephone number at the telephone triage service. This service is the nation's largest health care information, referral, and teleservices program, with more than 28 million enrolled members (>10% of the US population) and more than 1000 clients, including health plans, government organizations, self-insured employers, providers, and integrated delivery networks.19 It has 3 call centers employing more than 400 registered nurses who have received special training in telephone triage. The triage algorithms have been field tested and modified over 25 years. The triage service is also a commercial service, and is not based at a children's hospital.
Callers were not blinded and knew to which group they were assigned. The advice nurses were not aware of this study. The on-call pediatricians were aware that there was an ongoing study, but did not know which callers had consented to participate, unless the caller revealed they had already spoken to an advice nurse in the prior 72 hours.
All participating callers were called back between 5 PM and 9 PM 72 to 96 hours after the initial call for medical advice. If no contact was made, repeat calls were made at 4, 5, and 6 days after the initial call. In all cases, the follow-up questionnaire was completed with the person who made the original call for medical advice. The interview elicited information about the telephone triage advice provided by the on-call physician or advice nurse, and any health care visits made within 72 hours of the initial telephone call.
The physician on call provided a daily faxed record of advice for after-hours telephone calls, and the commercial advice nurse service provided a computer-generated record of the advice given by their nurses. When the advice nurse told the caller to page the on-call pediatrician, the advice of the on-call pediatrician was used; these callers were still included in the advice nurse group (intention-to-treat analysis). When the advice, as reported by the physician or the advice nurse service, was missing, we used the advice as reported by the caller. The categories of advice were combined to create 3 clearly distinct triage groups: (1) ED or urgent care (call 911, go to the ED, or obtain urgent care), (2) office care (visit a physician's office within 72 hours), and (3) self-care. When there was more than one call for advice within a 72-hour period, we used the highest level of triage advice.
All inpatient care was provided at the Mattel Children's Hospital. Most (>95%) outpatient care was provided at locations that are part of the UCLA Healthcare Enterprise, including the EDs at the UCLA Medical Center and the Santa Monica–UCLA Medical Center, the UCLA Children's Health Center, and 5 satellite clinics. We determined the occurrence and type of medical care (ED care, urgent care, or office care) from claims data obtained from the university billing system for all enrolled patients for all provider contact within 72 hours of the initial telephone call for medical advice. At the follow-up telephone interview, parents were asked to report all health care visits for their child within 72 hours of the advice call. With parental consent, we obtained the medical records for care not provided within the university health care enterprise.
The definitions of the outcome variables are as follows. (1) Determination of unadvised health care visits. These health care visits were considered unadvised if the level of care pursued by the caller was higher than that advised by the on-call physician or advice nurse. This included all ED or urgent care visits when only office care or self-care had been advised, and all office visits when only self-care had been advised. (2) Determination of significant medical care. The medical record for every unadvised health care visit was reviewed by a registered nurse (J.G.) to determine what types of treatment were given. Significant medical care included hospitalization or a significant treatment from a predetermined list that included intravenous fluids (≥20 mL/kg), parenteral antibiotics, nebulized bronchodilators, minor surgical procedures (laceration repair, burn care, fracture or joint reduction, or foreign body removal), or a prescription for antibiotics. A prescription for antibiotics was only considered unadvised significant care if the telephone advice was self-care.
The types of triage advice, the level of care pursued by callers for medical advice, the proportion of callers who sought unadvised medical care, and the proportion who received unadvised significant care were compared by calculating the 95% confidence interval of the difference between the 2 groups.
During the 44-week study period, from January 18, 2000, to November 20, 2000, there were 4085 calls to the page operator for the pediatrician on call (Figure 1). The page operators did not attempt to enroll 807 of these callers in the study and did not complete a study data form. When operators were busy, they did not have the time to obtain informed consent and collect enrollment data. Data were collected by the page operators for the remaining 3278 calls from 2227 individual callers. Of these calls, 3193 were medical advice calls. Eighty-five calls were not for telephone advice, including calls for a prescription refill (n = 52), authorization for medical treatment (n = 6), laboratory results (n = 5), and other reasons (n = 22). Of the 3193 calls for medical advice, 30 were excluded. Of the remaining 3163 calls for medical advice, 410 had already been enrolled in the prior 72 hours and were excluded as repeat callers. Of the 2753 eligible callers, 1569 declined to participate and 1184 agreed to participate. Nine hundred forty-four individual children were enrolled. Of the callers, 566 were enrolled in the on-call pediatrician group and 616 were enrolled in the advice nurse group; 2 erroneously consented. Data were available for health care visits for all callers in the physician group and for 99.7% of the callers in the advice nurse group.
On-call physician vs telephone advice nurse: recruitment of caller. Percentages may not total 100 because of rounding.
There were no significant differences in the mean age; sex; ethnic group; insurance status; timing of after-hours calls; location of callers, as determined by area code; or proportion of complaints between the 2 enrolled study groups (Table 1).
Table 2 demonstrates that the types of telephone triage advice in the physician and advice nurse groups were not significantly different. The physician was not reached by 18 (3.2%) of 566 in the on-call physician group, and the nurse was not reached by 23 (3.7%) of 616 in the advice nurse group. There was no record for 5 calls (0.9%) in the on-call physician group and for 5 calls (0.8%) in the advice nurse group. Other advice was given for 0 calls in the on-call physician group and for 10 calls (1.6%) in the advice nurse group. Eighty-three subjects (13.5%) in the advice nurse group were instructed to page the physician on call. These callers were still included in the advice nurse group (intention-to-treat analysis).
Table 3 demonstrates that there was no significant difference in the proportion of callers who sought unadvised care. Of the callers who pursued unadvised care, there was no significant difference in the proportion of callers who received significant medical care. A prescription for antibiotics was only considered unadvised significant care if the telephone advice was self-care.
Table 4 presents the types of significant care provided for unadvised visits. The most common type of unadvised significant care was a prescription for oral antibiotics; also included were parenteral antibiotics, parenteral fluids, and nebulized bronchodilators. Other prescriptions, most often for acetaminophen and ibuprofen, were not classified as significant care. No child underwent an unadvised minor surgical procedure. Two children whose parents were advised self-care were hospitalized within 72 hours of the advice call. Both were randomized to the advice nurse group. The first was a 6-year-old boy with a complaint of fever, headache, cough, and sore throat. His parents brought him to the ED the same day (unadvised). His vital signs were as follows: temperature, 39.0°C; pulse rate, 146/min; respiratory rate, 18/min; and blood pressure, 117/74 mm Hg. He was described as non–toxic appearing, active, and alert, and his lungs were clear. No diagnostic tests were performed. He was diagnosed as having an upper respiratory tract infection and was treated with acetaminophen. The following day, he was taken to another ED and was diagnosed as having left lower lobe pneumonia with questionable effusion. He was treated with intravenous fluids and antibiotics, and transferred to the Mattel Children's Hospital for a 3-day hospitalization. The second child was a 1-month-old girl with Down syndrome with a complaint of vomiting and diarrhea. The advice nurse referred the call to the on-call pediatrician, who advised self-care. The parent brought the child to the ED (unadvised) 2 days later with a 4-day history of watery stools with some blood and mucus 6 to 10 times per day. Her vital signs were as follows: temperature, 37.5°C; heart rate, 174 beats/min; and respiratory rate, 52/min. The patient was crying. Her stool tested negative for blood. She was admitted for observation and took oral fluids well. The results of laboratory studies, including a complete blood cell count and electrolyte counts, were normal. She was hospitalized for 2 days and was diagnosed as having diarrhea secondary to cow's milk intolerance. Both of these children were randomized to the advice nurse group and included as cases of unadvised and significant care.
Table 5 compares the age and reasons for calls between the enrolled and refused groups.
Call centers in which specially trained nurses provide medical advice using standardized protocols have the potential to improve the quality of advice and to reduce the demand for health care services. Although improved quality of advice may be a contributing factor, the recent and rapid growth in commercial advice nurse call centers has primarily been a market-driven response.
We found that the level of triage advice was remarkably similar in the on-call physician and advice nurse groups, with less than 1% difference for each category of advice (ED or urgent care, office visit, or self-care). Although almost 20% of callers in both groups pursued a more urgent level of care than had been advised, the majority of children did not receive significant treatment. Only 4% of all patients received significant treatment in both groups after seeking unadvised care. For those who did receive significant treatment, this consisted of a prescription for antibiotics in most cases. One could argue that many of these antibiotic prescriptions may not have been necessary. The inappropriate use of antibiotics in the outpatient setting is well documented, and efforts are under way to reduce the use of antibiotics for common respiratory tract infections.20-22 If antibiotic prescriptions were not included in the list of significant treatments, the basic results would not be changed: unadvised significant care was not common and did not differ between the 2 groups.
The difficulty with evaluating the quality of telephone medical advice without an accepted standard of care is exemplified in 2 recent studies.16,17 Both studies determined the appropriateness of after-hours referrals for urgent or emergency care by evaluating patients who were referred for urgent or emergency care and then subsequently had an urgent or emergency care provider visit. In one study,16 90% of these visits were judged appropriate (by the treating physician), and in another,17 80% were judged appropriate by medical record review by 3 physicians. In these studies, there is no clearly defined standard for the appropriateness of an urgent or emergency visit, and some might disagree with the reasons for judging an after-hours referral appropriate. These included parental anxiety (41%) and patient discomfort (53%) for many visits.16
Also, these studies only evaluated the appropriateness of telephone advice for patients who were referred for urgent or emergency care. Children who were referred for delayed care or self-care are perhaps the more important population to evaluate. Prior studies7,10 evaluating the quality of telephone triage service have suggested that no mistakes have been made resulting in mortality or serious morbidity without having conducted close follow-up of the children whose parents received advice. We conducted a close follow-up of patients referred for delayed office care or self-care to determine if they subsequently presented for urgent or emergency care or office care that had not been advised, and then received significant treatment as defined by a medical record–based audit for a defined list of significant treatments that reduced dependence on subjective physician judgment.
The outcome variable we chose was intended to detect if the advice nurses were telling callers who actually required significant treatment to stay home. We believe that our outcome variable essentially captures the intent of this study question. It is possible to construct cases (such as a mildly ill child who calls and is given good advice, but subsequently gets worse) that escape the intent of the outcome variable. However, if in one group, there were significantly more children who seemed to have mild illness and subsequently presented in the ED and received significant care, one would have to question the quality of the advice. The power of randomization is that the number of these cases that evade the intent of the outcome variable will be the same in both groups.
The 72-hour window, although somewhat arbitrary, was chosen to balance 2 concerns. In almost all cases, all care for a single acute pediatric illness will occur within 72 hours of the telephone call. A shorter interval could easily miss the consequences of poor triage advice. A longer interval increases the probability that the condition will change significantly and that provider contact will occur for a second condition unrelated to the initial telephone call.
We considered using more significant health outcomes, such as the incidence of adverse outcomes, including death, permanent disability, or another significant morbidity, as the dependent variable. However, the incidence of such events in a general pediatrics population is so low that the required sample size would be prohibitive. Therefore, we selected intermediate health outcomes that were measurable and that answered our primary question: does telephone triage delay significant medical treatment? We defined as significant only those treatments that, if delayed, could potentially result in significant morbidity, such as parenteral fluids or antibiotics. We included oral antibiotics only when self-care was advised, presuming that if a prescription for antibiotics was given, the child needed this treatment. Had we not, there would have been almost no unadvised significant care provided in the office setting.
The results of this study may not be generalizable to other telephone triage services. There may be unique aspects of any telephone triage service that could influence the quality of telephone medical advice and the potential for mistriage, such as the quality of triage protocols and the level of training of call takers. We attempted to mitigate this by using a well-established call center. Although the call center is not based at a children's hospital, approximately 70% of all telephone calls for medical advice are for children, making it doubtful that their nurses lack experience with pediatric calls.
There may also be aspects of the pediatricians studied that could influence the quality of telephone medical advice. However, although they belong to a university practice, most of the faculty taking calls in our study operate as community pediatricians. They spend 32 hours per week in a clinical practice and see numbers of patients comparable to a busy private practice.
The study population may differ from other populations receiving telephone triage services. The parent population is extremely well educated, with more than 80% having a college or greater degree. A higher level of education and a corresponding greater knowledge of health might better enable callers to overcome deficiencies in medical advice provided by telephone triage services, and mask differences in the quality of advice between physicians and triage nurses. Callers with a greater knowledge of medical problems might know when to question the advice of the triage nurse and to contact the physician on call.
Another potential limitation of the study was the low overall enrollment rate (43.0% of callers for whom consent was attempted and 28.9% of all calls). This enrollment rate is lowered by multiple refusals from the same parent. When these are eliminated, the enrollment rate was 48.8%. We hypothesized that the children of parents who refused may have had more serious or complicated medical problems, and that the callers who consented had simpler medical problems that would not illuminate differences in quality of advice between physicians and triage nurses. However, the children in the groups who consented and refused were similar; the children in the group who refused were slightly younger (mean age, 3.1 vs 3.5 years; 95% confidence interval of difference, 0.1-0.7 years), and did seem to have more serious reasons for calls. One reason for the low participation may be that our population was accustomed to receiving advice from a pediatrician, and that they would not choose to enroll in a study in which they might receive advice from a nurse. However, the low enrollment was also in part due to lesser satisfaction with the advice nurse service.23 It is possible that the inclusion of these less satisfied callers in the study could have changed our results.
A final limitation of the study was the lack of blinding of the pediatricians who participated. It is possible that because of the knowledge that they were being studied, they changed their usual behavior on call to improve the quality of their telephone triage advice. Although the pediatricians were aware of this study, they did not know whether a particular parent had consented to the study. The parents reported that 4.5% of calls were not returned, and that 20.2% of calls required more than 30 minutes for a callback, suggesting the physicians were not taking any extraordinary measures to increase parental satisfaction. The advice nurses were not informed that a study was being performed. However, their calls are regularly monitored for quality assurance.
Corresponding author: Larry J. Baraff, MD, Emergency Medicine Center, UCLA Medical Center, 924 Westwood Blvd, Suite 300, Los Angeles, CA 90024 (e-mail: email@example.com).
Accepted for publication February 3, 2003.
This study was supported by grant 3 RO1 HS10604 from the Agency for Healthcare Research and Quality, Rockville, Md.
Advice nurse call centers are used to ensure access to medical advice, thereby potentially reducing the costs of health services. Previous studies of call centers using registered nurses demonstrated that 80% to 90% of after-hour referrals for urgent or emergency care were appropriate. However, these studies only evaluated patients who had been referred for urgent or emergency care.
This study compared the occurrence of unadvised and significant care, including emergency care, in children referred for office care or self-care. We demonstrated that there was no significant difference in the proportion of callers who sought unadvised care, or in the proportion of callers who received unadvised significant medical care, in the physician and the advice nurse groups.
Lee TJ, Baraff LJ, Guzy J, Johnson D, Woo H. Does Telephone Triage Delay Significant Medical Treatment?Advice Nurse Service vs On-Call Pediatricians. Arch Pediatr Adolesc Med. 2003;157(7):635–641. doi:10.1001/archpedi.157.7.635