Effect of Clinical Decision Support on Diagnostic Imaging for Pediatric Appendicitis: A Cluster Randomized Trial | Clinical Decision Support | JAMA Network Open | JAMA Network
[Skip to Navigation]
Sign In
Visual Abstract. Effect of Clinical Decision Support on Diagnostic Imaging for Pediatric Appendicitis
Effect of Clinical Decision Support on Diagnostic Imaging for Pediatric Appendicitis
Figure.  Flowchart of Patients Who Were Eligible, Excluded, and Enrolled by Intervention and Usual Care Group
Flowchart of Patients Who Were Eligible, Excluded, and Enrolled by Intervention and Usual Care Group

Cost analyses were limited to enrolled subjects insured through the health system (HealthPartners or Kaiser Permanente, Northern California). CDS indicates clinical decision support; ED, emergency department.

aNot enrolled indicates that no patient information was entered in the clinical decision support system.

Table 1.  Study Population Characteristics by Intervention Group and Study Phase
Study Population Characteristics by Intervention Group and Study Phase
Table 2.  Appendicitis, Imaging Use, and Safety Outcomes During the Index Visit by Intervention Group and Study Phasea
Appendicitis, Imaging Use, and Safety Outcomes During the Index Visit by Intervention Group and Study Phasea
Table 3.  Imaging Use by pARC Risk Stratification, Study Group and Phase, at Kaiser Permanente Northern California Sites
Imaging Use by pARC Risk Stratification, Study Group and Phase, at Kaiser Permanente Northern California Sites
Table 4.  Costs of Care Within 14 Days After Index Emergency Department Visit by Study Group During Intervention Phasea
Costs of Care Within 14 Days After Index Emergency Department Visit by Study Group During Intervention Phasea
1.
Barrett  ML, Hines  AL, Andrews  RM. Trends in rates of perforated appendix, 2001-2010: Statistical Brief #159. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality; 2006. Accessed January 5, 2021. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb159.pdf
2.
Colvin  JM, Bachur  R, Kharbanda  A.  The presentation of appendicitis in preadolescent children.   Pediatr Emerg Care. 2007;23(12):849-855. doi:10.1097/pec.0b013e31815c9d7fPubMedGoogle ScholarCrossref
3.
Anderson  KT, Bartz-Kurycki  MA, Austin  MT,  et al.  Hospital type predicts computed tomography use for pediatric appendicitis.   J Pediatr Surg. 2019;54(4):723-727. doi:10.1016/j.jpedsurg.2018.05.018PubMedGoogle ScholarCrossref
4.
Raval  MV, Deans  KJ, Rangel  SJ, Kelleher  KJ, Moss  RL.  Factors associated with imaging modality choice in children with appendicitis.   J Surg Res. 2012;177(1):131-136. doi:10.1016/j.jss.2012.03.044PubMedGoogle ScholarCrossref
5.
Shah  SR, Sinclair  KA, Theut  SB, Johnson  KM, Holcomb  GW  III, St Peter  SD.  Computed tomography utilization for the diagnosis of acute appendicitis in children decreases with a diagnostic algorithm.   Ann Surg. 2016;264(3):474-481. doi:10.1097/SLA.0000000000001867PubMedGoogle ScholarCrossref
6.
Warner  BW, Kulick  RM, Stoops  MM, Mehta  S, Stephan  M, Kotagal  UR.  An evidenced-based clinical pathway for acute appendicitis decreases hospital duration and cost.   J Pediatr Surg. 1998;33(9):1371-1375. doi:10.1016/S0022-3468(98)90010-0PubMedGoogle ScholarCrossref
7.
Warner  BW, Rich  KA, Atherton  H, Andersen  CL, Kotagal  UR.  The sustained impact of an evidenced-based clinical pathway for acute appendicitis.   Semin Pediatr Surg. 2002;11(1):29-35. doi:10.1053/spsu.2002.29364PubMedGoogle ScholarCrossref
8.
Saucier  A, Huang  EY, Emeremni  CA, Pershad  J.  Prospective evaluation of a clinical pathway for suspected appendicitis.   Pediatrics. 2014;133(1):e88-e95. doi:10.1542/peds.2013-2208PubMedGoogle ScholarCrossref
9.
Wagenaar  AE, Tashiro  J, Wang  B,  et al.  Protocol for suspected pediatric appendicitis limits computed tomography utilization.   J Surg Res. 2015;199(1):153-158. doi:10.1016/j.jss.2015.04.028PubMedGoogle ScholarCrossref
10.
Bachur  RG, Levy  JA, Callahan  MJ, Rangel  SJ, Monuteaux  MC.  Effect of reduction in the use of computed tomography on clinical outcomes of appendicitis.   JAMA Pediatr. 2015;169(8):755-760. doi:10.1001/jamapediatrics.2015.0479PubMedGoogle ScholarCrossref
11.
Niles  LM, Goyal  MK, Badolato  GM, Chamberlain  JM, Cohen  JS.  US emergency department trends in imaging for pediatric nontraumatic abdominal pain.   Pediatrics. 2017;140(4):e20170615. doi:10.1542/peds.2017-0615PubMedGoogle Scholar
12.
Otero  HJ, Crowder  L.  Imaging utilization for the diagnosis of appendicitis in stand-alone children’s hospitals in the United States: trends and costs.   J Am Coll Radiol. 2017;14(5):603-608. doi:10.1016/j.jacr.2016.12.013PubMedGoogle ScholarCrossref
13.
Santillanes  G, Simms  S, Gausche-Hill  M,  et al.  Prospective evaluation of a clinical practice guideline for diagnosis of appendicitis in children.   Acad Emerg Med. 2012;19(8):886-893. doi:10.1111/j.1553-2712.2012.01402.xPubMedGoogle ScholarCrossref
14.
Depinet  H, von Allmen  D, Towbin  A, Hornung  R, Ho  M, Alessandrini  E.  Risk stratification to decrease unnecessary diagnostic imaging for acute appendicitis.   Pediatrics. 2016;138(3):e20154031. doi:10.1542/peds.2015-4031PubMedGoogle Scholar
15.
Kharbanda  AB, Madhok  M, Krause  E,  et al.  Implementation of electronic clinical decision support for pediatric appendicitis.   Pediatrics. 2016;137(5):e20151745. doi:10.1542/peds.2015-1745PubMedGoogle Scholar
16.
Gurien  LA, Smith  SD, Dassinger  MS, Burford  JM, Tepas  JJ, Crandall  M.  Suspected appendicitis pathway continues to lower CT rates in children two years after implementation.   Am J Surg. 2019;218(4):716-721. doi:10.1016/j.amjsurg.2019.07.024PubMedGoogle ScholarCrossref
17.
Binkovitz  LA, Unsdorfer  KM, Thapa  P,  et al.  Pediatric appendiceal ultrasound: accuracy, determinacy and clinical outcomes.   Pediatr Radiol. 2015;45(13):1934-1944. doi:10.1007/s00247-015-3432-7PubMedGoogle ScholarCrossref
18.
Schuh  S, Chan  K, Langer  JC,  et al.  Properties of serial ultrasound clinical diagnostic pathway in suspected appendicitis and related computed tomography use.   Acad Emerg Med. 2015;22(4):406-414. doi:10.1111/acem.12631PubMedGoogle ScholarCrossref
19.
Kotagal  M, Richards  MK, Flum  DR, Acierno  SP, Weinsheimer  RL, Goldin  AB.  Use and accuracy of diagnostic imaging in the evaluation of pediatric appendicitis.   J Pediatr Surg. 2015;50(4):642-646. doi:10.1016/j.jpedsurg.2014.09.080PubMedGoogle ScholarCrossref
20.
McDermott  KW, Stocks  C, Freeman  WJ. Overview of pediatric emergency department visits, 2015: Statistical Brief #242. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality; 2018.
21.
Kharbanda  AB, Vazquez-Benitez  G, Ballard  DW,  et al.  Development and validation of a novel pediatric appendicitis risk calculator (pARC).   Pediatrics. 2018;141(4):e20172699. doi:10.1542/peds.2017-2699PubMedGoogle Scholar
22.
Ekstrom  HL, Kharbanda  EO, Ballard  DW,  et al.  Development of a clinical decision support system for pediatric abdominal pain in emergency department settings across two health systems within the HCSRN.   EGEMS (Wash DC). 2019;7(1):15. doi:10.5334/egems.282PubMedGoogle Scholar
23.
Vinson  DR, Mark  DG, Chettipally  UK,  et al; eSPEED Investigators of the KP CREST Network.  Increasing safe outpatient management of emergency department patients with pulmonary embolism: a controlled pragmatic trial.   Ann Intern Med. 2018;169(12):855-865. doi:10.7326/M18-1206PubMedGoogle ScholarCrossref
24.
Ballard  DW, Vemula  R, Chettipally  UK,  et al; KP CREST Network Investigators.  Optimizing clinical decision support in the electronic health record: clinical characteristics associated with the use of a decision tool for disposition of ED patients with pulmonary embolism.   Appl Clin Inform. 2016;7(3):883-898. doi:10.4338/ACI-2016-05-RA-0073PubMedGoogle ScholarCrossref
25.
Simon  LE, Rauchwerger  AS, Chettipally  UK,  et al.  Text message alerts to emergency physicians identifying potential study candidates increase clinical trial enrollment.   J Am Med Inform Assoc. 2019;26(11):1360-1363. doi:10.1093/jamia/ocz118PubMedGoogle ScholarCrossref
26.
HealthPartners. Total Care Relative Resource Value (TCRRVTM): a measurement approach to achieve the triple aim. Updated September 21, 2017. Accessed June 12, 2018. https://www.healthpartners.com/ucm/groups/public/@hp/@public/documents/documents/cntrb_039627.pdf
27.
National Quality Forum. NQF endorses resource use measures. Published January 31, 2012. Accessed June 12, 2018. http://www.qualityforum.org/News_And_Resources/Press_Releases/2012/NQF_Endorses_Resource_Use_Measures.aspx
28.
Yan  Z, Ip  IK, Raja  AS, Gupta  A, Kosowsky  JM, Khorasani  R.  Yield of CT pulmonary angiography in the emergency department when providers override evidence-based clinical decision support.   Radiology. 2017;282(3):717-725. doi:10.1148/radiol.2016151985PubMedGoogle ScholarCrossref
29.
Khairat  S, Marc  D, Crosby  W, Al Sanousi  A.  Reasons for physicians not adopting clinical decision support systems: critical analysis.   JMIR Med Inform. 2018;6(2):e24. doi:10.2196/medinform.8912PubMedGoogle Scholar
30.
Young  PJ, Nickson  CP, Perner  A.  When should clinicians act on non-statistically significant results from clinical trials?   JAMA. 2020;323(22):2256-2257. doi:10.1001/jama.2020.3508PubMedGoogle ScholarCrossref
31.
O’Connor  SD, Sodickson  AD, Ip  IK,  et al.  Journal club: requiring clinical justification to override repeat imaging decision support: impact on CT use.   AJR Am J Roentgenol. 2014;203(5):W482-90. doi:10.2214/AJR.14.13017PubMedGoogle Scholar
32.
Raja  AS, Ip  IK, Dunne  RM, Schuur  JD, Mills  AM, Khorasani  R.  Effects of performance feedback reports on adherence to evidence-based guidelines in use of CT for evaluation of pulmonary embolism in the emergency department: a randomized trial.   AJR Am J Roentgenol. 2015;205(5):936-940. doi:10.2214/AJR.15.14677PubMedGoogle ScholarCrossref
33.
Cabana  MD, Rand  CS, Powe  NR,  et al.  Why don’t physicians follow clinical practice guidelines? a framework for improvement.   JAMA. 1999;282(15):1458-1465. doi:10.1001/jama.282.15.1458PubMedGoogle ScholarCrossref
34.
Hess  EP, Homme  JL, Kharbanda  AB,  et al.  Effect of the head computed tomography choice decision aid in parents of children with minor head trauma: a cluster randomized trial.   JAMA Netw Open. 2018;1(5):e182430. doi:10.1001/jamanetworkopen.2018.2430PubMedGoogle Scholar
35.
Davidson  KW, Ye  S, Mensah  GA.  Commentary: de-implementation science: a virtuous cycle of ceasing and desisting low-value care before implementing new high value care.   Ethn Dis. 2017;27(4):463-468. doi:10.18865/ed.27.4.463PubMedGoogle ScholarCrossref
36.
Norton  WE, Chambers  DA.  Unpacking the complexities of de-implementing inappropriate health interventions.   Implement Sci. 2020;15(1):2. doi:10.1186/s13012-019-0960-9PubMedGoogle ScholarCrossref
37.
Kharbanda  AB, Dudley  NC, Bajaj  L,  et al; Pediatric Emergency Medicine Collaborative Research Committee of the American Academy of Pediatrics.  Validation and refinement of a prediction rule to identify children at low risk for acute appendicitis.   Arch Pediatr Adolesc Med. 2012;166(8):738-744. doi:10.1001/archpediatrics.2012.490PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Pediatrics
    February 9, 2021

    Effect of Clinical Decision Support on Diagnostic Imaging for Pediatric Appendicitis: A Cluster Randomized Trial

    Author Affiliations
    • 1Department of Pediatric Emergency Medicine, Children’s Minnesota, Minneapolis
    • 2Division of Research, HealthPartners Institute, Minneapolis, Minnesota
    • 3The Permanente Medical Group, Oakland, California
    • 4The Kaiser Permanente Northern California Division of Research, Oakland, California
    JAMA Netw Open. 2021;4(2):e2036344. doi:10.1001/jamanetworkopen.2020.36344
    Key Points

    Question  Can an electronic health record–linked clinical decision support tool reduce the use of diagnostic imaging in children and adolescents with emergency department visits for acute abdominal pain?

    Findings  In this cluster randomized trial of 17 emergency departments caring for 5940 patients, the clinical decision support tool did not lead to an overall reduction in the use of computed tomography or ultrasonography. In 1 health care system, the clinical support tool was associated with a significant reduction in diagnostic imaging among patients at low or moderate risk of appendicitis.

    Meaning  These findings suggest that more research is needed to determine whether clinical decision support tools promote more appropriate imaging in patients with acute abdominal pain.

    Abstract

    Importance  Appendicitis is the most common pediatric surgical emergency. Efforts to improve efficiency and quality of care have increased reliance on computed tomography (CT) and ultrasonography (US) in children with suspected appendicitis.

    Objective  To evaluate the effectiveness of an electronic health record–linked clinical decision support intervention, AppyCDS, on diagnostic imaging, health care costs, and safety outcomes for patients with suspected appendicitis.

    Design, Setting, and Participants  In this parallel, cluster randomized trial, 17 community-based general emergency departments (EDs) in California, Minnesota, and Wisconsin were randomized to the AppyCDS intervention group or usual care (UC) group. Patients were aged 5 to 20 years, presenting for an ED visit with right-sided or diffuse abdominal pain lasting 5 days or less. We excluded pregnant patients, those with a prior appendectomy, those with selected comorbidities, and those with traumatic injuries. The trial was conducted from October 2016 to July 2019.

    Interventions  AppyCDS prompted data entry at the point of care to estimate appendicitis risk using the pediatric appendicitis risk calculator (pARC). Based on pARC estimates, AppyCDS recommended next steps in care.

    Main Outcomes and Measures  Primary outcomes were CT, US, or any imaging (CT or US) during the index ED visit. Safety outcomes were perforations, negative appendectomies, and missed appendicitis. Costs were a secondary outcome. Ratio of ratios (RORs) for primary and safety outcomes and differences by group in cost were used to evaluate effectiveness of the clinical decision support tool.

    Results  We enrolled 3161 patients at intervention EDs and 2779 patients at UC EDs. The mean age of patients was 11.9 (4.6) years and 2614 (44.0%) were boys or young men. RORs for CT (0.94; 95% CI, 0.75-1.19), US (0.98; 95% CI, 0.84-1.14), and any imaging (0.96; 95% CI, 0.86-1.07) did not differ by study group. In an exploratory analysis conducted in 1 health system, AppyCDS was associated with a reduction in any imaging (ROR, 0.82; 95% CI, 0.73- 0.93) for patients with pARC score of 15% or less and a reduction in CT (ROR, 0.58; 95% CI, 0.45-0.74) for patients with a pARC score of 16% to 50%. Perforations, negative appendectomies, and cases of missed appendicitis by study phase did not differ significantly by study group. Costs did not differ overall by study group.

    Conclusions and Relevance  In this study, AppyCDS was not associated with overall reductions in diagnostic imaging; exploratory analysis revealed more appropriate use of imaging in patients with a low pARC score.

    Trial Registration  ClinicalTrials.gov Identifier: NCT02633735

    Introduction

    Appendicitis is the most common pediatric surgical emergency, with more than 75 000 cases in the United States each year.1 Despite its high frequency, distinguishing appendicitis from other causes of acute abdominal pain remains a challenge.2 Pressures to improve emergency department (ED) efficiency, while preventing negative appendectomies and missed appendicitis, have contributed to heavy utilization of computed tomography (CT) and ultrasonography (US) in patients with suspected appendicitis.3,4

    During the past 2 decades, there have been incremental improvements in the evaluation of patients with suspected appendicitis, especially at pediatric EDs,5-9 where efforts to reduce ionizing radiation exposure have shifted imaging from CT to US.10-12 Clinical pathways for acute abdominal pain have demonstrated the safety and effectiveness5,13-16 of US as first-line imaging.14-16 However, US can be problematic if applied indiscriminately, as it is operator dependent, and equivocal or nondiagnostic results are common.17,18 Unfortunately, CT use remains high for children with abdominal pain at community-based, general EDs.3,4,19 Differential management between pediatric and general EDs is concerning, considering that community-based sites are where most children across the United States present for emergent evaluation.20

    We previously developed and validated the pediatric appendicitis risk calculator (pARC).21 Subsequently, we created a clinical decision support (CDS) system, AppyCDS, integrating pARC with the electronic health record (EHR). Using EHR and web-based algorithms, AppyCDS prompts health care clinicians to screen patients at risk of appendicitis and recommends next steps in care based on calculated risk or pARC score.22 The primary aim of this study was to evaluate the impact of AppyCDS on use of CT or US during an ED visit among pediatric and adolescent patients with acute abdominal pain. Secondary aims were to evaluate the impact of AppyCDS on health care costs and safety outcomes.

    Methods

    We evaluated AppyCDS in a pragmatic (ie, the intervention was tested in real-world clinical settings), parallel cluster randomized trial in 17 community-based general EDs within 2 large health systems. Of these, 6 were HealthPartners (HP) facilities in Minnesota and Western Wisconsin, with a combined annual ED census of 160 000 visits. Eleven EDs were Kaiser Permanente Northern California (KPNC) facilities, located across Northern California, with a combined annual ED census of 800 000 visits. This study was approved by institutional review boards from participating sites with a waiver of informed consent because it posed minimal risk to patients. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline. The trial protocol and statistical analysis plan are available in Supplement 1. For both health systems, patients aged 5 to 20 years old account for approximately 20% of ED visits. All EDs use the Epic EHR. Neither health system is university-based; 5 EDs have academic affiliations. Facilities were staffed by board-certified or board-eligible emergency physicians. Physician assistants also provided care at HP EDs. All EDs had access to CT and US during regular business hours; after-hours US availability varied by facility. Age thresholds to admit for observation or surgery (vs transfer to a dedicated pediatric facility) also varied by ED.

    Design, Study Population, and Randomization

    We conducted the trial between October 2016 and July 2019, with 2 study phases, ie, preintervention and intervention. Detailed descriptions of the AppyCDS intervention have been published.22 The timing of AppyCDS implementation varied by health system (eFigure in Supplement 2). At HP, following a 5-month pilot, the preintervention phase ran from November 2016 to April 2017. After a 1-month washout period, the intervention phase ran from June 2017 to July 2019. At KPNC, following a 2-month pilot, the preintervention phase ran from October 2016 to June 2017. After a 1-month washout period, the intervention phase ran from August 2017 to July 2019. Data from pilot and washout periods were not included in analyses.

    At all sites, patients aged 5 to 20 years presenting to the ED with a chief complaint of abdominal pain were initially assessed using 3 study-specific screening questions, displayed in the EHR, as follows: (1) is the patient’s abdominal pain diffuse (ie, generalized)? (2) does the patient have any right-sided abdominal pain? and (3) what is the duration of abdominal pain? Screening questions were completed by nursing staff (at HP) or treating physicians (at KPNC) prior to evaluation. Patients with 5 days or fewer of right-sided or diffuse abdominal pain were eligible and enrolled. Automated exclusions were applied in real time based on data in the EHR.22 Additional exclusions were applied during analysis (eTable 1 in Supplement 2).

    Sites were randomized to the AppyCDS intervention or usual care (UC) in January 2017. We stratified EDs by health system and then grouped them into clusters by patterns of pediatric referral and health care professional coverage. We sorted clusters into pairs by pediatric patient volume. Using a random-number generator, 6 clusters (9 EDs) were randomly assigned to AppyCDS intervention; 6 clusters (8 EDs) were assigned to UC.

    Description of Intervention

    The overall goals of AppyCDS were as follows: (1) to collect data to calculate risk of appendicitis (pARC score) and (2) to provide point-of-care decision support, recommending next steps in care based on pARC score. As previously described, AppyCDS platforms differed by health system to accommodate existing workflows and EHR-linked CDS infrastructure.22 At HP, clinicians at intervention sites were directed by best practice advisories (BPAs) to use AppyCDS; BPAs displayed for patients who were study eligible when the clinician opened the medical record or attempted to order a CT or US. At KPNC, physicians were trained to access AppyCDS through an existing EHR-linked CDS platform, RISTRA (Risk Stratification).23,24 In addition, KPNC physicians were notified of potentially eligible patients through automated text message alerts.25 At KPNC, physicians accessing AppyCDS completed initial screening questions, identified exclusions, and, for eligible patients, proceeded with data entry to calculate a pARC score. Across all intervention sites, recommendations for patient management displayed based on pARC score (eTable 2 in Supplement 2). Recommendations were tailored by health system, based on resource availability and consensus from clinical leaders.

    Description of UC

    At all HP sites preintervention and at HP UC sites during the intervention, no additional data were collected. At all KPNC sites during preintervention and at KPNC UC sites during the intervention, additional clinical information was entered by health care professionals into AppyCDS, but pARC scores and recommended next steps in care were not displayed.

    Outcomes

    The primary outcome was utilization of diagnostic imaging (ie, CT, US, or any imaging [CT or US]) during the index ED visit. Use of diagnostic imaging was identified through Current Procedural Terminology (CPT) or internal codes.

    Appendicitis was confirmed by manual review of pathology and operative notes for appendectomies within 7 days of the index ED visit with definitions based on our prior work.15,21 When operative or pathology reports were not available, diagnosis was based on data at the index ED visit.

    Safety outcomes included negative appendectomies, perforation, and missed appendicitis. Negative appendectomy was defined as a nonincidental appendectomy within 7 days of the index visit with no evidence for appendicitis on histopathology. Perforation was based on manual review of operative notes. Missed appendicitis was defined as appendicitis within 7 days of the index visit among patients discharged home. Appendicitis outcomes were reviewed by 2 study investigators (D.W.B. and E.O.K.) with adjudication by a third investigator (A.B.K.) as needed.

    Secondary outcomes included ED length of stay, disposition, ED visits and intensive care admissions within 7 days of the index visit, severe medical conditions (eg, pyelonephritis, nephrolithiasis), and other surgical conditions (eg, intussusception, ovarian torsion). Among patients insured through HP or KPNC, costs of care for the index ED visit and the subsequent 14 days were evaluated using claims data. Costs of care in 2018 US dollars were derived by matching billing codes to Total Care Relative Resource Values,26 a standardized set of measures endorsed by the National Quality Forum for calculating the total cost of care.27 Pharmacy costs were limited to antibiotics, antiemetics, and pain medications.

    Statistical Analysis

    First, we evaluated patient characteristics and use of imaging by study group and phase using frequency distributions, means, and standard deviations. We evaluated the effectiveness of AppyCDS by estimating the ratio of ratios (RORs) and 95% CIs of the intervention group, from intervention phase to preintervention phase, compared with the UC group for the 2 study phases for primary and safety outcomes. To estimate the ROR, we used a general estimating equation (GEE) Poisson model with a study group indicator, a study phase indicator, a study group–study phase interaction term, and health system as fixed effects. We evaluated whether adding patient characteristics into the model changed the ROR estimates. The GEE model accounted for the data structure, patients within each ED, and randomization by cluster, with a repeat statement and compound symmetry covariance structure. For safety and secondary binary outcomes, similar methods were used. Length of ED stay and hospitalization were estimated using a log normal distribution. Costs were estimated with a GEE model using a γ distribution with a log link for patients enrolled during the intervention phase. For KPNC, where pARC values were collected for all enrolled patients, we evaluated imaging outcomes by pARC strata in exploratory analyses, using a second-order interaction. Furthermore, we evaluated whether the AppyCDS intervention significantly reduced imaging in patients with pARC scores of 50% or less, consistent with the AppyCDS tool recommendations for this low-risk group (eTable 2 in Supplement 2). Analysis for effectiveness of the AppyCDS system was evaluated with a 2-sided test at P < .05 level of significance. Analysis was performed using SAS version 9.4 (SAS Institute). A priori power analysis assumed that with 12 clusters, baseline CT utilization of 30%, 600 patients per cluster during the intervention period, α of .05 with a 2-tailed test, and an intraclass cluster correlation of 0.05, the study would have 80% power to identify an ROR of 0.76, contrasting the post-pre ratios of CT in the intervention vs the UC group.

    Results

    Between October 2016 and July 2019 there were 40 283 patients aged 5 to 20 years with visits to participating EDs with abdominal pain. Of these, 8605 (21.4%) had an a priori exclusion (eg, prior appendectomy, pregnancy, trauma, other selected comorbidities, or visit during washout period). We excluded 949 patients (2.4%) at intervention sites and 703 (1.7%) at UC sites, based on data entered into AppyCDS indicating pain was not right-sided or generalized or had lasted for more than 5 days (Figure). An additional 12 945 (32.1%) at intervention sites and 11 140 (27.7%) at UC sites did not have data entered in AppyCDS, so their eligibility could not be evaluated. The final enrolled population with 5 days or less of right-sided or diffuse abdominal pain included 3161 patients at intervention EDs (1011 [32.0%] preintervention phase; 2150 [68.0%] intervention phase) and 2779 patients at UC EDs (788 [28.4%] preintervention phase; 1991 [71.6%] intervention phase). Mean age was 11.9 (4.6) years, and 2614 (44.0%) were boys or young men. Patients at UC sites were more likely to be White, non-Hispanic individuals compared with those at intervention sites (preintervention UC, 311 [39.5%]; intervention UC, 901 [45.2%] vs preintervention AppyCDS, 234 [23.2%]; intervention AppyCDS, 721 [34.5%]) (Table 1). Proportions with appendicitis ranged from 11.1% (112 of 1011) to 13.5% (290 of 2150) by study group and study phase (Table 2). Comparison of patients who were not enrolled vs those who were enrolled revealed a higher mean (SD) age (13.4 [5.0] years vs 11.8 [4.5] years) and a higher proportion of female patients (15 166 of 24 203 [62.7%] vs 3293 of 5940 [55.4%]).

    Primary Outcomes

    There was no statistically significant change in CT use from preintervention to intervention phase between study groups (ROR, 0.94; 95% CI, 0.75-1.19) (Table 2). A nonsignificant decrease in US use was noted between preintervention and intervention phases at intervention and UC sites (485 of 1011 [48.0%] to 986 of 2150 [45.9%] and 397 of 788 [50.4%] to 958 of 1991 [48.1%], respectively; ROR, 0.98; 95% CI, 0.84-1.14). At intervention sites, any imaging decreased nonsignificantly from 586 (58.0%) in the preintervention phase to 1237 (57.5%) in the intervention phase. At UC sites, there was a nonsignificant increase in any imaging, from 455 (57.7%) in the preintervention phase to 1163 (58.4%) in the intervention phase. The ratio of any imaging by study phase did not differ by study group (ROR, 0.96; 95% CI, 0.86-1.08). After adjusting for age, sex, and race, results were similar (CT: ROR, 0.92; 95% CI, 0.76-1.06; US: ROR, 0.98; 95% CI, 0.83-1.14; any imaging: ROR, 0.95; 95% CI, 0.84-1.06).

    Exploratory Analyses

    At KPNC, where pARC scores were calculated for all enrolled patients, AppyCDS was associated with a reduction in any imaging (ROR, 0.82; 95% CI, 0.73-0.93) for pARC scores of 15% or less and a reduction in CT (ROR, 0.58; 95% CI, 0.45-0.74) for pARC scores between 16% and 50% (Table 3). However, there was no overall association between pARC strata and imaging.

    Safety, Secondary Outcomes, and Cost Analysis

    Perforation rates decreased over time at intervention sites (20 [17.9%] to 43 [14.8%]) and increased at UC sites (17 [17.9%] to 50 [20.3%]); the ratio of perforation by phase did not differ significantly by study arm (ROR, 0.74; 95% CI, 0.41-1.32). Negative appendectomies decreased at both intervention sites (9 [7.4%] to 11 [3.7%]) and at UC sites (8 [7.8%] to 19 [7.3%]); however, the ratio by study arm was not significant (ROR, 0.51; 95% CI, 0.16-1.66). Missed appendicitis was uncommon in intervention and UC arms across study phases; changes in missed appendicitis rates between the preintervention and intervention phases, by study group, were not significant (ROR, 2.83; 95% CI, 1.0-7.7) (Table 2).

    Most patients were discharged home following their ED visit with minimal variation by study group and study phase. Mean ED length of stay increased minimally at intervention sites and remained constant at UC sites. Additional secondary outcomes did not differ by study phase by study group (eTable 3 in Supplement 2).

    A total of 1372 patients in the intervention arm and 1290 patients in the UC arm had HP or KPNC insurance and were included in cost analyses (Figure). Overall health care costs did not significantly differ by study group. Nevertheless, total outpatient costs on days 4 to 14 after the index visit were significantly lower at intervention sites compared with UC sites (difference, −$31; 95% CI, −$47 to −$15) (Table 4; eTable 4 in Supplement 2).

    Discussion

    In this multisite pragmatic cluster randomized trial, AppyCDS did not reduce CT or US use for the full spectrum of pediatric patients with suspected appendicitis. However, in an exploratory analysis, patients with a low (≤15%) or moderate (16%-50%) pARC scores underwent less imaging. Our study demonstrated the feasibility and safety of conducting a large randomized trial to improve care for children presenting for care in general, community-based EDs, while highlighting challenges and areas for improvement in future interventions.

    The pARC score was designed to quantify risk of appendicitis and maximize sorting of patients as having high or low risk of appendicitis.21 Each participating health system developed care recommendations based on pARC strata, with input from clinical leaders. At intervention sites, clinicians were significantly more likely to defer imaging for patients with a less than 15% risk of appendicitis, consistent with AppyCDS recommendations. For those with a risk of 16% to 50%, we recommended US as first-line imaging, and accordingly, we noted a significant decrease in use of CT at intervention sites. Consistent with data from the National Surgical Quality Improvement Program for Pediatrics, patients with high pARC scores (>75%) continued to undergo diagnostic imaging at high rates (>90%).3 These results suggest that intervention EDs moved toward more appropriate imaging.30

    Our findings are also consistent with prior research demonstrating low CDS use in acute health care settings.28,29 AppyCDS was designed to provide recommendations in real time to influence decision-making; however, alerts to complete the screening or use the CDS were easy to dismiss or ignore. The CDS system was developed in collaboration with end-users, yet fully automated estimation of pARC using EHR data may be preferred to improve efficiency and adherence. Requiring clinicians to provide a reason for overriding the CDS system or providing real-time feedback has also been demonstrated to improve use of CDS systems.31,32

    Lack of agreement with recommendations, along with system-level barriers, may have limited adherence with AppyCDS guidance.33 For example, calculation of appendicitis risk through pARC required a white blood cell count, yet parents may have preferred imaging to confirm a diagnosis without a blood draw. For patients at low to moderate risk (ie, pARC score of 16%-25%), the CDS recommended observation prior to imaging, but this may be impractical in busy general EDs with limited space for pediatric patients. For many community-based EDs, AppyCDS recommendations represented a deimplementation of care (ie, forgoing US in patients at low risk of appendicitis). Efforts to deimplement care in the ED may require multilevel approaches that cannot be delivered through CDS systems alone. Ultimately, shared decision-making between families and clinicians,34 based on pARC score, along with health system support,35 may be needed to optimize diagnostic imaging in pediatric patients with acute abdominal pain.36

    Limitations

    Several important limitations should be noted. There were a large number of patients who were not enrolled at intervention and control sites. As previously described, the rate of appendicitis among these patients was low22; it is likely that physicians ignored alerts to use AppyCDS because they had low suspicion for appendicitis. Improved capture of these patients may have increased the generalizability of our results but would also have resulted in a lower rate of appendicitis. The clinical characteristics of our enrolled population were consistent with prior publications describing children with acute abdominal pain.15,21,37 Regardless, our findings demonstrated the challenges we faced trying to evaluate and improve care for children in community-based general EDs. Barriers to CDS systems adherence may be amplified in settings where children are only a fraction of the patient population, and thus, alerts to complete CDS screening questions were infrequent. Additional barriers to CDS system use, such as pressure to reduce ED length of stay, lack of consistent availability of US, financial concerns of families, and fear of litigation due to missed appendicitis, were not measured. To conduct this trial in 2 health systems across 17 hospitals, it was necessary to provide system-level recommendations for care and differences in integration of the CDS system.22 However, this approach did not account for varying levels of risk tolerance among physicians and surgeons. Furthermore, our primary a priori study aim was to demonstrate an overall reduction in diagnostic imaging. A more nuanced approach to outcome assessment may have been more appropriate and was consistent with our findings. To achieve sustained reductions in diagnostic imaging, further studies should evaluate targeted approaches to US based on appendicitis risk.

    Conclusions

    In this cluster randomized trial, we demonstrated the feasibility and safety of an EHR-linked appendicitis CDS tool in community-based EDs. Although we did not show an overall reduction in imaging, exploratory analyses at 1 health system revealed more appropriate use of diagnostic imaging.

    Back to top
    Article Information

    Accepted for Publication: December 16, 2020.

    Published: February 9, 2021. doi:10.1001/jamanetworkopen.2020.36344

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Kharbanda AB et al. JAMA Network Open.

    Corresponding Author: Anupam B. Kharbanda, MD, MSc, Department of Pediatric Emergency Medicine, Children’s Minnesota, 2525 Chicago Ave S, Minneapolis, MN 55404 (anupam.kharbanda@childrensmn.org).

    Author Contributions: Drs Kharbanda and Vazquez-Benitez had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: A. B. Kharbanda, Vazquez-Benitez, Ballard, Vinson, Chettipally, Ekstrom, O’Connor, E. O. Kharbanda.

    Acquisition, analysis, or interpretation of data: A. B. Kharbanda, Vazquez-Benitez, Ballard, Vinson, Dehmer, Rauchwerger, McMichael, Cotton, Kene, Simon, Zhu, Warton, O’Connor, E. O. Kharbanda.

    Drafting of the manuscript: A. B. Kharbanda, Vazquez-Benitez, Ballard, Ekstrom, Warton.

    Critical revision of the manuscript for important intellectual content: Vazquez-Benitez, Ballard, Vinson, Chettipally, Dehmer, Rauchwerger, McMichael, Cotton, Kene, Simon, Zhu, O’Connor, E. O. Kharbanda.

    Statistical analysis: A. B. Kharbanda, Vazquez-Benitez, Dehmer, Cotton, Warton.

    Obtained funding: A. B. Kharbanda, Ballard, Vinson, Rauchwerger, E. O. Kharbanda.

    Administrative, technical, or material support: Ballard, Chettipally, Ekstrom, Rauchwerger, McMichael, Cotton, Kene, Simon, Zhu.

    Supervision: A. B. Kharbanda, Ballard, Rauchwerger, E. O. Kharbanda.

    Conflict of Interest Disclosures: Ms Ekstrom reported receiving grants from HealthPartners Institute during the conduct of the study. Ms McMichael reported receiving grants from the National Institutes of Health outside the submitted work. Dr Cotton reported grants from Kaiser Permanente during the conduct of the study. No other disclosures were reported.

    Funding/Support: This work was supported by grant R01HD079463 from the National Institute of Child Health and Human Development.

    Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Clinical Research on Emergency Services and Treatments Network (CREST): Site principal investigators: Dustin W. Ballard, MD, MBE, Uli K. Chettipally, MD, MPH, Dale M. Cotton, MD, Mamata V. Kene, MD, MPH, James S. Lin, MD, Dustin G. Mark, MD, D. Ian McLachlan, MD, MPH, Dana R. Sax, MD, MPH, David R. Vinson, MD. Statistical support: Mary E. Reed, DrPH, E. Margaret Warton, MPH. Project management: Adina. S. Rauchwerger, MPH, Laura E. Simon, BA (Kaiser Permanente Northern California Division of Research). Monetary support was provided to several of the study team members.

    Critical Care Research Center, HealthPartners Institute: Site leads: Mike Zwank, MD, Bjorn Peterson, MD, Levon Ohaodha, MD, Sandi Wewerka, RN. Advisors: Brad Gordon, MD, Kurt Isenburger, MD, Dave Dries, MD, Martin Richards, MD, Azhar Ali, MD. Monetary support was provided to several of the study team members.

    Data Sharing Statement: See Supplement 3.

    Additional Contributions: We would like to thank the site investigators, clinical leaders, champions, research managers and coordinators from the following networks for their assistance with developing care recommendations, workflow assessment, training, recruitment, and/or data collection. We would like to thank the Center for Acute Care Outcomes, Children’s Minnesota, David J. Schmeling, MD, and William Mize, MD. We would also like to thank our mentors and advisors for their guidance and support in developing this intervention: Nathan Kuppermann, MD, MPH (University of California, Davis School of Medicine), and Peter S. Dayan, MD, MSc (Columbia University). Each received a minimal stipend to participate in the study.

    References
    1.
    Barrett  ML, Hines  AL, Andrews  RM. Trends in rates of perforated appendix, 2001-2010: Statistical Brief #159. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality; 2006. Accessed January 5, 2021. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb159.pdf
    2.
    Colvin  JM, Bachur  R, Kharbanda  A.  The presentation of appendicitis in preadolescent children.   Pediatr Emerg Care. 2007;23(12):849-855. doi:10.1097/pec.0b013e31815c9d7fPubMedGoogle ScholarCrossref
    3.
    Anderson  KT, Bartz-Kurycki  MA, Austin  MT,  et al.  Hospital type predicts computed tomography use for pediatric appendicitis.   J Pediatr Surg. 2019;54(4):723-727. doi:10.1016/j.jpedsurg.2018.05.018PubMedGoogle ScholarCrossref
    4.
    Raval  MV, Deans  KJ, Rangel  SJ, Kelleher  KJ, Moss  RL.  Factors associated with imaging modality choice in children with appendicitis.   J Surg Res. 2012;177(1):131-136. doi:10.1016/j.jss.2012.03.044PubMedGoogle ScholarCrossref
    5.
    Shah  SR, Sinclair  KA, Theut  SB, Johnson  KM, Holcomb  GW  III, St Peter  SD.  Computed tomography utilization for the diagnosis of acute appendicitis in children decreases with a diagnostic algorithm.   Ann Surg. 2016;264(3):474-481. doi:10.1097/SLA.0000000000001867PubMedGoogle ScholarCrossref
    6.
    Warner  BW, Kulick  RM, Stoops  MM, Mehta  S, Stephan  M, Kotagal  UR.  An evidenced-based clinical pathway for acute appendicitis decreases hospital duration and cost.   J Pediatr Surg. 1998;33(9):1371-1375. doi:10.1016/S0022-3468(98)90010-0PubMedGoogle ScholarCrossref
    7.
    Warner  BW, Rich  KA, Atherton  H, Andersen  CL, Kotagal  UR.  The sustained impact of an evidenced-based clinical pathway for acute appendicitis.   Semin Pediatr Surg. 2002;11(1):29-35. doi:10.1053/spsu.2002.29364PubMedGoogle ScholarCrossref
    8.
    Saucier  A, Huang  EY, Emeremni  CA, Pershad  J.  Prospective evaluation of a clinical pathway for suspected appendicitis.   Pediatrics. 2014;133(1):e88-e95. doi:10.1542/peds.2013-2208PubMedGoogle ScholarCrossref
    9.
    Wagenaar  AE, Tashiro  J, Wang  B,  et al.  Protocol for suspected pediatric appendicitis limits computed tomography utilization.   J Surg Res. 2015;199(1):153-158. doi:10.1016/j.jss.2015.04.028PubMedGoogle ScholarCrossref
    10.
    Bachur  RG, Levy  JA, Callahan  MJ, Rangel  SJ, Monuteaux  MC.  Effect of reduction in the use of computed tomography on clinical outcomes of appendicitis.   JAMA Pediatr. 2015;169(8):755-760. doi:10.1001/jamapediatrics.2015.0479PubMedGoogle ScholarCrossref
    11.
    Niles  LM, Goyal  MK, Badolato  GM, Chamberlain  JM, Cohen  JS.  US emergency department trends in imaging for pediatric nontraumatic abdominal pain.   Pediatrics. 2017;140(4):e20170615. doi:10.1542/peds.2017-0615PubMedGoogle Scholar
    12.
    Otero  HJ, Crowder  L.  Imaging utilization for the diagnosis of appendicitis in stand-alone children’s hospitals in the United States: trends and costs.   J Am Coll Radiol. 2017;14(5):603-608. doi:10.1016/j.jacr.2016.12.013PubMedGoogle ScholarCrossref
    13.
    Santillanes  G, Simms  S, Gausche-Hill  M,  et al.  Prospective evaluation of a clinical practice guideline for diagnosis of appendicitis in children.   Acad Emerg Med. 2012;19(8):886-893. doi:10.1111/j.1553-2712.2012.01402.xPubMedGoogle ScholarCrossref
    14.
    Depinet  H, von Allmen  D, Towbin  A, Hornung  R, Ho  M, Alessandrini  E.  Risk stratification to decrease unnecessary diagnostic imaging for acute appendicitis.   Pediatrics. 2016;138(3):e20154031. doi:10.1542/peds.2015-4031PubMedGoogle Scholar
    15.
    Kharbanda  AB, Madhok  M, Krause  E,  et al.  Implementation of electronic clinical decision support for pediatric appendicitis.   Pediatrics. 2016;137(5):e20151745. doi:10.1542/peds.2015-1745PubMedGoogle Scholar
    16.
    Gurien  LA, Smith  SD, Dassinger  MS, Burford  JM, Tepas  JJ, Crandall  M.  Suspected appendicitis pathway continues to lower CT rates in children two years after implementation.   Am J Surg. 2019;218(4):716-721. doi:10.1016/j.amjsurg.2019.07.024PubMedGoogle ScholarCrossref
    17.
    Binkovitz  LA, Unsdorfer  KM, Thapa  P,  et al.  Pediatric appendiceal ultrasound: accuracy, determinacy and clinical outcomes.   Pediatr Radiol. 2015;45(13):1934-1944. doi:10.1007/s00247-015-3432-7PubMedGoogle ScholarCrossref
    18.
    Schuh  S, Chan  K, Langer  JC,  et al.  Properties of serial ultrasound clinical diagnostic pathway in suspected appendicitis and related computed tomography use.   Acad Emerg Med. 2015;22(4):406-414. doi:10.1111/acem.12631PubMedGoogle ScholarCrossref
    19.
    Kotagal  M, Richards  MK, Flum  DR, Acierno  SP, Weinsheimer  RL, Goldin  AB.  Use and accuracy of diagnostic imaging in the evaluation of pediatric appendicitis.   J Pediatr Surg. 2015;50(4):642-646. doi:10.1016/j.jpedsurg.2014.09.080PubMedGoogle ScholarCrossref
    20.
    McDermott  KW, Stocks  C, Freeman  WJ. Overview of pediatric emergency department visits, 2015: Statistical Brief #242. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality; 2018.
    21.
    Kharbanda  AB, Vazquez-Benitez  G, Ballard  DW,  et al.  Development and validation of a novel pediatric appendicitis risk calculator (pARC).   Pediatrics. 2018;141(4):e20172699. doi:10.1542/peds.2017-2699PubMedGoogle Scholar
    22.
    Ekstrom  HL, Kharbanda  EO, Ballard  DW,  et al.  Development of a clinical decision support system for pediatric abdominal pain in emergency department settings across two health systems within the HCSRN.   EGEMS (Wash DC). 2019;7(1):15. doi:10.5334/egems.282PubMedGoogle Scholar
    23.
    Vinson  DR, Mark  DG, Chettipally  UK,  et al; eSPEED Investigators of the KP CREST Network.  Increasing safe outpatient management of emergency department patients with pulmonary embolism: a controlled pragmatic trial.   Ann Intern Med. 2018;169(12):855-865. doi:10.7326/M18-1206PubMedGoogle ScholarCrossref
    24.
    Ballard  DW, Vemula  R, Chettipally  UK,  et al; KP CREST Network Investigators.  Optimizing clinical decision support in the electronic health record: clinical characteristics associated with the use of a decision tool for disposition of ED patients with pulmonary embolism.   Appl Clin Inform. 2016;7(3):883-898. doi:10.4338/ACI-2016-05-RA-0073PubMedGoogle ScholarCrossref
    25.
    Simon  LE, Rauchwerger  AS, Chettipally  UK,  et al.  Text message alerts to emergency physicians identifying potential study candidates increase clinical trial enrollment.   J Am Med Inform Assoc. 2019;26(11):1360-1363. doi:10.1093/jamia/ocz118PubMedGoogle ScholarCrossref
    26.
    HealthPartners. Total Care Relative Resource Value (TCRRVTM): a measurement approach to achieve the triple aim. Updated September 21, 2017. Accessed June 12, 2018. https://www.healthpartners.com/ucm/groups/public/@hp/@public/documents/documents/cntrb_039627.pdf
    27.
    National Quality Forum. NQF endorses resource use measures. Published January 31, 2012. Accessed June 12, 2018. http://www.qualityforum.org/News_And_Resources/Press_Releases/2012/NQF_Endorses_Resource_Use_Measures.aspx
    28.
    Yan  Z, Ip  IK, Raja  AS, Gupta  A, Kosowsky  JM, Khorasani  R.  Yield of CT pulmonary angiography in the emergency department when providers override evidence-based clinical decision support.   Radiology. 2017;282(3):717-725. doi:10.1148/radiol.2016151985PubMedGoogle ScholarCrossref
    29.
    Khairat  S, Marc  D, Crosby  W, Al Sanousi  A.  Reasons for physicians not adopting clinical decision support systems: critical analysis.   JMIR Med Inform. 2018;6(2):e24. doi:10.2196/medinform.8912PubMedGoogle Scholar
    30.
    Young  PJ, Nickson  CP, Perner  A.  When should clinicians act on non-statistically significant results from clinical trials?   JAMA. 2020;323(22):2256-2257. doi:10.1001/jama.2020.3508PubMedGoogle ScholarCrossref
    31.
    O’Connor  SD, Sodickson  AD, Ip  IK,  et al.  Journal club: requiring clinical justification to override repeat imaging decision support: impact on CT use.   AJR Am J Roentgenol. 2014;203(5):W482-90. doi:10.2214/AJR.14.13017PubMedGoogle Scholar
    32.
    Raja  AS, Ip  IK, Dunne  RM, Schuur  JD, Mills  AM, Khorasani  R.  Effects of performance feedback reports on adherence to evidence-based guidelines in use of CT for evaluation of pulmonary embolism in the emergency department: a randomized trial.   AJR Am J Roentgenol. 2015;205(5):936-940. doi:10.2214/AJR.15.14677PubMedGoogle ScholarCrossref
    33.
    Cabana  MD, Rand  CS, Powe  NR,  et al.  Why don’t physicians follow clinical practice guidelines? a framework for improvement.   JAMA. 1999;282(15):1458-1465. doi:10.1001/jama.282.15.1458PubMedGoogle ScholarCrossref
    34.
    Hess  EP, Homme  JL, Kharbanda  AB,  et al.  Effect of the head computed tomography choice decision aid in parents of children with minor head trauma: a cluster randomized trial.   JAMA Netw Open. 2018;1(5):e182430. doi:10.1001/jamanetworkopen.2018.2430PubMedGoogle Scholar
    35.
    Davidson  KW, Ye  S, Mensah  GA.  Commentary: de-implementation science: a virtuous cycle of ceasing and desisting low-value care before implementing new high value care.   Ethn Dis. 2017;27(4):463-468. doi:10.18865/ed.27.4.463PubMedGoogle ScholarCrossref
    36.
    Norton  WE, Chambers  DA.  Unpacking the complexities of de-implementing inappropriate health interventions.   Implement Sci. 2020;15(1):2. doi:10.1186/s13012-019-0960-9PubMedGoogle ScholarCrossref
    37.
    Kharbanda  AB, Dudley  NC, Bajaj  L,  et al; Pediatric Emergency Medicine Collaborative Research Committee of the American Academy of Pediatrics.  Validation and refinement of a prediction rule to identify children at low risk for acute appendicitis.   Arch Pediatr Adolesc Med. 2012;166(8):738-744. doi:10.1001/archpediatrics.2012.490PubMedGoogle ScholarCrossref
    ×