Effect of Cognitive Aids on Adherence to Best Practice in the Treatment of Deteriorating Surgical Patients: A Randomized Clinical Trial in a Simulation Setting | Clinical Decision Support | JAMA Surgery | JAMA Network
[Skip to Navigation]
Sign In
Figure 1.  Cognitive Aid: Cardiac Arrest
Cognitive Aid: Cardiac Arrest

BLS indicates basic life support; CPR, cardiopulmonary resuscitation; IV, intravenous.

Figure 2.  CONSORT Flow Diagram
CONSORT Flow Diagram
Figure 3.  Failure to Adhere to Best Practice Expressed as Percentage of Omitted Critical Management Steps
Failure to Adhere to Best Practice Expressed as Percentage of Omitted Critical Management Steps

aPerforming the scenarios without cognitive aids for the management of deteriorating surgical patients resulted in a significantly higher percentage of omitted critical management steps.

Table 1.  Characteristics of Study Groups
Characteristics of Study Groups
Table 2.  Perceived Usability (Likert Scale: 0, Strongly Disagree, to 4, Strongly Agree)
Perceived Usability (Likert Scale: 0, Strongly Disagree, to 4, Strongly Agree)
1.
Anderson  O, Davis  R, Hanna  GB, Vincent  CA.  Surgical adverse events: a systematic review.  Am J Surg. 2013;206(2):253-262. doi:10.1016/j.amjsurg.2012.11.009PubMedGoogle ScholarCrossref
2.
Ghaferi  AA, Birkmeyer  JD, Dimick  JB.  Hospital volume and failure to rescue with high-risk surgery.  Med Care. 2011;49(12):1076-1081. doi:10.1097/MLR.0b013e3182329b97PubMedGoogle ScholarCrossref
3.
Ghaferi  AA, Osborne  NH, Birkmeyer  JD, Dimick  JB.  Hospital characteristics associated with failure to rescue from complications after pancreatectomy.  J Am Coll Surg. 2010;211(3):325-330. doi:10.1016/j.jamcollsurg.2010.04.025PubMedGoogle ScholarCrossref
4.
Symons  NR, Almoudaris  AM, Nagpal  K, Vincent  CA, Moorthy  K.  An observational study of the frequency, severity, and etiology of failures in postoperative care after major elective general surgery.  Ann Surg. 2013;257(1):1-5. doi:10.1097/SLA.0b013e31826d859bPubMedGoogle ScholarCrossref
5.
Findlay  GP, Goodwin  APL, Protopapa  K, Mason  M. A review of the peri-operative care of surgical patients: a report by the National Confidential Enquiry into Patient Outcome and Death. https://www.ncepod.org.uk/2011report2/downloads/POC_fullreport.pdf. Published 2011. Accessed May 13, 2019.
6.
Garry  DA, McKechnie  SR, Culliford  DJ,  et al; PREVENT group.  A prospective multicentre observational study of adverse iatrogenic events and substandard care preceding intensive care unit admission (PREVENT).  Anaesthesia. 2014;69(2):137-142. doi:10.1111/anae.12535PubMedGoogle ScholarCrossref
7.
Johnston  M, Arora  S, King  D, Stroman  L, Darzi  A.  Escalation of care and failure to rescue: a multicenter, multiprofessional qualitative study.  Surgery. 2014;155(6):989-994. doi:10.1016/j.surg.2014.01.016PubMedGoogle ScholarCrossref
8.
Johnston  M, Arora  S, Anderson  O, King  D, Behar  N, Darzi  A.  Escalation of care in surgery: a systematic risk assessment to prevent avoidable harm in hospitalized patients.  Ann Surg. 2015;261(5):831-838. doi:10.1097/SLA.0000000000000762PubMedGoogle ScholarCrossref
9.
van Galen  LS, Struik  PW, Driesen  BE,  et al.  Delayed recognition of deterioration of patients in general wards is mostly caused by human related monitoring failures: a root cause analysis of unplanned ICU admissions.  PLoS One. 2016;11(8):e0161393. doi:10.1371/journal.pone.0161393PubMedGoogle Scholar
10.
Haynes  AB, Weiser  TG, Berry  WR,  et al; Safe Surgery Saves Lives Study Group.  A surgical safety checklist to reduce morbidity and mortality in a global population.  N Engl J Med. 2009;360(5):491-499. doi:10.1056/NEJMsa0810119PubMedGoogle ScholarCrossref
11.
de Vries  EN, Prins  HA, Crolla  RM,  et al; SURPASS Collaborative Group.  Effect of a comprehensive surgical safety system on patient outcomes.  N Engl J Med. 2010;363(20):1928-1937. doi:10.1056/NEJMsa0911535PubMedGoogle ScholarCrossref
12.
Pronovost  P, Needham  D, Berenholtz  S,  et al.  An intervention to decrease catheter-related bloodstream infections in the ICU.  N Engl J Med. 2006;355(26):2725-2732. doi:10.1056/NEJMoa061115PubMedGoogle ScholarCrossref
13.
Ghaferi  AA, Dimick  JB.  Importance of teamwork, communication and culture on failure-to-rescue in the elderly.  Br J Surg. 2016;103(2):e47-e51. doi:10.1002/bjs.10031PubMedGoogle ScholarCrossref
14.
Lingard  L, Espin  S, Whyte  S,  et al.  Communication failures in the operating room: an observational classification of recurrent types and effects.  Qual Saf Health Care. 2004;13(5):330-334. doi:10.1136/qshc.2003.008425PubMedGoogle ScholarCrossref
15.
Marshall  SD, Sanderson  P, McIntosh  CA, Kolawole  H.  The effect of two cognitive aid designs on team functioning during intra-operative anaphylaxis emergencies: a multi-centre simulation study.  Anaesthesia. 2016;71(4):389-404. doi:10.1111/anae.13332PubMedGoogle ScholarCrossref
16.
Marshall  SD, Mehra  R.  The effects of a displayed cognitive aid on non-technical skills in a simulated ‘can’t intubate, can’t oxygenate’ crisis.  Anaesthesia. 2014;69(7):669-677. doi:10.1111/anae.12601PubMedGoogle ScholarCrossref
17.
Hepner  DL, Arriaga  AF, Cooper  JB,  et al.  Operating room crisis checklists and emergency manuals.  Anesthesiology. 2017;127(2):384-392. doi:10.1097/aln.0000000000001731PubMedGoogle ScholarCrossref
18.
Weller  J, Boyd  M.  Making a difference through improving teamwork in the operating room: a systematic review of the evidence on what works.  Curr Anesthesiol Rep. 2014;4:77-83. doi:10.1007/s40140-014-0050-0Google ScholarCrossref
19.
Sacks  GD, Shannon  EM, Dawes  AJ,  et al.  Teamwork, communication and safety climate: a systematic review of interventions to improve surgical culture.  BMJ Qual Saf. 2015;24(7):458-467. doi:10.1136/bmjqs-2014-003764PubMedGoogle ScholarCrossref
20.
Buist  M, Marshall  S, Shearer  B,  et al.  Getting more efficient Rapid Response System (RRS) utilization by the use of a general ward based deteriorating patient contract.  States Health. 2015;1:8. doi:10.1186/2056-5917-1-8Google Scholar
21.
Arriaga  AF, Bader  AM, Wong  JM,  et al.  Simulation-based trial of surgical-crisis checklists.  N Engl J Med. 2013;368(3):246-253. doi:10.1056/NEJMsa1204720PubMedGoogle ScholarCrossref
22.
Soar  J, Nolan  JP, Böttiger  BW,  et al; Adult advanced life support section Collaborators.  European Resuscitation Council Guidelines for Resuscitation 2015, section 3: adult advanced life support.  Resuscitation. 2015;95:100-147. doi:10.1016/j.resuscitation.2015.07.016PubMedGoogle ScholarCrossref
23.
Perkins  GD, Handley  AJ, Koster  RW,  et al; Adult basic life support and automated external defibrillation section Collaborators.  European Resuscitation Council Guidelines for Resuscitation 2015, section 2: adult basic life support and automated external defibrillation.  Resuscitation. 2015;95:81-99. doi:10.1016/j.resuscitation.2015.07.015PubMedGoogle ScholarCrossref
24.
Truhlář  A, Deakin  CD, Soar  J,  et al; Cardiac arrest in special circumstances section Collaborators.  European Resuscitation Council Guidelines for Resuscitation 2015, Section 4: cardiac arrest in special circumstances.  Resuscitation. 2015;95:148-201. doi:10.1016/j.resuscitation.2015.07.017PubMedGoogle ScholarCrossref
25.
Nolan  JP, Soar  J, Cariou  A,  et al; European Resuscitation Council; European Society of Intensive Care Medicine.  European Resuscitation Council and European Society of Intensive Care Medicine 2015 guidelines for post-resuscitation care.  Intensive Care Med. 2015;41(12):2039-2056. doi:10.1007/s00134-015-4051-3PubMedGoogle ScholarCrossref
26.
Loftus  I.  Care of the Critically Ill Surgical Patient. London, England; The Royal College of Surgeons of England; 2010.
27.
Rhodes  A, Evans  LE, Alhazzani  W,  et al.  Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016.  Intensive Care Med. 2017;43(3):304-377. doi:10.1007/s00134-017-4683-6PubMedGoogle ScholarCrossref
28.
Koers  L, Eveleens  FM, Schlack  WS, Preckel  B.  Cognitive aid for emergencies in the OR: AMC bundle helps ensure that no steps are left out.  Ned Tijdschr Geneeskd. 2015;159:A832.Google Scholar
29.
Stanford Anesthesia Cognitive Aid Group. Emergency Manual: cognitive aids for perioperative critical events. http://emergencymanual.stanford.edu. Published 2016. Accessed May 13, 2019.
30.
Goldhaber-Fiebert  SN, Howard  SK.  Implementing emergency manuals: can cognitive aids help translate best practices for patient care during acute events?  Anesth Analg. 2013;117(5):1149-1161. doi:10.1213/ANE.0b013e318298867aPubMedGoogle ScholarCrossref
31.
Burden  AR, Carr  ZJ, Staman  GW, Littman  JJ, Torjman  MC.  Does every code need a “reader?” improvement of rare event management with a cognitive aid “reader” during a simulated emergency: a pilot study.  Simul Healthc. 2012;7(1):1-9. doi:10.1097/SIH.0b013e31822c0f20PubMedGoogle ScholarCrossref
32.
Ziewacz  JE, Arriaga  AF, Bader  AM,  et al.  Crisis checklists for the operating room: development and pilot testing.  J Am Coll Surg. 2011;213(2):212-217.e10. doi:10.1016/j.jamcollsurg.2011.04.031PubMedGoogle ScholarCrossref
33.
Harrison  TK, Manser  T, Howard  SK, Gaba  DM.  Use of cognitive aids in a simulated anesthetic crisis.  Anesth Analg. 2006;103(3):551-556. doi:10.1213/01.ane.0000229718.02478.c4PubMedGoogle ScholarCrossref
34.
Neal  JM, Hsiung  RL, Mulroy  MF, Halpern  BB, Dragnich  AD, Slee  AE.  ASRA checklist improves trainee performance during a simulated episode of local anesthetic systemic toxicity.  Reg Anesth Pain Med. 2012;37(1):8-15. doi:10.1097/AAP.0b013e31823d825aPubMedGoogle ScholarCrossref
35.
Pucher  PH, Aggarwal  R, Almond  MH, Darzi  A.  Surgical care checklists to optimize patient care following postoperative complications.  Am J Surg. 2015;210(3):517-525. doi:10.1016/j.amjsurg.2015.03.018PubMedGoogle ScholarCrossref
36.
Arriaga  AF, Lancaster  RT, Berry  WR,  et al.  The better colectomy project: association of evidence-based best-practice adherence rates to outcomes in colorectal surgery.  Ann Surg. 2009;250(4):507-513. doi:10.1097/SLA.0b013e3181b672bcPubMedGoogle Scholar
37.
Pucher  PH, Aggarwal  R, Qurashi  M, Singh  P, Darzi  A.  Randomized clinical trial of the impact of surgical ward-care checklists on postoperative care in a simulated environment.  Br J Surg. 2014;101(13):1666-1673. doi:10.1002/bjs.9654PubMedGoogle ScholarCrossref
38.
Gardner-Thorpe  J, Love  N, Wrightson  J, Walsh  S, Keeling  N.  The value of Modified Early Warning Score (MEWS) in surgical in-patients: a prospective observational study.  Ann R Coll Surg Engl. 2006;88(6):571-575. doi:10.1308/003588406X130615PubMedGoogle ScholarCrossref
39.
Ludikhuize  J, Brunsveld-Reinders  AH, Dijkgraaf  MG,  et al; Cost and Outcomes of Medical Emergency Teams Study Group.  Outcomes associated with the nationwide introduction of rapid response systems in the netherlands.  Crit Care Med. 2015;43(12):2544-2551. doi:10.1097/CCM.0000000000001272PubMedGoogle ScholarCrossref
40.
Koers  L, Stevens  MF, Eveleens  FM, Schlack  WS, Preckel  B.  Development of a paediatric emergency manual.  Eur J Anaesthesiol. 2017;34(55):129.Google Scholar
41.
Marshall  S.  The use of cognitive aids during emergencies in anesthesia: a review of the literature.  Anesth Analg. 2013;117(5):1162-1171. doi:10.1213/ANE.0b013e31829c397bPubMedGoogle ScholarCrossref
42.
Harvey  R, Foulds  L, Housden  T,  et al.  The impact of didactic read-aloud action cards on the performance of cannula cricothyroidotomy in a simulated ‘can’t intubate can’t oxygenate’ scenario.  Anaesthesia. 2017;72(3):343-349. doi:10.1111/anae.13643PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    November 27, 2019

    Effect of Cognitive Aids on Adherence to Best Practice in the Treatment of Deteriorating Surgical Patients: A Randomized Clinical Trial in a Simulation Setting

    Author Affiliations
    • 1Department of Anaesthesia, University of Amsterdam, Amsterdam, the Netherlands
    • 2Department of Critical Care, University of Amsterdam, Amsterdam, the Netherlands
    • 3Department of Surgery, University of Amsterdam, Amsterdam, the Netherlands
    JAMA Surg. 2020;155(1):e194704. doi:10.1001/jamasurg.2019.4704
    Key Points

    Question  Would cognitive aids improve adherence to best practice in the treatment of deteriorating postoperative surgical patients?

    Findings  In this randomized clinical trial, the use of cognitive aids decreased the omissions of critical management steps in the management of deteriorating surgical patients by 70%. In a multivariate analysis, the use of cognitive aids was the only significant factor in reducing the omission of critical management steps.

    Meaning  Cognitive aids for the management of deteriorating surgical patients have the potential to reduce failure to rescue and hereby improve patient outcomes.

    Abstract

    Importance  Failure to rescue causes significant morbidity and mortality in the surgical population. Human error is often the underlying cause of failure to rescue. Human error can be reduced by the use of cognitive aids.

    Objectives  To test the effectiveness of cognitive aids on adherence to best practice in the management of deteriorating postoperative surgical ward patients.

    Design, Setting, and Participants  Randomized clinical trial in a simulation setting. Surgical teams consisted of 1 surgeon and 2 nurses from a surgical ward from 4 different hospitals in Amsterdam, the Netherlands. Data were analyzed between February 2, 2017, and December 18, 2018.

    Interventions  The teams were randomized to manage 3 simulated deteriorating patient scenarios with or without the use of cognitive aids.

    Main Outcomes and Measures  The primary outcome of the study was failure to adhere to best practice, expressed as the percentage of omitted critical management steps. The secondary outcome of the study was the perceived usability of the cognitive aids.

    Results  Of the total participants, 93 were women and 51 were men. Twenty-five surgical teams performed 75 patient scenarios with cognitive aids, and 25 teams performed 75 patient scenarios without cognitive aids. Using the cognitive aids resulted in a reduction of omitted critical management steps from 33% to 10%, which is a 70% (P < .001) reduction. This effect remained significant (odds ratio, 0.63; 95% CI, −0.228 to −0.061; P = .001) in a multivariate analysis. Overall usability (scale of 0-10) of the cognitive aids was scored at a median of 8.7 (interquartile range, 8-9).

    Conclusions and Relevance  Failure to comply with best practice management of postoperative complications is associated with worse outcomes. In this simulation study, adherence to best practice in the management of postoperative complications improves significantly by the use of cognitive aids. Cognitive aids for deteriorating surgical patients therefore have the potential to reduce failure to rescue and improve patient outcome.

    Trial Registration  ClinicalTrials.gov identifier: NCT03812861

    Introduction

    Human failure in the postoperative trajectory must be addressed to improve outcomes in the surgical population. Diagnostic errors and delayed treatment of complications have been shown to cause surgical adverse events more than 3 times as often as surgical errors.1 The inability to effectively recognize and treat patients who develop complications has been termed failure to rescue.2,3 Misapplication of the early warning score, incorrect monitoring, failure to recognize a deteriorating patient, delays in seeking senior advice, delays in diagnostics, and delays in adequate management or inadequate resuscitation have all been identified as causes for failure to rescue.1,4-9 Human factors play a pivotal role in failure to rescue. By decreasing human failure in the complex environment of modern surgical medicine, cognitive aids have been shown to significantly reduce perioperative morbidity and mortality.10-12 Cognitive aids target all 3 key domains associated with the timely recognition and effective management of complications in the surgical population13: they improve communication,14,15 teamwork and leadership,15-17 and the surgical safety culture17-19 and are therefore likely to be effective in decreasing failure to rescue. More specifically, they accelerate escalation of care20 and optimize resuscitation by reducing the amount of omitted critical treatment steps.21

    To our knowledge, there are no cognitive aids for the management of deteriorating surgical patients. We hypothesized that cognitive aids would improve adherence to best practice in the management of deteriorating surgical patients, which was tested in a simulation setting.

    Methods

    A randomized clinical trial was performed comparing adherence to best practice in the management of deteriorating surgical patients in the ward with and without the use of cognitive aids. After review of the trial protocol, the medical ethics review committee of the Academic Medical Center Amsterdam declared the Act of Medical Research Involving Human Subjects (WMO) not applicable to this study (WMO W16_209#16.245), and need to comply with this legislation was therefore waived. Participants were surgeons and nurses working in surgical departments (general surgery, gynecologic surgery, and urology) in 2 tertiary teaching hospitals (Academic Medical Center Amsterdam and Vrije Universiteit Medical Center, Amsterdam, the Netherlands) and 2 general Dutch hospitals (Flevo Ziekenhuis, Almere, the Netherlands, and Tergooi Ziekenhuis, Hilversum, the Netherlands). Physicians and nurses of all levels of experience were included and were contacted by email to participate on a voluntary basis without compensation. Teams were formed on the basis of availability of the health care personnel. All participants gave written informed consent. Consolidated Standards of Reporting Trials (CONSORT) guidelines were followed to report this study. The formal trial protocols can be found in Supplement 1.

    Design of the Cognitive Aids

    The content of the cognitive aids for the management of deteriorating surgical patients (CAMDS) was created within an interdisciplinary expert team of anesthetic, critical care, and surgical staff of both physicians and nurses. First, conditions most associated with poor outcomes in the postoperative period were identified from the literature. Guidelines, (inter)national protocols, and relevant literature for these conditions were reviewed and incorporated in the cognitive aids.22-27 The cognitive aids were reviewed and revised by the expert panel until full consensus was reached regarding the medical accuracy and context appropriateness of the CAMDS. To minimize following the wrong algorithm when the CAMDS were used, the subject per cognitive aids was chosen to be about a symptom or a syndrome rather than a specific diagnosis. The final CAMDS manual consists of 16 symptom-specific cognitive aids (ie, airway problem, shortness of breath, allergic reaction, chest pain, hypotension, infection, and sepsis) and 6 general algorithms (ie, ABCD approach, blood gas, and electrocardiogram analysis). A sample of the cardiac arrest algorithm is shown in Figure 1. The user-centered design principles that were incorporated in the cognitive aids were taken from the Dutch cognitive aids for emergencies in the operating room that were created by the authors in collaboration with the Stanford Anesthesia Cognitive Aid Group as described previously.28-30 The cognitive aids for emergencies in the operating theater have been extensively used in both high-fidelity simulation as well as in clinical practice since 2013 in the Academic Medical Center Amsterdam. The CAMDS were tested for usability and face validity by a group of surgeons and nurses, and some minor changes were made after this process. No official pilot study was conducted prior to the start of the study; rather, it was concluded to proceed with the study on the basis of a feasibility review that was carried out after the first 6 sessions. Items that were included in the feasibility review were possibility to adhere to standardization of the scenarios, timeframe to conduct scenarios, interpretation of the patient brief, and quality of video recordings. The CAMDS are in A4 format, bundled, and have color coded tabs to facilitate navigation to the correct algorithm (eFigure in Supplement 2).

    Setting and Location

    The study was performed in a simulation laboratory with a standardized high-fidelity surgical ward setting. The simulation laboratory is equipped with 3 cameras and microphones. The study sessions were digitally recorded for data acquisition within Laerdal SimView (Laerdal USA). Three certified simulation operators were in charge of the simulation session and the Laerdal SimMan 3G mannequin (Laerdal USA). Ten preprogrammed standardized deteriorating surgical patient scenarios were created for the study: pneumonia with respiratory failure, pneumothorax, anaphylactic shock, postoperative bleeding, bradycardia, cardiac arrest caused by ventricular fibrillation, cardiac arrest caused by asystole, myocardial infarction, sepsis, and loss of consciousness.

    All participants were novices in the use of cognitive aids. First, a standardized video introduction was given about the aim of the study and the use of cognitive aids. Subsequently, participants underwent a familiarization with the cognitive aids, the simulation laboratory in ward setting, the SimMan 3G mannequin, materials that could be used during the session (eg, crash cart and its content, medication, oxygen masks, blood drawing materials, and intravenous fluids), and use of the telephone for requesting the resuscitation or rapid response team, radiographs, and electrocardiograms. Teams were encouraged to allocate a reader31 (someone to read the applicable algorithm out loud) in case they would be allocated to use the cognitive aids. Familiarization ended when participants felt confident to start with study sessions. Randomization occurred after study introduction and simulation laboratory familiarization to rule out biased teaching during the introduction and familiarization because both staff and participants were not aware of group allocation at this time. Teams were randomized to the intervention or control group and to 3 of 10 scenarios by opening a consecutively numbered (1-50) sealed opaque envelope that contained a computer-generated allocation code for each of the 50 teams. An independent physician had put the codes into envelopes prior to the start of the study. Every team was allocated to a cardiac arrest scenario (either a shockable or nonshockable rhythm), because this was considered a crucial clinical event to validate the CAMDS for, and 2 additional (of the remaining 8) scenarios.

    When a team was randomized to the CAMDS group, the cognitive aids were left in the simulation laboratory in a dedicated place next to the hypothetical patient’s bed. When the team was randomized to perform the session without the CAMDS, the cognitive aids were removed from the simulation laboratory. No comments were made about the use of other resources, such as the participant’s telephones or the available computer that allowed access to the internet and local protocols. Teams were simply instructed to manage the patient as they would normally do. One of the team members (alternately) started the scenario; they were given a written brief consisting of medical history of the patient, date and type of operation the patient underwent, and clinical course up to the present. Participants were told they came for routine recording of vital signs (nurses) or a routine chat with the patient (physician). All teams had to make the diagnosis in the scenario based on the information they had been given in the introduction and the information that they gathered at the bedside from the Laerdal manikin (vital signs displayed when monitoring attached) and additional investigations that they could order. The hypothetical patient could be asked questions when conscious (strict set of scripted answers from which simulation operator worked per scenario) and was able to cough, wheeze, or sweat. When teams called additional specialties for help, eg, the intensive care unit team or the cardiologist, the teams were given the standard answer that that specialist would come to help; they did not receive any additional information from this specialist, nor did the specialist actually come to help during the scenario. The complete simulation session took 2 hours, and teams were given around 10 minutes to complete each scenario.

    For all scenarios, 15 critical management steps were predefined by the expert team based on best practice recommendations from the literature.22-27 Team performance was measured on a total of 45 critical management steps (15 per scenario) across the 3 scenarios. Primary outcome of the study was failure to adhere to best practice, expressed as percentage of omitted critical management steps. Two independent observers scored the team’s performance in adhering to all the management steps on video playback of the recorded sessions. Blinding during scoring of these sessions was not possible owing to the nature of the study. Life support feedback (depth of chest compressions, timeframes during cardiopulmonary resuscitation that no chest compressions were given, adequacy of opening the airway, rescue breaths, and joules used to defibrillate) was recorded from the Session Viewer BLS report (Laerdal USA). Steps were scored in a binary or ternary manner for the critical steps defined within a time frame. For example, in the cardiac arrest scenario, a time frame of 2 minutes was defined for calling for help and attaching the automatic external defibrillator. When a management step was performed during the scenario but not within the indicated time, 1 point was awarded rather than 2 because a patient is likely to benefit from an action even if it is performed beyond the predefined time frame. To correct for learning curve bias, scores from all 3 scenarios per group were combined to assess failure to adhere to best practice. The effect of the individual type of scenario on the primary outcome was assessed in a multivariate analysis, as described in the statistical methods.

    In a postsession survey, participants scored 8 aspects of perceived usability (ease of use, logical order of described management steps, readability, whether the CAMDS provided overview, interrupted treatment, improved treatment, recommendation to use, and suitability for daily use) of the CAMDS on a 5-point Likert scale (0, strongly disagree, to 4, strongly agree). Overall usability was scored on a numerical rating scale from 0 to 10 (0, no use at all, to 10, extremely useful).

    Sample Size Calculation

    Sample size calculation was based on a cluster-randomized design. No previous data on the effectiveness of cognitive aids for deteriorating surgical patients in the ward were available. From data available on the use of cognitive aids in simulated crisis scenarios in the operating theater, we have seen a baseline reduction in omission of critical steps of about 75%.21 We made a more conservative estimate that the correct application of the CAMDS would have a 50% relative reduction in the omission of critical steps. Based on this effect size and an estimated intracluster correlation coefficient within teams of 0.1 and a mean cluster size of 45, with a 2-sided α level of .05 and 80% power, 25 surgical teams per study arm (CAMDS, control), all performing 3 (of 10) scenarios, were needed. No interim analysis was planned.

    Statistical Methods

    Statistical analysis was performed with IBM SPSS Statistics for Windows, version 25.0 (IBM Corp). A P value less than .05 was considered statistically significant. Agreement between 2 observers was assessed with a Cohen κ, and distribution of data was checked using the Kolmogorov-Smirnov test. Data are presented as mean and standard deviation or median with interquartile range (IQR), depending on the distribution of the data. The Mann-Whitney U test was used to assess between-group differences in failure rates of adherence to best practice. In a multivariate regression model including group allocation, the effect of the experience of the participants, number of participants, and type and duration of scenario on failure rate was assessed. Descriptive statistics were used to describe the perceived usability of the CAMDS.

    Results

    Fifty surgical teams were randomized to CAMDS or control group in 150 simulated deteriorating surgical patient cases in sets of 3 per team. The trial ran from February 2017 to December 2018, when the 50th study session was completed. There were a total of 144 participants: 50 physicians (7 consultants, 11 senior registrars, and 32 junior registrars), of whom 42 were from general surgery, 6 from gynecology, and 2 from urology. In addition, 94 nurses participated: 82 from a general surgical ward and 14 from an oncologic gynecologic ward. Characteristics are displayed in Table 1. There were no dropouts. The consort flow diagram is shown in Figure 2.

    During video review, a total of 2250 management steps (15 critical steps in each of 150 scenarios) were scored by 2 observers. Cohen κ for interrater reliability was 0.94 (97.2% agreement). In 64 management steps, there were discrepancies in scoring between the 2 reviewers. In all cases, agreement was achieved after reviewing the recorded session again.

    Adherence to Best Practice

    Using the CAMDS resulted in a significant decrease in the percentage of omitted critical management steps, from 33% (IQR, 22-43) to 10% (IQR, 5-16); P < .001 (Figure 3). This is a 70% reduction of missed steps (absolute risk reduction of 23%). This effect was still observed and was the only significant factor in the multivariate analysis (odds ratio [OR], 0.63; 95% CI, −0.228 to −0.061; P = .001), which included group allocation, experience of the participants, number of participants, and type and duration of scenario. Use of the CAMDS also resulted in a significant decrease in failure to adhere to best practice management for each scenario individually (eTable 2 in Supplement 2). In 8% of the sessions (6 of 75), the cognitive aids were not used despite being available in the scenario (intervention group). Per protocol analysis resulted in a decrease in the percentage of omitted critical management steps from 33% (IQR, 22-42) to 6% (IQR, 5-16; P < .001). This is an 82% reduction of missed steps (absolute risk reduction of 27%). In 1 case (0.6%), the wrong cognitive aid was followed; the cognitive aid with the sepsis algorithm was used in an actual anaphylaxis scenario.

    Perceived Usability

    Table 2 shows the responses to the perceived usability survey. Overall usability (0-10) of the CAMDS was scored at a median of 8.7 (IQR, 8-9). All participants rated the scenarios to be realistic (ie, resembling daily clinical work); 4% neutral (n = 6), 54% agreed (n = 78), and 42% strongly agreed (n = 60).

    Discussion

    Cognitive aids, as a tool to prevent human error, reduced the omissions of critical steps in the management of deteriorating surgical patients in the ward from 33% to 10% (relative reduction of 70%). The baseline omission of 33% of critical management steps found in this study corroborates previously reported incidences between 20% to 50%.21,32-34 A case note review in the care of surgical patients35 also revealed a noncompliance rate in best practice of 86% in the management of postoperative complications. Failure to comply with best practice was found to be the only significant factor in the development of further complications (OR, 6.75; 95% CI, 1.11-41.00) in this study.35 Another study36 also showed that failure to adhere to critical management steps in postoperative care significantly predicted the occurrence of a complication, and that each additional management step missed increased the odds of a postoperative complication by 60% (OR, 1.6; 95% CI, 1.2-2.2).36 Failure to adhere to best practice thus not only occurs in simulation sessions but in clinical practice every day, resulting in preventable morbidity and mortality. It is therefore critical to improve adherence to best practice management.

    Limitations

    A limitation of this study is that it was performed in a simulation setting; the effect of the use of cognitive aids for deteriorating surgical patients on clinical outcome measures is therefore yet to be established. The assessment rubric used to test the intervention was not validated prior to the study. However, this rubric is exclusively composed of 15 recommended management steps that could be scored performed or not, taken from the literature. The rationale to assess 15 management steps for all scenarios was a pragmatic decision for the purpose of the sample size calculation because 15 steps seemed the mean number of critical management steps per scenario during the development of the scoring cards. However, although the assessment tool describes the recommended treatment steps from the literature, not all steps are likely to have the same effect in the real world. Furthermore, it was not possible to blind the judges to the use of cognitive aids in this study. Although this is a potential source of bias, it is unlikely to affect the results of the study because the management steps are unambiguous to score (ie, feedback from the Laerdal manikin or administration of antibiotics; teams were awarded 0 points if they did not administer the antibiotics even if they mentioned that they were going to administer them).

    Another potential limitation of the study is the relatively small number of participating consultants. Because participation was on a voluntary basis, it was more difficult for consultants to find time to participate. This could have introduced bias by deficiency of senior input in a proportion of teams. However, in clinical practice, junior physicians are usually the ones in the frontline and the first to be confronted with a deteriorating patient as well. Strengths of this study are the large number of different teams from different backgrounds, augmenting the generalizability of the results of this study.

    Other trials in the use of cognitive aids for the surgical population also report improved adherence to best practice.13,23,28 Cognitive aids for the management of operating room crises reduce omission of critical management steps with 75% to 83%.13,23 A checklist-based intervention for the postoperative ward round reduced the noncompliance rate with best practice for postoperative complications from 60% to 0%.37 These studies thus show that adherence to best practice, and potentially patient outcome, can be improved significantly by the use of cognitive aids. The modified early warning score is another valuable tool in preventing failure to rescue in the surgical population,38 However, it has been shown that the rapid response team is not called in 70% of cases when modified early warning score criteria for activating the rapid response team for critical care input for a deteriorating patient are actually met.39 Reasons stated are an unclear escalation policy and self-doubt of the attending nurse or physician as well as hierarchical barriers.7,39 A clear escalation policy on the CAMDS, embraced by all the departments involved, can assist in the timely and uniform escalation of patient care and reduce hierarchical barriers. Designing cognitive aids for deteriorating surgical patients will therefore require support from all the departments involved in the care of these patients. Creating local CAMDS will result in reviewing and improving logistics and protocols regarding the care of these patients, and this process can already improve patient safety.40 The CAMDS assist in systematically assessing and treating a patient when a diagnosis is made or major symptom is identified. However, there is a risk of fixation error when a team follows a wrong algorithm. This study revealed an incidence of 0.6% of following the wrong algorithm to treat a patient. The design of a cognitive aid can help in preventing fixation error; users should be prompted to explore a differential diagnosis and test the accuracy of their diagnosis. When designing a cognitive aid, it should be taken into account that bad design and lack of training can potentially cause harm by interfering with teamwork or promoting the wrong action or the wrong sequence of actions.41 A study in the use of a cognitive aid for an emergency surgical airway showed that it took 35.4 seconds longer to achieve oxygenation with a cognitive aid.42 This emphasizes the point that a cognitive aid should be as clear and concise as possible and only critical steps should be incorporated, especially if the cognitive aid is intended for use during emergencies. Users should be familiar with the cognitive aids when used in clinical practice.41 In our study, all participants were novices in the use of cognitive aids. It is likely that with better training and familiarity into the use and the limitations of cognitive aids, the effectiveness of the cognitive aids will be improved. Because none of the teams without the cognitive aids were able to effectively use the resources (internet and local protocol database) from the available computer to manage the scenario, there is a probable benefit of hardcopy manuals close to the bed site, in addition to digital versions. Finally, it needs to be emphasized that cognitive aids are tools to assist medical staff, but they by no means replace the need for professional training and involvement of expert help, eg, from critical care staff.

    Conclusions

    Cognitive aids are tools to improve expert performance. However, despite the potential benefits, the widespread use of cognitive aids in clinical practice is still lacking. This study shows that the use of cognitive aids significantly reduces the number of omitted critical management steps in the treatment of deteriorating patients following surgery. Cognitive aids for deteriorating postoperative patients therefore have the potential to reduce failure to rescue and improve patient outcome. Further research should focus on identifying and targeting barriers to the use of cognitive aids and how to optimize their use in clinical practice.

    Back to top
    Article Information

    Corresponding Author: Jeroen Hermanides, MD, PhD, Department of Anaesthesia, Amsterdam UMC, location AMC, University of Amsterdam, Meibergdreef 9, 1105 AZ, Amsterdam, the Netherlands (j.hermanides@amc.uva.nl).

    Accepted for Publication: September 8, 2019.

    Published Online: November 27, 2019. doi:10.1001/jamasurg.2019.4704

    Author Contributions: Dr Koers had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Koers, van Haperen, Meijer, Waller, Boermeester, Preckel.

    Acquisition, analysis, or interpretation of data: Koers, Meijer, van Wandelen, Waller, Dongelmans, Boermeester, Hermanides, Preckel.

    Drafting of the manuscript: Koers, Meijer, Preckel.

    Critical revision of the manuscript for important intellectual content: Koers, van Haperen, van Wandelen, Waller, Dongelmans, Boermeester, Hermanides, Preckel.

    Statistical analysis: Koers, Hermanides.

    Obtained funding: Koers, Boermeester, Preckel.

    Administrative, technical, or material support: Koers, van Haperen, Meijer, van Wandelen, Waller, Preckel.

    Supervision: Koers, Boermeester, Hermanides, Preckel.

    Conflict of Interest Disclosures: Dr Boermeester reported grants from KCI, Bard, LifeCell, New Compliance, and Mylan outside the submitted work. Dr Preckel reported grants from Academic Medical Center Amsterdam during the conduct of the study and personal fees from Laboratoire Aguettant, France, and Sensium Healthcare UK outside the submitted work. No other disclosures were reported.

    Funding/Support: This study was supported by a local innovation grant from the Amsterdam University Medical Centre.

    Role of the Funder/Sponsor: The funding organization had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.

    Data Sharing Statement: See Supplement 3.

    Additional Contributions: We thank Kevin Smit, Registered Nurse Anesthetist, Department of Anesthesiology, Amsterdam UMC, and Bart Kroon, Registered Nurse Anesthetist, Department of Anesthesiology, Amsterdam UMC, for their help with running the scenarios. We also the Stanford Anesthesia Cognitive Aid group, in particular Larry Chu, MD, MS, Perioperative and Pain Medicine and Director of the Stanford Anesthesia Informatics and Media Lab, Stanford University, Stanford, California, and Sarah Goldhaber Fiebert, MD, Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Stanford, California, for sharing their knowledge and expertise in designing and implementing cognitive aids. No compensation was received from a funding sponsor for these contributions.

    References
    1.
    Anderson  O, Davis  R, Hanna  GB, Vincent  CA.  Surgical adverse events: a systematic review.  Am J Surg. 2013;206(2):253-262. doi:10.1016/j.amjsurg.2012.11.009PubMedGoogle ScholarCrossref
    2.
    Ghaferi  AA, Birkmeyer  JD, Dimick  JB.  Hospital volume and failure to rescue with high-risk surgery.  Med Care. 2011;49(12):1076-1081. doi:10.1097/MLR.0b013e3182329b97PubMedGoogle ScholarCrossref
    3.
    Ghaferi  AA, Osborne  NH, Birkmeyer  JD, Dimick  JB.  Hospital characteristics associated with failure to rescue from complications after pancreatectomy.  J Am Coll Surg. 2010;211(3):325-330. doi:10.1016/j.jamcollsurg.2010.04.025PubMedGoogle ScholarCrossref
    4.
    Symons  NR, Almoudaris  AM, Nagpal  K, Vincent  CA, Moorthy  K.  An observational study of the frequency, severity, and etiology of failures in postoperative care after major elective general surgery.  Ann Surg. 2013;257(1):1-5. doi:10.1097/SLA.0b013e31826d859bPubMedGoogle ScholarCrossref
    5.
    Findlay  GP, Goodwin  APL, Protopapa  K, Mason  M. A review of the peri-operative care of surgical patients: a report by the National Confidential Enquiry into Patient Outcome and Death. https://www.ncepod.org.uk/2011report2/downloads/POC_fullreport.pdf. Published 2011. Accessed May 13, 2019.
    6.
    Garry  DA, McKechnie  SR, Culliford  DJ,  et al; PREVENT group.  A prospective multicentre observational study of adverse iatrogenic events and substandard care preceding intensive care unit admission (PREVENT).  Anaesthesia. 2014;69(2):137-142. doi:10.1111/anae.12535PubMedGoogle ScholarCrossref
    7.
    Johnston  M, Arora  S, King  D, Stroman  L, Darzi  A.  Escalation of care and failure to rescue: a multicenter, multiprofessional qualitative study.  Surgery. 2014;155(6):989-994. doi:10.1016/j.surg.2014.01.016PubMedGoogle ScholarCrossref
    8.
    Johnston  M, Arora  S, Anderson  O, King  D, Behar  N, Darzi  A.  Escalation of care in surgery: a systematic risk assessment to prevent avoidable harm in hospitalized patients.  Ann Surg. 2015;261(5):831-838. doi:10.1097/SLA.0000000000000762PubMedGoogle ScholarCrossref
    9.
    van Galen  LS, Struik  PW, Driesen  BE,  et al.  Delayed recognition of deterioration of patients in general wards is mostly caused by human related monitoring failures: a root cause analysis of unplanned ICU admissions.  PLoS One. 2016;11(8):e0161393. doi:10.1371/journal.pone.0161393PubMedGoogle Scholar
    10.
    Haynes  AB, Weiser  TG, Berry  WR,  et al; Safe Surgery Saves Lives Study Group.  A surgical safety checklist to reduce morbidity and mortality in a global population.  N Engl J Med. 2009;360(5):491-499. doi:10.1056/NEJMsa0810119PubMedGoogle ScholarCrossref
    11.
    de Vries  EN, Prins  HA, Crolla  RM,  et al; SURPASS Collaborative Group.  Effect of a comprehensive surgical safety system on patient outcomes.  N Engl J Med. 2010;363(20):1928-1937. doi:10.1056/NEJMsa0911535PubMedGoogle ScholarCrossref
    12.
    Pronovost  P, Needham  D, Berenholtz  S,  et al.  An intervention to decrease catheter-related bloodstream infections in the ICU.  N Engl J Med. 2006;355(26):2725-2732. doi:10.1056/NEJMoa061115PubMedGoogle ScholarCrossref
    13.
    Ghaferi  AA, Dimick  JB.  Importance of teamwork, communication and culture on failure-to-rescue in the elderly.  Br J Surg. 2016;103(2):e47-e51. doi:10.1002/bjs.10031PubMedGoogle ScholarCrossref
    14.
    Lingard  L, Espin  S, Whyte  S,  et al.  Communication failures in the operating room: an observational classification of recurrent types and effects.  Qual Saf Health Care. 2004;13(5):330-334. doi:10.1136/qshc.2003.008425PubMedGoogle ScholarCrossref
    15.
    Marshall  SD, Sanderson  P, McIntosh  CA, Kolawole  H.  The effect of two cognitive aid designs on team functioning during intra-operative anaphylaxis emergencies: a multi-centre simulation study.  Anaesthesia. 2016;71(4):389-404. doi:10.1111/anae.13332PubMedGoogle ScholarCrossref
    16.
    Marshall  SD, Mehra  R.  The effects of a displayed cognitive aid on non-technical skills in a simulated ‘can’t intubate, can’t oxygenate’ crisis.  Anaesthesia. 2014;69(7):669-677. doi:10.1111/anae.12601PubMedGoogle ScholarCrossref
    17.
    Hepner  DL, Arriaga  AF, Cooper  JB,  et al.  Operating room crisis checklists and emergency manuals.  Anesthesiology. 2017;127(2):384-392. doi:10.1097/aln.0000000000001731PubMedGoogle ScholarCrossref
    18.
    Weller  J, Boyd  M.  Making a difference through improving teamwork in the operating room: a systematic review of the evidence on what works.  Curr Anesthesiol Rep. 2014;4:77-83. doi:10.1007/s40140-014-0050-0Google ScholarCrossref
    19.
    Sacks  GD, Shannon  EM, Dawes  AJ,  et al.  Teamwork, communication and safety climate: a systematic review of interventions to improve surgical culture.  BMJ Qual Saf. 2015;24(7):458-467. doi:10.1136/bmjqs-2014-003764PubMedGoogle ScholarCrossref
    20.
    Buist  M, Marshall  S, Shearer  B,  et al.  Getting more efficient Rapid Response System (RRS) utilization by the use of a general ward based deteriorating patient contract.  States Health. 2015;1:8. doi:10.1186/2056-5917-1-8Google Scholar
    21.
    Arriaga  AF, Bader  AM, Wong  JM,  et al.  Simulation-based trial of surgical-crisis checklists.  N Engl J Med. 2013;368(3):246-253. doi:10.1056/NEJMsa1204720PubMedGoogle ScholarCrossref
    22.
    Soar  J, Nolan  JP, Böttiger  BW,  et al; Adult advanced life support section Collaborators.  European Resuscitation Council Guidelines for Resuscitation 2015, section 3: adult advanced life support.  Resuscitation. 2015;95:100-147. doi:10.1016/j.resuscitation.2015.07.016PubMedGoogle ScholarCrossref
    23.
    Perkins  GD, Handley  AJ, Koster  RW,  et al; Adult basic life support and automated external defibrillation section Collaborators.  European Resuscitation Council Guidelines for Resuscitation 2015, section 2: adult basic life support and automated external defibrillation.  Resuscitation. 2015;95:81-99. doi:10.1016/j.resuscitation.2015.07.015PubMedGoogle ScholarCrossref
    24.
    Truhlář  A, Deakin  CD, Soar  J,  et al; Cardiac arrest in special circumstances section Collaborators.  European Resuscitation Council Guidelines for Resuscitation 2015, Section 4: cardiac arrest in special circumstances.  Resuscitation. 2015;95:148-201. doi:10.1016/j.resuscitation.2015.07.017PubMedGoogle ScholarCrossref
    25.
    Nolan  JP, Soar  J, Cariou  A,  et al; European Resuscitation Council; European Society of Intensive Care Medicine.  European Resuscitation Council and European Society of Intensive Care Medicine 2015 guidelines for post-resuscitation care.  Intensive Care Med. 2015;41(12):2039-2056. doi:10.1007/s00134-015-4051-3PubMedGoogle ScholarCrossref
    26.
    Loftus  I.  Care of the Critically Ill Surgical Patient. London, England; The Royal College of Surgeons of England; 2010.
    27.
    Rhodes  A, Evans  LE, Alhazzani  W,  et al.  Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016.  Intensive Care Med. 2017;43(3):304-377. doi:10.1007/s00134-017-4683-6PubMedGoogle ScholarCrossref
    28.
    Koers  L, Eveleens  FM, Schlack  WS, Preckel  B.  Cognitive aid for emergencies in the OR: AMC bundle helps ensure that no steps are left out.  Ned Tijdschr Geneeskd. 2015;159:A832.Google Scholar
    29.
    Stanford Anesthesia Cognitive Aid Group. Emergency Manual: cognitive aids for perioperative critical events. http://emergencymanual.stanford.edu. Published 2016. Accessed May 13, 2019.
    30.
    Goldhaber-Fiebert  SN, Howard  SK.  Implementing emergency manuals: can cognitive aids help translate best practices for patient care during acute events?  Anesth Analg. 2013;117(5):1149-1161. doi:10.1213/ANE.0b013e318298867aPubMedGoogle ScholarCrossref
    31.
    Burden  AR, Carr  ZJ, Staman  GW, Littman  JJ, Torjman  MC.  Does every code need a “reader?” improvement of rare event management with a cognitive aid “reader” during a simulated emergency: a pilot study.  Simul Healthc. 2012;7(1):1-9. doi:10.1097/SIH.0b013e31822c0f20PubMedGoogle ScholarCrossref
    32.
    Ziewacz  JE, Arriaga  AF, Bader  AM,  et al.  Crisis checklists for the operating room: development and pilot testing.  J Am Coll Surg. 2011;213(2):212-217.e10. doi:10.1016/j.jamcollsurg.2011.04.031PubMedGoogle ScholarCrossref
    33.
    Harrison  TK, Manser  T, Howard  SK, Gaba  DM.  Use of cognitive aids in a simulated anesthetic crisis.  Anesth Analg. 2006;103(3):551-556. doi:10.1213/01.ane.0000229718.02478.c4PubMedGoogle ScholarCrossref
    34.
    Neal  JM, Hsiung  RL, Mulroy  MF, Halpern  BB, Dragnich  AD, Slee  AE.  ASRA checklist improves trainee performance during a simulated episode of local anesthetic systemic toxicity.  Reg Anesth Pain Med. 2012;37(1):8-15. doi:10.1097/AAP.0b013e31823d825aPubMedGoogle ScholarCrossref
    35.
    Pucher  PH, Aggarwal  R, Almond  MH, Darzi  A.  Surgical care checklists to optimize patient care following postoperative complications.  Am J Surg. 2015;210(3):517-525. doi:10.1016/j.amjsurg.2015.03.018PubMedGoogle ScholarCrossref
    36.
    Arriaga  AF, Lancaster  RT, Berry  WR,  et al.  The better colectomy project: association of evidence-based best-practice adherence rates to outcomes in colorectal surgery.  Ann Surg. 2009;250(4):507-513. doi:10.1097/SLA.0b013e3181b672bcPubMedGoogle Scholar
    37.
    Pucher  PH, Aggarwal  R, Qurashi  M, Singh  P, Darzi  A.  Randomized clinical trial of the impact of surgical ward-care checklists on postoperative care in a simulated environment.  Br J Surg. 2014;101(13):1666-1673. doi:10.1002/bjs.9654PubMedGoogle ScholarCrossref
    38.
    Gardner-Thorpe  J, Love  N, Wrightson  J, Walsh  S, Keeling  N.  The value of Modified Early Warning Score (MEWS) in surgical in-patients: a prospective observational study.  Ann R Coll Surg Engl. 2006;88(6):571-575. doi:10.1308/003588406X130615PubMedGoogle ScholarCrossref
    39.
    Ludikhuize  J, Brunsveld-Reinders  AH, Dijkgraaf  MG,  et al; Cost and Outcomes of Medical Emergency Teams Study Group.  Outcomes associated with the nationwide introduction of rapid response systems in the netherlands.  Crit Care Med. 2015;43(12):2544-2551. doi:10.1097/CCM.0000000000001272PubMedGoogle ScholarCrossref
    40.
    Koers  L, Stevens  MF, Eveleens  FM, Schlack  WS, Preckel  B.  Development of a paediatric emergency manual.  Eur J Anaesthesiol. 2017;34(55):129.Google Scholar
    41.
    Marshall  S.  The use of cognitive aids during emergencies in anesthesia: a review of the literature.  Anesth Analg. 2013;117(5):1162-1171. doi:10.1213/ANE.0b013e31829c397bPubMedGoogle ScholarCrossref
    42.
    Harvey  R, Foulds  L, Housden  T,  et al.  The impact of didactic read-aloud action cards on the performance of cannula cricothyroidotomy in a simulated ‘can’t intubate can’t oxygenate’ scenario.  Anaesthesia. 2017;72(3):343-349. doi:10.1111/anae.13643PubMedGoogle ScholarCrossref
    ×