Effect of Systematic Physician Cross-checking on Reducing Adverse Events in the Emergency Department: The CHARMED Cluster Randomized Trial

Key Points

Question  Does the implementation of systematic physician cross-checking reduce the rate of adverse events in the emergency department?

Findings  In this cluster randomized trial that included 1680 patients, the implementation of systematic cross-checking between physicians resulted in a significant relative risk reduction for adverse events. The rate of adverse events was 10.7% in the control group vs 6.4% in the cross-checking group.

Meaning  Systematic physician cross-checking may be a key to reducing the high rate of adverse events in the emergency department.

Importance  Emergency departments (ED) are environments that are at high risk for medical errors. Previous studies suggested that the proportion of medical errors may decrease when more than 1 physician is involved.

Objective  To reduce the proportion of medical errors by implementing systematic cross-checking between emergency physicians.

Design, Setting, and Participants  This cluster randomized crossover trial includes a random sample of 14 adult patients (age ≥18 years) per day during two 10-day period in 6 EDs (n = 1680 patients) in France.

Interventions  Systematic cross-checking between emergency physicians, 3 times a day, which included a brief presentation of one physician’s case to another, followed by the second physician’s feedback to the first.

Main Outcomes and Measures  Medical error in the ED, defined as an adverse event (either a near miss or a serious adverse event). The primary end point was identified using a 2-level error detection surveillance system, blinded to the strategy allocation.

Results  Among the 1680 included patients (mean [SD] age, 57.5 [21.7] years), 144 (8.6%) had an adverse event. There were 54 adverse events among 840 patients (6.4%) in the cross-check group compared with 90 adverse events among 840 patients (10.7%) in the standard care group (relative risk reduction [RRR], 40% [95% CI, 12% to 59%]; absolute risk reduction [ARR], 4.3%; number needed to treat [NNT], 24). There was also a significant reduction rate of near misses (RRR, 47% [95% CI, 15% to 67%]; ARR, 2.7%; NNT, 37) but not of the rate of preventable serious adverse events (RRR, 29% [95% CI, −18% to 57%]; ARR, 1.2%; NNT, 83).

Conclusions and Relevance  The implementation of systematic cross-checking between emergency physicians was associated with a significant reduction in adverse events, mainly driven by a reduction in near misses.

Trial Registration  ClinicalTrials.gov Identifier: NCT02356926

Medical errors and adverse events remain a major cause of harm,^1,2 and have been described as the third leading cause of death in the United States.^1-3 In the Emergency department (ED), physicians are required to evaluate multiple patients at the same time and must make rapid decisions with incomplete information. For these reasons, EDs are high-risk environments for errors and subsequent adverse events, with a reported rate of errors up to 10%.^4,5

In the ED, the care of a patient is often managed by a sole physician. This contrasts with other clinical settings in which patients may benefit from multiple ward rounds, staff meetings, and handovers. Although previous studies have suggested that handovers in the ED can result in a greater risk of adverse events,^6-8 it has been reported that transitions of care may actually be associated with improved outcomes. This has been seen in critical care settings with nighttime cross-coverage,⁹ and in pediatric settings after the implementation of a handoff-improvement program.^10,11 In a previous pilot study,⁴ we reported that the involvement of more than 1 physician in the care of the patient, such as handoff between 2 emergency physicians, was associated with a reduced risk of adverse events and medical error. These findings are also in line with high-risk industry settings, where important actions need to be cross-checked by a peer (the closing of aircraft doors for example). Cross-checking between 2 emergency physicians is an opportunity to discuss their patients, receive feedback from their peers, and share the decision-making process.

The purpose of this randomized cluster crossover superiority trial was to test the hypothesis that implementation of systematic cross-checking between 2 emergency physicians would result in a decreased rate of medical errors.

The study methods have been described in detail previously.¹² The CHARMED (cross-checking to reduce adverse events resulting from medical errors in the emergency department) trial was a prospective, cluster randomized, 2-period crossover study in a convenience sample of 6 EDs (6 invited, 0 refused) in France (NCT02356926). Randomization of the order of exposure was performed by an independent statistician. The unit of randomization was the ED. One block of size 6 was used to generate the list. For each number on the list (from 1 to 6), the order of exposure to the intervention was randomly assigned (3 numbers for each order of exposure). Then this randomization list was combined with the blinded list of centers previously numbered.

The trial consisted of 2 periods. Each period lasted 10 days (Monday to Friday for 2 weeks), separated by a 1-month washout period. We chose a cluster crossover design instead of a parallel cluster to limit a “site effect” because we anticipated that the primary end point rate could vary between centers. In the first period, the strategy (intervention or control) was assigned randomly to each center (3 centers intervention, 3 control). After the 1-month washout, the alternative strategy was applied to each center.

The trial was approved by our institutional review board (Comité des Protection de Personnes, Paris île-de-France 6), with a waiver of individual signed informed consent. We followed the CONSORT extension for cluster randomized trial guidelines for the reporting of this study (Figure).¹³ The trial protocol is available in Supplement 1.

In our centers, all emergency patients are managed by a substantively employed senior emergency physician (consultant/attending grade equivalent), with or without a trainee. During weekdays, the number of emergency physicians working in the 6 EDs varies from 4 to 7. During the nights or weekends, 2 to 4 physicians are on call.

During the intervention periods, repeated systematic cross-checking between emergency physicians were implemented. Three times a day, between 8:30 am and 6:00 pm (corresponding to the period of the day-shift), a clinical research technician (CRT) sought all emergency physicians (not including trainees) working in the ED and asked them to meet with a peer (ie, another senior emergency physician) and discuss all his or her current patients. Each physician presented the patients he or she was actually taking care of with a brief description of the medical case. The physicians were not previously trained, and the sessions were standardized, with a written copy of the plan of cross-checking handed by the CRT to the physicians, including the following items⁹: sex, age, chief complaint and main medical history; main clinical findings; main investigation available or outstanding; treatment given in the ED; and a brief summary of the plan.

After presenting each of their patients, the emergency physician sought feedback and comments from his or her peer. In cases where a peer was not available (eg, when there were uneven numbers of emergency physicians present at the time of cross-checking) a trainee was chosen to be the cross-checker. The CRT witnessed the procedure and recorded the number and names of the cross-checked patients, the duration of the procedure, and whether the cross-checker peer was a trainee or not.

We screened all patients that visited the ED during a weekday from 1 of the 2 inclusion periods. We excluded patients that were not in the ED between 8:30 am and 4:00 pm, because they would not have been exposed to the intervention; patients whose care was not provided by an emergency physician (eg, patients with issues pertaining to mental health); scheduled return visits to the ED; and low-acuity patients. This latter group was excluded because their anticipated short length of stay in the ED meant that it would have been unlikely that they would be exposed to the intervention. These were patients referred to the “fast track” or “minor” unit, patients with a triage level 5 on a 1 to 5 scale (5 being the lowest acuity) or patients discharged home less than 1 hour after first contact with the emergency physician.¹⁴ A triage category is assigned to all patients during their initial encounter with a triage nurse.

We needed to include 140 patients in each center for each period (total of 1680). After the completion of each period, we listed all eligible patients for each day through the electronic medical record software (Urqual McKesson). Data were extracted by a third party on the shared electronic database and then randomly ordered using a computer-generated random list (alea function of Microsoft Excel; Microsoft). We then selected the first 14 patients for each of the 10 days in each center for analysis. If a patient could not be analyzed because he or she did not meet the inclusion criteria, we included the next eligible patient on the list of the day.

The primary end point was the rate of patients with an adverse event (preventable or not), defined as a near miss (a medical error that has the potential to cause an adverse event, but did not, either by chance or after an intervention) or a serious adverse event (preventable or not, an injury that might have resulted from medical care or lack thereof).

Before the start of the study, we slightly modified our primary end point that was initially “near miss or serious adverse event” to “near miss or preventable serious adverse event,” because we considered that nonpreventable adverse events could not be defined as a medical error. Unfortunately, the registry (NCT02356926) was not updated subsequently. However the methods overview, published before the start of the study, incorporated this change. For the purpose of transparency, we provide here the result for our initial registry-specified primary end point, and also provide additional results for the modified protocol-specified end point that was mentioned in the method paper.

Secondary end points included the rate of near miss, the rate of serious adverse events, and the rate of preventable serious adverse events. To detect adverse events resulting from medical error that did not occur during the ED stay, any return visit to the ED or new hospital admission in the same institution was sought in the 7 days after the index visit to detect any adverse event subsequent to the ED visit. We used a 2-level review standard methodology to detect medical errors.^5,10,11,15 First, the local investigator, who was an emergency physician, screened all selected charts from his or her center to detect any situation at risk of an adverse event or a near miss (eFigure in Supplement 2). These situations were identified using a questionnaire derived from the National Emergency Department Safety Study.^12,15 All medical records with at least 1 positive item in the questionnaire then underwent blinded assessment by 2 independent experts who were trained in detecting error. The analysis was blinded to the strategy allocation and names of the physicians involved in the patients’ management. In cases of disagreement between the 2 experts (80% agreement; Cohen κ = 0.35; 95% CI, 0.27-0.43), the pair of experts sought reconciliation by discussion. If they could not agree, a tie-breaker third expert was obtained (n = 15 cases [2%]). In the case where several adverse events or near misses were observed in the same patient, the most severe preventable adverse event or near miss was considered.

The experts subsequently adjudicated the preventability of the serious adverse event, and whether it was owing to an emergency physician or to another physician. Then the expert pair classified any medical error according to its severity using the National Coordinating Council on Medication Error and Reporting,¹⁶ a categorization system designed for medication errors, running from B (an error that did not reach the patient) to I (an error that may have contributed to the death of the patient).

To assess whether the first level of reviewing did not miss a significant proportion of medical error, 102 patient medical records were randomly selected (17 in each center), among those that were not screened positive in the first level, and sent to the second level assessment, which was not blinded to the result of the first screening. We defined the first level review reliable if fewer than 2% of patients were found to meet the primary end point. Eventually, only 1 medical record was adjudicated with a near miss (1%; 95% CI, 0%-5%).

Data were expressed as frequency and percentage for categorical variables and as mean (SD) or median (interquartile range [IQR]) for continuous variables. Medical errors rates were compared by a generalized linear mixed model with Poisson distribution, taking into account center as a random effect. Strategy, period, and strategy-by-period interaction were considered as fixed effects. The Cohen κ was used to study agreement between experts. Analyses were performed on an intention-to-treat basis. A per-protocol analysis on patients effectively cross-checked was also performed. All tests were 2-sided and P values <.05 were considered significant.

The P values reported for fixed effects were based on t tests with the denominator degrees of freedom specified using Kenward-Roger approximation. The parameters of the model were estimated using a full maximum likelihood method with adaptive Gaussian quadrature. The model was reduced using backward selection. To explore the robustness of the results, additional analyses using Fisher exact test and the Cochran-Mantel-Haenszel test were performed and an additional sensitivity analysis was performed using logistic random effect model. SAS V.9.3 software (SAS Institute Inc) was used for statistical analyses. The proportion of medical errors was estimated at 10%.^4,5 With an hypothesis of a 40% reduction, (10% control vs 6% cross-checking), 1584 medical records—140 per period in each cluster—were required.¹⁷

A total of 1680 patients (n = 840 in each group) were included. Main characteristics of patients were similar in the 2 groups (Table 1). Characteristics of the recruiting centers are reported in eTables 1 and 2 in Supplement 2. In the first level of medical record review, a total of 818 cases (49%) were identified as being at risk of a medical error and were subjected to the second level review by a pair of experts. Of these, 151 adverse events or near misses were identified in 144 patients (18%), including 9 (1%) nonpreventable adverse events.

An adverse event was identified in 144 (8.6%) patients. There were 54 adverse events among 840 patients (6.4%; 95% CI, 4.9%-8.4%) in the cross-check group compared with 90 adverse events among 840 patients (10.7%; 95% CI, 8.7%-13.0%) in the standard-care group. There was a relative risk reduction (RRR) of 40.0% (95% CI, 12.0%-59.0%) in the rate of adverse events (absolute risk reduction [ARR], 4.3%; number needed to treat [NNT], 24). With the modified protocol-specified primary end point of “preventable serious adverse event or near miss,” there was a similar RRR of 39.0% (95% CI, 10.0%-59.0%; P = .02). A near miss was identified in 75 (4.5%) patients (26 [3.1%; 95% CI, 2.1%-4.6%] in the cross-checking group vs 49 [5.8%; 95% CI, 4.4%-7.7%] in the standard care group; P = .009). There was a RRR of 47.0% (95% CI, 15.0%-67.0%) in the rate of near misses. There was no significant reduction in the rate of patients with a serious adverse event in the 7 days after ED visit, either preventable or not (Table 2).

The analysis of the 69 serious adverse events showed that the management of patients considered septic was prone to error, with 24 adverse events (34%) that were related to a lack of recognition or adequate resuscitation and antibiotic therapy in patients with infections (Table 3). Among the 60 preventable serious adverse events, 4 (7%) occurred after the care of a physician other than an emergency physician (eg, orthopedist or intensivist).

The CRTs recorded 382 cross-checking sessions during the intervention period in the 6 centers, which identified 573 patients (68%) that were cross-checked during the intervention period (median [range], 63% [44%-100%] between the different centers). In 7 cases (2%), physicians could not or refused to attend the session. The median [IQR] duration of the sessions was 9 [5-13] minutes, and a median (IQR) number of 7 (4-9) patients were cross-checked during each session. In 60 cases (16%), the cross-checker was a trainee.

The sequence order of the periods was not associated with a higher risk of medical error, and there was no significant period effect (eTable 4 in Supplement 2). Finally, the additional analyses using Fisher exact test and the Cochran-Mantel-Haenszel test confirmed our results with a risk ratio for the primary end point of 0.57 and 0.61 respectively for each test. Finally, sensitivity analyses using logistic random effect model showed similar results to those obtained with the initial model (preventable adverse events or near misses: RRR = 42%, 95% CI, 17%-60%; P = .003 and adverse events or near misses: RRR = 43%, 95% CI, 19%-60%; P = .002).

Since the cross-checking procedure occurred 3 times per day in the intervention group, some patients may have been treated outside these hours (seen and discharged between 2 cross-checking sessions for example). Indeed, 267 patients (32%) in the intervention groups did not actually receive a cross-checking. The per-protocol analysis found no significant reduction in the rate of medical error for patients that were actually cross-checked (n = 573) vs others (n = 1107), with a RRR of 19% (95% CI, −16% to 43%). However, as presented in eTable 3 in Supplement 2, patients that were cross-checked were older, stayed longer in the ED, and had a more severe presentation.

In this multicenter randomized cluster crossover study, the implementation of systematic cross-checking between emergency physicians was associated with a 40% relative reduction in the rate of adverse events, mostly by reducing the rate of near miss.

We used an active 2-level surveillance system derived from that described in other studies on medical errors.^2,18,19 The observed medical error rate of 10% in the ED, was similar to that reported in the NEDSS study and in our pilot study.^4,5 A less active process of screening for medical errors often resulted in a lower rate of medical errors in the ED, and may be seen as too optimistic.^20-24 The surveillance system for error detection seemed reliable. It sampled 818 of 1680 patients (49%) with a high incidence of medical error (135 of 818 [17%]), and subsequently underwent comprehensive review by a pair of experts, blinded to each other. On the other hand, the remaining half patients that did not undergo a second review seemed to have a very-low rate (1%) of medical error.

The 40% reduction rate in adverse events that is observed is in line with other studies in different settings that assessed the benefit of transition of care and simple interventions aimed at improving its quality.^9-11 The implementation of a handoff bundle was associated with a 30% reduction in the rate of adverse events in a pediatric ward.^10,11 In the ED, such interventions would only expose a small fraction of patients because most of them are seen and discharged or admitted outside of the time designated for handoff.⁴ Systematic cross-checking allows that a higher fraction of patients benefit from the opinion of a second physician. The implementation of checklists was also reported to reduce the rate of medical error.^25-27 However, due to the broad variety of pathology and patients in the ED, it seems difficult to transpose this intervention into the ED setting.

Owing to the retrospective nature of our event analysis, it was difficult to determine the exact nature of the error that led to an adverse event. Whether an adverse event resulted from a misdiagnosis or from the implementation of an inappropriate plan for the correct diagnosis could not be determined decisively. This included sepsis-associated medical errors, which may be particularly relevant, because the most commonly occurring serious adverse events were violations of the surviving sepsis campaign guidelines.²⁸ This is particularly interesting as a recent study²⁹ reported similar results on patients with a short-term unexpected death after an ED visit. Leisman et al³⁰ also recently highlighted the importance of compliance with the sepsis guidelines, with the report of an independent association between mortality and deviation from a sepsis bundle.

It is likely that systematic cross-checking reduced the rate of adverse events errors by allowing a reevaluation of the patient by the peer cross-checker, who is able to cast a fresh eye on the patient’s management. Moreover, the cross-checks may have forced the physician to reconsider his or her initial diagnosis and management plan and, in turn, intercept an error. The involvement of a second physician in the process of care may help intercepting adverse events and near misses, hence the reduction that we observed in our study.^4,9,31

In a per-protocol analysis, we found no significant reduction in the rate of adverse event for patients that were actually cross-checked (n = 573) vs others (n = 1107). This does not contradict our results because the 2 populations were not similar (eTable 3 in Supplement 2): cross-checked patients were sicker than per-protocol standard patients; therefore, confounding factors may have limited the effect of the cross-checking. Moreover, the study was not powered to detect a difference between these 2 groups.

Our study has some limitations. We were unable to determine decisively that the observed benefit was the result of the intervention. We did not account for potential undocumented cross-checking that might have occurred spontaneously between physicians. However we believe that this did not happen often in the control group because this was a novel intervention. The risk of contamination in the second period for centers that were assigned to start with the intervention is limited because the first period was short (10 days) and was followed by a wash-out period. It is possible that some cross-checking occurred in the control group. However, the absence of period effect and sequence order effect advocates for the impact of the intervention. We could not control all confounding variables, for instance the presence of the CRT, which may have led to extra caution during the cross-checking period. Because we did not collect the name of the emergency physicians working in the ED during the study period, it is possible that different physicians staffed the ED in the 2 periods. This could also have created a bias as different physicians may present different risks of medical errors. Furthermore, the wide confidence intervals we provide for our primary end point is compatible with a smaller difference than anticipated.

As we sought for 7 days reattendance in the same hospital, we may have missed some serious adverse events that precipitated a patient having a return visit to another hospital. The observed agreement (κ = 0.35) for the adjudication of the primary end point could be seen as a limitation but is nonetheless considered to be fair agreement.³² Furthermore, it is close to that observed in the study by Starmer et al,¹⁰ which used the same active surveillance process (κ = 0.44), as well as in the study by Camargo et al (κ = 0.34).⁵ In addition, in cases of disagreement between the 2 experts, a consensus was found in 91% of the cases. Another limitation of our study lies in its lack of generalizability, as we only included a convenience sample of 6 EDs, and 5 centers of 6 were in an urban academic hospital in the Paris metropolitan area. Furthermore, the fact that a CRT was present in the ED during the intervention period may have facilitated physician’s adherence to the cross-checks, and further studies are needed to assess the feasibility of this protocol in busy EDs. Finally, our study was underpowered to identify a significant effect on the rate of preventable adverse events, nor was it powered to identify a significant site-level effect.

In this multicenter cluster crossover superiority trial, we report that the implementation of systematic cross-checking between emergency physicians was associated with a significant reduction in adverse events, mostly driven by the reduction of near misses.

Corresponding Author: Yonathan Freund, MD, PhD, Service d’Accueil des Urgences, Hôpital Pitié-Salpêtrière, 83 Blvd de l’hôpital, 75013 Paris, France (yonathan.freund@aphp.fr).

Accepted for Publication: January 26, 2018.

Published Online: April 23, 2018. doi:10.1001/jamainternmed.2018.0607

Author Contributions: Dr Freund had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Freund, Goulet, Rousseau, Simon, Riou.

Acquisition, analysis, or interpretation of data: Freund, Goulet, Leblanc, Bokobza, Ray, Maignan, Guinemer, Truchot, Feral-Pierssens, Yordanov, Philippon, Rouff, Bloom, Cachanado, Rousseau.

Drafting of the manuscript: Freund, Bokobza, Ray, Cachanado, Simon.

Critical revision of the manuscript for important intellectual content: Freund, Goulet, Leblanc, Maignan, Guinemer, Truchot, Feral-Pierssens, Yordanov, Philippon, Rouff, Bloom, Rousseau, Simon, Riou.

Statistical analysis: Freund, Leblanc, Maignan, Cachanado, Rousseau.

Obtained funding: Freund, Simon.

Administrative, technical, or material support: Freund, Goulet, Bokobza, Maignan, Guinemer, Truchot, Yordanov, Philippon, Rouff, Simon.

Study supervision: Freund, Goulet, Truchot, Bloom, Riou.

Conflict of Interest Disclosures: None reported.

Funding/Support: The research was funded and sponsored by Assistance Publique - Hôpitaux de Paris (Département de la Recherche Clinique et du Développement; grant No. CRC13074).

Role of the Funder/Sponsor: The funder/sponsor had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: We thank Sarah Salhi for coordination from the Unité de Recherche Clinique de l’Est Parisien” (Assistance Publique-Hôpitaux de Paris, Université Pierre et Marie Curie, Paris 06) and Hélène Fromentin and Isabelle Peigney from the Centre de Recherche Clinique de l’Est Parisien (Assistance Publique - Hôpitaux de Paris) for their logistical support. We also thank Rabia Mokhtari (CRC URC Est, Paris) and Pr Frederic Adnet (Hôpital Avicenne, Bobigny) for their help and support. No compensation was received by any acknowledged colleagues for their role in the study.

Additional Information: The full dataset, technical appendix and statistical code are available upon request from Dr Freund. Data are anonymized, and there is no identifiable data.