Development of an Online Morbidity, Mortality, and Near-Miss Reporting System to Identify Patterns of Adverse Events in Surgical Patients

Objectives  To design a Web-based system to track adverse and near-miss events, to establish an automated method to identify patterns of events, and to assess the adverse event reporting behavior of physicians.

Design  A Web-based system was designed to collect physician-reported adverse events including weekly Morbidity and Mortality (M&M) entries and anonymous adverse/near-miss events. An automated system was set up to help identify event patterns. Adverse event frequency was compared with hospital databases to assess reporting completeness.

Setting  A metropolitan tertiary care center.

Main Outcome Measures  Identification of adverse event patterns and completeness of reporting.

Results  From September 2005 to August 2007, 15 524 surgical patients were reported including 957 (6.2%) adverse events and 34 (0.2%) anonymous reports. The automated pattern recognition system helped identify 4 event patterns from M&M reports and 3 patterns from anonymous/near-miss reporting. After multidisciplinary meetings and expert reviews, the patterns were addressed with educational initiatives, correction of systems issues, and/or intensive quality monitoring. Only 25% of complications and 42% of inpatient deaths were reported. A total of 75.2% of adverse events resulting in permanent disability or death were attributed to the nature of the disease. Interventions to improve reporting were largely unsuccessful.

Conclusions  We have developed a user-friendly Web-based system to track complications and identify patterns of adverse events. Underreporting of adverse events and attributing the complication to the nature of the disease represent a problem in reporting culture among surgeons at our institution. Similar systems should be used by surgery departments, particularly those affiliated with teaching hospitals, to identify quality improvement opportunities.

The weekly surgical Morbidity and Mortality (M&M) conference has long been recognized as a critical component of surgical education and quality improvement.^1,2 Individual cases are typically presented by a resident, and a discussion ensues addressing the issues in that single case. Recent efforts have attempted to make the discussion more evidence-based and less blame-oriented.^1,3 However, the focus has remained on individual cases, residents, and physicians rather than on the system and overall quality of patient care.

Although complications and mortalities are evaluated at the M&M conference, near-misses, events that have the potential to result in an adverse patient outcome, are rarely discussed.⁴ Moreover, identification and discussion of these near-misses has been described as a free lesson, as they can help identify system issues to prevent similar events from resulting in actual adverse patient outcomes in the future.^5,6

Adverse events are discussed at the M&M conference in isolation without consideration of similar complications that occurred in the past. Therefore, a mechanism to track these complications and to identify patterns of adverse events may improve patient safety. Our objectives were (1) to develop a user-friendly Web-based system to capture adverse and near-miss events, (2) to identify patterns of adverse events and automate the pattern recognition process, and (3) to examine the reporting culture at a large metropolitan tertiary care center.

Prior to August 1, 2005, it was required that all cases, complications, and morbidities within the Department of Surgery be reported weekly to the M&M moderator. These reports were unstructured and typically submitted via e-mail or in a hand-written format with little detail. The M&M moderator would then review the cases and select approximately 4 complications and/or mortalities each week to be presented by a resident at the conference.

With assistance from the Department of Surgery's Information Systems staff, a Web-based reporting interface, the Northwestern Online Surgical Quality Improvement (NOSQI) system, was created with the primary goals of keeping the system secure and user friendly. Three modules were developed to collect data for (1) the M&M conference, (2) anonymous/near-miss reporting, and (3) the Surgery Quality Management (QM) Committee's case reviews.

The M&M module allows residents to enter the mandatory weekly reports that are used for the M&M conference (Figure 1). All cases, irrespective of whether they are associated with a complication or death, must be reported. Definitions of what constitutes a complication and exactly what should be reported to the NOSQI system were standardized. All complications (within 30 days of surgery or at any time if related to the surgery), morbidities in nonoperative patients, deaths, readmissions, unanticipated reoperations, or transfers to the intensive care unit must be reported. The surgical services that are required to be reported to the NOSQI system are general surgery (general surgery, trauma, gastrointestinal surgery, endocrine surgery, surgical oncology), vascular surgery, transplant, and thoracic surgery.

Through review of previous complications and deaths and the adverse event definitions set forth in American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP), procedures and complications were categorized to allow the database to be efficiently queried (detailed categorization available from authors). For the first 4 months of the system's operation, the weekly reports were reviewed to assess whether the categorization scheme was sufficient.

Up to 2 complications can be entered per patient. In addition, error class (error in diagnosis, error in judgment, error in technique, systems error, nature of disease) and error grade (based on whether the complication is life threatening, requires invasive treatment, and/or results in permanent disability or death) must be assigned for each complication. A free text description of the event is also required. Although more fields could be collected, our goal was to keep the process simple and limited to one page of data entry per patient.

The Anonymous/Near-Miss module was designed to allow reporting of adverse events or near-misses without any associated accountability or fear of blame and/or punishment. A common password was provided to the Department of Surgery to maximize reporting anonymity for this module. The reporter can include more information if they choose, but to protect the reporter's anonymity, we elected to require only the minimal amount of information to potentially act on a report. Data entered into this module were reviewed quarterly.

The third module was designed after the first 2 modules had been operational for 1 year. The Surgery QM module allows all cases reviewed by the Department of Surgery's QM Committee to be entered into the system. These cases are selected according to various criteria by the surgical QM nurse or are recommended for Surgery QM Committee review by other departments in the hospital.

An administrator interface was also designed that allows the database to be queried and generates numerous standardized reports. The data from the NOSQI system were incorporated into the weekly M&M conference. The M&M moderator, when selecting cases for the week, can also require that residents include data from the NOSQI system in their presentation.

The NOSQI system was introduced to the department in July 2005 at the weekly M&M conference where residents and attending physicians were taught how to use the system and were instructed as to what should be reported. The standardized definitions of complications and near-misses were introduced. The effect on reporting was compared before and after each intervention.

The highly categorized classification of complications was designed to facilitate queries and to allow automated notification of adverse event patterns. As each complication is entered, the NOSQI system checks that complication against all of the complications previously entered into the database. If a match is identified, an automated e-mail is sent to the system administrator with a description of the complication, the number of previous complications, and descriptions of the previous cases to facilitate an expeditious analysis. In addition, if a question came up at the weekly M&M conference about the frequency of a specific complication or whether the event has occurred previously, the database was queried to assess the issue.

Once an adverse event pattern was identified, multiple mechanisms were used to address the issue including discussion at the M&M conference and involvement of the hospital's QM department. Interdisciplinary groups and expert panels were convened as needed to discuss the adverse event patterns.

All patients reported to the NOSQI system from September 1, 2005, to August 31, 2007, were identified. Data on readmission and mortality rates were obtained from the hospital's QM system, a database used to track quality issues within the hospital. The hospital also offers the Northwestern Event Tracking System (NETS) to all departments and hospital employees as an anonymous near-miss and adverse event reporting mechanism. The complications reported to the NOSQI system were compared with those reported to NETS. Next, NOSQI system results were compared with M&M data from ACS-NSQIP. The ACS-NSQIP assesses 30-day risk-adjusted postoperative complications and mortality based on data abstracted from patient medical records by a formally trained nurse and compares these outcomes with the other approximately 150 participating hospitals.^7,8

Categorical variables were compared using the χ² test. The Bonferroni correction was used for multiple comparisons. Reporting over time was compared using the χ² test for trend. All analyses were performed using SPSS, version 15 (SPSS Inc, Chicago, Illinois). The Northwestern University institutional review board approved this study.

From August 1, 2005, to August 31, 2007, 15 524 cases were entered into the NOSQI system; 15 428 (99.4%) were in the M&M module, 34 (0.2%) in the Anonymous/Near-Miss module, and 62 (0.4%) in the Surgery QM Committee module. Of the events reported to the M&M module, 957 (6.2%) patients had an adverse event, 264 (1.7%) patients were readmitted, and 218 (1.4%) patients required an unplanned return to the operating room (Table 1). Of the 15 428 cases reported to the M&M module, there were 89 (0.6%) deaths.

Of the 957 complications, 57.7% were gastrointestinal, hematologic, or infectious in etiology (Table 1). Hematomas were the most commonly reported complication followed by wound infections of abdominal incisions, abscesses, and surgical hemorrhage. The primary error class was reported as the nature of the disease in 475 (49.6%) patients, error in technique in 360 (37.6%) patients, error in judgment in 71 (7.4%) patients, error in diagnosis in 36 (3.8%) patients, and systems error in 15 (1.6%) patients (Table 1). Of those with a reported complication, 489 (51.1%) patients required invasive treatment (Table 1). Complications were most frequently categorized as the nature of the disease for cardiac, gastrointestinal, genitourinary, infectious, neurologic, and pulmonary adverse event categories (Table 2). Complications were classified most frequently as error in technique for anesthesia, biliary, hematologic, and vascular adverse event categories. Adverse events most frequently required invasive treatment except for cardiac, endocrine, genitourinary, and neurologic complications. Nature of the disease was also reported as the most frequent error class irrespective of error grade (Table 3). Of the complications that were reported to result in permanent disability or death, 66.7% and 76.4% were attributed to the nature of the disease by the reporting physicians.

Four adverse event patterns were identified from the M&M module, and 3 adverse event patterns were identified from the Anonymous/Near-Miss module (Table 4). The automated notification helped identify patterns of breast hematomas, chest tube placement complications, and complications in surgical patients in nonsurgical intensive care units. One pattern was identified manually after discussion of a case in the M&M conference regarding a laparotomy with negative findings based on an incorrect computed tomography scan read at night by a radiology senior resident (reread the next morning as normal by the attending radiologist). The NOSQI system was queried to identify whether this had been an issue previously. Three prior reports were identified discussing similar events, and one case was reported 3 weeks later. The 3 adverse event patterns identified from reports to the Anonymous/Near-Miss module concerned nursing assistant–to-nurse and nurse-to-physician communication and a series of complications involving nasogastric tubes.

From August 2005 through August 2007, the reporting of complications (proportion of complications to all cases) varied considerably, from 2.0% to 11.0% (Figure 2). When the online system was initiated, there was no appreciable difference in the monthly adverse event reporting rate. The effect of several interventions was investigated. The most appreciable increases in reporting came when the department chairman discussed the importance of complication reporting and what constitutes a reportable event (from 3.4% in May 2006 to 11.0% in June 2007, P < .001 and from 4.3% in April 2007 to 8.3% in May 2007, P < .001). In January 2007, all residents and attending physicians began receiving a weekly reminder to report complications and near-miss events; however, this did not have an appreciable effect on the reporting rate.

In comparison with the denominator of events from the hospital's administrative quality database, only 27.2% (264 of 970) of readmissions and 41.6% (89 of 214) of inpatient deaths were reported to the NOSQI system (Table 5). Only 5.2% of adverse events submitted to the NOSQI system were also entered in the hospitalwide event tracking system, NETS. When compared with ACS-NSQIP, the NOSQI system (for general and vascular surgery only) demonstrated significantly lower morbidity rates (16% vs 4.5%) but similar mortality rates (0.5% vs 0.4%). Compared with the hospital's discharge data for all surgical services, the mortality rate reported to the NOSQI system was considerably lower (1.4% vs 0.6%).

Traditionally, adverse events at the M&M conference have been anecdotal, discussed in isolation without consideration of prior similar events, and not focused on systems issues. In their 1999 report titled To Err is Human,⁹ the Institute of Medicine strongly recommended expanding voluntary reporting of adverse events and medical errors. If adverse event patterns can be identified, then action can be taken preemptively to prevent the repetition of similar events. We developed an online Web-based system to capture and analyze adverse and near-miss event data.

Although our hospital had a readily accessible online system by which to report near-miss and adverse events, there were several problems with the hospital's event tracking system, as have been reported by others, such as a lengthy and time-consuming report form and a lack of a surgery-specific classification system to facilitate queries.¹⁰ Importantly, only 5% of complications reported to the M&M conference were also reported to NETS; thus many events would be missed with the hospital's system alone.

Beyond creating an adverse event data repository, we were interested in identifying patterns of adverse events that otherwise might have gone unnoticed. We attempted to automate the process to reduce the work involved in identifying patterns. In our opinion, adverse event pattern recognition is the primary purpose and the most powerful aspect of the NOSQI system. The automated process identified 4 adverse event patterns, and we were also able to investigate possible patterns when prompted by a particular case discussed at the M&M conference.

Prior studies based on intensive chart reviews and prospective expert evaluation have shown that the adverse event rate in surgical patients ranges from 30% to 42%.^11,12 Our own ACS-NSQIP data shows that our complication rate is at least 16% for general and vascular surgery. Of the cases entered in the M&M module, only 6.2% were reported to have a complication. Multiple studies have demonstrated that 65% to 91% of adverse events fail to be reported to the M&M conference.^13-16 Based on reports to their M&M system, Hutter et al¹⁵ noted a similar adverse event rate of 6.4% at the Massachusetts General Hospital. They compared their M&M and ACS-NSQIP data and estimated that only 25% of complications and 50% of deaths were reported to the M&M conference. We also found that a considerable number of morbidities and even mortalities were not reported to the NOSQI system. Though not surprising, this underreporting was disappointing. During the study period, interventions aimed at increasing reporting had little effect. The only intervention to have even a short-term effect was the presentation from the department chair; however, the initial spike in complications reporting was short lived. It is important to note that the physician-reported complication rates underestimate the actual complication rates owing to this severe underreporting.

Also disappointing was the infrequent use of the near-miss reporting module. Despite offering reporting anonymity, only 34 cases in 2 years were reported to the Anonymous/Near-Miss module. Interestingly, this small number of cases led to the identification of 3 adverse event patterns. Thus, when the Near-Miss/Anonymous module was used, it was a high-yield opportunity to identify event patterns and institute quality improvement initiatives. At the University of Louisville, one near-miss event is discussed at their weekly conference.⁴ Incorporating a discussion of near-misses into the M&M conference may encourage residents and attending physicians to report such events.

Multiple reasons have been suggested regarding why physicians do not report their complications, including time constraints and concerns of punishment, litigation, and diminished reputation in the opinion of their peers.^6,17,18 Wu et al¹⁹ found that 54% of residents failed to inform their attending physicians when they made a serious error. In addition, the culture associated with the M&M conference may lead to underreporting and prevent identification of adverse event patterns. The reporting resident is put in the spotlight and questioned regarding the details of the case. Complications, near-misses, and even mortalities may not be reported because of the “blame and shame” mentality.¹ In addition, physicians may not report adverse events because of time pressures.⁶

Another aspect of reporting behavior that was examined involved how residents classified the complication. “Nature of disease” was listed as the error class in 75.2% of adverse events resulting in permanent disability or death. Studies have demonstrated that only 26% to 46% of adverse events are not preventable^12,20-22; thus assignment of the nature of the disease is likely too frequent and may reflect attitudes that attempt to deflect the blame for complications. Moreover, only 1.6% of cases were listed as a systems error. Additional resident education is clearly needed to shift the focus of adverse event causes from individuals to the overall system of care and away from placing the responsibility on the patient's illness.^5,23 In a culture that is attempting to move toward transparency and more systems-oriented quality improvement initiatives, honest and open reporting is critical to improve patient safety.^5,6,23-25 Further investigation is needed to improve the reporting culture among surgical residents.

Although the M&M conference at our institution makes a significant effort to discuss complications in the context of prior studies,³ cases were still discussed in isolation without consideration of similar past events. Recently, we have incorporated the data from the NOSQI system and from ACS-NSQIP into the weekly M&M conference.^7,8 For example, if a resident is presenting a postoperative wound infection, data from the NOSQI system, the Surgical Care Improvement Project, the University Health Consortium, and/or ACS-NSQIP can be presented to offer the audience an idea of how frequently these events happen and how our hospital compares with others across the country. These data, in conjunction with the case being presented, provide powerful quality assessment and improvement opportunities on a weekly basis.

An electronic physician-reported event tracking system should be incorporated into all surgery departments irrespective of whether the department is associated with a residency program; however, this is just one component of what should be a larger quality improvement effort. As the public and regulatory agencies are increasingly demanding evidence of oversight, having and using such a system may be mandated to demonstrate evidence of quality improvement activities. The NOSQI system is a valuable resource, as it serves to enhance the weekly M&M conference by identifying systems issues and areas for health care provider education. An online event tracking system is a feasible, promising, and potentially powerful initiative to improve surgical safety in the United States.

Correspondence: Karl Y. Bilimoria, MD, Department of Surgery, Northwestern Memorial Hospital, 251 E Huron St, Galter 3-150, Chicago, IL 60611 (k-bilimoria@northwestern.edu).

Accepted for Publication: March 15, 2008.

Author Contributions: Dr Bilimoria had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Bilimoria, Kmiecik, DaRosa, Bell, and Wayne. Acquisition of data: Bilimoria, Kmiecik, and Bell. Analysis and interpretation of data: Bilimoria, Kmiecik, DaRosa, Halverson, Bell, Soper, and Wayne. Drafting of the manuscript: Bilimoria and Wayne. Critical revision of the manuscript for important intellectual content: Bilimoria, Kmiecik, DaRosa, Halverson, Eskandari, Bell, Soper, and Wayne. Statistical analysis: Bilimoria and Eskandari. Administrative, technical, and material support: Bilimoria, Kmiecik, DaRosa, Eskandari, and Soper. Study supervision: DaRosa, Bell, Soper, and Wayne.

Financial Disclosure: None reported.

Previous Presentations: This study was presented at the 2007 American College of Surgeons 93rd Annual Clinical Congress; October 9, 2007; New Orleans, Louisiana.

Additional Contributions: The authors would like to acknowledge the Northwestern University Department of Surgery's Surgical Education staff (Sybil Boyajian, Betty Hunt, Erin Schembari, Jacquelyn Sostarics, Kristen Ward, and Megan Weltscheff) for their assistance in the development and operation of the NOSQI system. We also appreciate the support of Northwestern Memorial Hospital's administration and Quality Department, particularly Jay Anderson, Lois Huminiak, Scott Kerth, and Charles Watts, MD. We appreciate the assistance of Daniel E. Abbott, MD, for help with the development of the NOSQI system.