Figure 1. Selection of articles for review. QI indicates quality indicator.
Figure 2. Distribution of published quality indicators (QIs) in trauma care.
Henry Thomas Stelfox, Barbara Bobranska-Artiuch, Avery Nathens, Sharon E. Straus. Quality Indicators for Evaluating Trauma CareA Scoping Review. Arch Surg. 2010;145(3):286–295. doi:10.1001/archsurg.2009.289
To systematically review the literature on quality indicators (QIs) for evaluating trauma care, identify QIs, map their definitions, and examine the evidence base in support of the QIs.
We searched MEDLINE, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, Cochrane Database of Abstracts of Reviews of Effects, and Cochrane Central Register of Controlled Trials from the earliest available date through January 14, 2009. To increase the sensitivity of the search, we also searched the grey literature and select journals by hand, reviewed reference lists to identify additional studies, and contacted experts in the field.
Study Selection and Data Extraction
We selected all articles that identified or proposed 1 or more QIs to evaluate the quality of care delivered to patients with major traumatic injuries. Minimum inclusion criteria were a description of 1 or more QIs designed to evaluate patients with major traumatic injuries (defined as multisystem injuries resulting in hospitalization or death) and focused on prehospital care, hospital care, posthospital care, or secondary injury prevention.
The literature search identified 6869 citations. Review of abstracts led to the retrieval of 538 full-text articles for assessment, of which 192 articles were selected for review. Of these, 128 (66.7%) articles were original research, predominantly trauma database case series (57 [29.7%]) and cohort studies (55 [28.6%]), whereas 37 (19.3%) were narrative reviews and 8 (4.2%) were guidelines. A total of 1572 QIs in trauma care were identified and classified into 8 categories: non-American College of Surgeons Committee on Trauma (ACS-COT) audit filters (42.0%), ACS-COT audit filters (19.1%), patient safety indicators (13.2%), trauma center/system criteria (10.2%), indicators measuring or benchmarking outcomes of care (7.4%), peer review (5.5%), general audit measures (1.8%), and guideline availability or adherence (0.8%). Measures of prehospital and hospital processes (60.4%) and outcomes (22.8%) were the most common QIs identified. Posthospital and secondary injury prevention QIs accounted for less than 5% of QIs.
Many QIs for evaluating the quality of trauma care have been proposed, but the evidence to support these indicators is not strong. Practical recommendations to select QIs to measure the quality of trauma care will require systematic reviews of identified candidate indicators and empirical studies to fill the knowledge gaps for postacute QIs.
Injury is one of the leading causes of death in almost every country in the world. Each year, injuries affect 700 million people around the world and result in more than 5 million deaths.1 However, deaths are only the tip of the injury iceberg. In the United States alone, injured Americans annually make 30 million emergency department visits resulting in 1.9 million hospital admissions at direct medical costs of approximately $80 billion.2 The human and societal burden is even greater, with many survivors never returning to school, work, or their “regular” lives.3,4
Surgical services provide injured patients with pivotal treatment for what is a major cause of morbidity and mortality. However, there is growing evidence that the best treatments and strategies for these patients are not always implemented.5,6 Several studies have documented that as many as half of all patients with major traumatic injuries do not receive recommended care.7- 11 In addition, medical errors are common among critically ill trauma patients.12,13 Review of trauma deaths in hospitals has suggested that between 2.5% and 14% of medical errors are preventable.14- 19 As a result, the outcomes of patients with major traumatic injuries are not as good as they could be with better translation of the best research evidence at the bedside.
To improve trauma care, the quality of care first needs to be measured using evidence-based tools. However, it is unclear if the proper tools have yet been developed. The purpose of this scoping review was to systematically review the literature on quality indicators (QIs) in evaluating trauma care, identify QIs, map their definitions, and examine the evidence base in support of QIs.
Relevant articles were identified by searching the following databases on January 14, 2009, from the first date available: Ovid MEDLINE; Ovid EMBASE; Ovid CINAHL; and Cochrane Database of Systematic Reviews, Cochrane Database of Abstracts of Reviews of Effects, and Cochrane Central Register of Controlled Trials. Searches were performed with no language of publication restrictions. Combinations of the following search terms were used: trauma, injury, quality indicator, quality assurance, quality control, performance improvement, quality measure, best practice, and audit. The Cochrane Library was searched separately using the search term trauma. To increase the sensitivity of the search strategy, we also searched the grey literature. This search included identifying and searching Web sites of relevant trauma organizations (American College of Surgeons, American Association for the Surgery of Trauma, Eastern Association for the Surgery of Trauma, American Trauma Society, British Trauma Society, Trauma Association of Canada, Australasian Trauma Society, Orthopedic Trauma Association, Western Trauma Association, Trauma.org, The Society of Trauma Nurses, International Trauma Anesthesia, and Critical Care Society) and Google using combinations of the following search terms: trauma, injury, quality indicator, quality of care, quality assurance, and quality control. In addition, a query was sent to 2 medical librarian listservs (CANMED-LIB and MEDLIB) asking about relevant materials, and responses were collected for 1 week. Appropriate wildcards were used in all searches to account for plural words and variations in spelling. Additional articles were identified by searching the bibliographies of those articles identified in the searches and contacting experts in the field. Selected journals (Journal of Trauma: Injury, Infection and Critical Care, Injury: International Journal of the Care of the Injured, Annals of Surgery, Surgery, and Journal of the American College of Surgeons) were manually searched from August 1, 2006, through July 31, 2007, to ensure that important articles were not missed.
We selected all articles that identified or proposed 1 or more QIs for evaluating the quality of trauma care. We included all articles, both research and nonresearch. For this study, a QI was defined as a performance measure that compared actual care against ideal criteria, a tool to help assess quality of care.20 Minimum inclusion criteria were (1) the article included at least 1 QI; (2) the QI was designed to evaluate the care of patients with major traumatic injuries; and (3) the QI focused on trauma care specific to at least 1 of the following phases of patient care: prehospital, hospital, posthospital, or secondary injury prevention. Injury was defined as the “transfer of energy applied clinically.”21 A major injury was defined as an injury resulting in at least 1 of the following: a multisystem injury (involving ≥2 regions of the body), an Injury Severity Score higher than 9, patient hospitalization, or patient death.22,23 We did not include studies of patients with injuries secondary to poisonings, adverse effects of drugs or biological substances, and isolated hip fractures.22,24 These conservative criteria were chosen because we wanted to optimize our chances for identifying relevant QIs.
Eligible articles were identified through 2 phases. In the first phase, 2 of us (H.T.S. and B.B.-A.) independently reviewed the titles and abstracts of retrieved publications and selected relevant articles for possible inclusion in the review. Disagreements between the 2 assessors were discussed, and, if agreement could not be reached, the article was retained for further review.
In the second phase, the full texts of the remaining articles were independently reviewed by the same 2 authors using forms for determining eligibility criteria. Disagreements between the 2 assessors were discussed, and a third author (S.E.S.) was consulted if agreement could not be reached. Reviewers were not masked to author or journal name.25
Two of us (H.T.S. and B.B.-A.) independently reviewed all full-text articles that satisfied the minimum inclusion criteria and abstracted data using a standardized form. The agreement of the raters was very good (κ = 0.742). Extracted information included QI source, definition (8 components), data sources for measuring QIs, and the articles' authors' recommendations regarding the QI.26 Quality indicators were classified using a prespecified conceptual model that merged the descriptions of the components of a contemporary trauma system, prehospital, hospital, and posthospital care and secondary injury prevention with the 3 components of health care quality in Donabedian's framework: structure, process, and outcome (Table 1).27- 29 Reviewers also judged whether QIs were operational (yes vs no). Disagreements in assessment and data extraction were resolved by author consensus, and a third author (S.E.S.) was consulted if agreement could not be reached. Owing to the heterogeneity of the results, articles and QIs were summarized as counts and proportions using Stata statistical software, version 10.0 (Stata Corp, College Station, Texas).
The literature search identified 6869 articles. Review of abstracts led to retrieval of 402 full-text articles for assessment plus 72 articles from review of references and 64 articles from the hand search. We identified 192 articles written in 3 languages (English, German, and Spanish) for inclusion in the study (Figure 1).7- 10,12,13,16- 19,22,27,30- 209 The most common reason for excluding articles after full-text review was the absence of a QI.
Table 2 summarizes the characteristics of the articles (see eTable for descriptions of individual articles). Most articles were research studies (66.7%) primarily consisting of case series (29.7%), cohort studies (28.6%), cross-sectional studies (4.2%), and nonrandomized controlled studies (2.6%) using established trauma databases (Table 2). The non–original research articles (33.3%) consisted primarily of narrative review articles (19.3%). Most articles were published in the United States (57.8%) in the last 2 decades. For most articles, the academic status of the corresponding institution was unclear. Few were supported by grants, primarily from the government (14.6%) or private foundations (7.3%).
Most articles focused on QIs to evaluate the quality of trauma care for adult patients, whereas few focused exclusively on pediatric patients (7.8%). The mechanism of injury was not specified in most articles. However, the 3 mechanisms of injury most commonly reported were road traffic accidents (25.5%), falls (17.2%), and violence (16.1%). The anatomical location of patient injuries was reported in some articles (23.4%), with neurological injuries being the most common (14.6%).
Table 3 summarizes the characteristics of QIs in trauma care. We identified 1572 QIs in the 192 articles, a median of 3 QIs per article (interquartile range, 1-11) with a range of 1 to 60 QIs per article. The QIs were described in the articles using 103 different terms (eg, QI, audit filter, standards for trauma care, safety standard, annual feedback, avoidable factors, criteria deficiencies, etc). The most common source of QIs were authors of articles or local health care provider groups (56.9%), followed by the American College of Surgeons Committee on Trauma (ACS-COT) (26.5%) (Table 3). We reviewed 8 components of QI definitions.26 Virtually all QIs had a descriptive statement (1568 of 1572 [99.7%]), but only a few provided a list of data elements, specifications for data element collection, a description of the population and/or analytic models used to construct the measure, timing of data collection, the format for presentation of the results, and the timing of reporting. The data sources for measuring QIs were specified for 942 indicators (59.9%), with trauma registries (27.2%) and medical record review (24.6%) being the most common sources. After reviewing the definitions and data sources provided for the QIs, the 2 reviewers determined that 959 (61.0%) of the QIs were potentially operational.
Table 4 summarizes the 8 categories for classifying the 1572 QIs based on content matter. The 2 most frequent categories were non–ACS-COT (42.0%) and ACS-COT (19.1%) audit filters. Some QIs related to guideline availability or adherence were identified.
Figure 2 summarizes the distribution of the QIs according to our prespecified conceptual model. Measures of prehospital and hospital processes of care were the most common QIs identified. Posthospital and secondary injury prevention QIs accounted for less than 5% of QIs.
Table 5 summarizes author recommendations regarding use of QIs in trauma care. Authors of the articles recommended that 616 QIs (39.2%) be implemented and used routinely, whereas 78 QIs (5.0%) should not be used. The authors recommended that the QIs be used for quality evaluation in 377 cases (24.0%) and as a quality improvement tool in 308 cases (19.6%). Additional research was recommended for 94 QIs (6.0%) before their implementation in clinical practice. Table 6 lists the 10 QIs identified most frequently in original research studies, which are potential candidates for systematic review and further evaluation.
In our review of the current literature on QIs to evaluate the quality of trauma care, we identified 192 articles and 1572 QIs. A large body of literature exists for evaluating trauma care and may be helpful for guiding development of evidence-based QIs, although few indicators appear to be explicitly defined and supported by valid criteria for collection. The articles available are predominantly focused on prehospital and hospital processes of care for adult populations and provide opportunities for systematic reviews. Very few articles were identified that focused on QIs for evaluating the quality of trauma care for children or postacute care, suggesting a need for empirical research in these domains.
The single most important result from our review is that we did not find a common set of clearly defined, evidence-based, broadly accepted QIs for evaluating the quality of trauma care. Rather, our review identified a large group of heterogeneous indicators that could be broadly categorized into 8 groups. It is not possible to have a transparent, explicit, systematic, data-driven performance measurement feedback system if there are no generally agreed upon measures. How can we explain the contrast between our results and the interest and resources that are being directed toward measuring quality around the globe? Despite its theoretical appeal, developing quality measures is challenging. One, physicians, researchers, and administrators in a field have to communicate using the same language. We identified 103 terms used to identify or describe QIs in our review. Although the field of quality improvement is fundamentally a process of change in human behavior and is driven largely by experiential learning, it would be nearly impossible to effectively communicate with such a large variation in common terms. An agreement on a basic taxonomy is critical similar to the recently published SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines.210 Two, trauma care has until recently been intensively local, with quality improvement based on local surgical audits.211 Quality measures should be developed to serve national goals, but should be implemented locally.26 Our review highlights the frequency with which local QIs have been developed. Three, performance measurement is facilitated with a large, high-quality evidence base. For example, the American College of Cardiology and the American Heart Association have jointly published 11 evidence-based QIs for adults with myocardial infarction.212 These indicators are widely accepted because the underlying data arise from large, multicenter, randomized, controlled trials. Conversely, physicians caring for patients with traumatic injuries do not have the same evidence base to draw upon in developing QIs.213 Four, developing QIs for diseases or disorders that are homogeneous is easier than for those that are heterogeneous.214 Trauma is not a disease but a collection of injuries, and QIs may therefore need to focus on common pathways and outcomes.
Just because a common set of evidence-based QIs of trauma care have not been developed does not mean that they cannot be developed. Recognizing this challenge, the American College of Surgeons has proposed the Trauma Quality Improvement Project,168 modeled after the successful National Surgical Quality Improvement Program,215 a validated, risk-adjusted, outcomes-based program to measure and improve the quality of surgical care. The Trauma Quality Improvement Project is a proposal to measure risk-adjusted performance of trauma centers, identify institutional characteristics associated with good patient outcomes, and promote those institutional characteristics among trauma organizations. A feedback mechanism involving confidential report cards would be used to inform each trauma center of its performance. This proposal is potentially an important step in improving the quality of trauma care because it may provide for the first systematic measuring and reporting of trauma care quality. However, more is needed. Valid measures of structure and process of care are also important because they provide a direct means for programs to evaluate themselves. Finally, outcome measures other than hospital mortality will be needed to capture additional outcome dimensions such as health-related quality-of-life and to provide greater specificity for different components of patient care.141 Our review begins to outline where sufficient literature exists to evaluate candidate indicators (Table 6) and where gaps exist (Figure 2) and empirical research is needed.
There are limitations to this review. First, our search may not have been exhaustive, despite the search of multiple databases using comprehensive search strategies with the assistance of an information specialist and imposing no language restrictions. Nevertheless, it is unlikely that our search missed broad categories of important QIs. Second, it is difficult to extract accurate data from all publications. Some articles are difficult to obtain (3 in our study), some do not disclose all materials or methods used, and results are often unclear and difficult to interpret. Third, categorizing articles by type, even with the assistance of predefined data abstraction tools and classification schemes, is partly subjective.
We have shown that many QIs for evaluating the quality of trauma care have been proposed but that the distribution of these indicators is skewed across different components of health care quality and trauma systems. Our results should promote future research in 3 areas. First, systematic reviews are warranted to explore whether evidence-based QIs can be developed from the existing literature for acute-care measures. Second, empirical research is needed to fill the knowledge gaps for postacute QIs. Third, research is needed on how QIs are currently used by trauma systems and centers to measure and improve trauma care as a means to better understanding how to effectively incorporate newly developed indicators into performance-improvement initiatives. Identifying, developing, evaluating, and implementing evidence-based QIs is essential to measuring and improving trauma care.
Correspondence: Henry Thomas Stelfox, MD, PhD, Department of Critical Care Medicine, Foothills Medical Centre, University of Calgary, South Tower, Room 1105, 1403-29 St NW, Calgary, Alberta T2N 2T9, Canada (Tom.Stelfox@albertahealthservices.ca).
Accepted for Publication: July 23, 2009.
Author Contributions: Drs Stelfox and Straus had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Stelfox, Nathens, and Straus. Acquisition of data: Stelfox, Bobranska-Artiuch, and Straus. Analysis and interpretation of data: Stelfox, Bobranska-Artiuch, Nathens, and Straus. Draft of the manuscript: Stelfox. Critical revision of the manuscript: Stelfox, Bobranska-Artiuch, Nathens, and Straus. Statistical analysis: Stelfox. Administrative, technical, and material support: Stelfox and Straus. Study supervision: Stelfox.
Financial Disclosure: None reported.
Funding/Support: The project was supported by Synthesis Grant KRS-91770 from the Canadian Institutes of Health Research. Dr Stelfox is supported by a New Investigator award from the Canadian Institutes of Health Research.
Role of the Sponsors: The funding sources had no role in the design, conduct, or reporting of this study.
Additional Contributions: Laure Perrier, MS, assisted with the literature search strategy, and Kelly Mrklas, MSc, assisted with translation of Spanish and German language articles.