[Skip to Content]
[Skip to Content Landing]
Figure.
Risk-Adjusted Incidence of Complications With 95% CIs, by Trauma Center and Designation Level
Risk-Adjusted Incidence of Complications With 95% CIs, by Trauma Center and Designation Level

Diamonds indicate level I trauma centers; circles, level II trauma centers; triangles, level III trauma centers; and squares, level IV trauma centers.

Table 1.  
Sociodemographic and Injury Characteristics of the Study Population
Sociodemographic and Injury Characteristics of the Study Population
Table 2.  
In-Hospital Complications Stratified by Trauma Center Designation Level
In-Hospital Complications Stratified by Trauma Center Designation Level
Table 3.  
Model Coefficients and Adjusted Odds Ratios of In-Hospital Complications
Model Coefficients and Adjusted Odds Ratios of In-Hospital Complications
Table 4.  
Validity of the Quality Indicator to Benchmark In-Hospital Complications
Validity of the Quality Indicator to Benchmark In-Hospital Complications
1.
Parachute Canada. The cost of injury in Canada. http://www.parachutecanada.org/downloads/research/Cost_of_Injury-2015.pdf. Published 2015. Accessed December 29, 2015.
2.
National Trauma Institute. Trauma statistics. http://www.nationaltraumainstitute.org/home/trauma_statistics.html. Updated February 2014. Accessed September 21, 2015.
3.
World Health Organization. Guidelines for trauma quality improvement programmes. http://www.who.int/violence_injury_prevention/services/traumacare/traumaguidelines/en/. Published 2015. Accessed April 30, 2015.
4.
Moore  L, Turgeon  AF, Lauzier  F,  et al.  Evolution of patient outcomes over 14 years in a mature, inclusive Canadian trauma system.  World J Surg. 2015;39(6):1397-1405.PubMedGoogle ScholarCrossref
5.
Jurkovich  GJ, Mock  C.  Systematic review of trauma system effectiveness based on registry comparisons.  J Trauma. 1999;47(3)(suppl):S46-S55.PubMedGoogle ScholarCrossref
6.
Mackenzie  EJ, Rivara  FP, Jurkovich  GJ,  et al.  The national study on costs and outcomes of trauma.  J Trauma. 2007;63(6)(suppl):S54-S67.PubMedGoogle ScholarCrossref
7.
Stelfox  HT, Bobranska-Artiuch  B, Nathens  A, Straus  SE.  A systematic review of quality indicators for evaluating pediatric trauma care.  Crit Care Med. 2010;38(4):1187-1196.PubMedGoogle ScholarCrossref
8.
Moore  L, Lauzier  F, Stelfox  HT,  et al.  Validation of complications selected by consensus to evaluate the acute phase of adult trauma care: a multicenter cohort study.  Ann Surg. 2015;262(6):1123-1129.PubMedGoogle ScholarCrossref
9.
Hemmila  MR, Jakubus  JL, Wahl  WL,  et al.  Detecting the blind spot: complications in the trauma registry and trauma quality improvement.  Surgery. 2007;142(4):439-448.PubMedGoogle ScholarCrossref
10.
Santana  MJ, Stelfox  HT; Trauma Quality Indicator Consensus Panel.  Development and evaluation of evidence-informed quality indicators for adult injury care.  Ann Surg. 2014;259(1):186-192.PubMedGoogle ScholarCrossref
11.
Stelfox  HT, Straus  SE, Nathens  A, Bobranska-Artiuch  B.  Evidence for quality indicators to evaluate adult trauma care: a systematic review.  Crit Care Med. 2011;39(4):846-859.PubMedGoogle ScholarCrossref
12.
Moore  L, Lauzier  F, Stelfox  HT,  et al.  Complications to evaluate adult trauma care: an expert consensus study.  J Trauma Acute Care Surg. 2014;77(2):322-329.PubMedGoogle ScholarCrossref
13.
Moore  L, Stelfox  HT, Turgeon  AF.  Complication rates as a trauma care performance indicator: a systematic review.  Crit Care. 2012;16(5):R195.PubMedGoogle ScholarCrossref
14.
Krumholz  HM, Brindis  RG, Brush  JE,  et al; American Heart Association; Quality of Care and Outcomes Research Interdisciplinary Writing Group; Council on Epidemiology and Prevention; Stroke Council; American College of Cardiology Foundation.  Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council.  Circulation. 2006;113(3):456-462.PubMedGoogle ScholarCrossref
15.
American College of Surgeons Committee on Trauma.  Resources for Optimal Care of the Injured Patient. Chicago, IL: American College of Surgeons; 2006.
16.
Association for the Advancement of Automotive Medicine.  Abbreviated Injury Scale (AIS). Chicago, IL: AAAM Publications; 1990.
17.
Moore  L, Lavoie  A, Le Sage  N,  et al.  Using information on preexisting conditions to predict mortality from traumatic injury.  Ann Emerg Med.2008;52(4):356-364.e2. doi:10.1016/j.annemergmed.2007.09.007. PubMedGoogle ScholarCrossref
18.
Royston  P, Ambler  G, Sauerbrei  W.  The use of fractional polynomials to model continuous risk variables in epidemiology.  Int J Epidemiol. 1999;28(5):964-974.PubMedGoogle ScholarCrossref
19.
Austin  PC, Tu  JV.  Bootstrap methods for developing predictive models.  Am Stat. 2004;58(2):131-137.Google ScholarCrossref
20.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387.PubMedGoogle ScholarCrossref
21.
Hanley  JA, McNeil  BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve.  Radiology. 1982;143(1):29-36.PubMedGoogle ScholarCrossref
22.
Cox  DR.  Two further applications of a model for binary regression.  Biometrika.1958;45(3):562-565.Google ScholarCrossref
23.
Nagelkerke  NJ.  A note on a general definition of the coefficient of determination.  Biometrika. 1991;78(3):691-692.Google ScholarCrossref
24.
Moore  L, Lavoie  A, Turgeon  AF,  et al.  The trauma risk adjustment model: a new model for evaluating trauma care.  Ann Surg. 2009;249(6):1040-1046.PubMedGoogle ScholarCrossref
25.
Agency for Healthcare Research and Quality. Inpatient Quality Indicators: AHRQ Composite Measures Workgroup final report. http://www.qualityindicators.ahrq.gov/Downloads/Modules/IQI/IQI_Composite_Development.pdf. Published 2013. Accessed December 29, 2015.
26.
Iezzoni  L.  Risk Adjustment for Measuring Health Care Outcomes. 3rd ed. Chicago, IL: Health Administration Press; 2003.
27.
Moore  L, Hanley  JA, Turgeon  AF, Lavoie  A, Eric  B.  A new method for evaluating trauma centre outcome performance: TRAM-adjusted mortality estimates.  Ann Surg. 2010;251(5):952-958.PubMedGoogle ScholarCrossref
28.
Moore  L, Stelfox  HT, Turgeon  AF,  et al.  Derivation and validation of a quality indicator for 30-day unplanned hospital readmission to evaluate trauma care.  J Trauma Acute Care Surg. 2014;76(5):1310-1316.PubMedGoogle ScholarCrossref
29.
Moore  L, Stelfox  HT, Turgeon  AF,  et al.  Derivation and validation of a quality indicator of acute care length of stay to evaluate trauma care.  Ann Surg. 2014;260(6):1121-1127.PubMedGoogle ScholarCrossref
30.
Zar  JH.  Biostatistical Analysis. 3rd ed. Englewood Cliffs, NJ: Prentice Hall; 1996.
31.
Little  RJA, Rubin  DB.  Statistical Analysis With Missing Data. 2nd ed. New York, NY: Wiley; 2002. Wiley Series in Probability and Statistics.
32.
Moore  L, Lavoie  A, LeSage  N,  et al.  Multiple imputation of the Glasgow Coma Score.  J Trauma. 2005;59(3):698-704.PubMedGoogle Scholar
33.
Charlson  ME, Pompei  P, Ales  KL, MacKenzie  CR.  A new method of classifying prognostic comorbidity in longitudinal studies: development and validation.  J Chronic Dis. 1987;40(5):373-383.PubMedGoogle ScholarCrossref
34.
Baker  GR, Norton  PG, Flintoft  V,  et al.  The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada.  CMAJ. 2004;170(11):1678-1686.PubMedGoogle ScholarCrossref
35.
Wilson  RM, Runciman  WB, Gibberd  RW, Harrison  BT, Newby  L, Hamilton  JD.  The Quality in Australian Health Care Study.  Med J Aust. 1995;163(9):458-471.PubMedGoogle Scholar
36.
Davis  P, Lay-Yee  R, Briant  R, Ali  W, Scott  A, Schug  S.  Adverse events in New Zealand public hospitals, I: occurrence and impact.  N Z Med J. 2002;115(1167):U271.PubMedGoogle Scholar
37.
Brennan  TA, Leape  LL, Laird  NM,  et al.  Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I.  N Engl J Med. 1991;324(6):370-376.PubMedGoogle ScholarCrossref
38.
Vincent  C, Neale  G, Woloshynowych  M.  Adverse events in British hospitals: preliminary retrospective record review.  BMJ. 2001;322(7285):517-519.PubMedGoogle ScholarCrossref
39.
Drösler  SE, Romano  PS, Tancredi  DJ, Klazinga  NS.  International comparability of patient safety indicators in 15 OECD member countries: a methodological approach of adjustment by secondary diagnoses.  Health Serv Res. 2012;47(1, pt 1):275-292.PubMedGoogle ScholarCrossref
40.
Ingraham  AM, Richards  KE, Hall  BL, Ko  CY.  Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach.  Adv Surg. 2010;44:251-267.PubMedGoogle ScholarCrossref
41.
Agency for Healthcare Research and Quality. Patient safety indicators overview. http://www.qualityindicators.ahrq.gov/modules/psi_resources.aspx. Accessed March 31, 2014.
42.
Khuri  SF, Daley  J, Henderson  W,  et al; National VA Surgical Quality Improvement Program.  The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care.  Ann Surg. 1998;228(4):491-507.PubMedGoogle ScholarCrossref
43.
Glance  LG, Dick  AW, Meredith  JW, Mukamel  DB.  Variation in hospital complication rates and failure-to-rescue for trauma patients.  Ann Surg. 2011;253(4):811-816.PubMedGoogle ScholarCrossref
44.
Hemmila  MR, Jakubus  JL, Maggio  PM,  et al.  Real money: complications and hospital costs in trauma patients.  Surgery. 2008;144(2):307-316.PubMedGoogle ScholarCrossref
45.
O’Keefe  GE, Maier  RV, Diehr  P, Grossman  D, Jurkovich  GJ, Conrad  D.  The complications of trauma and their associated costs in a level I trauma center.  Arch Surg. 1997;132(8):920-924.PubMedGoogle ScholarCrossref
46.
Osler  T, Glance  LG, Hosmer  DW.  Complication-associated mortality following trauma: a population-based observational study.  Arch Surg. 2012;147(2):152-158.PubMedGoogle ScholarCrossref
47.
de Jongh  MA, Bosma  E, Verhofstad  MH, Leenen  LP.  Prediction models for complications in trauma patients.  Br J Surg. 2011;98(6):790-796.PubMedGoogle ScholarCrossref
48.
Hoyt  DB, Coimbra  R, Potenza  B,  et al.  A twelve-year analysis of disease and provider complications on an organized level I trauma service: as good as it gets?  J Trauma. 2003;54(1):26-36.PubMedGoogle ScholarCrossref
49.
Moore  L, Hanley  JA, Lavoie  A, Turgeon  A.  Evaluating the validity of multiple imputation for missing physiological data in the National Trauma Data Bank.  J Emerg Trauma Shock. 2009;2(2):73-79.PubMedGoogle ScholarCrossref
50.
Boyd  CR, Tolson  MA, Copes  WS.  Evaluating trauma care: the TRISS method: Trauma Score and the Injury Severity Score.  J Trauma. 1987;27(4):370-378.PubMedGoogle ScholarCrossref
Original Investigation
July 2016

Derivation and Validation of a Quality Indicator to Benchmark In-Hospital Complications Among Injury Admissions

Author Affiliations
  • 1Department of Social and Preventive Medicine, Université Laval, Quebec, Quebec, Canada
  • 2Population Health and Optimal Health Practices Research Unit, Trauma–Emergency–Critical Care Medicine, Centre de Recherche du Centre Hospitalier Universitaire de Québec, Hôpital de l’Enfant-Jésus, Université Laval, Quebec, Quebec, Canada
  • 3Department of Medicine, Université Laval, Quebec, Quebec, Canada
  • 4Department of Critical Care Medicine, University of Calgary, Calgary, Alberta, Canada
  • 5Department of Medicine, University of Calgary, Calgary, Alberta, Canada
  • 6Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada
  • 7Division of General Surgery, Department of Surgery, University of Calgary, Calgary, Alberta, Canada
  • 8Division of Critical Care, Department of Surgery, University of Calgary, Calgary, Alberta, Canada
  • 9Department of Surgery, University of British Columbia, Vancouver, British Columbia, Canada
  • 10Department of Family Medicine, Université Laval, Quebec, Quebec, Canada
  • 11Department of Surgery, Université Laval, Quebec, Quebec, Canada
  • 12Institut National d’Excellence en Santé et en Services Sociaux, Quebec, Quebec, Canada
  • 13Division of Critical Care Medicine, Department of Anesthesiology and Critical Care Medicine, Université Laval, Quebec, Quebec, Canada
JAMA Surg. 2016;151(7):622-630. doi:10.1001/jamasurg.2015.5484
Abstract

Importance  The rate of complications among injury admissions has been estimated to be more than 3 times that observed for general admissions, and complications have been targeted as an important quality-of-care metric. Despite the negative effect of complications on resource use and patient mortality and morbidity, there is no standardized method to benchmark trauma centers in terms of in-hospital complications, to our knowledge.

Objectives  To develop a quality indicator (QI) for in-hospital complications that can be used to evaluate the quality of acute injury care and to assess its validity.

Design, Setting, and Participants  Multicenter retrospective cohort study. The setting was a well-established inclusive trauma system in Canada. Participants included all 66 048 moderate or major injury admissions to an adult trauma center between April 1, 2006, and March 31, 2012. The dates of the analysis were January to April 2015.

Main Outcomes and Measures  The primary outcome was the occurrence of at least 1 in-hospital complication. We selected risk-adjustment variables by expert consultation and bootstrap resampling. We evaluated internal validity using measures of discrimination, construct validity, and forecasting.

Results  The study cohort comprised 66 048 patients. Their mean (SD) age was 59 (22) years, and 48.0% were female. Fifteen percent of patients had at least 1 in-hospital complication. The risk-adjustment model has excellent discrimination (area under the curve, 0.81) and calibration. The QI was correlated with the risk-adjusted incidence of mortality (r = 0.71), unplanned readmission (r = 0.43), and mean length of stay (r = 0.68). Hospital performance on the QI from 2007 to 2009 was predictive of performance from 2010 to 2012 (r = 0.82).

Conclusions and Relevance  We developed a QI to benchmark trauma centers on in-hospital complications among injury admissions. The QI is based on data that are routinely collected in most trauma systems and demonstrates good internal validity. The integration of this QI in trauma quality improvement programs will facilitate the identification of quality problems, the implementation of solutions, and the evaluation of their effectiveness. Therefore, the QI has the potential to lead to reductions in mortality, morbidity, and resource use after injury.

Introduction

Preventable injuries led to 232 000 hospital stays, 60 000 disabilities, and 16 000 deaths in 2010 in Canada, with direct costs of CaD $20 billion.1 In the United States, 2.3 million hospital stays and 192 000 deaths in 2014 were attributed to injuries.2 To address the economic and societal burden of injury, health care administrators require tools to measure trauma care quality.3 Benchmarking tools developed to assess mortality for injury admissions have led to significant improvements in patient survival.4-6 However, to further reduce the burden of injury, we need to develop tools to assess nonfatal outcomes.

In Canada, 7.5 complications are recorded for every 100 hospital admissions, with associated costs of CaD $406 billion.7 The in-hospital complication rate for injury admissions has recently been estimated to be 22 per 100 admissions, more than 3 times that observed for general admissions.8 Developing tools to measure complications has been pinpointed as a priority for injury care quality research.9-11 A group of experts recently reached consensus on in-hospital complications that should be monitored in line with injury care quality.12 The face, content, and predictive validity of these complications has been established.8,12,13 We aimed to develop a quality indicator (QI) to measure in-hospital complications among injury admissions and to assess its internal validity.

Methods

We performed a multicenter retrospective cohort study in line with published standards for statistical modeling of health outcomes.14 The study received research ethics approval and did not require patient informed consent, with a waiver granted by the Centre Hospitalier Universitaire research center research ethics committee in Quebec (Quebec, Canada).

Study Population

We selected patients admitted to any trauma center for adults in the inclusive trauma system of Quebec that met criteria for inclusion in the provincial trauma registry. These inclusion criteria were emergency department or in-hospital death, intensive care unit admission, acute care length of stay of at least 3 days, and transfer in from another acute care hospital. The trauma system includes 57 adult trauma centers (3 level I centers, 5 level II centers, 21 level III centers, and 28 level IV centers). Designation is based on American College of Surgeons criteria.15 We excluded patients who were 65 years or older with an isolated hip fracture (ie, no other Abbreviated Injury Scale score >1) and patients who died within 48 hours of injury.

Study Data

We extracted data from the provincial trauma registry, which is completed by each trauma center for all patients meeting the inclusion criteria listed above. Medical abstractors code provincial trauma registry data in each center using standardized coding protocols.15 The Abbreviated Injury Scale is used to describe injuries and is coded using published guidelines.16 All complications noted in patient files are coded. Hospital and ministerial directives encourage close monitoring and accurate recording of in-hospital complications in all provincial hospitals. Physicians note complications and comorbidities in 2 separate areas of the patient file. Admissions occurring 12 months before the injury admission were documented for risk adjustment, and those admissions up to 30 days after discharge from the injury admission were documented to evaluate unplanned readmission through linkage of the provincial trauma registry to the Quebec hospital administrative database.

Primary Outcome

Our primary outcome was the occurrence of at least 1 complication during the injury admission. Complications were selected using a systematic review,13 an expert consensus study,12 and empirical validation.8 The expert consensus was based on a 3-round Delphi study that included 17 physicians from North America and Australia. They represented the fields of emergency medicine, trauma surgery, critical care, orthopedics, and nursing, with an 89.5% participation rate. This study led to a standardized list of 24 complications with excellent face, content,12 and predictive8 validity. Empirical validation showed that the selected complications were associated with a 2.7-fold increase in the odds of mortality and a 2.2-fold increase in the hospital length of stay, whereas complications recorded in the provincial trauma registry but not on the consensus list were associated with no increase in mortality and only a 60% increase in the hospital length of stay.8 Complications were identified using International Statistical Classification of Diseases, 10th Revision codes selected by a panel of physicians with expertise in emergency medicine, surgery, and critical care, as well as a medical coder.8,12 We excluded hospitals reporting complication rates of less than 10% among patients with major trauma (690 patients at 11 trauma centers) to minimize the effect of underreporting.

Statistical Analysis

We used a software program (SAS, version 9.4; SAS Institute Inc) for statistical analyses, and α was established a priori at 5%.

Model Derivation

We first identified the potential risk-adjustment variables and interactions using a literature review and consultation with clinical experts. Comorbidities were quantified using the number of Charlson Comorbidity Index conditions present on arrival at the hospital.17 Fractional polynomials were used to model continuous variables when dummy variables on categories led to significantly lower model discrimination.18 Associations modeled using fractional polynomials were validated by clinical experts using visual inspection of probability plots (eg, probability of complications on age category). We used bootstrap resampling as a model selection technique.19,20 Variables that were statistically significant in more than 70% of 500 samples drawn with replacement from the original sample were kept in the model.19 The risk-adjustment model and QI were derived and validated for level I and II and for level III and IV trauma centers separately to address the fact that they treat different patient populations.

Model Validation

We evaluated the predictive accuracy of the risk-adjustment model using the area under the curve (AUC),21 Cox calibration intercept and slope,22 and Nagelkerke R2 coefficient,23 as well as a plot of observed vs predicted probabilities. We evaluated internal validity of the model by calculating optimism-corrected performance, whereby the AUC, Cox calibration intercept and slope, and Nagelkerke R2 were recalculated on 200 bootstrap samples.20

Quality Indicator

The QI was based on the risk-adjusted incidence (95% CI) of complications in each trauma center. These estimates were obtained by logistic regression modeling. The risk score from the risk-adjustment model was entered as a fixed effect, and the trauma center was entered as a random effect. Using P < .05, benchmarking was performed by comparing each trauma center with the global mean.24

We used discrimination, construct validity, and forecasting to assess the validity of the QI.25 The capacity of the QI to discriminate between hospitals was assessed with estimates of intercenter variance (95% CI).26 We evaluated construct validity by assessing the direction and strength of correlations between the QI and other QIs derived and validated previously, including risk-adjusted mortality,27 unplanned readmission,28 and mean length of stay.29 We evaluated forecasting properties by measuring the correlation between the risk-adjusted complication incidence estimates based on data from 2007 to 2009 and estimates based on data from 2010 to 2012. We assessed the correlation in all validation analyses using Pearson product moment correlation coefficient (95% CI) on complication incidence proportions that were transformed with arcsine square root transformation and weighted by the mean patient volume per year.30

Sensitivity Analyses

We evaluated whether hospital ranks changed when we classified patients who were readmitted within 30 days of discharge for a listed complication (principal diagnosis) as complication events. Correlations between hospital ranks were calculated using Spearman rank correlation coefficient (r), and r > 0.95 was defined as acceptable agreement.

The Glasgow Coma Scale (GCS) score evaluated on emergency department arrival was absent in 56.1% of admissions, and systolic blood pressure was absent in 11.3% of admissions. We used multiple imputation to manage missing data, as described previously.31,32 The imputation model included all independent and dependent variables in the analyses models.

Results

There were 83 226 adult injury admissions recorded in the provincial trauma registry between April 1, 2006, and March 31, 2012. We excluded 16 993 isolated hip fractures and 185 cases with missing injury information. We included 66 048 patients in the study population. Of these patients, 46.2% were 65 years or older with a mean (SD) age of 59 (22) years (Table 1), 48% were female, 22.4% had a New Injury Severity Score of at least 15, and 54.1% were admitted to a level I or II trauma center.

Primary Outcome

Overall, 16 969 complications were recorded during the study period, for a mean of 22 complications per 100 admissions, and 9800 patients (14.8%) had 1 or more in-hospital complications. The most frequent complications included hospital-acquired pneumonia, decubitus ulcer, delirium, deep vein thrombosis, and acute renal failure (Table 2).

Risk-Adjustment Model

The final risk-adjustment model included the following: age, transfer in, GCS score, mechanism of injury, systolic blood pressure, New Injury Severity Score, body region of the most severe injury, number of Charlson Comorbidity Index conditions,33 and number of admissions in the year before the injury admission (P < .001 for all) (Table 3). The potential interactions between age and comorbidities, between injury severity and age, and between comorbidities and injury severity specified a priori did not reach statistical significance.

The model AUC was 0.807, with a bootstrap 95% CI of 0.807 to 0.808 (eTable in the Supplement). The AUC remained unchanged after correction for optimism. The calibration intercept was close to 0, and the slope was close to 1, indicating excellent calibration. In addition, there was high agreement between observed and predicted probabilities throughout the risk range (eFigure 1 and eFigure 2 in the Supplement).

Quality Indicator

The risk-adjusted incidence of complications varied between 8.2% and 24.5% across level I and II trauma centers and between 5.9% and 15.8% across level III and IV trauma centers (Figure). We identified 17 trauma center outliers. The complication incidence was below the global mean for 8 centers and above the global mean for 9 centers.

The risk-adjusted complication incidence showed significant variation between trauma centers, indicating good discrimination (Table 4). Trauma center complication rates were positively correlated with mortality, unplanned readmission, and mean length of stay (all risk adjusted). Performance from 2007 to 2009 was strongly predictive of performance from 2010 to 2012.

Sensitivity Analyses

Among patients discharged alive, 1189 were rehospitalized within 30 days for a complication. The most common complications that led to hospital readmission were wound infection (n = 199) and hospital-acquired pneumonia (n = 163). The correlation coefficient measuring agreement between the QI shown in the Figure and the QI incorporating complications that led to rehospitalization was 0.99.

Discussion

We developed a QI that can be used to monitor in-hospital complications among injury admissions. The QI has excellent discrimination (AUC, 0.81), construct validity, and forecasting properties and can be applied with data collected routinely in most trauma systems. Complications were selected by consensus and validated empirically, and the model used for risk adjustment demonstrates excellent internal and temporal validity. Validation analyses revealed that hospitals with a high risk-adjusted incidence of complications also tended to have higher mortality and readmission rates and longer mean length of stay.

In line with previous studies,34-39 there was significant interhospital variation in the incidence of complications. The complication rate of the highest outlier hospital was more than 3-fold that of the lowest outlier hospital. This observation supports the hypothesis that there is room for significant improvement in complication rates for injury admissions. Indeed, after the implementation of a QI based on complications by the American College of Surgeons National Surgical Quality Improvement Program, 82% of hospitals observed an improvement in surgical complication rates.40 The fact that level I and II trauma centers had a higher risk-adjusted incidence of complications than level III and IV trauma centers could be explained by underreporting of complications in lower-level centers (see the Strengths and Limitations subsection below) or by higher intensity of care, leading to higher exposure to complications in level I and II trauma centers. However, this finding is probably largely explained by the different referral patterns to level I and II centers vs level III and IV centers, which justifies using separate benchmarks for different levels of care. The strong correlations observed between hospital rates of complications and mortality, unplanned readmission, and mean length of stay suggest that reductions in complication rates have the potential to improve other patient outcomes.

Complications are routinely used to evaluate the quality of care for general admissions.34-36,38,39,41 In particular, the US Agency for Healthcare Research and Quality41 proposes a series of safety indicators based on adverse events and complications that are used to monitor the quality of US hospitals using administrative data adjusted for age, sex, modified diagnosis related groups, and comorbidities. Like our QI, their patient safety indicators were based on an initial literature review, followed by an expert consensus process and derivation and validation of a model for risk adjustment. The US National Veterans Affairs Surgical Quality Improvement Program has also developed and validated a complication QI for surgical admissions based on a literature review and expert consensus.42 Although injury care quality has been widely evaluated using complication rates,5,43-46 we are the first to propose a validated complication QI specific to injury admissions. A Trauma and Injury Severity Score–like model for complications has been proposed specifically for surgical trauma admissions.47 This model was based on data from 1 US level I trauma center and an exhaustive list of more than 70 complications previously observed in a trauma population.48 Performance of our model (AUC, 0.82) compares favorably with this surgical trauma model (AUC, 0.64 and 0.74), probably because risk adjustment was limited to age, the Injury Severity Score, and the GCS score in the latter.

Strengths and Limitations

We used standards on statistical modeling of health outcomes to derive and validate the QI.14 Strengths include the following: the risk-adjustment model was derived using a clearly defined population, risk-adjustment variables were selected using expert opinion, the data used are timely and of high quality, comorbidities are differentiated from complications, and we used a model that accounts for the multilevel organization of data. Furthermore, the risk-adjustment model has excellent discrimination and calibration.20 Finally, the selection of complications was based on a systematic literature review,13 followed by a rigorous expert consensus process12 and empirical validation.8

The potential limitations include missing data, external validity, data quality, heterogeneity in the types of complications observed, and the difficulty in identifying complications that are a consequence of low quality of care. First, half of the patients had missing GCS scores.31 However, most patients with missing GCS scores have minor extracranial trauma (with 16.2% in the present study missing GCS scores for traumatic brain injury), and the results of trauma registry simulation studies32,49 suggest that the imputation of physiological data used for case mix adjustment leads to valid effect estimates if the imputation model is correctly specified. Second, our study population resembles injury admissions across Canada and in the United Kingdom, Australia, and rural United States, but we have fewer cases of penetrating trauma and more elderly patients than urban US hospitals. However, the model performance is good in the subpopulation of patients admitted for penetrating injury (AUC, 0.88) and is better for patients younger than 65 years (AUC, 0.85) than for patients 75 years or older (AUC, 0.71). The external validity of the model needs to be evaluated in a completely independent sample. Third, all complications were considered to have the same weight (severity) despite the fact that they do not have the same consequences in terms of morbidity or resource use. However, it would be difficult to identify a criterion on which weights should be based, particularly because complications that led to the largest increase in the hospital length of stay are not associated with increased mortality.8 Fourth, underreporting of complications in patient files is a well-documented problem. Trauma system administrators looking to monitor complication rates must take precautions to ensure accurate reporting and should be aware that interhospital variation may occur because of data quality problems. Fifth, complications were defined as conditions that could plausibly be a consequence of low quality of care because identifying conditions directly imputable to caregiver actions or inactions on a patient level would not be feasible. However, we are confident that if the data quality is high (in particular, accurate reporting of complications), systematic interhospital variations in complication rates identified by the QI are at least in part explained by issues related to the quality of injury care.

Implications

The QI may be implemented in trauma system quality assurance programs to compare complication rates across trauma centers or to monitor complication rates in the system or in specific centers over time. To do so, the risk-adjustment model can be recalibrated using data collected locally. Trauma systems or centers that want to compare their complication rates with an external standard can use the coefficients listed in Table 2 to derive ratios of observed to expected complications.50 The validity of the model should be evaluated in a completely independent sample before its widespread use as a benchmarking tool.

Conclusions

In-hospital complications are common among injury admissions and have a major effect on resource use and patient morbidity. We propose a QI to monitor and benchmark in-hospital complications among injury admissions. This information can be used to identify outlying hospitals, drill down the data to identify reasons behind observed differences, and facilitate the implementation of targeted quality improvement interventions, possibly based on strategies used by trauma centers with low incidences of complications. If used as part of a quality assurance program, the QI could lead to reductions in resource use and improvements in injury outcomes.

Back to top
Article Information

Accepted for Publication: October 19, 2015.

Corresponding Author: Lynne Moore, PhD, Population Health and Optimal Health Practices Research Unit, Trauma–Emergency–Critical Care Medicine, Centre de Recherche du Centre Hospitalier Universitaire de Québec, Hôpital de l’Enfant-Jésus, Université Laval, 1401 18e Rue, Local H-012a, Quebec, QC G1J 1Z4, Canada (lynne.moore@fmed.ulaval.ca).

Published Online: February 3, 2016. doi:10.1001/jamasurg.2015.5484.

Author Contributions: Dr Moore had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Moore.

Critical revision of the manuscript for important intellectual content: All authors.

Administrative, technical, or material support: All authors.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was funded by a Canadian Institutes of Health Research New Investigator Award (Drs Moore and Stelfox), by research grant 110996 from the Canadian Institutes of Health Research (Dr Moore), and by a clinician-scientist award from the Fonds de la Recherche du Québec–Santé (Drs Lauzier and Turgeon).

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Previous Presentation: The abstract was presented at theTrauma 2015 Trauma Association of Canada Annual Scientific Meeting; April 10, 2015; Calgary, Alberta, Canada.

References
1.
Parachute Canada. The cost of injury in Canada. http://www.parachutecanada.org/downloads/research/Cost_of_Injury-2015.pdf. Published 2015. Accessed December 29, 2015.
2.
National Trauma Institute. Trauma statistics. http://www.nationaltraumainstitute.org/home/trauma_statistics.html. Updated February 2014. Accessed September 21, 2015.
3.
World Health Organization. Guidelines for trauma quality improvement programmes. http://www.who.int/violence_injury_prevention/services/traumacare/traumaguidelines/en/. Published 2015. Accessed April 30, 2015.
4.
Moore  L, Turgeon  AF, Lauzier  F,  et al.  Evolution of patient outcomes over 14 years in a mature, inclusive Canadian trauma system.  World J Surg. 2015;39(6):1397-1405.PubMedGoogle ScholarCrossref
5.
Jurkovich  GJ, Mock  C.  Systematic review of trauma system effectiveness based on registry comparisons.  J Trauma. 1999;47(3)(suppl):S46-S55.PubMedGoogle ScholarCrossref
6.
Mackenzie  EJ, Rivara  FP, Jurkovich  GJ,  et al.  The national study on costs and outcomes of trauma.  J Trauma. 2007;63(6)(suppl):S54-S67.PubMedGoogle ScholarCrossref
7.
Stelfox  HT, Bobranska-Artiuch  B, Nathens  A, Straus  SE.  A systematic review of quality indicators for evaluating pediatric trauma care.  Crit Care Med. 2010;38(4):1187-1196.PubMedGoogle ScholarCrossref
8.
Moore  L, Lauzier  F, Stelfox  HT,  et al.  Validation of complications selected by consensus to evaluate the acute phase of adult trauma care: a multicenter cohort study.  Ann Surg. 2015;262(6):1123-1129.PubMedGoogle ScholarCrossref
9.
Hemmila  MR, Jakubus  JL, Wahl  WL,  et al.  Detecting the blind spot: complications in the trauma registry and trauma quality improvement.  Surgery. 2007;142(4):439-448.PubMedGoogle ScholarCrossref
10.
Santana  MJ, Stelfox  HT; Trauma Quality Indicator Consensus Panel.  Development and evaluation of evidence-informed quality indicators for adult injury care.  Ann Surg. 2014;259(1):186-192.PubMedGoogle ScholarCrossref
11.
Stelfox  HT, Straus  SE, Nathens  A, Bobranska-Artiuch  B.  Evidence for quality indicators to evaluate adult trauma care: a systematic review.  Crit Care Med. 2011;39(4):846-859.PubMedGoogle ScholarCrossref
12.
Moore  L, Lauzier  F, Stelfox  HT,  et al.  Complications to evaluate adult trauma care: an expert consensus study.  J Trauma Acute Care Surg. 2014;77(2):322-329.PubMedGoogle ScholarCrossref
13.
Moore  L, Stelfox  HT, Turgeon  AF.  Complication rates as a trauma care performance indicator: a systematic review.  Crit Care. 2012;16(5):R195.PubMedGoogle ScholarCrossref
14.
Krumholz  HM, Brindis  RG, Brush  JE,  et al; American Heart Association; Quality of Care and Outcomes Research Interdisciplinary Writing Group; Council on Epidemiology and Prevention; Stroke Council; American College of Cardiology Foundation.  Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council.  Circulation. 2006;113(3):456-462.PubMedGoogle ScholarCrossref
15.
American College of Surgeons Committee on Trauma.  Resources for Optimal Care of the Injured Patient. Chicago, IL: American College of Surgeons; 2006.
16.
Association for the Advancement of Automotive Medicine.  Abbreviated Injury Scale (AIS). Chicago, IL: AAAM Publications; 1990.
17.
Moore  L, Lavoie  A, Le Sage  N,  et al.  Using information on preexisting conditions to predict mortality from traumatic injury.  Ann Emerg Med.2008;52(4):356-364.e2. doi:10.1016/j.annemergmed.2007.09.007. PubMedGoogle ScholarCrossref
18.
Royston  P, Ambler  G, Sauerbrei  W.  The use of fractional polynomials to model continuous risk variables in epidemiology.  Int J Epidemiol. 1999;28(5):964-974.PubMedGoogle ScholarCrossref
19.
Austin  PC, Tu  JV.  Bootstrap methods for developing predictive models.  Am Stat. 2004;58(2):131-137.Google ScholarCrossref
20.
Harrell  FE  Jr, Lee  KL, Mark  DB.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  Stat Med. 1996;15(4):361-387.PubMedGoogle ScholarCrossref
21.
Hanley  JA, McNeil  BJ.  The meaning and use of the area under a receiver operating characteristic (ROC) curve.  Radiology. 1982;143(1):29-36.PubMedGoogle ScholarCrossref
22.
Cox  DR.  Two further applications of a model for binary regression.  Biometrika.1958;45(3):562-565.Google ScholarCrossref
23.
Nagelkerke  NJ.  A note on a general definition of the coefficient of determination.  Biometrika. 1991;78(3):691-692.Google ScholarCrossref
24.
Moore  L, Lavoie  A, Turgeon  AF,  et al.  The trauma risk adjustment model: a new model for evaluating trauma care.  Ann Surg. 2009;249(6):1040-1046.PubMedGoogle ScholarCrossref
25.
Agency for Healthcare Research and Quality. Inpatient Quality Indicators: AHRQ Composite Measures Workgroup final report. http://www.qualityindicators.ahrq.gov/Downloads/Modules/IQI/IQI_Composite_Development.pdf. Published 2013. Accessed December 29, 2015.
26.
Iezzoni  L.  Risk Adjustment for Measuring Health Care Outcomes. 3rd ed. Chicago, IL: Health Administration Press; 2003.
27.
Moore  L, Hanley  JA, Turgeon  AF, Lavoie  A, Eric  B.  A new method for evaluating trauma centre outcome performance: TRAM-adjusted mortality estimates.  Ann Surg. 2010;251(5):952-958.PubMedGoogle ScholarCrossref
28.
Moore  L, Stelfox  HT, Turgeon  AF,  et al.  Derivation and validation of a quality indicator for 30-day unplanned hospital readmission to evaluate trauma care.  J Trauma Acute Care Surg. 2014;76(5):1310-1316.PubMedGoogle ScholarCrossref
29.
Moore  L, Stelfox  HT, Turgeon  AF,  et al.  Derivation and validation of a quality indicator of acute care length of stay to evaluate trauma care.  Ann Surg. 2014;260(6):1121-1127.PubMedGoogle ScholarCrossref
30.
Zar  JH.  Biostatistical Analysis. 3rd ed. Englewood Cliffs, NJ: Prentice Hall; 1996.
31.
Little  RJA, Rubin  DB.  Statistical Analysis With Missing Data. 2nd ed. New York, NY: Wiley; 2002. Wiley Series in Probability and Statistics.
32.
Moore  L, Lavoie  A, LeSage  N,  et al.  Multiple imputation of the Glasgow Coma Score.  J Trauma. 2005;59(3):698-704.PubMedGoogle Scholar
33.
Charlson  ME, Pompei  P, Ales  KL, MacKenzie  CR.  A new method of classifying prognostic comorbidity in longitudinal studies: development and validation.  J Chronic Dis. 1987;40(5):373-383.PubMedGoogle ScholarCrossref
34.
Baker  GR, Norton  PG, Flintoft  V,  et al.  The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada.  CMAJ. 2004;170(11):1678-1686.PubMedGoogle ScholarCrossref
35.
Wilson  RM, Runciman  WB, Gibberd  RW, Harrison  BT, Newby  L, Hamilton  JD.  The Quality in Australian Health Care Study.  Med J Aust. 1995;163(9):458-471.PubMedGoogle Scholar
36.
Davis  P, Lay-Yee  R, Briant  R, Ali  W, Scott  A, Schug  S.  Adverse events in New Zealand public hospitals, I: occurrence and impact.  N Z Med J. 2002;115(1167):U271.PubMedGoogle Scholar
37.
Brennan  TA, Leape  LL, Laird  NM,  et al.  Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I.  N Engl J Med. 1991;324(6):370-376.PubMedGoogle ScholarCrossref
38.
Vincent  C, Neale  G, Woloshynowych  M.  Adverse events in British hospitals: preliminary retrospective record review.  BMJ. 2001;322(7285):517-519.PubMedGoogle ScholarCrossref
39.
Drösler  SE, Romano  PS, Tancredi  DJ, Klazinga  NS.  International comparability of patient safety indicators in 15 OECD member countries: a methodological approach of adjustment by secondary diagnoses.  Health Serv Res. 2012;47(1, pt 1):275-292.PubMedGoogle ScholarCrossref
40.
Ingraham  AM, Richards  KE, Hall  BL, Ko  CY.  Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach.  Adv Surg. 2010;44:251-267.PubMedGoogle ScholarCrossref
41.
Agency for Healthcare Research and Quality. Patient safety indicators overview. http://www.qualityindicators.ahrq.gov/modules/psi_resources.aspx. Accessed March 31, 2014.
42.
Khuri  SF, Daley  J, Henderson  W,  et al; National VA Surgical Quality Improvement Program.  The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care.  Ann Surg. 1998;228(4):491-507.PubMedGoogle ScholarCrossref
43.
Glance  LG, Dick  AW, Meredith  JW, Mukamel  DB.  Variation in hospital complication rates and failure-to-rescue for trauma patients.  Ann Surg. 2011;253(4):811-816.PubMedGoogle ScholarCrossref
44.
Hemmila  MR, Jakubus  JL, Maggio  PM,  et al.  Real money: complications and hospital costs in trauma patients.  Surgery. 2008;144(2):307-316.PubMedGoogle ScholarCrossref
45.
O’Keefe  GE, Maier  RV, Diehr  P, Grossman  D, Jurkovich  GJ, Conrad  D.  The complications of trauma and their associated costs in a level I trauma center.  Arch Surg. 1997;132(8):920-924.PubMedGoogle ScholarCrossref
46.
Osler  T, Glance  LG, Hosmer  DW.  Complication-associated mortality following trauma: a population-based observational study.  Arch Surg. 2012;147(2):152-158.PubMedGoogle ScholarCrossref
47.
de Jongh  MA, Bosma  E, Verhofstad  MH, Leenen  LP.  Prediction models for complications in trauma patients.  Br J Surg. 2011;98(6):790-796.PubMedGoogle ScholarCrossref
48.
Hoyt  DB, Coimbra  R, Potenza  B,  et al.  A twelve-year analysis of disease and provider complications on an organized level I trauma service: as good as it gets?  J Trauma. 2003;54(1):26-36.PubMedGoogle ScholarCrossref
49.
Moore  L, Hanley  JA, Lavoie  A, Turgeon  A.  Evaluating the validity of multiple imputation for missing physiological data in the National Trauma Data Bank.  J Emerg Trauma Shock. 2009;2(2):73-79.PubMedGoogle ScholarCrossref
50.
Boyd  CR, Tolson  MA, Copes  WS.  Evaluating trauma care: the TRISS method: Trauma Score and the Injury Severity Score.  J Trauma. 1987;27(4):370-378.PubMedGoogle ScholarCrossref
×