Derivation and Validation of a Quality Indicator to Benchmark In-Hospital Complications Among Injury Admissions

Importance  The rate of complications among injury admissions has been estimated to be more than 3 times that observed for general admissions, and complications have been targeted as an important quality-of-care metric. Despite the negative effect of complications on resource use and patient mortality and morbidity, there is no standardized method to benchmark trauma centers in terms of in-hospital complications, to our knowledge.

Objectives  To develop a quality indicator (QI) for in-hospital complications that can be used to evaluate the quality of acute injury care and to assess its validity.

Design, Setting, and Participants  Multicenter retrospective cohort study. The setting was a well-established inclusive trauma system in Canada. Participants included all 66 048 moderate or major injury admissions to an adult trauma center between April 1, 2006, and March 31, 2012. The dates of the analysis were January to April 2015.

Main Outcomes and Measures  The primary outcome was the occurrence of at least 1 in-hospital complication. We selected risk-adjustment variables by expert consultation and bootstrap resampling. We evaluated internal validity using measures of discrimination, construct validity, and forecasting.

Results  The study cohort comprised 66 048 patients. Their mean (SD) age was 59 (22) years, and 48.0% were female. Fifteen percent of patients had at least 1 in-hospital complication. The risk-adjustment model has excellent discrimination (area under the curve, 0.81) and calibration. The QI was correlated with the risk-adjusted incidence of mortality (r = 0.71), unplanned readmission (r = 0.43), and mean length of stay (r = 0.68). Hospital performance on the QI from 2007 to 2009 was predictive of performance from 2010 to 2012 (r = 0.82).

Conclusions and Relevance  We developed a QI to benchmark trauma centers on in-hospital complications among injury admissions. The QI is based on data that are routinely collected in most trauma systems and demonstrates good internal validity. The integration of this QI in trauma quality improvement programs will facilitate the identification of quality problems, the implementation of solutions, and the evaluation of their effectiveness. Therefore, the QI has the potential to lead to reductions in mortality, morbidity, and resource use after injury.

Preventable injuries led to 232 000 hospital stays, 60 000 disabilities, and 16 000 deaths in 2010 in Canada, with direct costs of CaD $20 billion.¹ In the United States, 2.3 million hospital stays and 192 000 deaths in 2014 were attributed to injuries.² To address the economic and societal burden of injury, health care administrators require tools to measure trauma care quality.³ Benchmarking tools developed to assess mortality for injury admissions have led to significant improvements in patient survival.^4-6 However, to further reduce the burden of injury, we need to develop tools to assess nonfatal outcomes.

In Canada, 7.5 complications are recorded for every 100 hospital admissions, with associated costs of CaD $406 billion.⁷ The in-hospital complication rate for injury admissions has recently been estimated to be 22 per 100 admissions, more than 3 times that observed for general admissions.⁸ Developing tools to measure complications has been pinpointed as a priority for injury care quality research.^9-11 A group of experts recently reached consensus on in-hospital complications that should be monitored in line with injury care quality.¹² The face, content, and predictive validity of these complications has been established.^8,12,13 We aimed to develop a quality indicator (QI) to measure in-hospital complications among injury admissions and to assess its internal validity.

We performed a multicenter retrospective cohort study in line with published standards for statistical modeling of health outcomes.¹⁴ The study received research ethics approval and did not require patient informed consent, with a waiver granted by the Centre Hospitalier Universitaire research center research ethics committee in Quebec (Quebec, Canada).

We selected patients admitted to any trauma center for adults in the inclusive trauma system of Quebec that met criteria for inclusion in the provincial trauma registry. These inclusion criteria were emergency department or in-hospital death, intensive care unit admission, acute care length of stay of at least 3 days, and transfer in from another acute care hospital. The trauma system includes 57 adult trauma centers (3 level I centers, 5 level II centers, 21 level III centers, and 28 level IV centers). Designation is based on American College of Surgeons criteria.¹⁵ We excluded patients who were 65 years or older with an isolated hip fracture (ie, no other Abbreviated Injury Scale score >1) and patients who died within 48 hours of injury.

We extracted data from the provincial trauma registry, which is completed by each trauma center for all patients meeting the inclusion criteria listed above. Medical abstractors code provincial trauma registry data in each center using standardized coding protocols.¹⁵ The Abbreviated Injury Scale is used to describe injuries and is coded using published guidelines.¹⁶ All complications noted in patient files are coded. Hospital and ministerial directives encourage close monitoring and accurate recording of in-hospital complications in all provincial hospitals. Physicians note complications and comorbidities in 2 separate areas of the patient file. Admissions occurring 12 months before the injury admission were documented for risk adjustment, and those admissions up to 30 days after discharge from the injury admission were documented to evaluate unplanned readmission through linkage of the provincial trauma registry to the Quebec hospital administrative database.

Our primary outcome was the occurrence of at least 1 complication during the injury admission. Complications were selected using a systematic review,¹³ an expert consensus study,¹² and empirical validation.⁸ The expert consensus was based on a 3-round Delphi study that included 17 physicians from North America and Australia. They represented the fields of emergency medicine, trauma surgery, critical care, orthopedics, and nursing, with an 89.5% participation rate. This study led to a standardized list of 24 complications with excellent face, content,¹² and predictive⁸ validity. Empirical validation showed that the selected complications were associated with a 2.7-fold increase in the odds of mortality and a 2.2-fold increase in the hospital length of stay, whereas complications recorded in the provincial trauma registry but not on the consensus list were associated with no increase in mortality and only a 60% increase in the hospital length of stay.⁸ Complications were identified using International Statistical Classification of Diseases, 10th Revision codes selected by a panel of physicians with expertise in emergency medicine, surgery, and critical care, as well as a medical coder.^8,12 We excluded hospitals reporting complication rates of less than 10% among patients with major trauma (690 patients at 11 trauma centers) to minimize the effect of underreporting.

We used a software program (SAS, version 9.4; SAS Institute Inc) for statistical analyses, and α was established a priori at 5%.

We first identified the potential risk-adjustment variables and interactions using a literature review and consultation with clinical experts. Comorbidities were quantified using the number of Charlson Comorbidity Index conditions present on arrival at the hospital.¹⁷ Fractional polynomials were used to model continuous variables when dummy variables on categories led to significantly lower model discrimination.¹⁸ Associations modeled using fractional polynomials were validated by clinical experts using visual inspection of probability plots (eg, probability of complications on age category). We used bootstrap resampling as a model selection technique.^19,20 Variables that were statistically significant in more than 70% of 500 samples drawn with replacement from the original sample were kept in the model.¹⁹ The risk-adjustment model and QI were derived and validated for level I and II and for level III and IV trauma centers separately to address the fact that they treat different patient populations.

We evaluated the predictive accuracy of the risk-adjustment model using the area under the curve (AUC),²¹ Cox calibration intercept and slope,²² and Nagelkerke R² coefficient,²³ as well as a plot of observed vs predicted probabilities. We evaluated internal validity of the model by calculating optimism-corrected performance, whereby the AUC, Cox calibration intercept and slope, and Nagelkerke R² were recalculated on 200 bootstrap samples.²⁰

The QI was based on the risk-adjusted incidence (95% CI) of complications in each trauma center. These estimates were obtained by logistic regression modeling. The risk score from the risk-adjustment model was entered as a fixed effect, and the trauma center was entered as a random effect. Using P < .05, benchmarking was performed by comparing each trauma center with the global mean.²⁴

We used discrimination, construct validity, and forecasting to assess the validity of the QI.²⁵ The capacity of the QI to discriminate between hospitals was assessed with estimates of intercenter variance (95% CI).²⁶ We evaluated construct validity by assessing the direction and strength of correlations between the QI and other QIs derived and validated previously, including risk-adjusted mortality,²⁷ unplanned readmission,²⁸ and mean length of stay.²⁹ We evaluated forecasting properties by measuring the correlation between the risk-adjusted complication incidence estimates based on data from 2007 to 2009 and estimates based on data from 2010 to 2012. We assessed the correlation in all validation analyses using Pearson product moment correlation coefficient (95% CI) on complication incidence proportions that were transformed with arcsine square root transformation and weighted by the mean patient volume per year.³⁰

We evaluated whether hospital ranks changed when we classified patients who were readmitted within 30 days of discharge for a listed complication (principal diagnosis) as complication events. Correlations between hospital ranks were calculated using Spearman rank correlation coefficient (r), and r > 0.95 was defined as acceptable agreement.

The Glasgow Coma Scale (GCS) score evaluated on emergency department arrival was absent in 56.1% of admissions, and systolic blood pressure was absent in 11.3% of admissions. We used multiple imputation to manage missing data, as described previously.^31,32 The imputation model included all independent and dependent variables in the analyses models.

There were 83 226 adult injury admissions recorded in the provincial trauma registry between April 1, 2006, and March 31, 2012. We excluded 16 993 isolated hip fractures and 185 cases with missing injury information. We included 66 048 patients in the study population. Of these patients, 46.2% were 65 years or older with a mean (SD) age of 59 (22) years (Table 1), 48% were female, 22.4% had a New Injury Severity Score of at least 15, and 54.1% were admitted to a level I or II trauma center.

Overall, 16 969 complications were recorded during the study period, for a mean of 22 complications per 100 admissions, and 9800 patients (14.8%) had 1 or more in-hospital complications. The most frequent complications included hospital-acquired pneumonia, decubitus ulcer, delirium, deep vein thrombosis, and acute renal failure (Table 2).

The final risk-adjustment model included the following: age, transfer in, GCS score, mechanism of injury, systolic blood pressure, New Injury Severity Score, body region of the most severe injury, number of Charlson Comorbidity Index conditions,³³ and number of admissions in the year before the injury admission (P < .001 for all) (Table 3). The potential interactions between age and comorbidities, between injury severity and age, and between comorbidities and injury severity specified a priori did not reach statistical significance.

The model AUC was 0.807, with a bootstrap 95% CI of 0.807 to 0.808 (eTable in the Supplement). The AUC remained unchanged after correction for optimism. The calibration intercept was close to 0, and the slope was close to 1, indicating excellent calibration. In addition, there was high agreement between observed and predicted probabilities throughout the risk range (eFigure 1 and eFigure 2 in the Supplement).

The risk-adjusted incidence of complications varied between 8.2% and 24.5% across level I and II trauma centers and between 5.9% and 15.8% across level III and IV trauma centers (Figure). We identified 17 trauma center outliers. The complication incidence was below the global mean for 8 centers and above the global mean for 9 centers.

The risk-adjusted complication incidence showed significant variation between trauma centers, indicating good discrimination (Table 4). Trauma center complication rates were positively correlated with mortality, unplanned readmission, and mean length of stay (all risk adjusted). Performance from 2007 to 2009 was strongly predictive of performance from 2010 to 2012.

Among patients discharged alive, 1189 were rehospitalized within 30 days for a complication. The most common complications that led to hospital readmission were wound infection (n = 199) and hospital-acquired pneumonia (n = 163). The correlation coefficient measuring agreement between the QI shown in the Figure and the QI incorporating complications that led to rehospitalization was 0.99.

We developed a QI that can be used to monitor in-hospital complications among injury admissions. The QI has excellent discrimination (AUC, 0.81), construct validity, and forecasting properties and can be applied with data collected routinely in most trauma systems. Complications were selected by consensus and validated empirically, and the model used for risk adjustment demonstrates excellent internal and temporal validity. Validation analyses revealed that hospitals with a high risk-adjusted incidence of complications also tended to have higher mortality and readmission rates and longer mean length of stay.

In line with previous studies,^34-39 there was significant interhospital variation in the incidence of complications. The complication rate of the highest outlier hospital was more than 3-fold that of the lowest outlier hospital. This observation supports the hypothesis that there is room for significant improvement in complication rates for injury admissions. Indeed, after the implementation of a QI based on complications by the American College of Surgeons National Surgical Quality Improvement Program, 82% of hospitals observed an improvement in surgical complication rates.⁴⁰ The fact that level I and II trauma centers had a higher risk-adjusted incidence of complications than level III and IV trauma centers could be explained by underreporting of complications in lower-level centers (see the Strengths and Limitations subsection below) or by higher intensity of care, leading to higher exposure to complications in level I and II trauma centers. However, this finding is probably largely explained by the different referral patterns to level I and II centers vs level III and IV centers, which justifies using separate benchmarks for different levels of care. The strong correlations observed between hospital rates of complications and mortality, unplanned readmission, and mean length of stay suggest that reductions in complication rates have the potential to improve other patient outcomes.

Complications are routinely used to evaluate the quality of care for general admissions.^{34-36,38,39,41} In particular, the US Agency for Healthcare Research and Quality⁴¹ proposes a series of safety indicators based on adverse events and complications that are used to monitor the quality of US hospitals using administrative data adjusted for age, sex, modified diagnosis related groups, and comorbidities. Like our QI, their patient safety indicators were based on an initial literature review, followed by an expert consensus process and derivation and validation of a model for risk adjustment. The US National Veterans Affairs Surgical Quality Improvement Program has also developed and validated a complication QI for surgical admissions based on a literature review and expert consensus.⁴² Although injury care quality has been widely evaluated using complication rates,^5,43-46 we are the first to propose a validated complication QI specific to injury admissions. A Trauma and Injury Severity Score–like model for complications has been proposed specifically for surgical trauma admissions.⁴⁷ This model was based on data from 1 US level I trauma center and an exhaustive list of more than 70 complications previously observed in a trauma population.⁴⁸ Performance of our model (AUC, 0.82) compares favorably with this surgical trauma model (AUC, 0.64 and 0.74), probably because risk adjustment was limited to age, the Injury Severity Score, and the GCS score in the latter.

We used standards on statistical modeling of health outcomes to derive and validate the QI.¹⁴ Strengths include the following: the risk-adjustment model was derived using a clearly defined population, risk-adjustment variables were selected using expert opinion, the data used are timely and of high quality, comorbidities are differentiated from complications, and we used a model that accounts for the multilevel organization of data. Furthermore, the risk-adjustment model has excellent discrimination and calibration.²⁰ Finally, the selection of complications was based on a systematic literature review,¹³ followed by a rigorous expert consensus process¹² and empirical validation.⁸

The potential limitations include missing data, external validity, data quality, heterogeneity in the types of complications observed, and the difficulty in identifying complications that are a consequence of low quality of care. First, half of the patients had missing GCS scores.³¹ However, most patients with missing GCS scores have minor extracranial trauma (with 16.2% in the present study missing GCS scores for traumatic brain injury), and the results of trauma registry simulation studies^32,49 suggest that the imputation of physiological data used for case mix adjustment leads to valid effect estimates if the imputation model is correctly specified. Second, our study population resembles injury admissions across Canada and in the United Kingdom, Australia, and rural United States, but we have fewer cases of penetrating trauma and more elderly patients than urban US hospitals. However, the model performance is good in the subpopulation of patients admitted for penetrating injury (AUC, 0.88) and is better for patients younger than 65 years (AUC, 0.85) than for patients 75 years or older (AUC, 0.71). The external validity of the model needs to be evaluated in a completely independent sample. Third, all complications were considered to have the same weight (severity) despite the fact that they do not have the same consequences in terms of morbidity or resource use. However, it would be difficult to identify a criterion on which weights should be based, particularly because complications that led to the largest increase in the hospital length of stay are not associated with increased mortality.⁸ Fourth, underreporting of complications in patient files is a well-documented problem. Trauma system administrators looking to monitor complication rates must take precautions to ensure accurate reporting and should be aware that interhospital variation may occur because of data quality problems. Fifth, complications were defined as conditions that could plausibly be a consequence of low quality of care because identifying conditions directly imputable to caregiver actions or inactions on a patient level would not be feasible. However, we are confident that if the data quality is high (in particular, accurate reporting of complications), systematic interhospital variations in complication rates identified by the QI are at least in part explained by issues related to the quality of injury care.

The QI may be implemented in trauma system quality assurance programs to compare complication rates across trauma centers or to monitor complication rates in the system or in specific centers over time. To do so, the risk-adjustment model can be recalibrated using data collected locally. Trauma systems or centers that want to compare their complication rates with an external standard can use the coefficients listed in Table 2 to derive ratios of observed to expected complications.⁵⁰ The validity of the model should be evaluated in a completely independent sample before its widespread use as a benchmarking tool.

In-hospital complications are common among injury admissions and have a major effect on resource use and patient morbidity. We propose a QI to monitor and benchmark in-hospital complications among injury admissions. This information can be used to identify outlying hospitals, drill down the data to identify reasons behind observed differences, and facilitate the implementation of targeted quality improvement interventions, possibly based on strategies used by trauma centers with low incidences of complications. If used as part of a quality assurance program, the QI could lead to reductions in resource use and improvements in injury outcomes.

Accepted for Publication: October 19, 2015.

Corresponding Author: Lynne Moore, PhD, Population Health and Optimal Health Practices Research Unit, Trauma–Emergency–Critical Care Medicine, Centre de Recherche du Centre Hospitalier Universitaire de Québec, Hôpital de l’Enfant-Jésus, Université Laval, 1401 18e Rue, Local H-012a, Quebec, QC G1J 1Z4, Canada (lynne.moore@fmed.ulaval.ca).

Published Online: February 3, 2016. doi:10.1001/jamasurg.2015.5484.

Author Contributions: Dr Moore had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: All authors.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Moore.

Critical revision of the manuscript for important intellectual content: All authors.

Administrative, technical, or material support: All authors.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was funded by a Canadian Institutes of Health Research New Investigator Award (Drs Moore and Stelfox), by research grant 110996 from the Canadian Institutes of Health Research (Dr Moore), and by a clinician-scientist award from the Fonds de la Recherche du Québec–Santé (Drs Lauzier and Turgeon).

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Previous Presentation: The abstract was presented at theTrauma 2015 Trauma Association of Canada Annual Scientific Meeting; April 10, 2015; Calgary, Alberta, Canada.