Compliance With Evidence-Based Guidelines and Interhospital Variation in Mortality for Patients With Severe Traumatic Brain Injury

Importance  Compliance with evidence-based guidelines in traumatic brain injury (TBI) has been proposed as a marker of hospital quality. However, the association between hospital-level compliance rates and risk-adjusted clinical outcomes for patients with TBI remains poorly understood.

Objective  To examine whether hospital-level compliance with the Brain Trauma Foundation guidelines for intracranial pressure monitoring and craniotomy is associated with risk-adjusted mortality rates for patients with severe TBI.

Design, Setting, and Participants  All adult patients (N = 734) who presented to a regional consortium of 14 hospitals from January 1, 2009, through December 31, 2010, with severe TBI (ie, blunt head trauma, Glasgow Coma Scale score of <9, and abnormal intracranial findings from computed tomography of the head). Data analysis took place from December 2013 through January 2015. We used hierarchical mixed-effects models to assess the association between hospital-level compliance with Brain Trauma Foundation guidelines and mortality rates after adjusting for patient-level demographics, severity of trauma (eg, mechanism of injury and Injury Severity Score), and TBI-specific variables (eg, cranial nerve reflexes and findings from computed tomography of the head).

Main Outcomes and Measures  Hospital-level risk-adjusted inpatient mortality rate and hospital-level compliance with Brain Trauma Foundation guidelines for intracranial pressure monitoring and craniotomy.

Results  Unadjusted mortality rates varied by site from 20.0% to 50.0% (median, 42.6; interquartile range, 35.5-46.2); risk-adjusted rates varied from 24.3% to 56.7% (median, 41.1; interquartile range, 36.4-47.8). Overall, only 338 of 734 patients (46.1%) with an appropriate indication underwent placement of an intracranial pressure monitor and only 134 of 335 (45.6%) underwent craniotomy. Hospital-level compliance ranged from 9.6% to 65.2% for intracranial pressure monitoring and 6.7% to 76.2% for craniotomy. Despite widespread variation in compliance across hospitals, we found no association between hospital-level compliance rates and risk-adjusted patient outcomes (Spearman ρ = 0.030 [P = .92] for ICP monitoring and Spearman ρ = −0.066 [P = .83] for craniotomy).

Conclusions and Relevance  Hospital-level compliance with evidence-based guidelines has minimal association with risk-adjusted outcomes in patients with severe TBI. Our results suggest that caution should be taken before using compliance with these measures as independent quality metrics. Given the complexity of TBI care, outcomes-based metrics, including functional recovery, may be more accurate than current process measures at determining hospital quality.

Traumatic brain injury (TBI) remains a substantial source of morbidity and mortality in the United States, accounting for nearly one-third of all injury-related deaths.^1-3 Following reports of widespread variation in care,⁴ the Brain Trauma Foundation (BTF) published the first set of clinical guidelines for the treatment of TBI in 1995. In their most recent revision, the BTF guidelines include management strategies, treatment thresholds, and indications for the use of invasive procedures, namely, intracranial pressure (ICP) monitoring and craniotomy.^5-7 In addition to receiving more streamlined clinical care, individuals who are treated based on BTF guidelines appear to have better clinical outcomes, including lower risk-adjusted mortality rates and higher rates of functional recovery.^8-10 As a result, compliance with these guidelines has been proposed as a marker of hospital quality, and hospitals throughout the country are attempting to improve their TBI outcomes by increasing their levels of guideline-compliant care.¹¹

However, despite substantial face validity, there is growing evidence that guideline compliance alone is an inaccurate and inadequate measure of hospital quality.^12,13 Multiple studies have demonstrated a loose association between compliance with Medicare’s Hospital Compare measures (eg, giving aspirin within 24 hours of an acute myocardial infarction) and inpatient mortality rates.^14,15 More specific to surgery, hospitals that score well on perioperative safety measures do not appear to have lower rates of surgical site infection,¹⁶ venous thromboembolism,¹⁷ or mortality.^18-20 To our knowledge, there remains no definitive evaluation of the connection between guideline compliance and hospital quality in severe TBI.

Our study had the following 2 aims: (1) to document levels of compliance with BTF guidelines for ICP monitoring and craniotomy in a large regional trauma system and (2) to examine the association between hospital-level compliance with these guidelines and risk-adjusted mortality rates. In so doing, we sought to determine whether compliance with BTF guidelines represents a viable marker of hospital quality in the treatment of patients with severe TBI.

The Los Angeles (LA) County Trauma Consortium was formed in 2013 as a collaboration between LA County’s 14 trauma centers, the county’s Emergency Medical Services Agency, and health services researchers from 2 local universities. A description of the consortium and its research objectives has been published previously.²¹Quiz Ref ID Briefly, we developed a prospective registry of all patients with severe TBI in LA County during a 2-year period (January 1, 2009, through December 31, 2010). Inclusion was based on 3 criteria: (1) blunt head trauma, (2) a Glasgow Coma Scale (GCS) score of less than 9 on arrival, and (3) abnormal intracranial findings from the initial computed tomographic scan of the head. These criteria were chosen based on the definition of severe TBI in the BTF guidelines. We excluded patients who died on arrival or were younger than 18 years at the time of injury. Of the 14 hospitals in the consortium, 1 is a designated children’s hospital and was effectively eliminated from our analysis based on the age restriction.

Trained trauma-program managers at each center prospectively identified patients who met the inclusion criteria and abstracted relevant patient-level data into our protected electronic registry. Hospital characteristics were obtained from 2 sources. Basic structural characteristics were taken from the American Hospital Association website (http://www.ahadataviewer.com). Trauma-program managers were also surveyed regarding care practices at their institution during the study period, specifically whether there was a designated neuro–critical care unit, a general or neurosurgery residency program, or a clinical protocol for the treatment of severe TBI. This study was approved by the Office of Human Research Protection Program at UCLA.

Our primary outcome at the patient level was death due to any cause during the initial hospitalization for TBI (inpatient mortality). Placement of an ICP monitor in the first 72 hours after arrival was recorded as an independent field in our clinical registry. Craniotomy during the first 72 hours was determined based on International Classification of Diseases, Ninth Edition, procedure codes (eTable 1 in the Supplement).

Covariates were selected to parallel the risk-adjustment techniques used by the Trauma Quality Improvement Program (TQIP), the largest clinical registry of traumatically injured patients in the United States, for patients with TBI. The TQIP includes the following 23 variables in its regression models: demographics, vital signs, mechanism of injury, Injury Severity Score, GCS score, and 14 medical comorbidities. Three variables (dialysis, concurrent corticosteroid use, and cardiac arrest) were recorded but not used in the statistical analyses because of their low prevalence and high collinearity.

In addition to the 20 TQIP variables, we included the following 11 TBI-specific variables captured by our registry: pupil reactivity, international normalized ratio, and 9 separate intracranial findings on the initial computed tomographic scan of the head (Table 1). Pupil reactivity was defined as present or absent based on the initial neurologic or trauma surgery consultation. International normalized ratio values were divided into normal (≤1.4) and elevated (>1.4) based on the initial measurements in the emergency department. All 9 findings from computed tomography of the head were coded independently as present or absent based on the official radiology report.

We determined compliance with guidelines for each hospital by dividing the number of patients who received a particular therapy (ie, ICP monitoring or craniotomy) by the number of patients with an indication for that therapy based on BTF guidelines. The BTF guidelines currently recommend ICP monitoring for all patients with TBI who have a GCS score of less than 9⁵; because this group included all patients in our sample, we divided the number of patients who underwent ICP monitoring at a given hospital by that hospital’s sample size. The BTF guidelines also recommend craniotomy for patients with a GCS score of less than 9 in the setting of epidural hematoma, subdural hematoma with signs of midline shift or mass effect, or intraparenchymal contusion with signs of mass effect.⁷ We identified patients at each hospital who met these criteria and then calculated the rate of patients who underwent craniotomy in this group.

We developed a hierarchical mixed-effects logistic regression model to predict inpatient mortality after controlling for all 31 patient-level covariates (20 TQIP + 11 TBI-specific variables) and hospital-specific random effects. We predicted the risk-adjusted mortality rates for each hospital after controlling for patient mix and then used empirical Bayesian techniques to adjust each estimate for its reliability.^22-24 This approach, also referred to as shrinkage adjustment, filters out statistical noise due to the small sample size of certain clusters and reweights values toward the overall sample mean based on each estimate’s reliability.²⁵

Bivariate tables were generated to compare patients who died during their hospitalization with those who survived until discharge. For unadjusted data, we used χ² tests of independence for categorical variables and either 2-sample t tests or Mann-Whitney tests for continuous variables, depending on the distribution. We also ranked hospitals by their compliance with BTF guidelines and then divided hospitals into terciles based on these rankings. This process was conducted separately for ICP monitoring and craniotomy such that the distribution of hospitals within terciles could differ for each quality metric.

We evaluated the association between hospital-level compliance rates and risk-adjusted mortality rates in 2 manners. First, we plotted hospitals’ compliance rates against their risk-adjusted mortality rates and calculated Spearman rank correlations. This analysis was conducted separately for each quality metric. Second, we added hospitals’ guideline compliance in terciles to our hierarchical regression model and predicted the risk-adjusted mortality rate for each tercile. This analysis was also conducted separately for each quality metric using only patients who were eligible for that measure. We then compared unadjusted mortality rates across terciles using the Kruskal-Wallis test and risk-adjusted mortality rates using Wald tests with standard errors generated via the delta method. All P values were 2-sided and P ≤ .05 was considered significant.

We conducted several sensitivity analyses to test the robustness of our estimates. First, we calculated the reliability of our registry data by comparing our independent data field for ICP monitoring with the International Classification of Diseases, Ninth Edition, procedure codes (eTable 2 in the Supplement). Because this method demonstrated a high level of consistency (κ = 0.852), we used our independent data field. Second, 43 patients (5.7%) had missing data in the following 2 variables: international normalized ratio (36 patients) and heart rate (7 patients). Multiple analyses demonstrated no informative pattern to the missing data. To use all the available data, we imputed missing values via a multivariate normal regression technique after controlling for the full set of covariates, including placement of an ICP monitor, craniotomy, and mortality. As an additional test of our findings, we conducted our analyses using only complete cases. This analysis did not affect our results, and we reported regression results for the full sample with imputed values for patients with missing data. Third, we restricted our analyses to patients with an isolated head injury (Abbreviated Injury Score <2 in all body regions except the head and neck); this restriction did not affect our findings and we report the data for the entire sample. Finally, we conducted our regression analyses using hospital compliance as a continuous variable. Because this method also did not affect our results, we reported terciles for interpretability. Statistical analyses were conducted from December 2013 through January 2015 using STATA/IC, version 13.0 (StataCorp LP).

During the study period, 753 adult patients sustained severe TBI; 19 patients (2.5%) died on arrival and were excluded from further analysis. Of the 734 patients who survived to admission, the mean age was 46.3 years, 177 patients (24.1%) were female, and 296 (40.3%) were Hispanic (Table 1). The median Injury Severity Score was 26 (interquartile range [IQR], 21-35) and the median GCS score was 3 (IQR, 3-6). Falls were the most common mechanism of injury (235 [32.0%]) followed by auto vs pedestrian (211 [28.8%]). Nearly two-thirds of patients had at least 1 reactive pupil on arrival (482 [65.7%]). Overall, 296 patients (40.3%) died during their inpatient hospitalization.

Most trauma centers were not for profit (9 [69.2%]) and maintained an academic affiliation (5 [38.5%] major and 4 [30.8%] minor) (Table 2). Less than half had a neurocritical care unit (5 [38.5%]), a general or neurosurgery resident program (6 [46.2%]), or a protocol for treating severe TBI during the study period (6 [46.2%]). Unadjusted mortality rates ranged from 20.0% to 50.0% by hospital (median, 42.6%; IQR, 35.5%-46.2%); risk-adjusted mortality rates ranged from 24.3% to 56.7% (median, 41.1%; IQR, 36.4%-47.8%).

Only 338 patients (46.1%) underwent placement for an ICP monitor despite all patients meeting BTF criteria. Quiz Ref IDRates of compliance with ICP monitoring ranged by center from 9.6% to 65.2% (median, 47.4; IQR, 40.0-51.7); the lowest tercile of hospitals had a median compliance rate of 38.7% compared with 54.8% in the highest tercile (Table 2). Quiz Ref IDIn our sample, 335 patients (45.6%) had an indication for craniotomy and 134 of these patients (40.0%) underwent the procedure. Rates of compliance with craniotomy also varied by center from 6.7% to 76.2% (median, 41.7%; IQR, 33.3%-50.0%); the lowest tercile of hospitals had a median compliance rate of only 9.5% compared with 61.7% in the highest tercile.

The Figure presents each hospital according to its rates of guideline compliance and risk-adjusted mortality. Quiz Ref IDThere was no correlation between compliance rates and mortality for either ICP monitoring (Spearman ρ = 0.030; P = .92) (Figure, A) or craniotomy (Spearman ρ = −0.066; P = .83) (Figure, B). Similarly, there was no association between compliance with BTF guidelines and either unadjusted or risk-adjusted mortality based on our hierarchical regression model (Table 3). For both quality metrics, there appeared to be a trend toward lower mortality rates in hospitals with higher compliance based on unadjusted data; however, this trend disappeared after risk adjustment. The middle tercile on compliance for both ICP monitoring and craniotomy had the lowest risk-adjusted mortality rates (33.8% and 47.1%, respectively).

In response to widespread variation in care, the BTF established clinical practice guidelines for TBI in the hope of promoting high-quality health care. Despite their efforts, we found no relationship between hospitals’ compliance with 2 BTF guidelines and risk-adjusted mortality in a large regional trauma system. Quiz Ref IDOur results suggest that caution should be taken before using compliance with these measures as independent quality metrics.

While ours is the first study, to our knowledge, to explore differences in hospitals’ use of craniotomy, other studies have investigated the association between hospital compliance with ICP monitoring and risk-adjusted mortality in patients with severe TBI.^26,27 Using data from TQIP, Alali and colleagues²⁶ reported improved survival rates for patients who were treated at hospitals in the highest vs the lowest quartile of ICP monitor use after controlling for patient- and hospital-level covariates. There are several possible explanations for our differing results. First, Alali and colleagues²⁶ excluded patients with either “nonsurvivable”^26(p1738) head injuries or significant nonhead injuries. Because we included all patients with severe TBI, our patient sample was more severely injured by comparison, as is evident by the difference in unadjusted mortality rates (35.5% in TQIP vs 40.3% in our sample). As a result, certain invasive treatments may have been less effective in our population, making interhospital differences in their use a less significant predictor of patient outcomes. Second, TQIP includes mostly large academic hospitals, which tend to be more similar to one another than to other hospitals in a given geographic region. Of our 13 adult trauma centers, only 3 currently participate in TQIP (all level I academic centers) and fewer than half have access to many of the hallmarks of academic medical centers, including specialized intensive care units or residency programs. Given the variety of hospitals involved, differences in compliance with BTF guidelines may simply play a smaller role in determining hospital quality for regional trauma systems, especially when one considers the fact that differences in ICP monitoring explained less than 10% of the interhospital variation in mortality rates in TQIP.²⁶

Our results also appear to conflict with multiple patient-level analyses, including one from our consortium, that have consistently demonstrated a survival benefit for ICP monitoring in patients with severe TBI.^21,28,29 Again, there are at least 2 possible explanations for this discrepancy: (1) differences in who receives ICP monitoring and (2) how effectively information from invasive monitors is incorporated into clinical management. First, even among the subset of patients with severe TBI, there is considerable heterogeneity such that not all patients benefit equally from invasive monitoring.³⁰ Therefore, 2 hospitals with similar rates of ICP monitoring might effectively be providing different levels of care—and achieving different patient outcomes—if the subset of patients being monitored differs significantly between the 2 institutions.

Second, by recording only whether an ICP monitor was placed, guidelines may insufficiently capture critical aspects in the management of severe TBI, including how health care professionals react to changes in ICP or use related medical therapies (eg, hypertonic saline solution) to treat intracranial hypertension. Many publications regarding ICP monitoring suggest that clinical outcomes depend more on the timely management of intracranial hypertension than the use of particular medications or procedures.^31-33 This idea is typified by the only randomized controlled trial³⁴ of ICP monitoring, which showed no difference in mortality between monitored and nonmonitored patients but took advantage of trained neurologists and neurosurgeons at the bedside acting as “functional” ICP monitors. Therefore, if hospitals that place more ICP monitors do not also use the additional information to adapt their treatment of intracranial hypertension, then the rates of placement may be uncorrelated (or even inversely correlated) with risk-adjusted mortality rates.

Our study has several limitations. First, our sample included the most severely injured patients in LA County and, as such, our results may not generalize to other regions or to less severely injured patients. Second, because of the delay between data collection and analysis, it is possible that health care practices have changed. However, as neither the guidelines nor the members of the trauma system changed during this period, we believe our results remain applicable to the current state of trauma care in LA County. Third, our sample size was relatively small at the hospital level, which limited our ability to compare hospitals directly and to control for hospital-level structural variables, such as teaching status or patient volume. Because other studies have suggested these factors may affect quality,^35,36 our results may be confounded if differences in structural characteristics drive variation in both compliance and mortality rates. However, because the BTF guidelines apply equally to patients regardless of their treating hospital, we believe that controlling for certain hospital characteristics is not only irrelevant but may bias our results by explaining away meaningful differences in patient outcomes. Additional studies using a larger hospital-level sample are needed to better understand the associations between structural variables, inpatient mortality, and guideline compliance. Fourth, we had no information on the rates of withdrawal of care and therefore could not control for the ways in which different withdrawal practices between centers might have affected center-specific mortality rates. Finally, we were unable to characterize patients’ functional status at the time of discharge, and it is unclear whether differences in the use ICP monitoring and craniotomy are associated with differences in functional recovery.

These limitations notwithstanding, we believe our results have important implications for quality assessment in severe TBI. The debate between process and outcome has continued since the earliest publications on quality measurement.³⁷ Process measures, including guideline compliance, offer the ease of measurement, sensitivity to change, and real-time feedback demanded by many proponents of quality improvement. Their downside, however, rests in the difficulty of determining which processes actually lead to better care.³⁸ Our results suggest that current BTF guidelines fail to improve hospital quality both by measuring the wrong processes (ie, if a monitor is placed rather than how a monitor is used in management) and by doing so in the wrong group of patients (all patients with severe TBI rather than only those who would benefit from invasive monitoring).

However, recent experiences from several large multi-institutional quality-improvement programs suggest that even tracking the “perfect” process measure in the most appropriate subset of patients may be insufficient to actually improve patient outcomes.^16,39,40 These findings have 3 implications for quality measurement, especially for conditions as complex as TBI. First, multiple related process measures may be needed for each clinical outcome (eg, placement of an ICP monitor plus thresholds for treatment plus goals for maintaining cerebral perfusion pressure).^41,42 Second, process measurement must be combined with clinically important outcome metrics, such as mortality and functional recovery, to prevent hospitals from improving performance on one metric at the expense of others. Finally, it is not enough to simply measure performance and track progress over time.⁴³ Instead, hospitals must build systems that use these data to influence physician behavior and improve patient care.

Despite improvements in care, mortality due to TBI remains both common and variable from hospital to hospital. Our results demonstrate no association between hospitals’ compliance with 2 BTF guidelines and risk-adjusted mortality, suggesting that neither measure should be used as an independent marker of hospital quality.

Accepted for Publication: April 3, 2015.

Corresponding Author: Aaron J. Dawes, MD, Ronald Reagan UCLA Medical Center, Department of Surgery, David Geffen School of Medicine, UCLA, 757 Westwood Plaza, Room B7-11, Los Angeles, CA 90095 (adawes@mednet.ucla.edu).

Published Online: July 22, 2015. doi:10.1001/jamasurg.2015.1678.

Author Contributions: Drs Dawes and Ko had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Dawes, Sacks, Gruen, Preston, Gorospe, Ko.

Acquisition, analysis, or interpretation of data: Dawes, Sacks, Cryer, Gruen, Preston, Gorospe, Cohen, McArthur, Russell, Maggard-Gibbons.

Drafting of the manuscript: Dawes, Sacks.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Dawes, Sacks, Cryer, Preston, Gorospe, McArthur, Russell.

Administrative, technical, or material support: Cryer, Preston, Gorospe, Cohen.

Study supervision: Cryer, Gruen, Russell, Maggard-Gibbons, Ko.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was supported by the Veterans Affairs Robert Wood Johnson Clinical Scholars Program (Drs Dawes and Sacks) and the Veterans Affairs Office of Academic Affiliations (Dr Dawes).

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: The Los Angeles County Trauma Consortium members were Cynthia Marin, RN, MSN, and Pavel Petrik, MD, Antelope Valley Hospital, Lancaster, California; Laura Schneider, RN, and Gudata Hinika, MD, California Hospital Medical Center, Los Angeles; Elizabeth Cleek, RN, and Jeffrey Upperman, MD, Children’s Hospital Los Angeles, Los Angeles, California; Heidi Hotz, RN, and Daniel Margulies, MD, Cedars-Sinai Medical Center, Los Angeles, California; Kimberly Murphy, RN, MSN, and David Hanpeter, MD, Providence Holy Cross Medical Center, Mission Hills, California; Robin Tyler, RN, and Brant Putnam, MD, Harbor-UCLA Medical Center, Torrance, California; Susan Thompson, RN, MSN, and Amal Obaid, MD, Huntington Memorial Hospital, Pasadena, California; Gilda Cruz-Manglapus, RN, and Ranbir Singh, MD, Henry Mayo Newhall Memorial Hospital, Valencia, California; Desiree Thomas, RN, and Brian Acker, MD, Long Beach Memorial Medical Center, Long Beach, California; Melanie Crowley, RN, MSN, and Shawki Saad, MD, Northridge Hospital Medical Center, Northridge, California; Renee Smith, RN, MSN, and Tchaka Shepherd, MD, St Francis Medical Center, Lynwood, California; Patricia Meier, RN, and James Murray, MD, St Mary Medical Center, Long Beach, California; Marilyn Cohen, RN, and H. Gill Cryer, MD, PhD, Ronald Regan UCLA Medical Center, Los Angeles, California; and Sixta Navarrete, RN, and Demetrios Demetriades, MD, LAC + USC (Los Angeles County and University of Southern California) Medical Center, Los Angeles.

Previous Presentation: This study was presented in part at the 86th Annual Meeting of the Pacific Coast Surgical Association; February 20, 2015; Monterey, California.

Additional Contributions: Nader Pouratian, MD, and Matt Garrett, MD, Department of Neurosurgery, UCLA, and Paul Vespa, MD, Departments of Neurosurgery and Neurology, UCLA, contributed to the project and Susan Paddock, PhD, RAND Pardee Graduate School, and John Adams, PhD, MS, Kaiser Permanente Department of Research and Evaluation, provided statistical review. None received financial compensation.