The validation data were acquired from the Brabant region of the Netherlands. Error bars indicate 95% CIs.
EMS indicates emergency medical services; TT app, Trauma Triage App (Synappz).
eMethods. The National Field Triage Protocol of the Netherlands
Customize your JAMA Network experience by selecting one or more topics from the list below.
Identify all potential conflicts of interest that might be relevant to your comment.
Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.
Err on the side of full disclosure.
If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.
Not all submitted comments are published. Please see our commenting policy for details.
van Rein EAJ, van der Sluijs R, Voskens FJ, et al. Development and Validation of a Prediction Model for Prehospital Triage of Trauma Patients. JAMA Surg. 2019;154(5):421–429. doi:10.1001/jamasurg.2018.4752
Can a new prehospital prediction model for trauma triage lower the undertriage rate to approximately 10%, with a maximum overtriage rate of 50%?
In this multicenter cohort study that included 4950 patients with trauma, 8 highly significant predictors associated with injury severity were selected for the prediction model. The new prehospital trauma triage prediction model was externally validated and may lower the undertriage rate to 11.2% with an overtriage rate of 50.0%.
This prediction model can be integrated in a mobile app for emergency medical services professionals to improve the prehospital trauma triage.
Prehospital trauma triage protocols are used worldwide to get the right patient to the right hospital and thereby improve the chance of survival and avert lifelong disabilities. The American College of Surgeons Committee on Trauma set target levels for undertriage rates of less than 5%. None of the existing triage protocols has been able to achieve this target in isolation.
To develop and validate a new prehospital trauma triage protocol to improve current triage rates.
Design, Setting, and Participants
In this multicenter cohort study, all patients with trauma who were 16 years and older and transported to a trauma center in 2 different regions of the Netherlands were included in the analysis. Data were collected from January 1, 2012, through June 30, 2014, in the Central Netherlands region for the design data cohort and from January 1 through December 31, 2015, in the Brabant region for the validation cohort. Data were analyzed from May 3, 2017, through July 19, 2018.
Main Outcomes and Measures
A new prediction model was developed in the Central Netherlands region based on prehospital predictors associated with severe injury. Severe injury was defined as an Injury Severity Score greater than 15. A full-model strategy with penalized maximum likelihood estimation was used to construct a model with 8 predictors that were chosen based on clinical reasoning. Accuracy of the developed prediction model was assessed in terms of discrimination and calibration. The model was externally validated in the Brabant region.
Using data from 4950 patients with trauma from the Central Netherlands region for the design data set (58.3% male; mean [SD] age, 47  years) and 6859 patients for the validation Brabant region (52.2% male; mean [SD] age, 51  years), the following 8 significant predictors were selected for the prediction model: age; systolic blood pressure; Glasgow Coma Scale score; mechanism criteria; penetrating injury to the head, thorax, or abdomen; signs and/or symptoms of head or neck injury; expected injury in the Abbreviated Injury Scale thorax region; and expected injury in 2 or more Abbreviated Injury Scale regions. The prediction model showed a C statistic of 0.823 (95% CI, 0.813-0.832) and good calibration. The cutoff point with a minimum specificity of 50.0% (95% CI, 49.3%-50.7%) led to a sensitivity of 88.8% (95% CI, 87.5%-90.0%). External validation showed a C statistic of 0.831 (95% CI, 0.814-0.848) and adequate calibration.
Conclusions and Relevance
The new prehospital trauma triage prediction model may lower undertriage rates to approximately 10% with an overtriage rate of 50%. The next step should be to implement this prediction model with the use of a mobile app for emergency medical services professionals.
In first-world countries, systems of trauma care substantially reduce mortality associated with injury.1-3 Multiple studies2,3 focused on optimizing such trauma systems by balancing timely access to expert care, the ability of health care professionals and teams to attain and sustain the necessary expertise, and the cost-effectiveness of the overall trauma system. Fundamental to the trauma system is prehospital trauma triage, the goal of which is to identify at-risk patients and provide early and resuscitative care while transporting the patient to the highest appropriate level of care.4 Identification of severely injured patients is challenging. Only 0.5% of those injured are severely injured.4 Other challenges include assessment of the incident scene and the patient’s physiological state and risk of deterioration, identification of obvious injuries, and consideration of adjuvant factors, such as age.
Emergency medical services (EMS) professionals must differentiate between patients on the scene, often in adverse situations, without use of advanced diagnostic tools. Therefore, the EMS professionals are often forced to perform the prehospital trauma triage based on incomplete data. The importance of prehospital trauma triage cannot be understated; a structured and reliable process is crucial.
Worldwide, protocols are used to help identify severely injured patients. However, none of the existing protocols can achieve the recommended triage rates.5-7 All are simplistic and static tools, whereas patients are dynamic, and more advanced methods are available to use on the scene, such as a prediction model in a mobile app.8,9
Undertriage occurs when severely injured patients are not transported to a higher-level trauma center and results in delayed care and increased mortality and morbidity.1,2 Overtriage occurs when patients without severe injuries are taken to a higher-level trauma center, often incurring preventable cost and resource consumption.3,10 In the Netherlands, level I trauma centers are considered higher-level trauma centers and levels II and III are considered lower-level trauma centers. The American College of Surgeons Committee on Trauma set target levels of undertriage at less than 5% and overtriage as great as 25% to 35%.11 The National Health Institute of the Netherlands recommends an undertriage rate of less than 10%.12 A target for the overtriage rate was not set; the overtriage rate, however, should depend on the regional circumstances and could be as great as 50%.
In the Netherlands, a protocol based on the Field Triage Decision Scheme is used nationwide (eMethods in the Supplement).13 The mean undertriage rate was 33% in 2015.14 A recent study15 showed an undertriage rate of 22% and an overtriage rate of 31% in a single inclusive trauma system. At present, existing protocols achieve undertriage rates ranging from 22% to 44%, with overtriage rates ranging from 11% to 22% in a general trauma population.6,16,17 The aim of the present study was therefore to develop and validate a new prehospital trauma triage prediction model that attempts to lower the undertriage rate to approximately 10%, with a maximum overtriage rate of 50%.
In the Netherlands, each ambulance service serves a region. In this prospective, multicenter cohort study, all adult patients with trauma and all trauma centers in 2 different regions were included. Data from the Central Netherlands region were used to develop a diagnostic prediction model that was externally validated using the data of the Brabant region. This study was judged by the Medical Ethical Committee of University Medical Center Utrecht, Utrecht, the Netherlands, as not subject to the Medical Research Involving Human Subjects Act and therefore exempt from the need for informed consent.
The Central Netherlands region has 1 level I trauma center, the University Medical Center Utrecht, and 7 level II or III trauma centers. The region covers 535 square miles and has 1.2 million residents. Brabant has 1 level I trauma center, Elisabeth-TweeSteden Hospital, Tilburg, and 10 level II or III trauma centers. The region covers 1343 square miles and has 1.7 million residents. In both regions, the Dutch National Ambulance Care Protocol is used (eMethods in the Supplement).13 Patients transported across the borders of these regions were excluded because data were unavailable.
All patients with trauma who were 16 years or older, determined to be highest priority (with flashing lights and siren) by the dispatch center, and transported to a trauma center in 1 of the 2 regions underwent evaluation. In the Central Netherlands, patients were included from January 1, 2012, through June 30, 2014.15 In Brabant, patients were included from January 1 through December 31, 2015.
Independent predictors associated with severe injury were identified to create a prediction model consisting of a limited number of variables. A severely injured patient was defined as a patient with an Injury Severity Score (ISS) greater than 15 (range, 0-75).
Undertriage was defined as the proportion of severely injured patients transported to a level II or III trauma center. Overtriage was defined as the proportion of patients without severe injuries transported to a level I trauma center. The protocol allows for these patients to be transported to a level I trauma center if it happens to be the nearest hospital.
Prehospital reports from EMS professionals were prospectively collected and included patient demographics, vital signs, description of the trauma mechanism, on-scene physical examination data (including suspected injured body region), and the receiving hospital. The body region suspected of injury by EMS professionals was divided into the head and neck, face, thorax, abdomen, extremities, or skin and others. These regions were chosen based on the categorization of the Abbreviated Injury Scale (AIS) regions that make up the ISS.
The Dutch National Trauma Database registered injuries for all patients admitted to a hospital. The Central Netherlands patient data were also extracted from the electronic patient documentation for patients discharged directly from the emergency department (ED). The injuries were recorded after discharge or 30 days after admission and coded by trained data managers in both regions (including E.A.J.v.R. and F.J.V.). Before 2015, all injuries were coded using the AIS 1990 Update 98 (AIS98); from 2015 and on, the AIS 2005 Update 2008 (AIS08) was used. Therefore, the injuries were coded according to the AIS98 for the Central Netherlands database and according to the AIS08 for the Brabant database. The ISS was calculated based on AIS scores to determine injury severity.
Missing data were analyzed and considered to be missing at random. Multiple imputation by chained equations was used for both regions separately to account for missing prehospital variables rather than deleting patients who had the most data available.18 Missing values were imputed based on all other predictors as well as the ISS. For both regions, respiratory rate, systolic blood pressure, oxygen saturation level, and Glasgow Coma Scale (GCS) score were imputed. Pulse was imputed for the Central Netherlands region only because this variable was missing in the Brabant data. For the Brabant region, ISS was only available for admitted patients. An ISS less than 15 was assumed for patients discharged from the ED because a previous study15 demonstrated that all severely injured patients (ISS >15) were admitted or died in the ED.
Data were analyzed from May 3, 2017, through July 19, 2018. Frequencies with percentages were used to describe nominal and ordinal variables, whereas means and SDs were used to describe continuous variables. Bivariable binary logistic regression was used to explore potential predictors associated with severe injury (ISS >15). Analyses were performed on 5 imputed data sets independently and pooled using Rubin’s rules, if applicable.19
To ensure practical applicability, the maximum number of predictors was limited to 8. A full-model strategy with clinically relevant variables was used to develop the prediction model. To improve the accuracy for future patients and other regions, penalized maximum likelihood estimation was used.20 Penalized maximum likelihood estimation is a rigorous estimation method that potentially results in better generalizability, model reduction, and differential shrinkage of coefficients.21 The functional forms of all continuous predictors were defined before modeling. Restricted cubic splines were used to model nonlinear predictors.
The performance of the final prediction model was expressed in terms of discrimination and calibration. Discriminative value was quantified by the C statistic. The receiver operating characteristic curve was plotted and a predefined value for specificity—an overtriage rate of 50%—was used to determine a cutoff point. Calibration was graphically assessed using a calibration plot.
The final model was externally validated with the Brabant data. Owing to heterogeneity between trauma regions (eg, prevalence of severe injury), the model needed to be recalibrated by updating the intercept for the Brabant region.22,23 Calibration and discrimination of the prediction model were assessed in the validation process. Bootstrapped 95% CIs based on percentiles and 1000 resamples were calculated for C statistics and accuracy metrics. Two-sided P values were calculated using univariable logistic regression, and P < .05 indicated significance. All statistical analyses were performed using R software (version 3.2.4; R Foundation for Statistical Computing).24
A total of 4950 adult patients with trauma were included for the Central Netherlands region constituting the design data set. To account for missing data, multiple imputation was used for respiratory rate in 324 (6.5%), systolic blood pressure in 345 (7.0%), oxygen saturation level in 662 (13.4%), and GCS score in 230 (4.6%). Mean patient age was 47 (21) years; 2887 (58.3%) were male and 2063 (41.7%) were female; and 435 (8.8%) had an ISS greater than 15 (Table 1). In this cohort, the undertriage rate was 21.6%; the overtriage rate, 30.6%.
In the Brabant region, a total of 6859 adult trauma patients were included in the validation data set. To account for missing data, multiple imputation was used for respiratory rate in 1973 patients (28.8%), systolic blood pressure in 1145 (16.7%), oxygen saturation in 1431 (20.9%), and GCS score in 288 (4.2%). Mean patient age was 51 (22) years; 3583 (52.2%) were male and 3276 (47.8%) were female; and 165 (2.4%) had an ISS greater than 15 (Table 1). The ISS was only available for the admitted patients. In this cohort, the undertriage rate was 27.3%; the overtriage rate, 26.3%.
To develop the prediction model, 43 potential prehospital predictors from the Central Netherlands database were explored using bivariable analysis (Table 2). The following 8 predictors were chosen for the final model, based on clinical reasoning: age; systolic blood pressure; GCS score; mechanism criteria; penetrating injury to the head, thorax, or abdomen; signs and/or symptoms of head or neck injury; expected injury in the AIS thorax region; and expected injury in 2 or more AIS regions. The optimal cutoff point with a minimum specificity of 50.0% (95% CI, 49.3%-50.7%) led to a sensitivity of 88.8% (95% CI, 87.5%-90.0%).
This prediction model resulted in an undertriage rate of 11.2% (Δ difference, 10.4%) and an overtriage rate of 50.0% (Δ difference, 19.4%) for the Central Netherlands region. Robust estimation using penalized maximum likelihood showed that all variables in the model were significant independent predictors (Table 3). The model had a good discrimination with a C statistic of 0.823 (95% CI, 0.813-0.832). The recalibration method led to an intercept of 0.894 for the Brabant region. External validation using the Brabant region database showed that the model with the new intercept was well calibrated (Figure 1) and had a good discrimination with a C statistic of 0.831 (95% CI, 0.814-0.848).
In this prospective, multicenter cohort study, we present a ready-to-use prehospital trauma triage prediction model for the presence of severe injury on the scene in patients with trauma. The model performed well in the derivation set and in external validation. To our knowledge, this is the first externally validated protocol showing acceptable triage rates, with a potential undertriage rate of 11.2% and overtriage rate of 50.0% depending on the chosen threshold.
Worldwide, triage protocols are based on a simple flowchart that includes vital signs, injury type, and mechanism of injury criteria.6,25-28 These triage protocols are simplistic and static: transport to a higher-level trauma center should be considered if just 1 criterion is present. In reality, some factors have a greater association with injury severity than others, and the combination of factors indicates the need for higher-level trauma care. In addition, current protocols often use cutoff points for continuous variables, whereas the prediction model uses coefficients for each predictor to represent each variable’s distinct association with injury severity to increase predictive ability.
The prediction model was based on the following 3 key elements: (1) inclusion of all adult patients with trauma transported by an ambulance to (2) all trauma centers of an entire geographic region, with (3) prehospital variables measured on the scene by EMS professionals. Previous studies have attempted to develop protocols but have not included these 3 key elements. For example, Dihn and colleagues27 developed a triage protocol based on patients taken to a higher-level trauma center only, thereby excluding the undertriaged patients. Others included admitted patients only,29,30 thus excluding patients discharged from the ED or the potentially overtriaged patients. These models would not be reliable in a general trauma population because they fail to include the patient populations in which improvement is of utmost importance: the undertriaged and overtriaged patients.
Eight predictors were included in the model based on clinical reasoning to achieve the best accuracy while keeping it user friendly without too many factors. Age was included because previous studies showed a higher undertriage rate in elderly patients.6,31-33 Two continuous predictors of the condition of a patient are systolic blood pressure and GCS score.34-36 Penetrating injury is an obvious predictor associated with potential severe trauma. The brain and thorax are 2 of the most commonly injured body regions, with both associated with a high prevalence of severe injury.37-39 Also, multiregional injury was previously found as a strong predictor associated with severe injury.27 Therefore, these 8 predictors were included in the current model. The prediction model resulted in a undertriage rate of 11.2% and an overtriage rate of 50.0% in the Central Netherlands region.
After penalized estimation, the updated diagnostic prediction model was externally validated with data from the Brabant region. The prediction model requires an update primarily owing to the difference in prevalence of severe injury, resulting in a difference in baseline risk.40 To account for this difference, the constant value (intercept) in the equation was altered. The constant value can be altered for other regions before applying the prediction model based on prehospital and hospital data of the specific region. External validation—with the altered intercept—showed good discrimination and calibration, indicating that the prediction model would likely be accurate in a region that is heterogeneous with respect to population, prevalence of severe injury, and mechanism of injury. In addition, the injuries were coded differently in both regions: AIS98 was used in the Central Netherlands and AIS08 in the Brabant regions. When using the AIS08, the overall ISS is lower compared with injuries scored with the AIS98.41 External validation using the AIS08 showed that the prediction model functions well using the most recent AIS.
The prediction model did not achieve the goal of an undertriage rate of less than 5% as targeted by the American College of Surgeons Committee on Trauma.42 However, the model is a significant improvement compared with the existing protocols.5,6,16,43 Whether further improvement of undertriage is achievable by solely improving the protocol is unclear. Previous studies have shown that addition of the judgment of EMS professionals can be useful in the identification of severely injured patients.44-47 In this study, EMS professional judgment could not be quantified because it was not recorded. Including EMS professional judgment may improve the undertriage rate even more as well as increase adherence to the protocol. Other factors could have influenced transport decisions, such as geographical distance to a higher-level trauma center. Although distances are relatively small in the Netherlands compared with other countries, distance could have influenced the destination decision, especially in Brabant, because distances are larger in this region. Unfortunately, the effect of distance on the triage quality could not be evaluated in this study, and the distance at which deviation to the nearest hospital is best remains unclear.48
In practice, it is not feasible to calculate the risk of severe injury based on an equation that must be memorized and applied on the scene. This problem could be solved by implementing the prediction model in a mobile app; such triage tools are increasingly being developed and used in the prehospital process.8,9 This mobile app includes every variable, calculates the chance of severe injury, and gives advice on where to transport the patient, which is much more practical for EMS professionals compared with an equation. With a mobile device available on every ambulance, the EMS professionals can calculate the risk of severe injury using the prediction model in the app to decide quickly and more accurately where to transport the patient. The EMS professional judgment could be included in the app. A mobile app with the described prediction model is currently being implemented in different regions in the Netherlands. The implementation aims at reducing undertriage specifically.
This study has several limitations. In the final prediction model, missing data were present for the systolic blood pressure and GCS variables. The data were considered to be missing at random, and multiple imputation was used to minimize selection bias. Second, for Brabant, the ISS was only available for the patients who were admitted or who died in the ED. In the Central Netherlands region—where the ISS was available for all patients—all severely injured patients (ISS >15) were admitted or died in the ED. Accordingly, an ISS less than 15 was assumed for patients discharged from the ED. Last, debate remains on the most accurate definition of a severely injured patient. Legitimate classification is difficult and dependent on multiple factors, such as regional circumstances and trauma center level. An ISS greater than 15 might not represent all patients in need of higher-level trauma center resources. However, an ISS greater than 15 is the most used surrogate marker for a severely injured patient when evaluating prehospital trauma triage; therefore, we chose this variable to define a severely injured patient.5
Future research should focus on the validation of the prediction model in other regions. Differences in incidence of severely injured patients, and consequently baseline risk, can be large, and therefore different baseline risks should be determined in other populations. A possible solution is to validate the prediction model in other regions using our methods.
The mobile app has been developed and is currently being implemented in the Netherlands (Figure 2). In this mobile app, the equation is integrated in addition to EMS professional judgment. This combination could be optimal to improve triage rates. In addition, the combination could give insight in the value of EMS professional judgment.
To our knowledge, this study is the first to develop and validate a prehospital trauma triage protocol based on all adult patients with trauma to be transported to a trauma center within a region and that can lower the undertriage rate to 11.2%, with an overtriage rate of 50.0%, from 21.6% and 30.6%, respectively. This protocol, based on an equation in which each predictor has its own coefficient, can be implemented with a mobile app for EMS professionals and could be of great help to lower undertriage rates.
Accepted for Publication: September 14, 2018.
Corresponding Author: Eveline A. J. van Rein, MD, Department of Traumatology, University Medical Center Utrecht, Ste G04.228, Heidelberglaan 100, 3584 CX, Utrecht, the Netherlands (email@example.com).
Published Online: February 6, 2019. doi:10.1001/jamasurg.2018.4752
Author Contributions: Dr van Rein had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: van Rein, van der Sluijs, Voskens, Lansink, Houwert, Lichtveld, Leenen, van Heijl.
Acquisition, analysis, or interpretation of data: van der Sluijs, Voskens, Lansink, de Jongh, Dijkgraaf, Champion, Beeres, van Heijl.
Drafting of the manuscript: van Rein, van der Sluijs, Lichtveld, Beeres, van Heijl.
Critical revision of the manuscript for important intellectual content: van der Sluijs, Voskens, Lansink, Houwert, de Jongh, Dijkgraaf, Champion, Beeres, Leenen, van Heijl.
Statistical analysis: van Rein, van der Sluijs, Dijkgraaf, van Heijl.
Obtained funding: van Heijl.
Administrative, technical, or material support: van Rein, Voskens, Lansink, de Jongh, Beeres, van Heijl.
Supervision: Voskens, Lansink, Houwert, Lichtveld, Beeres, Leenen, van Heijl.
Conflict of Interest Disclosures: None reported.
Additional Contributions: The authors thank Jillian Gruber, MD, School of Medicine and Dentistry, University of Rochester Medical Center, Rochester, New York, for copy editing the manuscript. She was compensated for this work.
Create a personal account or sign in to: