eFigure. Overview of Nominal Group Technique (NGT) Process
eAppendix. FACTS Consortium Expert Panel
Customize your JAMA Network experience by selecting one or more topics from the list below.
Cunningham RM, Carter PM, Ranney ML, et al. Prevention of Firearm Injuries Among Children and Adolescents: Consensus-Driven Research Agenda from the Firearm Safety Among Children and Teens (FACTS) Consortium. JAMA Pediatr. Published online June 10, 2019173(8):780–789. doi:10.1001/jamapediatrics.2019.1494
Firearm injuries are the second leading cause of death among US children and adolescents. Because of the lack of resources allocated to firearm injury prevention during the past 25 years, research has lagged behind other areas of injury prevention. Identifying timely and important research questions regarding firearm injury prevention is a critical step for reducing pediatric mortality.
The Firearm Safety Among Children and Teens (FACTS) Consortium, a National Institute for Child Health and Human Development–funded group of scientists and stakeholders, was formed in 2017 to develop research resources for the field, including a pediatric-specific research agenda for firearm injury prevention to assist future researchers and funders, as well as to inform cross-disciplinary evidence-based research on this critical injury prevention topic.
A nominal group technique process was used, including 4 key steps (idea generation, round-robin, clarification, and voting and consensus). During idea generation, stakeholders and workgroups generated initial research agenda topics after conducting scoping reviews of the literature to identify existing gaps in knowledge. Agenda topics were refined through 6 rounds of discussion and survey feedback (ie, round-robin, and clarification steps). Final voting (using a 5-point Likert scale) was conducted to achieve consensus (≥70% of consortium ranking items at 4 or 5 priority for inclusion) around key research priorities for the next 5 years of research in this field. Final agenda questions were reviewed by both the stakeholder group and an external panel of research experts not affiliated with the FACTS Consortium. Feedback was integrated and the final set of agenda items was ratified by the entire FACTS Consortium.
Overall, 26 priority agenda items with examples of specific research questions were identified across 5 major thematic areas, including epidemiology and risk and protective factors, primary prevention, secondary prevention and sequelae, cross-cutting prevention factors, policy, and data enhancement.
Conclusions and Relevance
These priority agenda items, when taken together, define a comprehensive pediatric-specific firearm injury prevention research agenda that will guide research resource allocation within this field during the next 5 years.
Firearms are the second leading cause of death1 among children and adolescents aged 1 to 18 years in the United States and responsible for more than 2570 deaths and nearly 12 000 nonfatal injuries requiring emergency department treatment in 2017.1,2 Pediatric firearm injuries result from a range of causes, including the unintentional discharge of a firearm, self-inflicted wounds, or the escalation of interpersonal violence. Nearly 265 million firearms are in civilian hands in the United States,3 and a 44% increase in pediatric firearm mortality rates has been documented during the past 5 years.2
Despite recent strides by private foundations and states, research funding, publications, and evidence-based programs to decrease pediatric firearm injuries have lagged substantially behind those for other forms of injury.4,5 Successes in reducing significant child and adolescent health threats from communicable diseases and motor vehicles required extensive scientific resources and translation of epidemiologic evidence to inform practical solutions and change policy. When considering road traffic safety, the great success of these evidence-based strategies has reduced pediatric motor vehicle crash injury and death without interfering with the ability of US citizens to own and responsibly operate cars. A 2013 Institute of Medicine report6 described the need for innovative research that addresses firearm injuries. This seminal report that was focused on firearm injuries brought substantial attention to this issue but included all age groups without specific discussion of issues affecting pediatric populations. Currently, there is a substantial deficit of data for pediatric firearm injuries. Such data are essential for developing and implementing efficacious firearm injury prevention strategies and policies. The development of a pediatric-specific research agenda is a needed next step to focus efforts to decrease the second leading cause of pediatric mortality.
Building on this critical need, the National Institute for Child Health and Human Development funded the Firearm Safety Among Children and Teens (FACTS) Consortium in 2017. The primary goal of FACTS, which includes 25 scientists from 12 academic institutions and 12 stakeholders (eAppendix in the Supplement), is to reduce pediatric firearm injury and death while also respecting firearm ownership as a part of US society. FACTS includes multidisciplinary expertise from medicine (emergency medicine, pediatrics, pediatric trauma surgery, and psychiatry), public health (health behavior and education, health policy, and epidemiology), criminology, psychology, sociology, and the fields of data and implementation science. Stakeholder members included a diverse group of interested parties, including firearm owners, firearm safety trainers, educators, faith-based leaders, law enforcement professionals, and hunting enthusiasts, who share a common goal of decreasing firearm injury and deaths among children. The consortium is developing research resources to inform the prevention of pediatric firearm injury and death, including the following: (1) training a cadre of postdoctoral trainees and graduate and undergraduate students to increase the pipeline of firearm researchers; (2) creating conferences, webinar series, and other resources to educate researchers and policy makers about firearm injury science; (3) creating an accessible data archive on pediatric firearm injury (https://www.childfirearmsafety.org); and (4) funding projects to generate preliminary data that inform future research to decrease the toll of child and adolescent injury and death from firearms.
The first focus of the consortium was to conduct substantive scoping reviews of the existing literature7-11 to inform the current state of the science and to identify and prioritize gaps in research. This article presents a rigorously generated comprehensive research agenda specific to pediatric firearm injury that is intended to aid in guiding the next 5 years of research for the field. This research agenda describes priorities for preventing both unintentional (accidental) and intentional (ie, homicide and suicide) firearm injury and death, as well as for preventing long-term consequences of pediatric firearm injury and mortality.
A modified nominal group technique (NGT)12-17 was used to develop the research agenda (eFigure in the Supplement). Nominal group technique is a widely accepted, highly structured process for collecting data and developing consensus among a panel of content experts.12-18 The technique involves the following 4 phases: silent idea generation, round-robin presentation of ideas and further idea generation, clarification (or narrowing and grouping) of ideas, and voting or ranking of preferred ideas to generate a prioritized agenda.12-18 Nominal group technique was selected rather than other approaches (eg, Delphi technique) as it generates a larger number of ideas than do traditional consensus approaches and the structured process serves to mitigate potential sources of bias.16,18,19 Agenda development occurred between January and October 2018, and included 45 workgroup meetings, 2 consortium-wide video conference meetings, and 4 surveys administered via Qualtrics. The NGT process was facilitated by the FACTS leadership team (R.M.C., P.M.C., and M.A.Z.), all of whom have prior NGT experience. Written minutes and/or audio recordings of all consortium meetings were maintained by nonvoting research staff.
Expert panelists (eAppendix in the Supplement) have all published pediatric firearm research and are recognized leaders within their field of expertise. Panelists were organized into the following 6 thematic workgroups: surveillance and epidemiology, risk and protective factors, primary prevention, secondary prevention, policy, and data and methods.
Each workgroup conducted a scoping literature review with the objective of identifying current research knowledge and gaps in a specific topic area.7-11 The reviews were conducted across the range of firearm outcomes that were defined at the outset for clarity (Box). Concurrently, research staff and expert panelists conducted semistructured interviews with 12 stakeholders (eAppendix in the Supplement) to identify independently formulated priorities for future research. Stakeholder interviews were recorded and transcribed by research staff with summaries shared with each workgroup. Stakeholder input was sought at this stage to ensure inclusion of their priorities at the outset and avoid biasing the agenda. After performing the literature reviews and receiving stakeholder input, each workgroup generated a preliminary internal list of up to 10 research agenda priorities (51 items), with a focus on those that would be most important for the next 5 years of research.
Attitudes toward and knowledge of firearms
Access and storage of firearms
Exposure to and witnessing firearm violence
Intentional firearm injury
Self-inflicted firearm injury (suicide attempts)
Firearm perpetration and aggression (includes nonpartner and partner)
Firearm victimization (includes nonpartner and partner)
Mass shootings, including school shootings
Unintentional (accidental) firearm injury
Next, we conducted a 2-phase round-robin process to obtain feedback from the expert panel on the 51-item preliminary research agenda. In the first phase, the expert panel completed a structured survey to allow for the generation of novel research agenda ideas and to obtain initial feedback on preliminary agenda items. In the second phase, the expert panel participated in a video discussion to modify or identify new items. Feedback was obtained in a structured format from the panel. The NGT facilitator was tasked with raising discussion points from the first survey that did not surface during the discussion. The FACTS leadership team (R.M.C., P.M.C., and M.A.Z.) integrated the feedback to generate a revised agenda of 52 items for the clarification phase.
During the clarification phase, the agenda items were refined. The expert panel completed a Qualtrics survey reviewing the preliminary list of agenda items with the objective of identifying items with common themes and establishing greater clarity. No items were deleted at this stage, but those with similar themes were combined and streamlined.
The final NGT phase was conducted in 3 stages. First, the expert panel and stakeholders completed an anonymous quantitative survey to establish a priority ranking for agenda items. Items were ranked on a 5-point Likert scale identifying how important each item was for a 5-year research agenda (1 = definitely not a priority, 2 = not a priority, 3 = important but low priority, 4 = should be a priority, and 5 = highest priority). Qualitative feedback was obtained on why items were chosen for inclusion or exclusion. Predetermined consensus for agenda inclusion was defined as more than 70% of the panelists ranking the item as at least a 4 (of 5); agenda exclusion was defined as less than 50% of the panelists ranking the item as at least a 4 (of 5). Second, items not reaching consensus were discussed at a second consortium-wide video meeting (14 items). The expert panel then recommended, defended, and refined these items for inclusion in the agenda. Third, a final round of anonymous voting was conducted on the items that had not reached consensus for inclusion or exclusion in the prior round (n = 12). Based on the final vote, 25 items were included in the semifinal agenda.
These 25 agenda items were then presented for feedback to the 12 stakeholders and a panel of 6 external researchers who were not included in the initial consortium process but have established expertise in the field (eAppendix in the Supplement). Stakeholder ratings were concordant with FACTS expert panel ratings for all agenda items (r = 0.77) and, with their input, 1 item that had previously been excluded was retained, bringing the total number of research agenda items to 26. External expert feedback was concordant with the last round of internal consortium ranking and suggestions were integrated. Finally, the full agenda was presented at an in-person consortium meeting in October 2018 and ratified by the entire FACTS Consortium.
We identified 26 priority agenda items (with samples of urgent research questions that clarify the items) across 5 major thematic areas. The final list of agenda items and research questions (Tables 1, 2, and 3)20,21 represent the areas that the FACTS Consortium defined as being the key priority areas for research and investment for research during the next 5 years.
We defined an agenda to guide future research priorities for prevention of pediatric firearm injury. This research agenda is intended to serve as a guide for future research efforts to decrease pediatric death and injury across the thematic areas of: (1) epidemiology, surveillance, and risk and protective factors; (2) primary, secondary, and cross-cutting prevention; (3) policy-related issues; and (4) data enhancement priorities.
More information is needed that characterizes the contextual factors associated with firearm injuries and the associations between firearm availability, storage, and presence and use of a firearm in the home. Research that extends beyond individual-level factors to include risk assessment across multiple ecological levels, with a focus on the intersection between family, peer, and community-level factors, is also needed. Most research beyond the individual level has focused on peers and families,22,23 with community factors lacking.22,23 The few community studies that have addressed more macro-level issues such as neighborhood socioeconomic variables have not focused on social relationships (eg, social capital and neighborhood monitoring). Only 2 studies have examined the intersection of peer relationships with community macro-level variables.22,23 Most research has focused attention on risk factors,7,8 while few have examined factors that may reduce the negative effects of risk exposure. Peer firearm carriage, for example may be a risk factor for youth firearm carriage,24 but the role that family firearm safety practices may play in reducing the risk of exposure to peer firearm carriage is unknown.22,25
Comprehensive evidence-based programs that address decreasing firearm carriage by children and adolescents and increasing firearm safe storage among caregivers are needed across the ecological spectrum, including in homes, schools, health care settings, and the broader community.9 We also identified that the adoption of evidence-based injury and violence prevention strategies to be applied specifically to decreasing firearm injury risk is a priority. The panel identified engaging caregivers who own firearms as critical to achieve lasting effects and enhance acceptance of prevention strategies focused on decreasing firearm access by children and adolescents, as well as programs focused on increasing safety skills for activities such as hunting. Such intervention development should also consider fidelity and scalability of firearm prevention programs, including cost-benefit analyses for intervention research.26,27 Finally, research on the outcome of existing and emerging firearm safety technologies (eg, trigger locks and personalized “smart” guns) is lacking specific pediatric outcomes.
The existing literature focuses largely on the prevalence of posttraumatic stress disorder as a consequence of pediatric mass shootings or firearm assault.28-37 Attention to prevalence, correlates, and prevention of the full range of physical, mental, educational, and behavioral effects after a pediatric firearm injury, for youths and their families, is needed. For example, the prevalence of depression and substance use, as well as recurrent injury, after firearm injury is poorly described. Rigorous development and validation of prevention strategies to reduce behavioral and physical health consequences after firearm injury is needed. This secondary prevention work must be conducted in a wide range of settings (eg, primary care, families, and schools) and capitalize on innovative technologies (eg, remote therapy and social media) to enhance reach and delivery across socioeconomic and geographic strata. Finally, research on the prevalence, correlates, and prevention of firearm injury sequelae must be conducted across the full range of firearm injury types (eg, not just mass shootings, but also self-harm and peer violence) and the full range of firearm outcomes, including direct victimization and indirect exposure (ie, witnessing violence or hearing about violence).
Firearm policies remain understudied, particularly as they relate to pediatric outcomes. Among the scant research studying the association between firearm policies and outcomes in children,11,38 most studies have tested Child Access Prevention laws or a legislative strength variable that represents how many specific types of firearm laws a state has implemented.39-50 Almost nothing is known about the association of individual laws with firearm outcomes among pediatric populations, let alone how various firearm laws work synergistically to reduce pediatric morbidity and mortality. It is also critical to consider the extent to which laws vary from state to state and how these variations may affect their reach and implementation. Furthermore, we have found inconsistency in how policies are defined and measured, making direct comparison across studies difficult. This inconsistency is most relevant to policy categories that have differing policy elements across jurisdictions, such as Child Access Prevention laws that differ in the criminal liability imposed (felony or misdemeanor), the criteria for criminal liability being imposed (negligent storage or reckless provision of the gun), and the age of the child with real or potential access to the handgun (<14 years in some states, to <18 years in others). An overarching priority is the need to rigorously define a measure of policy strength that is both specific to pediatric outcomes and considers the complexities noted above. Finally, we need to better understand how firearm policies are implemented, enforced, and communicated to the public so that we can determine how these factors may be associated with firearm morbidity and mortality.
Firearm research has been constrained by restrictions placed on data collection, availability of data elements, and access to data. Research has also been limited by a lack of linkage between data sources to increase awareness of available data and to allow for a more comprehensive analysis. Augmenting existing data sets with variables associated with firearm injury, and increasing researcher access to existing variables that are currently collected but not easily available, will greatly enhance our ability to study firearm injury outcomes, determine risk factors for firearm outcomes, and inform preventive and therapeutic interventions. Leveraging advances in health information technology to access and link patient-centric clinical data across the spectrum of care (eg, prehospital, hospital, outpatient, and rehabilitation) could improve our understanding of patterns of firearm outcomes among children. The richness of electronic health record data provides a unique opportunity for comparative effectiveness research, which investigates the association between diagnostic and therapeutic interventions and outcomes in heterogeneous patient populations in real-world settings. Integrating such approaches within health care settings, as well as involving data sources not related to health care, would aid in developing a robust understanding of firearm outcomes. Finally, we can maximize current investments by facilitating accessibility of existing and newly collected data in repositories, as is commonly done in other fields.51
In addition to the research areas noted above, we identified additional considerations for future research. First, research guided by cutting-edge theoretical models used in other injury prevention, public health, and behavioral research is needed to develop unbiased conclusions. Theoretically driven research can inform prevention by identifying modifiable risk and protective factors associated with firearm behavior that interact across socioecologic settings. Second, research that uses rigorous designs, including qualitative data collection, longitudinal data collection, and randomized clinical trial designs, is needed. In parallel, the field requires that more sophisticated and robust data analytic approaches (eg, spatial analysis and growth mixture modeling) be applied to prospective longitudinal data to understand causal relationships. Qualitative data collection is needed to inform quantitative findings, identify new issues to study, and inform the content and implementation of prevention programs. Third, we need research studies with more representative samples. Fourth, we need to improve the measurement and coding of outcomes (conceptual and operational) to capture the complexity of firearm outcomes. Fifth, we need to apply cutting-edge digital tools to advance measurement and intervention to capitalize on advances in technology (eg, mobile health tools), innovative analytic methods (eg, SMS [text messaging], social media, apps, biosensors, and self-quantification technologies), and novel screening methods (eg, self-administered patient portals).
Although bias is always a possibility when using consensus techniques, the NGT process is designed to minimize the influence of group dynamics or single individuals on the outcome. This bias was minimized by using a structured process with participation from all consortium experts, and by both open and anonymous feedback via multiple formats (eg, group discussion, qualitative surveys, email and individual feedback). We also included feedback from researchers beyond the field of firearm injury prevention, including firearm enthusiasts and other stakeholders at multiple points during the NGT process. Finally, feedback from an external expert panel that was not engaged in the iterative NGT group process was used to establish a source of external validity to the recommendations included within this report.
In the mid-1980s after the seminal publication of Injury in America: A Continuing Health Problem,52 the United States committed to reducing the burden of injury among our population, with substantial investment in injury prevention. Despite declines from the peak of firearm deaths noted in 1994, rates of firearm deaths have been rising recently, with a 44% increase in 2017 relative to 2013.2 Today, children and adolescents are more likely to die by a firearm then by any other cause except motor vehicle crash before the end of their teen years.1 Through our consensus group process, we identified key priority areas of firearm injury prevention research for children and adolescents to help guide increasing foundation and federal investment in this area during the next 5 years. We also posed a range of questions across these priority areas that are answerable with rigorous research methods and analysis used across the health and behavioral sciences, public health, economics, and criminal justice fields. By focusing research and investment in these identified priority areas, investigators and entities that sponsor research can make substantial strides toward decreasing firearm injury and death among US children and adolescents.
Accepted for Publication: March 12, 2019.
Corresponding Author: Rebecca M. Cunningham, MD, Department of Emergency Medicine, Michigan Medicine, University of Michigan, North Campus Research Complex, 2800 Plymouth Rd, Bldg 10-G080, Ann Arbor, MI 48109 (email@example.com).
Published Online: June 10, 2019. doi:10.1001/jamapediatrics.2019.1494
Author Contributions: Drs Cunningham and Carter had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Cunningham, Carter, Zimmerman, Ranney, Walton, Zeoli, Alpern, Branas, Ehrlich, Goyal, Goldstick, Hargarten, King, Massey, Pizarro, Rowhani-Rahbar, Rivara, Rupp, Savolainen, Sigel.
Acquisition, analysis, or interpretation of data: Cunningham, Carter, Zimmerman, Ranney, Walton, Beidas, Goldstick, Hemenway, King, Massey, Ngo, Pizarro, Prosser, Rowhani-Rahbar, Rivara, Rupp.
Drafting of the manuscript: Cunningham, Carter, Zimmerman, Ranney, Ehrlich, Massey, Ngo, Sigel.
Critical revision of the manuscript for important intellectual content: Cunningham, Carter, Zimmerman, Ranney, Walton, Zeoli, Alpern, Branas, Beidas, Ehrlich, Goyal, Goldstick, Hemenway, Hargarten, King, Massey, Ngo, Pizarro, Prosser, Rowhani-Rahbar, Rivara, Rupp, Savolainen.
Statistical analysis: Cunningham, Goldstick.
Obtained funding: Cunningham, Carter, Zimmerman, Ranney, Walton, Alpern, Branas, Ehrlich, Hemenway, Hargarten, Rivara, Savolainen.
Administrative, technical, or material support: Cunningham, Zimmerman, Ranney, Branas, Goyal, Hargarten, Massey, Ngo, Prosser, Rowhani-Rahbar, Rupp, Savolainen, Sigel.
Supervision: Cunningham, Zimmerman, Ranney, Branas, Ehrlich, Massey.
Conflict of Interest Disclosures: All authors reported receiving grants from the National Institutes of Health (NIH)/National Institute of Child Health and Human Development (NICHD) during the conduct of this study. No other disclosures were reported.
Funding/Support: This article was funded through National Institutes of Health/National Institute of Child Health and Human Development grant 1R24HD087149-01A1.
Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the funding agencies.
Additional Contributions: Interviews to inform agenda development and rigorous review and rating of agenda items were conducted by the following practitioner stakeholders: Adelyn Allchin, MPH, Brigadier General (Retired) James Anderson, PhD, James Berlin, Vicka Chaplin, MA, MPH, Rochelle A. Dicker, MD, Greg Dorfman, JD, Joseph V. Erardi Jr, PhD, Chris Harris, Geraldine Hills, MNps, Ben Hoffman, MD, Joneigh Khaldun, MD, Tom O’Connor, BA, and LokMan Sung, MD. Rigorous review and ranking of agenda items was performed by the following external expert panelists: Emmy Betz, MD, MPH, University of Colorado School of Medicine; Joel Fein, MD, MPH, The Perelman School of Medicine, University of Pennsylvania; Shannon Frattaroli, PhD, Johns Hopkins Bloomberg School of Public Health; Jesus Ramirez-Valles, PhD, MPH, San Francisco State University; Daniel Webster, ScD, Johns Hopkins Bloomberg School of Public Health; and Doug Wiebe, PhD, The Perelman School of Medicine, University of Pennsylvania. Assistance conducting scoping reviews and writing review articles across 6 work groups that provided a critical foundation for this agenda for research was conducted by the following Firearm Safety Among Children and Teens (FACTS) Consortium Trainees: Kira Bromwich, BA, Brown University; Jonathan Jay, JD, DrPh, Harvard University; Rebecca Karb, MD, PhD, Brown University; Amanda Mauri, MPH, University of Michigan; Allante Moon, MPH, University of Michigan; Charles Mouch, MD, University of Michigan; Stephen Oliphant, MPP, University of Michigan; Carissa Schmidt, PhD, MPH, University of Michigan; and Mikaela Wallin, MA, Michigan State University; and the following FACTS research assistants: Zahra Asghar, BS, Brown University; Mikala Cox, MSW, University of Michigan; Melissa Goodman, University of Michigan; Jhuree Hong, University of Michigan; Stephanie Kostolansky, University of Michigan; Jaemin Park, University of Michigan; and Max Ozer-Staton, University of Michigan. They received no additional compensation for their work.
Create a personal account or sign in to: