Assessment of Barriers and Enablers to Implementation of a Pediatric Early Warning System in Resource-Limited Settings

Key Points

Question  What barriers and enablers to pediatric early warning systems (PEWS) implementation in resource-limited hospitals are reported by health care professionals?

Findings  In this qualitative study including 5 resource-limited pediatric oncology centers in 4 countries in Latin America, many barriers to PEWS implementation were identified, including inadequate resources and staff resistance to change. Most barriers were successfully converted to enablers during implementation through strategies such as early stakeholder engagement, adapting PEWS to the local context, and changing the hospital setting to support use of PEWS.

Meaning  The findings of this study suggest that barriers to implementation of evidence-based interventions in resource-limited settings are not immutable and can be converted to enablers through targeted implementation strategies.

Importance  Pediatric early warning systems (PEWS) aid with early identification of clinical deterioration and improve outcomes in children with cancer hospitalized in resource-limited settings; however, there may be barriers to implementation.

Objective  To evaluate stakeholder-reported barriers and enablers to PEWS implementation in resource-limited hospitals.

Design, Setting, and Participants  In this qualitative study, semistructured stakeholder interviews were conducted at 5 resource-limited pediatric oncology centers in 4 countries in Latin America. Hospitals participating in a multicenter collaborative to implement PEWS were purposefully sampled based on time required for implementation (fast vs slow), and stakeholders interviewed included physicians, nurses, and administrators, involved in PEWS implementation. An interview guide was developed using the Consolidated Framework for Implementation Research (CFIR). Interviews were conducted virtually in Spanish, audiorecorded, and professionally transcribed and translated into English. A codebook was developed a priori using the CFIR and supplemented with codes inductively derived from transcript review. Two coders independently analyzed all transcripts, achieving a κ of 0.8 to 0.9. The study was conducted from June 1 to August 31, 2020.

Main Outcomes and Measures  Thematic analysis was conducted based on CFIR domains (inner setting, characteristics of individuals, outer setting, intervention characteristics, and implementation process) to identify barriers and enablers to PEWS implementation.

Results  Seventy-one staff involved in PEWS implementation were interviewed, including 32 physicians (45%), 32 nurses (45%), and 7 administrators (10%). Of these, 50 were women (70%). Components of the 5 CFIR domains were mentioned by participants as barriers and enablers to PEWS implementation at both fast- and slow-implementing centers. Participants emphasized barriers at the level of the clinical staff, hospital, external factors, and PEWS intervention. These barriers included staff resistance to change, inadequate resources, components of health systems, and the perceived origin and complexity of PEWS. At all centers, most barriers were successfully converted to enablers during the implementation process through targeted strategies, such as early stakeholder engagement and adaptation, including adapting PEWS to better fit the local context and changing the hospital setting to support ongoing use of PEWS.

Conclusions and Relevance  To date, this is the first multicenter, multinational study describing barriers and enablers to PEWS implementation in resource-limited settings. Findings suggest that many barriers are not immutable and can be converted to enablers during the implementation process. This work can serve as a guide for clinicians looking to implement evidence-based interventions to reduce global disparities in patient outcomes.

Prevention and management of critical illness are integral to improve survival for children globally, particularly for those at high risk for clinical deterioration, such as children with cancer.¹ However, resources for pediatric critical care vary worldwide, including limitations to funding, equipment, medicine, physical space, and staff needed to provide optimum patient care.² Pediatric early warning systems (PEWS) comprise bedside assessment tools associated with an action algorithm for early identification of deterioration.^3-5 PEWS have been validated to identify critical illness,^6,7 including in children with cancer.^8,9 Implementation of PEWS improves patient outcomes, reduces the cost of care, and optimizes interdisciplinary communication in resource-limited hospitals.^10-16

Despite evidence of their benefits, PEWS are not widely used in hospitals with resource limitations.⁵ Although reasons for this practice gap are multidimensional, resource-limited hospitals likely experience specific barriers to implementation of evidence-based practices, and challenges implementing PEWS may discourage their use in these settings. A deeper understanding of the barriers and enablers to PEWS implementation may help guide strategies to overcome these challenges. However, to our knowledge, the current literature is limited to single-institution studies conducted in high-income countries.^17-21 To reduce global disparities in patient outcomes, research is needed to understand factors affecting implementation of interventions such as PEWS in resource-limited settings.

The Consolidated Framework for Implementation Research (CFIR) is used to describe factors associated with successful implementation of evidence-based practices across 5 domains: inner setting, characteristics of individuals, outer setting, intervention characteristics, and implementation process,^22,23 with evidence supporting its use in resource-limited settings.²⁴ This study used the CFIR framework to evaluate barriers and enablers to PEWS implementation in resource-limited hospitals in Latin America and explore strategies that support implementation in these settings.

Escala de Valoración de Alerta Temprana (EVAT) is a Spanish-language PEWS validated in children with cancer.⁹ Proyecto EVAT is a collaborative led by St Jude Children’s Research Hospital (St Jude), which has supported PEWS implementation at more than 40 pediatric oncology hospitals in Latin America.^25,26 Hospitals are recruited through collaboration with the St Jude Global Alliance²⁶ or via learning about the program from colleagues. These centers self-identify as resource-limited owing to challenges including inadequate nursing and physician staff, limited equipment and physical space, and patients with low socioeconomic, educational, and nutritional indicators.^27-30 New Proyecto EVAT centers assemble a multidisciplinary implementation leadership team of physicians and nurses who are mentored by St Jude and regional PEWS experts through a standardized implementation process, including planning, training staff, piloting, implementation, and assessment of outcomes. Although all hospitals follow the same process, centers have required variable time (range, 3-12 months) to achieve high-quality PEWS use and complete PEWS implementation.³¹

This study was approved by the institutional review board of St Jude as an exempt, minimal risk study. Additional approvals were obtained by participating centers as needed. As an exempt study, written participant consent was waived; verbal consent was provided at the start of each interview. The Consolidated Criteria for Reporting Qualitative Research (COREQ) reporting guideline was followed to ensure rigor of qualitative reporting.

To evaluate barriers and enablers to PEWS implementation, centers were purposefully sampled to include those that implemented PEWS quickly (3-4 months between pilot start and implementation completion) and those that took longer (10-11 months), aiming for regional representation from Mexico, Central America, and South America. All centers completed PEWS implementation before the start of the COVID-19 pandemic (ie, before March 2020). Each center selected a study lead who identified 10 to 15 participants who were (1) implementation leaders (physicians and nurses responsible for local implementation of PEWS), (2) hospital directors (clinicians or administrators with a leadership role in the hospital), or (3) indirectly involved in PEWS implementation. This target sample was chosen as the estimated number needed to reach thematic saturation at each center.³²

The interview guide (eFigure in the Supplement) was developed using the CFIR^22,23 with adaptations suggested for resource-limited settings.²⁴ The preliminary guide was translated to Spanish, iteratively edited by the study team for relevancy and comprehension, piloted with 3 individuals from centers not participating in this study but representative of target participants, and revised based on feedback.

Interviews were conducted in Spanish via a video conferencing platform (WebEx; Cisco) by bilingual members of the study team (S.R.G. and P.E.) from June 1 to August 31, 2020. The interviewers were not previously known to the participants, did not work in their center, and were not involved in PEWS implementation. Interviews were audiorecorded, professionally transcribed and translated into English, and deidentified for analysis.

A codebook (eTable 1 in the Supplement) was developed using a priori codes from the CFIR²² and novel codes derived inductively by 2 of us (A.A. and G.F.) through review of 9 transcripts. Two of us (A.A. and G.F.) independently coded all transcripts using MAXQDA software (VERBI Software GmbH). These coders ultimately achieved a κ of 0.8 to 0.9 and met regularly to resolve discrepancies through consensus with a third team member (D.E.G.). Thematic content analysis focused on identifying barriers and enablers to PEWS implementation using the CFIR domains (eTable 2 in the Supplement), with constant comparative analysis across transcripts.³³ Identified themes were compared across slow- and fast-implementing centers.

Five pediatric oncology centers in 4 countries in Latin America (Ecuador, El Salvador, Mexico, and Peru) were included (eTable 3 in the Supplement). These centers underwent a similar PEWS implementation process but differed in their hospital type, funding structure, pediatric oncology volume, oncology and intensive care unit (ICU) service organization, and time required for implementation. We interviewed 71 participants (Table 1), including physicians (32 [45%]), nurses (32 [45%]), and administrators and data managers (7 [10%]); 21 men (30%) and 50 women (70%) were included.

Identified barriers to PEWS implementation were similar across all 5 institutions and involved all CFIR domains; however, most were converted to enablers during the PEWS implementation process. The Figure describes a modified CFIR framework based on identified themes.

Elements of the inner setting contributing to PEWS implementation included hospital characteristics, available resources, staff turnover, the role of hospital leaders, culture, and an understanding of why PEWS was needed. Examples of how these factors manifested as barriers or enablers are provided in Table 2.

Participants described hospital characteristics that influenced the ease of PEWS implementation, including the type of center (subspecialty vs general, academic vs not, and public vs private), size, unit organization, and patient population served. These characteristics determined whether centers had adequate available resources to support PEWS implementation: “We have more scarcity because we are not even part of public health, we are decentralized hospital and sometimes we have to find resources” (nurse director; San Louis Potosi, Mexico). Having inadequate human (staff, specialists) and material (supplies, space, money) resources was a major barrier to PEWS implementation. Staff turnover, the rotation of staff between units or recruitment of new staff, presented another barrier by requiring retraining in PEWS use.

Hospital leaders, such as clinical and administrative directors, served an important role in slowing or promoting PEWS implementation: “There are chiefs that have a lot of enthusiasm and motivation to establish these types of preventive measures and maybe some people don’t have that” (physician director; Lima, Peru). Hospital leaders unconvinced of the importance of PEWS placed barriers to implementation: “The chief of the department is still indecisive...he doesn’t get involved as we would like, it’s difficult, in that sense we have that big barrier” (nurse director; San Louis Potosi, Mexico). However, when supportive, hospital leaders facilitated implementation.

One implementation enabler was a strong culture of safety: “the [staff’s] desire to do something better for the patient, that helped a lot…everything is focused on the benefit of the patient, so the child can go home with good results and no complications” (implementation leader; San Salvador, El Salvador). Similarly, institutional experience with patient deterioration highlighted why PEWS was needed and facilitated implementation: “There was a moral situation…a little girl had just died of a common situation…was not assisted in the general room, died and she could have been saved” (implementation leader; Xalapa, Mexico). Although most participants described a dedication to safety, many also mentioned institutional resistance to change or a medical culture that discouraged interdisciplinary collaboration as a barrier to ongoing PEWS use.

Specific characteristics of clinical staff using PEWS influenced the ease of implementation. These characteristics included resistance to change (stage of change), skill calculating PEWS and measuring vital signs, and other characteristics (Table 2).

Participants at all centers mentioned staff resistance to change as a major barrier to implementation of quality improvement initiatives such as PEWS. A particular challenge was the perception that PEWS would increase workload: “Initially it was very hard…they said they studied already, they were already formed and that it was more work for them” (implementation leader; Xalapa, Mexico). When staff were convinced of the importance of PEWS, however, their conviction became a strong enabler: “I think the attitude of the staff, the commitment, which was the most important thing that made the project possible…they realize this is beneficial and that work becomes easier and faster, they end up accepting it” (foundation administrator; San Salvador, El Salvador).

Other characteristics of clinical staff perceived to impact PEWS implementation included their age, subspecialty training, motivation, and dedication to their work. The staff’s inability to measure vital signs, perform a patient assessment, and correctly calculate PEWS were identified as specific barriers to implementation. With practice, staff who lacked these skills were able to acquire them: “[nurses] forgot to check the pupils…at the beginning we had more than 15% [errors] but then we’re always low, that means the measurements were done correctly” (implementation leader; Xalapa, Mexico).

The outer setting was mentioned less frequently than other domains. However, the outer setting, or factors external to the hospital, including health systems and the hospital’s experience with collaboration with other institutions, affected implementation (Table 2).

Participants discussed the effects of health systems on their ability to provide high-quality care and implement interventions such as PEWS, including challenges of being in a resource-limited country with low levels of funding to support improvement initiatives, rigid laws and regulations that affected clinical work, and a small national workforce: “Our country is a poor country, our hospital is a public hospital, we don’t have many resources and it’s difficult to request them” (implementation leader; Lima, Peru). Some participants emphasized that aligning the objective of PEWS with criteria for hospital certification and national priorities served as a major enabler: “We moved forward because this is a new way of work and that was certified not only by the institution but also as quality system of management in the Ministry of Health” (nurse director; Lima, Peru).

In addition to health systems, a center’s experience with collaboration with other institutions affected PEWS implementation. Collaborations included those with other hospitals, professional organizations, and philanthropic foundations on projects related to quality improvement, research, education, and patient care. Participants described collaborative experiences as sources of new improvement ideas for the hospital, which was helpful for PEWS implementation: “The experience you gain when participating and collaborating with projects, that helps” (quality improvement coordinator; San Salvador, El Salvador).

Participants reported specific characteristics of PEWS as barriers or enablers to implementation. The characteristics included its origin, strength of supporting evidence, and perceived complexity (Table 2).

PEWS was commonly introduced to a center (PEWS origin) by a clinician who heard about the project from another hospital. The perception that PEWS belongs to one person was a barrier that had to be overcome for successful implementation: “That was very important…from knowing it was my project, my study, to know it was a project for this hospital” (implementation leader; Cuenca, Ecuador). Clarity that PEWS came from an international collaborative also helped win support: “When we heard [the study was being led by] St Jude, this is a world reference in the treatment of cancer in children…we knew we could participate in that project, because it’s a project that comes from a serious staff” (physician director; Cuenca, Ecuador).

All participants were motivated by published evidence supporting PEWS, including its validity, effect on patient outcomes, teamwork, and hospital costs. Many participants were reassured by anecdotal stories of successful implementation from centers with similar resources: “the results were good in other hospitals so that made us think that if other hospitals in the same level could get better, we could definitely get better as well” (implementation leader; Lima, Peru). Seeing how PEWS worked at other centers was particularly valuable to inform local implementation strategies: “it’s very important to know the experiences of other centers because they already made their mistakes and we can prevent those same errors” (implementation leader; Xalapa, Mexico).

Another important PEWS characteristic was its perceived complexity. Initially, staff were concerned that using PEWS would be challenging. Experience, however, alleviated these concerns: “at the beginning it was difficult because we felt like we’re wasting a lot of time…right now is very easy” (nurse director; San Louis Potosi, Mexico). Ultimately, participants believed PEWS was a simple intervention that did not require many resources, facilitating implementation: “a project which doesn’t require a great amount of money and the benefit[s] are huge…we’re not going to invest money, we’re just going to use the resources we have…this influenced a lot in the authorities to support this project” (implementation leader; Xalapa, Mexico).

All hospitals faced barriers. Most barriers, however, were resolved during the standardized implementation process, including adaptation, engagement, piloting, evaluation, and obtaining outside help, allowing all centers to successfully implement PEWS.

All centers planned for PEWS implementation by adapting both the PEWS intervention and their center context (Table 3). Minimal adaptations were made to the PEWS scoring tool, focused on medical terminology and the PEWS algorithm to reflect local processes for escalation of care. All participants, however, described changes to their hospital setting to address barriers from their hospital’s inner and outer settings to support PEWS use. Adaptations included changes to the physical space (posting PEWS information and patient tracking boards), documentation (nursing flowsheet, physician notes), hospital processes (frequency of vital signs, care escalation process), policies, and culture (interdisciplinary teamwork and communication). Challenges making these adaptations delayed implementation: “To implement the [modified nursing] sheet, how to do it…I think those aspects delayed the project…made the implementation process slower” (nurse director; San Salvador, El Salvador).

All participants also emphasized the need for early engagement of a diverse set of hospital stakeholders through training, informational meetings, and one-on-one conversations to address specific clinical skill deficits and general reluctance to adopt PEWS (Table 4). Important stakeholders included the PEWS implementation leaders, all clinical staff, hospital leadership, and others who supported PEWS adoption and use (champions), including the department of quality and safety, staff educators, and families. Late engagement of these stakeholders resulted in delayed implementation: “Maybe one of the mistakes was not to involve the medical staff, the pediatricians, earlier…it was something unknown for them” (implementation leader; Xalapa, Mexico).

As part of the implementation process, centers were mentored to pilot PEWS, which participants viewed as necessary to identify areas for improvement: “A pilot will always show you the limitations you have and how [to] maximize the benefits you’ve seen” (physician director; Lima, Peru). Lessons learned during the pilot identified additional opportunities to reduce barriers to implementation, including the need to further adapt PEWS, nursing flowsheets, or equipment available to support PEWS use, learning strategies to engage staff in PEWS, the importance of teamwork, increasing skill using PEWS through practice, and seeing the effect of PEWS on patient outcomes (Table 4).

Following the pilot, centers implemented PEWS with ongoing evaluation of PEWS use: “We have to register the PEWS errors…and have meetings to check how PEWS is working, we see if the staff still needs training” (implementation leader, San Louis Potosi, Mexico). These evaluations identified common errors, demonstrated staff satisfaction, and highlighted effect on patient outcomes, further enabling implementation by justifying the ongoing need for PEWS at the centers.

Centers were guided throughout PEWS implementation with outside help from St Jude, other Proyecto EVAT centers, and philanthropic foundations. This assistance informed all components of the implementation process, including obtaining the necessary training and addressing potential barriers from inadequate resources: “The clip from the oximeter is broken…this is a priority for the foundation, we must repair or buy a new one. So, with the equipment maintenance the foundation has been great help” (implementation leader, San Salvador, El Salvador).

We evaluated barriers and enablers to PEWS implementation in resource-limited settings, identifying multiple barriers at the level of the hospital, clinical staff, outer setting, and PEWS intervention. Despite geographic and organizational differences in the characteristics of participating hospitals, barriers were similar across centers, including having inadequate resources, staff resistance to change, components of health systems, and the perceived origin and complexity of PEWS. All centers, however, were able to overcome these barriers during the implementation process using outside help, adaptation, staff engagement, piloting, and evaluation to convert barriers to enablers and successfully implement PEWS. These findings can be used to guide clinicians on strategies to implement evidence-based interventions in resource-limited hospitals to improve patient outcomes. Specific recommendations informed by this work include early engagement of all relevant stakeholders before starting implementation and using a time-limited pilot followed by an evaluation and further adaptation (using quality improvement methods) to proactively identify and address challenges to the adoption and ongoing use of evidence-based interventions in these settings.

In addition, this study provides further evidence for the CFIR framework and supports modifications suggested for use in resource-limited settings.²⁴ Similar to prior work,²⁴ participants emphasized the importance of the implementation process, including early stakeholder engagement, overcoming resistance to change, contextually piloting the intervention, and evaluating outcomes. Our results, however, identified several differences from what has been reported in high-resource settings. The CFIR describes an intervention as having a rigid core and an adaptable periphery.^22,23 Although participants mentioned the need to make minor adjustments to PEWS to fit their context, most adaptations occurred in the hospital setting to support PEWS use, including leveraging external collaborations and advocating hospital leadership to overcome existing resource limitations. These findings correspond with evolving evidence that hospitals develop clinical capacity to sustain an intervention during the early implementation process.^34,35 Factors influencing PEWS sustainability should be explored in future work.

Another unique finding was the importance participants placed on the external origin of PEWS (originating outside their center), the support of the PEWS international collaborative (Proyecto EVAT), and the role of other institutions (St Jude, other Proyecto EVAT centers, and local foundations) in facilitating implementation. Resource-limited hospitals may be more willing to adopt interventions used in high-resource settings and that come with external multicenter, international support. The mentorship and shared experience of other geographically and organizationally similar resource-limited centers served as an enabler to implementation. These findings support recent global efforts to improve outcomes for children with cancer through local empowerment, regional collaborations, and international partnerships,^26,36 and are a model for the global scale of strategies to support implementation of evidence-based interventions such as PEWS.

This work has several limitations. This study included only 5 centers; however, inclusion of in-depth interviews with a variety of key stakeholders adds credibility to our findings. This study is further strengthened by a sample size sufficient to reach thematic saturation, regional and organizational diversity of participating centers, and the variable time required for PEWS implementation. Quantitative analysis of a larger sample comparing center characteristics with time required for PEWS implementation would build on our current findings and should be conducted in future work. Similarly, this study focused on evaluating the implementation of one intervention at pediatric oncology centers, potentially limiting its generalizability to other interventions or other patient populations. The identified barriers and enablers, however, are supported by findings from other contexts,^19,21 are not specific to PEWS or pediatric oncology, and can broadly inform strategies to implement evidence-based interventions globally. This study was conducted during the COVID-19 pandemic, which affected pediatric oncology care delivery worldwide.^37-39 Although participating centers completed PEWS implementation before March 2020 and experienced barriers independent of the pandemic, sustainability and scale-up of PEWS was likely affected during the pandemic, and this effect should be explored in future work.

We present the first multicenter, multinational study describing barriers and enablers to PEWS implementation in resource-limited hospitals. The findings suggest that many identified barriers can be converted to enablers through targeted effort during the implementation process. These findings provide guidance on strategies to support implementation of evidence-based interventions to reduce global disparities in patient outcomes.

Accepted for Publication: January 10, 2022.

Published: March 9, 2022. doi:10.1001/jamanetworkopen.2022.1547

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2022 Agulnik A et al. JAMA Network Open.

Corresponding Author: Asya Agulnik, MD, MPH, St Jude Children’s Research Hospital, 262 Danny Thomas Pl, Mail Stop 721, Memphis, TN 38105 (asya.agulnik@stjude.org).

Author Contributions: Dr Agulnik had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Agulnik, Armenta, Barra, Mendez, Graetz.

Acquisition, analysis, or interpretation of data: Agulnik, Ferrara, Puerto-Torres, Gillipelli, Elish, Muniz-Talavera, Gonzalez-Ruiz, Díaz Coronado, Hernandez, Juárez Tobias, Loeza, Mendez, Montalvo, Penafiel, Pineda Urquilla, Graetz.

Drafting of the manuscript: Agulnik, Hernandez, Mendez.

Critical revision of the manuscript for important intellectual content: Agulnik, Ferrara, Puerto-Torres, Gillipelli, Elish, Muniz-Talavera, Gonzalez-Ruiz, Armenta, Barra, Díaz Coronado, Juárez Tobias, Loeza, Mendez, Montalvo, Penafiel, Pineda Urquilla, Graetz.

Statistical analysis: Muniz-Talavera, Mendez.

Obtained funding: Agulnik.

Administrative, technical, or material support: Agulnik, Ferrara, Puerto-Torres, Elish, Gonzalez-Ruiz, Armenta, Mendez.

Supervision: Agulnik, Juárez Tobias, Loeza, Mendez, Graetz.

Conflict of Interest Disclosures: No disclosures were reported.

Funding/Support: This study was funded by the American Lebanese-Syrian Associated Charities. Dr Agulnik was funded by the Conquer Cancer Foundation Global Oncology Young Investigator Award for this work.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: We thank the Pediatric Early Warning Systems Implementation Team at all Proyecto EVAT centers, including those who participated in this study, as well as the Proyecto EVAT Steering Committee for oversight of this work.