Evaluation of a Novel Web-Based Pediatric Advanced Life Support Course

Objective  To assess the educational efficacy of a Web-based pediatric advanced life support course (Web-PALS).

Design  Nonrandomized, prospective, cohort study.

Setting  University medical center.

Participants  Health care providers (includes physicians, nurses, paramedics, and respiratory therapists) taking either the Web-PALS or a traditional PALS course (Trad-PALS).

Main Exposure  Web-PALS.

Main Outcome Measures  Postcourse written examination scores and scored videotapes of students performing 5 PALS procedures were compared between study groups. Students completed precourse and postcourse questionnaires, rating on a 5-point Likert scale their self-confidence to perform PALS assessments and procedures. A structured, course satisfaction survey was given after students had taken the Web-PALS course.

Results  Eighty-six students completed the study (44 Web-PALS and 42 Trad-PALS). All students achieved a passing score on the written examination on their first attempt. Compared with students in the Trad-PALS group, students in the Web-PALS group scored slightly lower (97.1% vs 95.4%; difference, 1.7%; 95% confidence interval, 0.1-3.2). Mean overall videotape scores were similar among the Web-PALS and Trad-PALS groups (75.0% vs 73.0%; difference, 2.0%; 95% confidence interval, −2.0 to 6.0). After completing the Web-PALS course, the mean level of confidence improved from 3.77 to 4.28 (difference, 0.51; 95% confidence interval, 0.33-0.69). Ninety-six percent of respondents indicated that Web-PALS met all of the stated objectives of the PALS course. All respondents indicated that they would recommend Web-PALS to a colleague.

Conclusions  Students perceive Web-PALS as a positive educational experience. Though not identical to students taking the Trad-PALS course, they performed well on postcourse cognitive and psychomotor testing. These findings support Web-PALS as an acceptable format for administering the PALS course.

In cooperation with the American Heart Association (AHA), in July 2002, we introduced a Web-based pediatric advanced life support course (Web-PALS) that teaches the cognitive portion of the PALS material via an Internet Web site. Through the Web site, students learn PALS concepts by completing 6 interactive, case-based modules. After completing online modules, students attend a 1-day course that provides the psychomotor training and testing components of the course. This format is similar to other self-directed, distance-taught life support courses such as the CD-ROM–based HeartCode advanced cardiac life support Anywhere course¹ and the online basic life support for health care providers² and electronic advanced cardiac life support³ renewal courses. Web-PALS is offered to students taking an initial course and to those seeking recertification.

The Web-PALS course allows students to review the cognitive portion of the PALS material at a convenient time and at their own pace. The reduced onsite time lessens time away from home and office and has the potential to decrease travel-related expenses. The Web-PALS course offers advantages for PALS instructors and administrators by providing cognitive material online, thus decreasing the number of instructor-hours and related expenses to teach the PALS course. It ensures that the cognitive material is provided in a consistent manner, including the dissemination of PALS updates to all students. Finally, with the increasing availability of broadband Internet access, in the future, Web-PALS will be ideally suited to present AHA-developed instructional videos to students taking the PALS course.

We describe the development of the Web-PALS course and a pilot study to assess its educational efficacy. The primary objective of the study was to compare the cognitive and psychomotor performance of students who completed the Web-PALS course with that of students who completed a traditional, classroom-based PALS course (Trad-PALS). Secondary goals were to assess the effect of the Web-PALS course on student self-confidence to perform PALS assessments and procedures and their overall satisfaction with the course.

Core material from the PALS course was incorporated into 6 interactive, case-based modules: respiratory failure, shock, newly born, rhythm disturbances, trauma, and cardiac arrest. Each module contains a list of objectives, an interactive case, links to pertinent algorithms, and postmodule questions. Postmodule questions differ from those on the postcourse written examination. For each module, students select from a prehospital or hospital setting. Each module is designed to take approximately 1 to 1½ hours to complete.

Students access the Web site at http://myamericanheart.org. The current charge for taking Web-PALS, including online and onsite components, is comparable to the charge for a traditional 2-day PALS course. A link for purchasing a hard-copy PALS textbook is provided on the Web site. Students must satisfactorily complete the AHA PALS pretest (minimum score, 85%) online before entering the modules. To complete each module, students must correctly answer 85% of the postmodule questions.

After completing the modules, students are eligible to take a 1-day skills/testing course, which is an 8-hour onsite class formatted to meet all of the AHA requirements for instruction of the psychomotor components of the PALS course.⁴ During the 1-day skills/testing course, students rotate through skills instructional stations and case scenario practice stations. At the end of the course, students attend practical evaluation stations and complete the postcourse written examination. After satisfactory completion of the course, students receive an AHA course completion card.

Between July 16, 2002, and September 4, 2003, health care providers (includes physicians, nurses, paramedics, and respiratory therapists) taking either a Web-PALS or Trad-PALS course at the Saint Louis University Health Sciences Center were recruited into the study. The Trad-PALS course taught at our institution is a 3-day pediatric advanced cardiorespiratory and trauma support (PACTS)–PALS course that integrates all of the PALS elements with additional PACTS material. The PACTS curriculum, developed at our institution, consists of classroom lectures on topics such as surgical, neurosurgical, and toxicologic emergencies. In addition, students rotate through a practical trauma skills and scenario station.⁵

During the study, subjects were recruited from 14 PALS courses (8 Web-PALS and 6 Trad-PALS). All students, regardless of profession or previous PALS experience, were eligible to participate. Subjects were excluded from consideration if they were enrolled in a 1-day PALS recertification course. As an incentive, study subjects were reimbursed $50.00. The study was approved by the Institutional Review Board at Saint Louis University. All subjects provided written informed consent.

Before the start of each PALS course, study subjects completed questionnaires to provide demographic and self-confidence data. The self-confidence portion, adapted from a questionnaire developed by Craven and Froman,⁶ consisted of 39 items on which students rated their confidence level to perform various PALS assessments and procedures on a 5-point Likert scale (1, very little; 5, quite a lot). Students taking the Web-PALS course completed the questionnaire online before being allowed into the educational modules. Students taking the Trad-PALS course completed the questionnaire at the beginning of the first day of each course. The self-confidence questionnaire was readministered to subjects immediately after completion of the course. Changes in self-confidence scores were assessed for students within each study group.

To assess cognitive performance, scores from the AHA postcourse written examination were collected. The postcourse written examination consists of 33 multiple-choice questions developed and validated by the Pediatric Resuscitation Subcommittee of the AHA (Mary Fran Hazinski, MSN, written communication, January 2004). Successful completion of the written examination (minimum score, 84%) is required to receive PALS certification. Version A of the 2002 AHA PALS examination was used for both groups throughout the study. The frequency of correct responses for each examination question and overall mean examination scores were compared between groups.

To assess psychomotor skills, videotapes were made of each subject performing a rapid cardiopulmonary assessment (RCPA) and 4 PALS procedures in a mock resuscitation setting. The videotaping protocol was modified from a protocol previously described by White et al⁷ and Quan et al.⁸ The procedures included bag-mask ventilation, tracheal intubation, intraosseous needle insertion, and rhythm assessment/defibrillation. Subjects participated in the videotaping sessions immediately after completion of each PALS course. The videotaping session was an additional station that study subjects participated in individually after completing the usual practical evaluation and written testing stations. Students were given a single opportunity to perform each task. Skill performance was not timed.

The videotaping sessions took place in a classroom setting with 4 Laerdal mannequins (Laerdal Medical Corp, Wappingers Falls, NY), a cardiac monitor/defibrillator with Laerdal 2000 HeartSim Rhythm Simulator, and standard PALS resuscitation equipment. Mannequin features included inflatable lungs, an intubatable airway, and an artificial tibia for intraosseous needle insertion. The room setup and equipment was identical for each videotaping session. One of 5 certified PALS instructors, trained in the study protocol, served as the moderator during each session. For practical considerations, moderators (including J.M.G., A.J. S. and S.P.L.) were not blinded to which course subjects had taken.

Each subject was given an identical scenario for the RCPA and each procedure. Reading from a prepared, written script, the moderator described the mock scenario, the age of the patient, and the specific task to be performed, for example, “You are asked to see a 6-month-old infant with a 2-day history of cough and trouble breathing. Please perform a rapid cardiopulmonary assessment.” Moderators provided identical, scripted answers for all of the RCPA assessments and procedures. Moderators were instructed not to lead or prompt the students in any way during the videotaping sessions.

At the end of the study, videotapes were scored by a group of reviewers blinded to which course subjects had taken. The review group consisted of 3 certified PALS instructors or medical directors with extensive experience in this field. Reviewers were not involved in developing the Web-PALS program and were not instructors of any of the PALS courses.

A 41-item score sheet identifying key subcomponents of the RCPA and procedures was designed for this study. Each of these subcomponents is listed as a specific teaching point within the PALS instructor manual and is considered important for the optimal performance of the RCPA and each procedure.⁴ Reviewers used this score sheet to grade each subject's performance. To score a subcomponent as being performed properly, the subcomponent had to be performed correctly and at an appropriate time in the given scenario. No partial credit was given.

To aid in scoring agreement, before scoring the videotapes, the reviewers met as a group with study investigators. Videotapes of 5 subjects were extensively reviewed and discussed for the scoring process. Disagreements on scoring of individual subcomponents were discussed and a consensus was reached on how subcomponents would be scored.

Over a 2-day period, each subject's videotape was independently scored by a single reviewer. These scores were used for comparisons between the study groups. After scoring the subjects' videotapes, each reviewer was given a copy of the videotapes of 10 randomly selected subjects. Each reviewer independently scored these same 10 subjects' videotapes, and the results were used to assess interrater agreement.

After completion of videotape scoring, study investigators (J.M.G. and P.S.) tabulated the score sheets. For each subject, an overall score (the number of all correctly performed subcomponents) as well as individual scores for the RCPA and each procedure were recorded. The frequency of correct performance for each subcomponent was compared between the study groups. In addition, group comparisons were made for mean overall score, RCPA score, and individual procedure score.

After completion of each Web-PALS course, to elicit feedback on the course, all students were given a 24-item, structured survey that addressed 3 main themes: technical aspects and usability of the Web-PALS site, content and flow of the online material, and content and flow of the 1-day skills/testing course. Many of the response formats used a continuous scale of 1 to 5 (“strongly disagree” or “really disliked” to “strongly agree or really liked,” respectively) by which students rated the various aspects of the course. Partially completed surveys were included in the analyses. No attempt was made to elicit feedback from nonrespondents.

For written examination scores, based on data from previously taught Trad-PALS courses at our institution, we calculated that to detect a 2.5% difference in mean scores a sample size of 40 subjects in each study group would be needed to achieve a power of 80% at the .05 significance level. The study was not powered for other outcome measures.

Comparisons of proportions were made using the χ² test or Fisher exact test. Because of a nonnormal distribution, the Mann-Whitney test was used to compare written examination scores. For the videotape component, mean overall scores and mean scores for the RCPA and each procedure were compared using independent-samples t tests. Cohen's multirater κ values were calculated to assess interrater agreement on videotape scoring. Paired-samples t tests were used to compare precourse and postcourse self-confidence data. A 2-tailed α level of .05 was considered statistically significant for all analyses. Power analyses were performed using S-Plus Version 4.5 (Insightful Co, Seattle, Wash). All other analyses were performed using SPSS version 12 with the Exact Tests add-on module (SPSS Inc, Chicago, Ill). A macro titled mkappasc.sps was downloaded from the SPSS Web site that enables the base version of SPSS to calculate the Cohen multiple rater κ values.

During the study, excluding students enrolled in recertification classes, 349 students attended PALS courses (80 Web-PALS and 269 Trad-PALS) at our institution. Of this group, 88 subjects (46 Web-PALS and 42 Trad-PALS) were enrolled in the study. Two subjects from the Web-PALS group were excluded from the statistical analyses. One was excluded because of incomplete self-confidence data. The other was excluded because, after the course, it was discovered that the student had not completed all of the online modules; the student was inadvertently allowed to take the 1-day skills/testing session and mistakenly allowed in the study. Eighty-six subjects (44 Web-PALS and 42 Trad-PALS), therefore, make up the study groups for comparisons. There were no significant differences in demographic characteristics between the study groups (Table 1). Thirty-two students (73%) in the Web-PALS group were taking an online educational course for the first time.

All students in both groups successfully passed the postcourse written examination on the first attempt. Compared with students in the Trad-PALS group, students in the Web-PALS group scored slightly lower on the written examination (97.1% vs 95.4%; difference, 1.7%; 95% confidence interval [CI], 0.1-3.2). No significant differences were seen between groups for frequency of correct responses for any of the individual examination questions.

For interrater agreement of videotape scoring, using the classification scheme of Landis and Koch,⁹ we rated agreement as poor or slight for 6 subcomponents, fair for 3, moderate for 7, substantial for 8, and nearly perfect for 17. Interrater agreement on overall videotape scores was substantial (κ = 0.75). Agreement was nearly perfect for the RCPA (κ = 0.86), substantial for the tracheal intubation, intraosseous needle insertion, and rhythm assessment/defibrillation procedures (κ = 0.74, 0.69, and 0.75, respectively), and moderate for the bag-mask ventilation procedure (κ = 0.58).

Students in the Web-PALS and Trad-PALS groups scored similarly on 38 of 41 skills subcomponents (Table 2). Differences between groups were seen for 3 subcomponents. During the tracheal intubation procedure, students in the Web-PALS group more often inserted the tracheal tube to the proper depth (61% vs 38%; difference, 23%; P = .03) and more often properly secured the tracheal tube (77% vs 57%; P = .046). During bag-mask ventilation, students in the Web-PALS group used the proper-sized resuscitation bag less often than did students in the Trad-PALS group (34% vs 60%; P = .02).

Mean overall videotape scores were similar among the Web-PALS and Trad-PALS groups (75% vs 73%; difference, 2%; 95% CI, −2 to 6; Table 3). Students in the Web-PALS and Trad-PALS groups performed similarly on RCPA, bag-mask ventilation, and rhythm assessment/defibrillation procedures. Compared with students in the Trad-PALS group, those in the Web-PALS group scored higher on tracheal intubation (76% vs 68%; difference, 8%; 95% CI, 0.3-16) and intraosseous needle insertion (73% vs 64%; difference, 9%; 95% CI, 0.1-18).

Self-confidence to perform PALS assessments and procedures significantly improved in both groups after completion of the PALS course. In the Web-PALS group, mean self-confidence improved from 3.77 before the course to 4.28 after the course (difference, 0.51; 95% CI, 0.33-0.69). Among students in the Trad-PALS group, mean self-confidence improved from 3.57 before the course to 4.24 after the course (difference, 0.67; 95% CI, 0.52-0.81).

During the study, 60 students (75% of potential respondents) completed the Web-PALS postcourse survey (Table 4 and Table 5). The results reveal that students seem to view the Web-PALS course as a positive educational experience. When asked to assess the overall educational experience, students gave a mean ± SD rating of 4.7 ± 0.5. Ninety-six percent of respondents believed that the Web-PALS course met all of the stated objectives of the PALS course. All of the respondents indicated that they would recommend the Web-PALS course to a colleague. The lowest ratings were given for the registration process (4.3 ± 1.0) and flow of the online modules (4.3 ± 0.8) (Table 5).

Despite their availability, the educational efficacy of distance-taught life support courses has largely not been studied. A limited number of studies have shown mixed results on self-instruction of basic life support and advanced cardiac life support material.^10-16 In the present study, students in the Web-PALS group compared favorably with students in the Trad-PALS group for cognitive (postcourse written examination scores) and psychomotor performance. The AHA-validated written examination is proportionally weighted to reflect the main teaching objectives of the PALS course and is an ideal tool to assess student knowledge of the most fundamental PALS concepts. Although their scores were slightly lower than those of students in the Trad-PALS group, students in the Web-PALS group performed well on the examination, achieving a mean ± SD score of 95.4% ± 3.9%. All students in the Web-PALS group successfully passed the written examination on the first attempt and scored well above the minimum passing score of 84% mandated by the AHA.

Previous studies of Trad-PALS courses have reported somewhat lower written examination scores than those found in our study. In a study of 370 Israeli health care workers, Waisman et al¹⁷ reported a postcourse mean score of 87%. In a smaller study of pediatric residents, Nadel et al¹⁸ reported a mean score of 93% among pediatric residents who had taken a PALS course 6 months earlier. To our knowledge, however, this is the first report using the 2002 version of the examination. It is impossible, therefore, to accurately compare examination scores from the present study with any published data from courses administered outside of our institution.

Few studies have examined skills performance of students who have taken the PALS course.^7,8,17 To assess psychomotor skills, we modified a protocol previously described in studies by White et al⁷ in 1998 and by Quan et al⁸ in 2001. In addition to the 4 procedures examined in these studies, we assessed student performance on the RCPA. Our study is unique in its inclusion of the RCPA in the skills assessments.

Unlike these previous studies, we did not time students during psychomotor testing. The speed with which assessments and procedures are performed is critical during real resuscitations. In the present study, students typically performed tasks within a reasonable amount of time, consistent with what we would expect in a real setting. As an initial evaluation of the Web-PALS course, however, we were more interested in determining whether students had learned all of the concepts needed to properly perform psychomotor skills. By not timing students, we allowed them to focus their attention on being thorough and technically correct during the skills evaluations. This enabled us to better determine if the concepts needed to perform each task were sufficiently presented in the Web-PALS course.

In our study, psychomotor scores were somewhat lower than expected in both groups. We attribute this to the strict criteria that reviewers used when scoring videotaped performances. Students in the Web-PALS and Trad-PALS groups performed similarly in overall skills performance. Students in the Web-PALS group performed similarly or better than those in the Trad-PALS group on all but one of the skills subcomponents. When individual procedures were compared, students in the Web-PALS group performed better on tracheal intubation and intraosseous needle insertion. By enhancing the review of cognitive material before the day of skills training, students may have been able to better focus on skills acquisition during these sessions.

Several studies have examined student satisfaction with distance-taught life support courses.^12,16 Similar to these, we found that students who took the Web-PALS course generally seem to like this mode of studying the course material, especially because of its flexibility and self-directed learning format.

There are a number of limitations to our study. The study groups are a mixture of subjects from different professions and with different previous PALS experience. Though this demographic profile reflects what is typically seen in most PALS courses, the small number of subjects in our study prevents us from reaching any meaningful conclusions about the performance of individual subgroups. The study is underpowered to show how first-time students who have taken a Web-PALS course compare with first-time students who have taken a Trad-PALS course.

The potential for selection bias in our study cannot be overlooked. Students with previous PALS experience were more likely to participate in the study. This contributed, in part, to the disparity between the groups for the percentage of eligible students who participated in the study (44 [55%] in the Web-PALS group vs 42 [16%] in the Trad-PALS group). Such bias, coupled with nonrandomization of subjects, could potentially lead to overestimation of the true educational effect of the Web-PALS course.

Students in the Trad-PALS group received PACTS in addition to standard PALS training. We believe there is little overlap between the PACTS and PALS curriculae. It is possible, however, that, to some extent, PACTS reinforces some of the general PALS concepts. This may have led to somewhat higher scores than would otherwise be expected among traditional students. We believe, however, that the effect of this potential confounder does not change the main conclusions reached in our study.

Our study was limited by the artificial nature of mock resuscitation scenarios. Though previous studies have shown positive effects of life support courses on performance during actual resuscitations, it is unknown with certainty how subjects would perform in a real setting.^19,20 Inasmuch as moderators were not blinded to which course subjects had taken, we sought to overcome this potential for bias by adhering to a strict videotaping protocol and scoring process. Moderators read from a prepared, written script and were instructed not to lead or direct the students in any way during the videotaping sessions. Despite these safeguards, the potential for moderator bias in leading students during the skills evaluations cannot be excluded.

Because we did not directly observe students studying online, it is unknown how much time students spent reviewing cognitive material. Moreover, our study does not longitudinally address the retention of knowledge, skills ability, or perceived self-confidence among students who have taken the Web-PALS course. Long-term retention of learned concepts is a vital measure in the assessment of life support courses. Retention of knowledge and skills has been shown to diminish over time among students who have taken Trad-PALS courses.^21,22 We are unable to comment on this aspect of the Web-PALS course.

Despite its limitations, the present study contributes to the evolving knowledge of distance-taught life support courses. The findings in our study support the continued development and promotion of the Web-PALS course. Future studies, with larger numbers of subjects, are needed to address questions that are unanswered by our study. Of greatest importance is the need to determine whether the Web-PALS course is appropriate for first-time PALS students and the effect of the course on long-term retention of knowledge and skills performance.

Correspondence: James M. Gerard, MD, Department of Pediatrics, Division of Pediatric Emergency Medicine, Saint Louis University School of Medicine, 1465 S Grand Blvd, St Louis, MO 63104 (gerardjm@slu.edu).

Accepted for Publication: November 19, 2005.

Author Contributions: Dr Gerard had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Financial Disclosure: Since the completion of this study, Ms Seratti has received financial compensation from the American Heart Association as a consultant to further develop the 1-day skills/testing course at PALS training sites.

Funding/Support: This study was funded by grant 1 H33 MC 00103 from the Health Resources and Services Administration and by a grant from the American Heart Association.

Previous Presentation: This study was presented at the Pediatric Academic Societies Meeting; May 15, 2005; Washington, DC.

Acknowledgment: We thank Steve Schexnayder, MD, for his insightful review of the manuscript.