Copyright 2000 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2000
To determine the effectiveness of a safety education program, Safety City, that is designed to teach kindergarten and first grade children how to cross the street, call 911 in an emergency, and avoid strangers.
Kindergarten students at 10 urban elementary schools.
Each school was randomized to either the intervention or control group. An evaluation tool was administered to all participants as a pretest. The Safety City program was then presented to the intervention schools. Afterward, the same evaluation tool was used as a posttest. The posttest was administered to the intervention group 6 months after the Safety City program was presented. The control group took the posttest 6 months after the pretest.
Main Outcome Measure
Change in individual test scores.
One hundred eighty-one children completed the pretest and posttest evaluations. There was no statistical difference in the change between pretest and posttest scores of children who participated in the Safety City program and those in the control group (crossing the street, P = .29; calling 911, P = .41; stranger avoidance, P = .57).
Exposure to the Safety City program did not achieve the desired changes in safety knowledge among participants. This is most likely owing to the fact that Safety City attempts to convey a large amount of relatively complex information to young children in a brief period. We conclude that programs such as Safety City are not sufficient to teach children these behaviors. This report also emphasizes the importance of building an evaluation component into educational programs.
IN 1989, the American Red Cross of Greater Columbus (Columbus, Ohio) initiated the Safety City program. Safety City is a half-day course designed to teach kindergarten and elementary school children basic rules regarding several safety issues. Individual schools invite the Red Cross of Greater Columbus to present the Safety City program to their students. Kindergarteners and first graders receive education on how to cross the street, avoid strangers, and call 911 in an emergency. Second and third graders are taught fire and electrical safety as well as gun avoidance.
During the Safety City program, participants are divided into groups of 25 to 30 students. These groups rotate between the topics just described. Approximately 20 minutes is devoted to each topic. Individual sessions consist of a lecture during which the teacher reviews the main learning objectives (Table 1). Afterward, there is a practice session or game that reinforces these objectives. For instance, students practice crossing the street at a mock intersection that includes a working traffic light, crossing signal, and street. A board game is used to reinforce the stranger avoidance curriculum, and students pretend to dial 911 using a fake telephone with one of the instructors serving as the operator. At the end of the program, students are given coloring books and are treated to a rock concert given by Safety City instructors. The coloring book and lyrics to the music reinforce Safety City teaching objectives.
Every year, more than 20,000 Columbus-area children participate in the Safety City program. Volunteers from local fire departments, police departments, high schools, the electric company, and a television station provide the instructors for Safety City. The program receives funding from corporate sponsors.
Representatives from the American Red Cross of Greater Columbus approached the authors to help them formally evaluate the kindergarten and first grade curriculum. The purpose of this study was to determine whether exposure to the Safety City curriculum had an effect on the participant's knowledge of these safety behaviors. While Safety City is unique to the Columbus area, there are many similar programs and curricula designed to teach children these skills. To our knowledge, few have been formally evaluated. This study addresses this important need.
During the summer of 1996, the principals of 10 Columbus elementary schools that include the Safety City program in their curricula were contacted. Each school was invited and subsequently agreed to participate in the evaluation. The schools were block-randomized so that there were 5 schools in the intervention group and 5 in the control group. Within each school, 25 kindergarten students in one class were randomly chosen to participate. Children in kindergarten were selected because they had not been previously exposed to the Safety City curriculum through the school system.
The investigators and Safety City administrators constructed an evaluation tool. The questions used in this tool were specifically designed to test key knowledge points taught during Safety City classes. These key knowledge points were taken directly from the scripts written for Safety City. The test consisted of 3 sections: how to cross the street, how to recognize an emergency and dial 911, and how to avoid strangers. A pediatric psychologist reviewed the test to ensure it was appropriate and could be understood by a 5-year-old child. The maximum number of points that could be scored for each of the sections on crossing the street and dialing 911 was 16. Fourteen points were possible for the section on stranger avoidance. Individual questions were weighted with regard to their importance to the Safety City curriculum.
The evaluation tool was administered to all participants as a pretest. All testing was done during normal school hours at a time agreed on by each school's principal and the investigators. The testing took approximately 15 minutes per student. The Safety City program was then administered to the intervention group. The Safety City staff noted which children involved in the evaluation were present for the program. Six months after the program (intervention group) or pretest (control group), the same evaluation materials were administered to all participants as a posttest.
The Wilcoxon rank sum test was used to compare the differences of pretest and posttest scores between the 2 groups. The t test was used to determine statistical power. The Mann-Whitney U test was used to compare pretest scores of those children who did not take the posttest evaluation (dropouts) with those who completed the study. Statistical significance was set at P≤.05. The Human Subjects Research Committee at Children's Hospital in Columbus, Ohio, approved the study.
The evaluation consisted of verbal questioning and skill demonstration. All subjects were evaluated individually. The questions used for the evaluation were as follows:
Question 1: Name the colors on a traffic light. What do each of these colors mean? 0.5 points per answer, or 3 points total, were awarded for the following answers: red means stop, green means go, and yellow means slow down.
Question 2: (A drawing of a city street with an intersection was used for this question.) This is a picture of a street. These are buildings, this is a traffic light, this is the sidewalk, and these are cars. Let's pretend that this is you. (Show the child a chess pawn.) You are standing on this X. (Place pawn on X located on the sidewalk.) Let's say you want to go to this store with the circle in front of it (point to circle). Show me where you would walk to get from the X to the circle. Four points were awarded if the child indicated he or she would cross at the crosswalk.
Question 3: (A mock intersection was set up for this section. The intersection had a street and traffic light. The colors on the light were changed by holding red and green cards over the appropriate areas on the traffic light.) I want you to stand here. (Evaluator positions child in front of traffic light then moves back to the traffic light and holds up the red card.) Let's pretend that this is a busy street. I want you to show me how you cross the street. Be sure to tell me what you are doing while you do it. (The evaluator waited at least 15 seconds before holding up the green card to see if the child knew when to cross.)
The following points were awarded for this section (9 total points were possible): Three points were awarded if the child said he or she would find an adult to help them cross the street. Two points were awarded if the child waited for the traffic light to change to green, 2 points were awarded if the child indicated he or she would listen for traffic, and 2 points were awarded if the child looked "left-right-left" before crossing.
Question 1: Let's pretend you are at home. You are doing your homework when you hear a loud noise. You look out of the window and see that a car has crashed into a telephone pole. You also see that the car is on fire. What would you do? Five points were awarded if the child said that he or she would tell an adult or call 911.
Question 2: What else could you do? Three points were awarded if the child said that he or she would tell an adult or call 911 if the child did not give that answer the first time the question was asked.
Question 3: Let's pretend you decide to call 911. (Evaluator takes out 2 telephones and places 1 in front of the child and 1 in front of the evaluator.) Show me how you would do it. Pretend I'm the 911 person on the other end. (The evaluator observes how they dial 911, asks them what the emergency is, and what their name is.)
The following points were awarded for this section (8 total points were possible): No points were awarded if the child did not dial 911 correctly. Two points were awarded if the child dialed 911 correctly. Two points were awarded if the child told the evaluator what happened. Two points were awarded if the child told the evaluator his or her name. Two points were awarded if the child did not hang up until the operator instructed him or her to do so.
Question 1: (The evaluator leaves the telephones where they are.) Let's pretend you are home alone. The phone rings and you answer it. I will talk to you on the phone but pretend you don't know me. (Have child pick up the phone.) Hello, this is Mr (Mrs) Smith. I'm a friend of your parents. Is your mom or dad home? Two points were awarded if the child did not tell the evaluator that he or she was home alone.
Question 2: What is your name? Two points were awarded if the child did not tell the evaluator his or her name.
Question 3: Listen, I need to drop something off for your parents but I can't remember your address. Can you tell me where you live? Two points were awarded if the child did not tell the evaluator his or her address.
Question 4 : Let's pretend a man you don't know comes up to you while you are playing on the playground at school. He tells you your mother has been hurt in a car accident and he is supposed to take you to the hospital to see her. What would you do?
The following points were awarded for this section (8 total points were possible): Eight points were awarded if the child indicated he or she would do any of the following: run away, tell a teacher or another adult, not get in the car, ask if the stranger knows the code word, or not go near the stranger.
One hundred twenty-four controls and 122 intervention subjects were initially enrolled in the evaluation. Ninety children in the experimental group and 91 control subjects completed the posttest evaluation. Dropout rates in each group were due to individual subjects being absent the day of the posttesting or having transferred to another school. Only the data for subjects completing both the pretesting and posttesting were used in the evaluation.
We did not collect demographic data for the children who participated in the project. The schools were all inner-city elementary schools that were believed to be homogeneous in regard to the socioeconomic status of the students. Because only kindergartners participated, the ages of the children were similar.
There was no statistical difference in the change between pretest and posttest scores of children exposed to the Safety City program and those of the control group. A summary of the pretest and posttest data is presented in Table 2, Table 3, and Table 4. The SDs in the differences between pretest and posttest scores of the control group were 3.5 points for crossing the street, 5.0 for dialing 911, and 4.2 for avoiding strangers. The results of the study were similar when the mean difference between pretest and posttest scores at individual schools were compared (P = .35 for crossing the street, P = .69 for dialing 911, and P = .89 for avoiding strangers).
Power calculations were based on a 2-point, or 10% to 15%, improvement in individual test scores for each section after exposure to the Safety City program. Based on our sample size and a 2-tailed α of .05, the study had approximately 95% power to detect a 2-point difference in test scores for the crossing the street section. For the dialing 911 and stranger avoidance sections, the power calculations for a 2-point difference in test scores were approximately 80% and 90%, respectively.
Forty-eight children (53%) in the intervention group and 46 (51%) in the control group demonstrated at least a 2-point improvement in test scores for the crossing the street section of the evaluation. In regard to the dialing 911 section, 60 children (67%) in the intervention group achieved at least a 2-point increase in test scores, compared with 57 children (63%) in the control group. Thirty-four children (38%) in the intervention group had at least a 2-point increase in test scores for the stranger avoidance section. Thirty children (33%) in the control group showed the same improvement.
Within the intervention group, a statistical difference was noted between the pretest scores of children who completed the study and those who dropped out for the stranger avoidance section (P = .04). No other differences in pretest scores between subjects and dropouts were noted.
There was no evidence of a beneficial effect of the Safety City program on participants' test scores. To conclude that Safety City is not effective in teaching the desired safety behaviors to children requires 2 assumptions. The first assumption is that the subjects understood the test questions. The second is that a child's performance on the test accurately predicts his or her behavior in the real world.
Poor performances on the evaluation tool could have resulted from either a knowledge deficit or not understanding the test questions. Children were asked several questions, such as naming the colors on a traffic light and demonstrating how to dial 911. Most of the test, however, required more abstract thinking. These items included indicating on a picture the proper place to cross the street, pretending they witnessed a car crash, or pretending a stranger approached them. It is possible that the children could not understand these more abstract questions. For instance, a large number of children scored zero on the stranger avoidance section. This section contained the most abstract questions. The testing instrument, therefore, represents a potential limitation of the study. The investigators and Safety City administrators developed it for the specific purpose of evaluating the program. Consequently, it has not been validated as a reliable method for assessing a child's knowledge of these safety practices.
Several steps were taken to maximize the clarity of the testing tool. The questions were constructed using Safety City scripts and were reviewed by a child psychologist for their appropriateness. Safety City employs scenarios similar to those used in the testing. If the children do not understand the questions, then they also may not understand or know how to apply the information presented during Safety City classes. It may be more reasonable, therefore, to present this curriculum to older children who have the ability to understand the material.
It is impossible to determine whether good test scores correlate with proper safety behaviors. A child may know that one should always cross the street at a crosswalk and may indicate this during a test. Whether the same child consistently exercises this practice at a busy street is another matter. The mean scores for all 3 sections of our evaluation instrument were fairly low, however. This may indicate that many of the children who participated in the study did not understand the learning objectives of Safety City. Unless they did not understand the test questions, it is hard to believe that the subjects would practice correct behaviors when they did not have knowledge of them. In the context of crossing the street, one could even argue that Safety City may increase the risk of a pedestrian injury by empowering children to cross the street when they are ill-prepared to do so.
Another limitation of the study is that any education the children received independent of the Safety City program could have influenced the results. Shortly before posttesting, a child attending one of the control schools was struck by a car. A teacher at this school then reviewed crossing the street with the class involved in the evaluation. The scores for this school were not significantly different than other schools in the control group despite the additional education. Furthermore, the study design included a control group to minimize the influence of outside factors on the study findings.
The scores for recognizing an emergency and calling 911 increased in both the control and experimental groups. This difference is a result of more children being able to correctly dial 911 when asked to do so. It is reasonable to assume that the study schools include number recognition as part of their kindergarten curriculum and this may have accounted for the rise in test scores.
There are few published data that examine the correct method for teaching injury prevention knowledge and skills to children. The studies we identified focus mainly on behaviors related to crossing the street. Programs that have been reported to be successful devote up to 6 hours in multiple sessions to teach this subject.1,2 Safety City, by comparison, devotes 20 minutes to teaching this complex behavior to a relatively large group of children. Other studies suggest the need for retraining to ensure adequate knowledge retention.3,4 Additionally, the ability of children in this age group to make safe street-crossing decisions has been questioned. Poor street-crossing skills have been associated with an inability to judge safe distances between oncoming traffic and the crosswalk,5 as well as lapses in attention while crossing.6
Appropriate safety behaviors will only be used if a child understands the skill, can correctly perform it, and is motivated to practice it. While Safety City may be useful for initially presenting this material, continued reinforcement by teachers and parents is likely to be necessary for success.7 After the Safety City program is administered, teachers should intermittently review the material. There should also be an opportunity to practice the desired skills. Teachers could supervise and provide feedback to students as they practice crossing a mock street on the playground. Parents could also mentor their children while performing this skill outside of the school setting. Another strategy is to have adult crossing guards review correct practices for crossing the street with children on their way to and from school at regular intervals throughout the year. Mock scenarios for calling 911 or avoiding strangers could also be reviewed intermittently with students to help reinforce these skills. When coupled with appropriate feedback, repeated practice of the desired skills will also help develop the self-confidence necessary to change a child's behavior.8
Motivating young children to consistently practice appropriate behaviors is a difficult task. Desired behaviors are usually practiced because of a perception of personal gain or decreased risk to an adverse outcome.9 During Safety City lectures, children are told a car could strike them if they do not cross the street correctly. The stranger avoidance curriculum stresses that some strangers may try to hurt or do "bad things" to children. While it is important to explain these concepts, some children in this age group may not fully appreciate the consequences of incorrect behaviors if their development favors more concrete thinking. Instead, motivation may come from explaining that by correctly performing the desired behaviors, they may gain some independence. Parents could tell their children that when they know how to cross the street and avoid strangers, they will be permitted to go to the playground or a friend's house without waiting for a caregiver to accompany them. Such independence would undoubtedly occur when a child is older than the participants of our study.
Injury prevention strategies can be either active or passive. Passive strategies do not rely on human action and therefore tend to be more effective. Education is an active strategy. A child can be taught to cross the street in a manner that minimizes the risk of pedestrian injury. If the child does not choose to exercise these practices, the effectiveness of the intervention is lost. When possible, education should be coupled with passive strategies such as product redesign and environmental modification. Such strategies might include moving bus stop locations so that children do not have to cross the street after leaving the vehicle, constructing sidewalks, and placing barriers such as chains or fences to physically separate children from traffic.10 Organizations that conduct programs like Safety City could work with schools and municipalities to identify potentially hazardous environments and develop strategies to reduce injuries.
The American Red Cross of Greater Columbus has implemented several changes to the Safety City program since the completion of the evaluation. The curriculum has been reviewed to insure the use of age-appropriate language. In addition, an information packet is provided to all schools prior to the administration of Safety City. The packet contains curriculum outlines so educators can become familiar with the material presented during Safety City classes. Also included is a list of activities that teachers can employ to reinforce Safety City's learning objectives. The American Red Cross of Greater Columbus also provides participating schools with a letter that can be reproduced and distributed to parents or other caregivers. The letter outlines strategies that parents can use to encourage proper safety behaviors. The coloring book each child receives as part of the program now contains a cover letter to parents. The letter describes strategies that may guide children toward the practice of appropriate safety behaviors.
This evaluation demonstrated that Safety City did not achieve the desired changes in safety knowledge among participants. Therefore, there is no evidence that the program is an effective way to teach injury prevention to this age group. Based on the evaluation, the American Red Cross of Greater Columbus has made several changes to Safety City. Subsequent evaluation will be necessary to determine if these changes have been effective.
Resources to teach injury prevention are relatively scarce and should be spent on programs with documented success. The support of developing programs is also essential. Injury prevention curricula should be developed with the aid of evidence-based studies and ongoing evaluation. This report emphasizes the importance of building an evaluation component into educational programs.
Editor's Note: Another safety education intervention down the drain. Keep on swimming.—Catherine D. DeAngelis, MD
Accepted for publication August 16, 1999.
We thank the American Red Cross of Greater Columbus, Columbus, Ohio, for their help in completing this study. We also thank Keith Yeates, PhD, for his review of the evaluation tool.
Corresponding author: Joseph Luria, MD, Division of Emergency Medicine, Children's Hospital Medical Center, 3333 Burnet Ave, OSB-4, Cincinnati, OH 45229-3039.
Luria JW, Smith GA, Chapman JI. An Evaluation of a Safety Education Program for Kindergarten and Elementary School Children. Arch Pediatr Adolesc Med. 2000;154(3):227-231. doi:10.1001/archpedi.154.3.227