Copyright 2007 American Medical Association. All Rights Reserved. Applicable FARS/DFARS Restrictions Apply to Government Use.2007
To evaluate reliability and validity of The Injury Prevention Project Safety Survey (TIPP-SS) of the American Academy of Pediatrics in measuring injury prevention practices.
Reliability was measured using the test-retest method. Validity is measured comparing results of parent-completed TIPP-SSs and a home safety audit conducted in the participants' homes at the time of survey.
Two Chicago Public School Early Childhood Education program sites.
Eighty-eight families (44 English speaking and 44 Spanish speaking) with a child aged 3 to 5 years attending a site A or B Chicago Public School Early Childhood Education program. Participants were split evenly between sites.
For the reliability study, primary caregivers completed TIPP-SS twice, 14 to 24 days apart. For the validity study, primary caregivers completed TIPP-SS during a home visit in which a research assistant completed a home safety audit. A total of 44 home visits were completed, 22 in Spanish and 22 in English.
Test-retest reliability and validity of TIPP-SS. Results are compared for agreement of individual items and the whole survey.
The Injury Prevention Project Safety Survey is reliable but not valid. The Injury Prevention Project Safety Survey is a good measure of the concept of injury prevention knowledge and practice (Cronbach α = 0.869). External reliability was statistically supported as well (P = .40). The Injury Prevention Project Safety Survey is not a valid measure of injury prevention behaviors. Validity was poor for items based on observed data (Pearson r = 0.287, in English; Pearson r = − 0.449, in Spanish). Validity was much stronger for parent report data (Pearson r = 0.689, in English; Pearson r = 1.00, in Spanish).
Results suggest that TIPP-SS measures knowledge and attitudes rather than behavior. Parents are often aware of the desired behavior or condition and report those instead of actual conditions or behaviors. This suggests that the quest to develop a valid home-based, injury prevention, behavior assessment tool should continue and be done in a way that carefully addresses potential instruments' validity and reliability.
Unintentional injury is the leading cause of death among children and youth between the ages of 1 and 19 years in the United States.1 Nonfatal injuries are also a major concern. In 2005, there were 9.3 million nonfatal, unintentional injuries to US children and youth aged 1 to 19 years.2 Therefore, injury prevention practices are an important aspect of child health in the United States.
To assess injury prevention practices, a validated and reliability-tested instrument measuring injury prevention practices is needed. A review of available instruments identified The Injury Prevention Program Safety Survey (TIPP-SS) of the American Academy of Pediatrics (AAP) as a candidate instrument. The Injury Prevention Project is an educational program for parents of newborns and children aged through 12 years. Its goal is to help prevent common injuries from the leading causes of unintentional childhood injury (eg, motor vehicle crashes, drowning, firearms, falls, bicycle crashes, pedestrian hazards, burns, poisoning, and choking). Initially developed for young children in 1983, TIPP-SS was expanded in 1988 to include children aged from 5 to 12 years. The Injury Prevention Program Safety Survey was revised again in 1994 and 2001 to better reflect leading causes of childhood injury. The Injury Prevention Program Safety Survey has been used widely since the mid-1980s, is sanctioned and supported by the AAP, is available in Spanish, and offers simple language and a short format. However, there are no published data regarding its validity or reliability. We report an evaluation of the reliability and validity of TIPP-SS in measuring injury prevention practices in a sample of 88 urban families (44 English speaking and 44 Spanish speaking) with a child aged 3 to 5 years.
The study, conducted between January and June of 2003, involved parents with at least 1 child between the ages of 3 and 5 years attending 1 of 2 Chicago Public School Early Childhood Education program sites. Site A had a racially and ethnically diverse student body: 38% white, 15% black, 43% Hispanic, and 4% other. Most parents (> 90%) at site A were native born. Nearly half (44%) of the site A student body comes from low-income households. Site B is less diverse. More than 99% of students enrolled are Mexican American and 95% are from low-income households. Most parents at site B were Spanish speaking.
At each site, 2 separate samples of 22 parents were recruited by a bilingual research assistant (RA) for a total of 44 parent subjects per site (a total of 88 parent subjects). The Figure depicts the study's group structure. These sample sizes allow for detection of correlations greater than 50% at 80% power and reporting by parents' nativity status.3
Study group structure.
To recruit for the reliability study, all parents of children attending the Early Childhood Education program at each site were invited to participate in the study when they came to drop off or pick up their children. Recruitment lasted approximately 3 weeks. A trained RA approached families and, using a script, asked them to participate in the study. At that time, parents were presented with information about the study and an informed consent form. If parents agreed to participate, they signed the informed consent and completed 2 surveys (TIPP-SS for parents of children aged 1-4 years and another survey on community walkability, used to de-emphasize attention on the home injury and safety focus of the study). Parents were told that they would be contacted again within the month to complete another survey. The second (time 2) surveys were completed at drop-off or pick-up time within 14 to 24 days after completion of the first (time 1) survey. A total of 35 parents (80%) completed the survey at both time 1 and time 2.
For the validity study, parents of children attending the Early Childhood Education program at each site who had not participated in the reliability study were invited to participate in the home-visit study when they came to drop off or pick up their children and via a presentation and information table at a school event. Recruitment again lasted approximately 3 weeks. Parents indicating an interest in participating received a more detailed explanation of the study and were scheduled for a home-visit appointment. At the home visit, parents were provided with additional information and asked to sign a consent form and complete 2 surveys. A total of 44 home visits were conducted, 22 with Spanish speakers and 22 with English speakers.
None of the parents participated in both stages of the study. Only 1 parent from each household was eligible to participate. In selecting the participating parent, we chose the parent who was the child's primary caregiver. The project was approved by the institutional review board at the Children's Memorial Research Center (project No. 2003-12053).
The study used (1) the AAP's TIPP-SS for children aged 1 to 4 years, parts 1 and 2, and (2) the Home Safety Audit. The safety audit form was based on one used for the Children's Safety Project, an AAP project funded by the Centers for Disease Control and Prevention (S. LeBailly, PhD, unpublished data, 1990).
The AAP offers instructions for practitioners for use of the survey (http://www.aap.org). For this study, TIPP-SS was modified to include parents' country of origin and age, annual household income, primary language spoken at home, parents' willingness to participate in further research, relationship of respondent to the child, and the ages of all persons living in the household. The modified survey was transferred to a scannable format using TeleForm software (Cardiff, Vista, California). The modified version of TIPP-SS maintains all of the questions in the original The Injury Prevention Project surveys, including questions about child supervision, presence/use of safety equipment, and safety practices.
The Home Safety Audit was modified for this study by elimination of items that had no direct match to items in TIPP-SS (presence of radiators; presence of bows, arrows, darts, and trajectory toys; and child use of a pacifier). The RA received 2 hours of training in survey administration. She was instructed to observe parents completing surveys and to offer assistance by reading the survey aloud if the parent appeared to struggle with reading (indicated by extreme slowness in completing the survey, look of puzzlement, and/or reading aloud or mouthing words).
The safety audit checklist included storage of chemicals and poisons; presence and accessibility of strangulation, suffocation, and choking hazards; presence of smoking/smoking materials; presence of electrical cords and appliances near water; uncovered electrical outlets; burn and fall hazards; presence of safety equipment, eg, smoke detector or fire extinguisher; and safety practices, eg, presence of a fire plan, parental rules for outdoor play and vehicle travel, and presence/storage of firearms.
The RA received 4 hours of training on use of the audit instrument. Training was provided by a researcher who used the audit in the Children's Safety Research Project. Training included review of the audit instrument, use of tools for measurement used with the audit (water thermometer, choke tube, and ruler for measuring mattress spacing), and completion of 2 practice audits (not included in the analysis). The audit data were obtained using 2 methods: (1) data that were observed and recorded by the RA during the home visit and (2) data that were obtained from parent reports in answers to questions posed by the RA during the home visit.
Completed surveys and audits were scanned using TeleForm software, version 9.0 (Cardiff), and downloaded into SPSS, version 12.0 (SPSS Inc, Chicago, Illinois), data files. Data were checked against hard copy instruments and cleaned for accuracy. Variables were recoded for consistency in direction (safest behaviors and conditions received higher values) and continuous variables were recoded into categories when necessary for analysis. Two separate files were created: 1 for the reliability analysis (matched times 1 and 2 survey results) and 1 for the validity analysis (matched survey and audit results). Reliability and validity analyses were done separately. In addition, we analyzed results by the language in which TIPP-SS was taken (English or Spanish). Half of the study subjects completed TIPP-SS in English (n = 44) and half completed the survey in Spanish (n = 44).
For the whole survey, reliability analysis data were restructured and composite values were computed by summing all individual item scores. Composite values were compared at survey times 1 and 2 to determine internal reliability. Internal reliability measures how well the individual items in a survey combine to provide a single measure reflecting a latent concept (or constructed concept, in this case), ie, the concept of home-based injury prevention knowledge and practice. We used the Cronbach α coefficient of reliability, a function of the number of test items (TIPP-SS questions) and the average intercorrelation among the questions, to determine how well TIPP-SS items fit together to represent the concept of home-based injury prevention knowledge and practice. The higher the interitem correlations among questions, the better the evidence that the questions are measuring the same construct, in this case, injury prevention knowledge and practice.4 Paired sample t tests are used to determine external reliability with language in which the survey is completed as a covariate.
For individual item reliability analysis, data were recoded for direction (eg, highest score was safest). We then compared individual item responses at survey times 1 and 2 using percentage agreement (percentage of same answers at time 1 and time 2) for each item. Although the κ statistic is often used for test-retest comparisons, it was not suitable for this analysis because, for most items, there was very little variation in the categorical distribution of the data across the repeated surveys.
For the validity analysis, data were restructured as follows: each item in the safety audit recorded a separate value for each room in the household and, as a result, there were multiple values for each item, eg, if the multiple values were totaled for each item, the resulting score was dependent on the number of rooms in the household (households with more rooms could receive higher [safer] scores). To adjust for the number of rooms in a household, we created a total item score by summing all room values for each item and recoded the summed item scores into a dichotomous variable (home safety score) using the following rules: if all conditions met safety recommendations in all rooms, the variable received a 1. If all conditions did not meet safety recommendations in all rooms, the variable received a 0. We used the dichotomous summary variables for the validity analysis.
For validity analysis, individual item comparisons for agreement were done using the percentage agreement (percentage of same answers for matched audit and survey items). We used the Franzblau classification system for interpretation of Pearson r: 0.00 to 0.20, no correlation; 0.21 to 0.40, low degree of correlation; 0.41 to 0.60, moderate degree of correlation; 0.61 to 0.80, a marked degree of correlation; and 0.81 to 1.00, high correlation.5 For individual items, reliability and validity were judged to be acceptable if agreement was greater than 75%.
Our results indicate that TIPP-SS is a good measure of the concept of injury prevention knowledge and practice (Cronbach α = 0.869). If individual items with less than 75% agreement (Table 1) were dropped, internal reliability improved for those taking TIPP-SS in English (Cronbach α = 0.952) as well as for those taking TIPP-SS in Spanish (Cronbach α = −0.830).
External reliability (the extent to which a measure varies from one use to another) for TIPP-SS was statistically supported, as well. We determined the overall external reliability for TIPP-SS using the paired sample t test. Here, we found the means at survey times 1 and 2 were not significantly different (P = .40). When we added language in which TIPP-SS was completed (English or Spanish) as a covariate using the repeated measures analysis of variance procedure in SPSS, we again found no statistically significant difference at survey times 1 and 2 (P = .48). In addition, no significant statistical differences were found when items with unacceptable percentage agreement were excluded (P = .33, Spanish; P = .43, English).
Percentage agreement (percentage of cases that answered the same way at times 1 and 2) was calculated for each of the 41 individual survey items. Results are reported in Table 1. Note that reliability of individual items varied by the language (English or Spanish) in which TIPP-SS was taken.
The variable measuring whether any of the children in the respondent's home had ever had an injury requiring a visit to a physician or hospital was deleted from the analyses. This was done because we recoded the number of physician or hospital visits for injuries variable to include a zero option, making the “Have any of your children ever had an injury requiring a visit to the doctor or hospital?” variable redundant.
For the validity analysis, all individual survey and audit items that were directly comparable (17 of 40) were tested for agreement using percentage agreement (percentage of answers that were the same for matched audit and survey items). The survey-audit item agreement results are presented by the method in which the audit data were obtained, ie, observed or reported.
Table 2 presents data showing that survey-audit agreement was generally poor for audit items based on observed data. For those taking the survey in Spanish, agreement was less than 75% for all but 3 of 8 observed items (window guards in place, medicines without safety caps, and medicine not in locked cabinets). For those taking the survey in English, all but 1 of 8 observed items (medicines without safety caps) had less than 75% agreement (Table 2).
In contrast, Table 3 shows that survey-audit item agreement for audit items based on the parent report data was generally much stronger. For those taking the survey in Spanish, only 3 of 9 items (working fire extinguisher, restraining child in car, and child plays in driveway) had agreement of less than 75% between the survey and observed audit. For those taking the survey in English, only 2 of 9 items (working fire extinguisher and restraining child in car) had agreement of less than 75% between the survey and observed audit.
Agreement results between TIPP-SS and the home safety audit varied widely by the type of comparisons made. Table 4 presents the results of several comparisons. When the total scores for the 17 items that are directly comparable between the survey and audit were compared, results indicated a moderate degree of association.
If the comparisons were further refined based on the method of obtaining audit data, the results showed statistically significant differences. For audit items for which data were obtained through observation, there was no agreement between matched TIPP-SS and audit items. For audit items for which data were obtained through the parent report, our results indicate moderate, but statistically insignificant, agreement between matched TIPP-SS and audit items for subjects taking TIPP-SS in English and strong and statistically significant agreement for subjects taking TIPP-SS in Spanish.
We often think that knowledge leads to positive behavioral practices. Given this, one would expect that knowledge of injury prevention would be correlated with safety prevention practice. The results of this study indicate the assumptions linking knowledge and behavior are not borne out with regard to the safety practices measured by TIPP-SS. We found that TIPP-SS is reliable but not valid. This is similar to findings from other studies looking at linkages between health knowledge and behavior.6
Our findings suggest that TIPP-SS measures knowledge and attitudes rather than behavior, that is, parents are often aware of the desired behavior or condition and report those instead of actual conditions or behaviors. While knowledge and attitudes toward injury-risk prevention can be important to injury prevention strategies, behavior is more important. These findings suggest that the quest to develop a valid tool to assess home-based injury prevention behaviors should continue and should be done in a way that carefully addresses validity and reliability of potential instruments.
Additionally, these findings suggest cautions for, and rethinking of, the use of TIPP-SS results in patient counseling. While research must be done on the effectiveness of and techniques for counseling approaches using TIPP-SS results, our results imply that (1) narrowing counseling to items with incorrect answers may miss important opportunities for addressing behaviors and conditions that do not meet safety prevention standards and (2) further comment on items with correct answers in counseling may offer the opportunity to impact behavior in areas in which the patient has a knowledge and behavior mismatch. For example, a physician may say, “I am happy to see that you do X because it is important because of Y. Just to remind you, the easiest/best way to do this is Z.”
We chose a home-based audit to validate TIPP-SS, because home-based injuries are a major cause of injury fatality for children younger than 15 years of age.7 The use of a survey to measure home-based injury-related behaviors can be problematic because TIPP-SS, like all surveys unless proven otherwise, is a measure of opinion rather than fact. In particular, the validity of self-reported home-based safety practices has been widely debated, yet aided by relatively few empirical data. Recent studies assessing the validity of reported safety behaviors show mixed results. A study by Hatfield et al8 reports generally high sensitivities for 16 safety practices when comparing face-to-face interview results with home inspections. Another study by Yorkston et al9 reports findings of poor validity when comparing a self-report survey with direct in-home observations. The research to date (of which we are aware) that assesses the validity of self-report safety practices concludes that relying on self-reported safety behaviors is problematic. Self-report measures are especially questionable in cases where subjects report the presence of desired safety behaviors or products.10 Results for cases in which subjects report not having a safety product or not practicing a safety behavior are more accurate.11 Despite the lack of validated self-report instruments, studies of pediatric injury continue to use self-report data because of the technical and logistical limitations of home visits.
Furthermore, TIPP-SS and audit instruments are not perfectly comparable. For example, TIPP-SS asks “Do you keep household products, medicines (including acetaminophen and iron), and sharp objects out of the reach of your children and in locked cabinets?” Whereas the audit includes variables: (1)“Cleaning supplies not in locked cabinet over 5 ft?” (2)“Medicines without safety caps or in locked cabinets over 5 ft?” and (3)“Sharp objects not in locked cabinets over 5 ft?” The audit was structured in a way that combined some of the single items found in TIPP-SS.
Additionally, the way in which TIPP-SS was implemented is not consistent with its intended use. The Injury Prevention Project Safety Survey is part of a system of counseling and information gathering. This analysis looks at TIPP-SS as a stand-alone instrument. This report did not explore the ways in which TIPP-SS is used as a counseling tool or the variety of other possible ways in which TIPP-SS is used in private practices. Although beyond the scope of this study, the implications of this study's findings may be better understood if we knew more about the ways in which TIPP-SS is implemented in the office setting, especially within the low-income, minority population on which this study was focused.
The findings reported here are based on results from a study of urban, low-income, minority households. Further testing of other families is needed to assess applicability of the following results to other populations: (1) reliable questions within TIPP-SS for children aged 0 to 4 years can be used to assess parental knowledge and attitudes regarding home-based safety practices, (2) for TIPP-SS topics with unreliable questions, replacement questions with acceptable reliability results must be identified and TIPP-SS should be subsequently modified to include them, (3) TIPP-SS, for children aged 0 to 4 years, even if modified by omitting unreliable questions, cannot be used as a proxy for actual home safety practices, and (4) a valid proxy for home safety practices is needed.
Correspondence: Maryann Mason, PhD, Child Health Research, Children's Memorial Hospital Research Center, 2300 Children's Plaza, Box 157, Chicago, IL 60614 (firstname.lastname@example.org).
Accepted for Publication: January 25, 2007.
Author Contributions: Dr Mason had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Mason, Christoffel, and Sinacore. Acquisition of data: Mason. Analysis and interpretation of data: Mason, Christoffel, and Sinacore. Drafting of the manuscript: Mason, Christoffel, and Sinacore. Critical revision of the manuscript for important intellectual content: Mason and Christoffel. Statistical analysis: Sinacore. Obtained funding: Mason. Administrative, technical, and material support: Mason and Christoffel. Study supervision: Christoffel.
Financial Disclosure: None reported.
Funding/Support: Funding was provided by a seed grant from Children's Memorial Research Center.
Mason M, Kaufer Christoffel K, Sinacore J. Reliability and Validity of The Injury Prevention Project Home Safety Survey. Arch Pediatr Adolesc Med. 2007;161(8):759-765. doi:10.1001/archpedi.161.8.759