Is Human Immunodeficiency Virus Sexual Risk Prevention Intervention Effective?

THE DESIGN used in this study¹–quasi-experimental–is unfamiliar to clinical readers. There are few easily available worksheets or prescriptive methods for reviewing such an article. We used 2 excellent resources to generate questions used in the evaluation of this article,^2,3 modified by 2 of us (M.H. and W.B.), and reordered here to match the sequence of the study. The graphic (Figure 1) helps to arrange the methodological issues we raise throughout the article. In the Web-based version we provide links, depicted in the graphic, to provide further detail about the various methodological considerations and threats to validity.

Yes. There have been few long-term evaluation studies of educational interventions to reduce the high rates of pregnancy and sexually transmitted diseases (STDs) (including human immunodeficiency virus) in adolescents in the United States. These public health problems are clearly linked to high-risk sexual behaviors in adolescents, specifically in those adolescents living in urban environments. In this study Siegel et al¹ address this problem by evaluating a sexual risk prevention intervention for middle school (MS) and high school (HS) students in an urban population.

The investigators used a quasi-experimental design with 3 school-based sexual risk prevention intervention groups and 1 school-based health education control group.

Yes. The study is based on the hypothesis that the rates of teen pregnancy and STDs are determined by the high-risk sexual behaviors of adolescents. Using findings from prior work in this area, the authors make a reasonable assumption that by providing sexual risk prevention education to youth, we can reduce the prevalence of high-risk behaviors and, thus, reduce the incidence of pregnancy and STDs.

Yes. Teen pregnancy and STDs are more prevalent in adolescents living in urban environments. To assess the effect of a sexual risk prevention education program for this target population, the investigators sampled a diverse group of MS and HS students living in a large urban northeastern US city. The authors clearly describe the distribution of students for race, ethnicity, and socioeconomic status (based on available census tract data). This large study sample seems to be well-representative of a US urban school-based population.

No. To our knowledge, this is the first study to assess the effect of this specific sexual risk prevention education curriculum on long-term knowledge, attitudes, and practices. However, the intervention in the study was implemented in 10 schools within geographic confines, with students of similar ages and backgrounds.

Yes. Demographic characteristics were obtained from the adolescents participating in the study. These characteristics included age, sex, grade, ethnicity, history of sexual intercourse experience, and proxies for socioeconomic status.

Yes. All independent and dependent variables were assessed on the participants prior to receiving the intervention or the control group's educational curriculum. These variables included baseline scores for knowledge, sex self-efficacy, safe behavior intention, sexual intercourse history, and sexual risk behaviors.

The best statistical measure available to answer this question is R² —the percentage of variability explained by the independent (purportedly "explanatory") variables. The models in this study had a wide range of values for R² (0.12-0.54). This variability indicates that there are other factors related to the outcome that were not included in the analysis.

Some factors at the individual respondent level that might contribute to the outcome include the adolescent's religion,⁴ his or her strength of support groups (ie, number and quality of friendships, as well as the quality of the relationships with parents and other family members),^5,6 the adolescent's perception of risky behaviors of his or her friends,^7,8 and the influence of the media's portrayal of sexuality.⁹ Another factor for adolescent girls could include a participant's intention to become pregnant. Although the investigators included this factor in the sex risk behaviors score, it may be a stronger factor than the others assessed in the construct.

Factors related to the outcome at the school level include the rate of transience in the school, the school's atmosphere (this was controlled for in the study by including a class climate variable), and the rates of pregnancy and STDs in the schools. Understandably, many of these factors are difficult to assess in a self-administered survey, and the authors did an excellent job at attempting to cover the most appropriate variables.

Yes. The investigators measured the subjects' scores prior to the intervention, and each individual's pretest score was included as a covariate in the analysis. It is understandable that there were probably no data for trends in high-risk sexual behaviors in the study sample prior to the intervention. However, it would have been helpful to see data on any trends in the rates of pregnancy or STDs for students in the schools included in the study. We are, therefore, unable to assess the effect of any maturational changes that may influence the outcome of interest. For example, if during the study period high-risk sexual behaviors were trending upward and the authors find a decrease in self-reported behaviors in the intervention group (compared with control subjects), the improvement would have even greater significance.

No. The study was quasi-experimental. As noted by the authors, randomization would not have been possible for conducting a large school-based study without interfering with routine school practice. The assignment of classes to intervention or control group was based on feasibility and availability of peer educators. To strengthen the external validity of the findings, however, all health classes in each of the MSs and HSs were assigned to both the experimental and control groups. In addition, although the authors were unable to randomize students, they appropriately assessed baseline demographics and pretest scores and controlled for differences in their analysis.

Yes.

Yes. Differences in group characteristics were adjusted for in the analysis. Demographics (age, sex, socioeconomic status, and ethnicity), the proportion of male and female students who reported sexual experience, and all other study variables (knowledge, self-efficacy scores, and others) were analyzed after stratifying by MS and HS groups, as well as by sex.

Yes. The survey was pilot tested on 450 students prior to the study.

The intervention needed to be implemented without too much disruption of the routine MS and HS curricula. The investigators went to great efforts to ensure consistent and reliable implementation at all sites participating in the study. There could have been variability in the teaching styles and effectiveness of the educators in the participating schools. However, the Rochester AIDS Prevention Project for Youth (RAPP) adult health educators were described as highly trained and the RAPP peer educators received 50 hours of intensive training. Both RAPP adult and peer educators taught classes as pairs. While attempts were made to maintain consistency and reliability, it is not clear if the educators' teaching ability was assessed prior to the study implementation. In addition, an observer could have ensured consistency in the content received in the intervention and control groups across participating sites.

Because the study occurred at multiple sites, and the long-term evaluation lasted from 14 to 80 weeks after the intervention, students may have moved to different schools. As such, it could have been possible for students to receive education from both the intervention and control groups. The investigators minimized this problem by assigning unique identifiers for both the students and the schools. Therefore, if a student did change schools, the investigators would be aware of it. In addition, at no time did intervention and control classes occur in the same school during a given semester.

No. It is not clear in the article if a process evaluation was performed. We are given information about attrition rates in the MSs and HSs, but we do not know much about the day-to-day activities that occurred during the study. It would have been helpful to know the average number of sessions each student attended. Perhaps this factor could not only be used to monitor the study, but also it could have an effect on the outcome evaluation. We are also not told about the average number of students in each class. This factor may be included in the climate score variable, but it would seem important to determine class size as a process variable.

We were unable to determine this from the information provided in the article.

Yes. The authors clearly specified the expected outcomes for the study. The outcomes included knowledge, sex self-efficacy, behavior intention, and behaviors including the onset of sexual intercourse experience and engagement in risky sexual behaviors.

Yes. Due to a scarcity of reliable and valid measures in the existing literature, the authors designed scales that were pretested in a pilot study of 450 students. The scales were based on prior research, previously validated instruments (Youth Risk Behavior Survey), consultation with experts in adolescent health, and the Theory of Reasoned Action.¹⁰ Reliabilities were calculated for the MS and HS students separately using the Cronbach α to assess internal consistency and test-retest reliability. Both statistical measures revealed high reliability.

Yes.

Yes. The authors used repeated-measures analyses of variance to assess differences between intervention and control groups stratified by sex and type of school (MS or HS). In addition to being in the intervention or control group, the investigators wished to investigate ethnicity and history of sexual intercourse experience as major factors associated with the outcomes. As such, each of these factors was entered one at a time controlling for the following covariates: demographics, general life risk score, mean score for length of time since the intervention, the class climate score, and the pretest score for each individual. Considering that the quasi-experimental design is not frequently described in the pediatric literature, it would have been helpful if the authors included issues related to interpreting the particular statistical tests used in their analysis.

Yes. As mentioned previously, the authors conducted a pretest with all participants to assess baseline knowledge and behavior. Since the same scales were used in the long-term follow-up survey, the measures can appropriately be compared to assess differences from the intervention.

The authors describe a positive long-term effect of the RAPP intervention (compared with the control group) in knowledge, self-efficacy, behavior intention, and self-reported behaviors. Although statistical significance was not achieved in all of these areas, scores in the intervention groups were in the "safer" direction. They found that the participant's pretest score and report of sexual history to be major predictors of posttest scores. The authors do suggest that the positive effects from the intervention could have been due to the high attrition rates (most notable in the 12th graders). Those participants who were lost to follow-up may have had no change, or possibly a worsening in their outcome scores at long-term follow-up. Appropriately, the authors examined the pretest data from these students and found them to be higher sexual risk students than students who remained in the study.

Yes.

As mentioned previously, a major concern we have relates to the large number of dropouts in this study. Attrition rates were 45% for HS students and 28% for MS students. As noted, many of these students at pretest were in the higher-risk sexual behavior group, and losing them may have made the positive results of the study look more impressive. Clearly, those students who were lost to follow-up represent a truly different group than those that were maintained in the study. In fact these dropouts may contribute disproportionately more to the problems of teen pregnancy and STDs than those students who remain in school. A future challenge, therefore, would be the development of sexual risk prevention interventions to specifically target these most at-risk youth.

The major conclusion of the study is that when implemented and evaluated in a large MS and HS sample of urban youth, the RAPP intervention was most helpful in improving safer sex knowledge, self-efficacy, behavior intention, and self-reported behaviors in students who were not engaging in high-risk behaviors prior to the intervention. The authors illustrate this point in their final statement of the article, and we look forward to seeing the RAPP intervention implemented and evaluated on a younger, sexually inexperienced sample of students.

Our own conclusion from this article is that the investigators used a thoroughly planned design, a difficult but successful implementation, and a comprehensive and intensive evaluation. Although the generalizability of quasi-experimental designs is always in question, in this case the statistical analysis used is appropriate, the interpretation of results is clear and understandable, and the overall findings are compelling and believable.

Behi R, Nolan M. Quasi-experimental research designs. Br J Nurs. 1996;5:1079-1081.

Cook TD, Campbell DT. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston, Mass: Houghton Mifflin; 1979.

Freemantle N, Wood J, Crawford F. Evidence into practice, experimentation and quasi-experimentation: are the methods up to the task? J Epidemiol Community Health. 1998;52:75-81.

Morgan GA, Gliner JA, Harmon RJ. Quasi-experimental designs. J Am Acad Child Adolesc Psychiatry. 2000;39:794-796.

We acknowledge the other faculty and fellows in the Department of Pediatrics who participated in the journal club session and those who greatly contributed to our learning process: Mark Adler, MD; Anne Duggan, ScD; Samer El-Kamary, MD, MPH; Elizabeth Jacobs, MD; Michael Johnston, MD; David Lang, MD; Iris Mabry, MD; Karen Schneider, MD; Janet Serwint, MD; Joyce Wong, MD. Also, thanks to Bonnie Cosner, BS, for transcribing the session.

Correspondence may be addressed to Dr Lehmann.