Effect of a Health Care Professional Communication Training Intervention on Adolescent Human Papillomavirus Vaccination: A Cluster Randomized Clinical Trial | Adolescent Medicine | JAMA Pediatrics | JAMA Network
[Skip to Navigation]
Figure.  CONSORT Diagram for the Study
CONSORT Diagram for the Study

Clinics were randomized by (1) count of patients who were aged 9 to 17 years, (2) percentage of patients who were eligible for the Vaccines for Children program, (3) percentage of health care professionals who strongly recommend human papillomavirus vaccine to girls aged 11 to 12 years, and (4) human papillomavirus vaccine initiation rates among patients aged 11 to 12 years. CONSORT indicates Consolidated Standards of Reporting Trials.

Table 1.  Characteristics of Participating Practices and Patients in the Trial Assessed for Human Papillomavirus (HPV) Vaccination
Characteristics of Participating Practices and Patients in the Trial Assessed for Human Papillomavirus (HPV) Vaccination
Table 2.  HVP Vaccine Series Initiation and Completion, All Ages and Sexes Combined: Control vs Intervention Difference-in-Differences Comparison of Baseline to Postimplementation Periodsa
HVP Vaccine Series Initiation and Completion, All Ages and Sexes Combined: Control vs Intervention Difference-in-Differences Comparison of Baseline to Postimplementation Periodsa
Table 3.  HPV Vaccine Series Initiation by Clinical Characteristics: Control/Intervention Difference-in-Differences Comparison of Baseline to Postimplementation Periodsa
HPV Vaccine Series Initiation by Clinical Characteristics: Control/Intervention Difference-in-Differences Comparison of Baseline to Postimplementation Periodsa
Table 4.  HPV Vaccine Series Initiation by Patient Factors: Control vs Intervention Difference-in-Differences Comparison of Baseline to Postimplementation Periodsa
HPV Vaccine Series Initiation by Patient Factors: Control vs Intervention Difference-in-Differences Comparison of Baseline to Postimplementation Periodsa
1.
Viens  LJ, Henley  SJ, Watson  M,  et al.  Human papillomavirus–associated cancers: United States, 2008-2012.  MMWR Morb Mortal Wkly Rep. 2016;65(26):661-666.PubMedGoogle ScholarCrossref
2.
Markowitz  LE, Dunne  EF, Saraiya  M, Lawson  HW, Chesson  H, Unger  ER; Centers for Disease Control and Prevention (CDC); Advisory Committee on Immunization Practices (ACIP).  Quadrivalent Human Papillomavirus Vaccine: Recommendations of the Advisory Committee on Immunization Practices (ACIP).  MMWR Recomm Rep. 2007;56(RR-2):1-24.PubMedGoogle Scholar
3.
US Food and Drug Administration. Approved products: Gardasil. https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM094042. Last updated January 9, 2018. Accessed December 1, 2017.
4.
US Food and Drug Administration. Gardasil package insert. http://www.fda.gov/downloads/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM111263.pdf. Revised April 2015. Accessed November 28, 2017.
5.
Centers for Disease Control and Prevention (CDC).  FDA licensure of quadrivalent human papillomavirus vaccine (HPV4, Gardasil) for use in males and guidance from the Advisory Committee on Immunization Practices (ACIP).  MMWR Morb Mortal Wkly Rep. 2010;59(20):630-632.PubMedGoogle Scholar
6.
Walker  TY, Elam-Evans  LD, Singleton  JA,  et al.  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2016.  MMWR Morb Mortal Wkly Rep. 2017;66(33):874-882.PubMedGoogle ScholarCrossref
7.
Keim-Malpass  J, Mitchell  EM, DeGuzman  PB, Stoler  MH, Kennedy  C.  Legislative activity related to the human papillomavirus (HPV) vaccine in the United States (2006-2015): a need for evidence-based policy.  Risk Manag Healthc Policy. 2017;10:29-32.PubMedGoogle ScholarCrossref
8.
President’s Cancer Panel.  Accelerating HPV Vaccine Uptake: Urgency for Action to Prevent Cancer: A Report to the President of the United States from the President’s Cancer Panel. Bethesda, MD: National Cancer Institute; 2014.
9.
Lau  M, Lin  H, Flores  G.  Factors associated with human papillomavirus vaccine–series initiation and healthcare provider recommendation in US adolescent females: 2007 National Survey of Children’s Health.  Vaccine. 2012;30(20):3112-3118.PubMedGoogle ScholarCrossref
10.
Rahman  M, Laz  TH, McGrath  CJ, Berenson  AB.  Provider recommendation mediates the relationship between parental human papillomavirus (HPV) vaccine awareness and HPV vaccine initiation and completion among 13- to 17-year-old U.S. adolescent children.  Clin Pediatr (Phila). 2015;54(4):371-375.PubMedGoogle ScholarCrossref
11.
Vadaparampil  ST, Malo  TL, Sutton  SK,  et al.  Missing the target for routine human papillomavirus vaccination: consistent and strong physician recommendations are lacking for 11- to 12-year-old males.  Cancer Epidemiol Biomarkers Prev. 2016;25(10):1435-1446.PubMedGoogle ScholarCrossref
12.
Vadaparampil  ST, Malo  TL, Kahn  JA,  et al.  Physicians’ human papillomavirus vaccine recommendations, 2009 and 2011.  Am J Prev Med. 2014;46(1):80-84.PubMedGoogle ScholarCrossref
13.
Stokley  S, Jeyarajah  J, Yankey  D,  et al; Immunization Services Division, National Center for Immunization and Respiratory Diseases, CDC; Centers for Disease Control and Prevention (CDC).  Human papillomavirus vaccination coverage among adolescents, 2007-2013, and postlicensure vaccine safety monitoring, 2006-2014: United States.  MMWR Morb Mortal Wkly Rep. 2014;63(29):620-624.PubMedGoogle Scholar
14.
Gilkey  MB, Malo  TL, Shah  PD, Hall  ME, Brewer  NT.  Quality of physician communication about human papillomavirus vaccine: findings from a national survey.  Cancer Epidemiol Biomarkers Prev. 2015;24(11):1673-1679.PubMedGoogle ScholarCrossref
15.
Gilkey  MB, Moss  JL, Coyne-Beasley  T, Hall  ME, Shah  PD, Brewer  NT.  Physician communication about adolescent vaccination: how is human papillomavirus vaccine different?  Prev Med. 2015;77:181-185.PubMedGoogle ScholarCrossref
16.
Weinstein  N, Sandman  PM. The precaution adoption process model and its application. In: DiClemente  R, Crosby  RA, Kegler  MC, eds.  Emerging Theories in Health Promotion Research and Practice. San Francisco, CA: Jossey-Bass; 2002:16-39.
17.
Farmar  AM, Love-Osborne  K, Chichester  K, Breslin  K, Bronkan  K, Hambidge  SJ.  Achieving high adolescent HPV vaccination coverage.  Pediatrics. 2016;138(5):e20152653.PubMedGoogle ScholarCrossref
18.
Dickinson  LM, Beaty  B, Fox  C,  et al.  Pragmatic cluster randomized trials using covariate constrained randomization: a method for practice-based research networks (PBRNs).  J Am Board Fam Med. 2015;28(5):663-672.PubMedGoogle ScholarCrossref
19.
Glynn  RJ, Brookhart  MA, Stedman  M, Avorn  J, Solomon  DH.  Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers.  Med Care. 2007;45(10)(suppl 2):S38-S43.PubMedGoogle ScholarCrossref
20.
Allison  MA, Hurley  LP, Markowitz  L,  et al.  Primary care physicians’ perspectives about HPV vaccine.  Pediatrics. 2016;137(2):e20152488.PubMedGoogle ScholarCrossref
21.
Oster  NV, McPhillips-Tangum  CA, Averhoff  F, Howell  K.  Barriers to adolescent immunization: a survey of family physicians and pediatricians.  J Am Board Fam Pract. 2005;18(1):13-19.PubMedGoogle ScholarCrossref
22.
Raab  GM, Butcher  I.  Balance in cluster randomized trials.  Stat Med. 2001;20(3):351-365.PubMedGoogle ScholarCrossref
23.
Opel  DJ, Heritage  J, Taylor  JA,  et al.  The architecture of provider-parent vaccine discussions at health supervision visits.  Pediatrics. 2013;132(6):1037-1046.PubMedGoogle ScholarCrossref
24.
Kenward  MG, Lesaffre  E, Molenberghs  G.  An application of maximum likelihood and generalized estimating equations to the analysis of ordinal data from a longitudinal study with cases missing at random.  Biometrics. 1994;50(4):945-953.PubMedGoogle ScholarCrossref
25.
Hedeker  D, Gibbons  RD.  MIXREG: a computer program for mixed-effects regression analysis with autocorrelated errors.  Comput Methods Programs Biomed. 1996;49(3):229-252.PubMedGoogle ScholarCrossref
26.
Murray  DM, Varnell  SP, Blitstein  JL.  Design and analysis of group-randomized trials: a review of recent methodological developments.  Am J Public Health. 2004;94(3):423-432.PubMedGoogle ScholarCrossref
27.
Varnell  SP, Murray  DM, Janega  JB, Blitstein  JL.  Design and analysis of group-randomized trials: a review of recent practices.  Am J Public Health. 2004;94(3):393-399.PubMedGoogle ScholarCrossref
28.
Reagan-Steiner  S, Yankey  D, Jeyarajah  J,  et al.  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2015.  MMWR Morb Mortal Wkly Rep. 2016;65(33):850-858.PubMedGoogle ScholarCrossref
29.
Dempsey  A, Cohn  L, Dalton  V, Ruffin  M.  Patient and clinic factors associated with adolescent human papillomavirus vaccine utilization within a university-based health system.  Vaccine. 2010;28(4):989-995.PubMedGoogle ScholarCrossref
30.
Snijders  T, Bosker  RJ.  Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. Thousand Oaks, CA: SAGE Publications; 1999.
31.
Fu  LY, Zook  K, Gingold  J,  et al.  Frequent vaccination missed opportunities at primary care encounters contribute to underimmunization.  J Pediatr. 2015;166(2):412-417.PubMedGoogle ScholarCrossref
32.
Kepka  D, Spigarelli  MG, Warner  EL, Yoneoka  Y, McConnell  N, Balch  A.  Statewide analysis of missed opportunities for human papillomavirus vaccination using vaccine registry data.  Papillomavirus Res. 2016;2:128-132.PubMedGoogle ScholarCrossref
33.
Brewer  NT, Hall  ME, Malo  TL, Gilkey  MB, Quinn  B, Lathren  C.  Announcements versus conversations to improve HPV vaccination coverage: a randomized trial.  Pediatrics. 2017;139(1):e20161764.PubMedGoogle ScholarCrossref
34.
Perkins  RB, Zisblatt  L, Legler  A, Trucks  E, Hanchate  A, Gorin  SS.  Effectiveness of a provider-focused intervention to improve HPV vaccination rates in boys and girls.  Vaccine. 2015;33(9):1223-1229.PubMedGoogle ScholarCrossref
35.
Centers for Disease Control and Prevention (CDC).  National and state vaccination coverage among adolescents aged 13-17 years: United States, 2011.  MMWR Morb Mortal Wkly Rep. 2012;61(34):671-677.PubMedGoogle Scholar
36.
Centers for Disease Control and Prevention (CDC).  National and state vaccination coverage among adolescents aged 13-17 years: United States, 2012.  MMWR Morb Mortal Wkly Rep. 2013;62(34):685-693.PubMedGoogle Scholar
37.
Elam-Evans  LD, Yankey  D, Jeyarajah  J,  et al; Immunization Services Division, National Center for Immunization and Respiratory Diseases; Centers for Disease Control and Prevention (CDC).  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2013.  MMWR Morb Mortal Wkly Rep. 2014;63(29):625-633.PubMedGoogle Scholar
38.
Reagan-Steiner  S, Yankey  D, Jeyarajah  J,  et al.  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2014.  MMWR Morb Mortal Wkly Rep. 2015;64(29):784-792.PubMedGoogle ScholarCrossref
39.
Gowda  C, Schaffer  SE, Dombkowski  KJ, Dempsey  AF.  Understanding attitudes toward adolescent vaccination and the decision-making dynamic among adolescents, parents and providers.  BMC Public Health. 2012;12:509.PubMedGoogle ScholarCrossref
40.
McSherry  LA, Dombrowski  SU, Francis  JJ,  et al; ATHENS Group.  “It’s a can of worms”: understanding primary care practitioners’ behaviours in relation to HPV using the Theoretical Domains Framework.  Implement Sci. 2012;7:73.PubMedGoogle ScholarCrossref
41.
Dempsey  AF, Abraham  LM, Dalton  V, Ruffin  M.  Understanding the reasons why mothers do or do not have their adolescent daughters vaccinated against human papillomavirus.  Ann Epidemiol. 2009;19(8):531-538.PubMedGoogle ScholarCrossref
42.
Dempsey  AF, Freed  GL.  Health care utilization by adolescents on Medicaid: implications for delivering vaccines.  Pediatrics. 2010;125(1):43-49.PubMedGoogle ScholarCrossref
43.
Rand  CM, Shone  LP, Albertin  C, Auinger  P, Klein  JD, Szilagyi  PG.  National health care visit patterns of adolescents: implications for delivery of new adolescent vaccines.  Arch Pediatr Adolesc Med. 2007;161(3):252-259.PubMedGoogle ScholarCrossref
Original Investigation
May 7, 2018

Effect of a Health Care Professional Communication Training Intervention on Adolescent Human Papillomavirus Vaccination: A Cluster Randomized Clinical Trial

Author Affiliations
  • 1Adult and Child Consortium for Outcomes Research and Dissemination Science (ACCORDS), University of Colorado Denver, Aurora
  • 2Division of General Pediatrics, Department of Pediatrics, University of Colorado Denver, Aurora
  • 3Center for Public Health Practice, Colorado School of Public Health, University of Colorado Denver, Aurora
  • 4National Center for Immunization and Respiratory Diseases, Centers for Disease Control and Prevention, Atlanta, Georgia
  • 5Department of Biostatistics, Colorado School of Public Health, University of Colorado Denver, Aurora
  • 6Division of Infectious Diseases, Department of Pediatrics, University of Colorado Denver, Aurora
JAMA Pediatr. 2018;172(5):e180016. doi:10.1001/jamapediatrics.2018.0016
Key Points

Question  Does implementation of an intervention to improve primary care professionals’ human papillomavirus (HPV) vaccine communication lead to increases in adolescent human papillomavirus vaccination?

Finding  Among 43 132 patients at 16 practices participating in this cluster randomized clinical trial, a 5-component intervention significantly increased HPV vaccine series initiation, stopped decline of completion, and was effective for both boys and girls. Two specific intervention components, communication training and customized HPV fact sheets, were the most used and useful based on health care professionals’ report.

Meaning  Disseminating this intervention widely among primary care professionals could substantially increase national adolescent HPV vaccination levels, particularly among boys.

Abstract

Importance  The incidence of human papillomavirus (HPV)–related cancers is more than 35 000 cases in the United States each year. Effective HPV vaccines have been available in the United States for several years but are underused among adolescents, the target population for vaccination. Interventions to increase uptake are needed.

Objective  To evaluate the effect of a 5-component health care professional HPV vaccine communication intervention on adolescent HPV vaccination.

Design, Setting, and Participants  A cluster randomized clinical trial using covariate-constrained randomization to assign study arms and an intent-to-treat protocol was conducted in 16 primary care practices in the Denver, Colorado, metropolitan area. Participants included 188 medical professionals and 43 132 adolescents.

Interventions  The 5 components of the intervention were an HPV fact sheet library to create customized information sheets relevant to each practice’s patient population, a tailored parent education website, a set of HPV-related disease images, an HPV vaccine decision aid, and 2½ hours of communication training on using a presumptive vaccine recommendation, followed by motivational interviewing if parents were resistant to vaccination. Each practice participated in a series of 2 intervention development meetings over a 6-month period (August 1, 2014, to January 31, 2015) before the intervention.

Main Outcomes and Measures  Differences between control and intervention changes over time (ie, difference in differences between the baseline and intervention period cohorts of patients) in HPV vaccine series initiation (≥1 dose) and completion (≥3 doses) among patients aged 11 to 17 years seen at the practices between February 1, 2015, and January 31, 2016. Vaccination data were obtained from the practices’ records and augmented with state immunization information system data.

Results  Sixteen practices and 43 132 patients (50.3% female; median age, 12.6 years [interquartile range, 10.8-14.7 years] at the beginning of the study period) participated in this trial. Adolescents in the intervention practices had significantly higher odds of HPV vaccine series initiation (adjusted odds ratio [aOR], 1.46; 95% CI, 1.31-1.62) and completion (aOR, 1.56; 95% CI, 1.27-1.92) than those in the control practices (a 9.5–absolute percentage point increase in HPV vaccine series initiation and a 4.4–absolute percentage point increase in HPV vaccine series completion in intervention practices). The intervention had a greater effect in pediatric practices compared with family medicine practices and in private practices compared with public ones. Health care professionals reported that communication training and the fact sheets were the most used and useful intervention components.

Conclusions and Relevance  A health care professional communication intervention significantly improved HPV vaccine series initiation and completion among adolescent patients.

Trial Registration  clinicaltrials.gov Identifier: NCT02456077

Introduction

The incidence of human papillomavirus (HPV)–related cancers is more than 35 000 cases in the United States each year.1 Highly effective vaccines against HPV have been available in the United States since 2006 for girls2,3 and 2009 for boys4,5 yet are largely underused. As of 2016, only 60.4% of children aged 13 to 17 years had started the HPV vaccination series, and only approximately two-thirds of those starting the series completed it.6 Attempts to use policy changes to increase uptake, such as mandating HPV vaccination for school entry, have been largely unsuccessful and ineffective.7 Interventions to improve adolescent HPV vaccine uptake by other means are a national priority.8

A key factor influencing adolescent HPV vaccination is whether and how a health care professional recommends it.9,10 Numerous studies11-15 demonstrate that medical professionals often fail to communicate effectively about the vaccine with patients and parents. The President’s Cancer Panel8 has indicated that interventions to improve health care professionals’ communication about adolescent HPV vaccination are needed.

Our group developed a 5-component health care professional communication intervention based on the precaution-adoption-process model16 to improve medical professionals’ ability to effectively communicate about HPV vaccines with their adolescent patients and their parents. The present study tests our hypothesis that implementation of the intervention increases practices’ adolescent HPV vaccine uptake compared with providing usual care.

Methods
Study Design

This was a 2-arm, controlled cluster randomized clinical trial that was performed between February 1, 2015, and January 31, 2016. Because the intervention was at the practice level, cluster randomization was performed at the practice level in a 1:1 ratio. All study activities were approved by the Colorado Multiple Institutional Review Board. The study is registered with clinicaltrials.gov (identifier NCT02456077). Informed consent was waived.

Study Sites

Twenty-four practices in the Denver, Colorado, metropolitan area that were part of a 30-clinic practice-based research network were invited to participate to represent a diverse cross-section of patient and practice demographics. Inclusion criteria were being a pediatrics or family medicine practice with at least 400 active (seen within the last 2 years) adolescent patients (age range, 11-17 years). There were no exclusion criteria. Of these 24, one practice withdrew from the study before randomization owing to new competing time demands (electronic medical record implementation). Seven practices that were part of one safety-net hospital system were dropped before randomization owing to very high baseline adolescent vaccination rates (approximately 90% for series initiation17). The final cohort for randomization included 16 practices (4 family medicine and 12 pediatrics) that included 188 medical professionals. Each practice participated in a series of 2 intervention development meetings over a 6-month period (August 1, 2014, to January 31, 2015) before officially launching the intervention.

All health care professionals who ordered vaccines for patients (ie, physicians, nurse practitioners, medical assistants, and physician assistants) in these practices could participate. Of these, 5 physicians in a single family medicine practice declined study participation owing to seeing few adolescent patients. This practice was ultimately assigned to the intervention arm, and these medical professionals’ data are included in the analyses, which used an intent-to-treat protocol (Supplement 1).

Data Sources

Vaccination data were retrieved from each practice’s electronic medical record. The following 2 periods with different but overlapping patient cohorts were compared: baseline (September 1, 2013, to August 30, 2014) and intervention implementation phase (February 1, 2015, to January 31, 2016). The 6-month interval between these 2 time points was for “on-boarding” intervention practices in implementation procedures. To ensure completeness, vaccination data were augmented with data from the Colorado Immunization Information System, to which all practices in the study actively reported. Data on all adolescents seen at least once during the study period were included in the analyses. Adolescents who were seen at multiple practices, who were deceased, or who were pregnant at the time of the visit (a contraindication for HPV vaccination) were excluded. Waivers of consent and Health Insurance Portability and Accountability Act of 1996 authorization were obtained to view adolescent vaccination records.

Cluster Randomization

Covariate-constrained randomization18 was used to assure balance in potential confounding factors across practices, including the following: baseline HPV vaccination rates among individuals aged 11 to 12 years, percentage of pediatric patients reported by health care professionals as eligible for the Vaccines for Children program (a proxy for low-income status and Medicaid insurance), proportion of medical professionals reporting “strongly” recommending the HPV vaccine for those aged 11 to 12 years, number of adolescent patients,19 and medical specialty (family medicine and pediatrics).20,21 Because most practices did not collect patient race and ethnicity data, these were not used as a balance criterion. All possible combinations of eligible practices that would create 2 equal groups (8 intervention and 8 control) were generated using the IML procedure in SAS (version 9.4; SAS Institute Inc). The distribution of the balance criterion was then used to define an acceptable set of study groups that were reasonably balanced in terms of the selected variables.22 From this, one set was randomly chosen and used to assign study arms. Health care professionals and the study team were not masked to the randomization category, but patients and analysts were.

Health Care Professional Communication Intervention

Intervention practices received a 5-component intervention that was designed based on the precaution-adoption-process model,16 which distinguishes between various stages of the decision-making continuum (ie, unaware, aware but unengaged, undecided, etc) and was developed to provide tools and training that could be used before, during, or at the end of the clinical encounter. The intervention included the following: (1) a fact sheet library that practices used to create practice-specific fact sheets about HPV infection and vaccination (eFigure 1 in Supplement 2), (2) a parent education website called “iVac” that created individually customized information about HPV vaccination (eFigure 2 in Supplement 2), (3) a series of disease images depicting diseases associated with HPV, (4) a decision aid for HPV vaccination (eFigure 3 in Supplement 2), and (5) communication training to improve health care professionals’ vaccine recommendation practices. The communication training consisted of a self-guided, 30-minute webinar, plus 2 in-person, group training sessions that lasted 1 hour each. These sessions focused on opening the HPV vaccine conversation with a “presumptive approach,” as defined by Opel et al,23 followed by the use of motivational interviewing techniques for parents perceived as resistant to vaccination. Intervention practices worked with the study team over a series of 2 meetings lasting 1 hour each to develop and plan for implementation of the intervention within their practice, and each intervention practice chose a study champion to help facilitate the study activities. Medical professional surveys were administered quarterly to collect self-reported use of each tool kit component. Health care professionals in intervention practices received 25 Maintenance of Certification Part IV credits for participation. No other incentives were provided. A detailed description of intervention components, study planning meetings, and implementation procedures is provided in eMethods in Supplement 2.

Control Group

Practices in the control arm continued usual care with regard to communication about HPV vaccines. Health care professionals in the control arm did not receive any incentives for participation.

Sample Size Estimation

Sample size estimates were based on an assumed final sample size of 16 000 adolescents (8000 per arm), and vaccine use centered around 50% (the most conservative estimate). In a mixed-effects analysis with practice as a random effect and an estimated intraclass correlation coefficient of 2%, this sample size would provide 86% power (α = .05) to detect a 10–percentage point (PP) difference in differences in changes in HPV vaccination over time from the baseline to intervention implementation periods between control and intervention groups. Based on practice data, a period of 1 year was allocated to achieve this sample.

Primary and Secondary Study Outcome Measures

The primary outcome was the difference between control and intervention groups in changes over time in the proportion of eligible adolescents initiating (≥1 dose) the HPV vaccine series. Secondary outcomes were uptake of 2 other adolescent vaccines, the meningococcal conjugate vaccine (MenACWY) and the tetanus-diphtheria-acellular pertussis vaccine (Tdap). Completion (≥3 doses) of the HPV vaccine series was also assessed post hoc. The denominator for this analysis was the number of adolescents in the practice who had received the prior 2 doses of vaccine.

Covariates

Patient-level covariates included patient age (11-12 or 13-17 years), sex, race (white, black, or other), ethnicity (Hispanic or non-Hispanic), and insurance at the most recent visit (private, public, other, or none). Medical specialty (pediatrics or family medicine) and practice type (public or private) were hypothesized to be potential effect modifiers and were included in subgroup analyses to examine heterogeneity of treatment effects.

Statistical Analysis

We used an intent-to-treat analysis and generalized linear mixed models,24,25 as is recommended for cluster randomized trials.26,27 Clustering of patients within practices was accounted for with a random intercept for each practice. Models are presented as unadjusted and adjusted for covariates significantly associated with the outcome (P < .05) or variables representing factors known from prior research to be associated with HPV vaccination (medical specialty, practice type, age, sex, and insurance).20,21,28,29 The intraclass correlation coefficient was calculated using an intercept-only model.30 Patients with unknown sex or with non–private or public insurance were excluded from this analysis. All P values are from 2-sided hypothesis tests. Statistical significance was defined at α = .05. Adjustments for multiplicity were not performed. All analyses used SAS software (version 9.4; SAS Institute Inc). With the exception of the missed opportunities analyses, which were visit-level analyses, all analyses were patient level.

Heterogeneity of Treatment Effects

A series of moderator (effect modification) analyses assessed whether there were differential effects of the intervention by selected practice (medical specialty and practice type) and patient (sex, race/ethnicity, and insurance) characteristics by examining the 3-way interaction term of time × study group × moderator and performing stratified analyses. Additional subgroup analyses were performed using (1) patients with a vaccination-eligible visit at age 11 to 12 years, (2) patients with a vaccination-eligible visit at age 13 to 17 years, (3) well-child care visits, and (4) sick visits (see eMethods in Supplement 2 for the definition of well or sick). Also assessed was the proportion of visits where adolescents had a missed opportunity for vaccination, defined as a clinic visit during the study period at which an adolescent was eligible for an HPV vaccine dose but did not receive it.31,32

Health Care Professional Use and Perceptions

Descriptive statistics were generated from 7 quarterly health care professional surveys assessing the use of intervention components over time, given to 85 to 107 medical professionals (response rate, 85.5%-100%), depending on the staffing of the clinic at the time. A study describing the use of each tool kit component in detail is in preparation.

Results
Study Participants

All 16 practices in the study were assessed for vaccination outcomes (Figure). As summarized in Table 1, patient and practice variables were mostly evenly distributed between groups except that a higher proportion of patients were reported as having public insurance in the intervention arm. Baseline vaccination rates among individuals aged 11 to 12 years, the variable used in the randomization process, were identical between arms. However, when the trial was completed, it became apparent that HPV vaccination rates among those aged 11 to 17 years differed slightly between arms (37.1% vs 31.6%) (Table 2).

Effect on HPV Vaccination Rates

Both the control and intervention groups significantly increased the proportion of eligible adolescents initiating the vaccine series over time (Table 2 and eFigure 4 in Supplement 2). However, in both unadjusted and adjusted models, these increases were significantly larger in the intervention group compared with control (1.8% increase control vs 11.3% increase intervention; 9.5–absolute PP difference, P < .001), with adjusted odds ratios (aORs) of 1.46 for initiation and 1.56 for completion (Table 2). In contrast, series completion significantly decreased in the control practices, while remaining stable in the intervention practices, resulting in intervention practices having significantly higher odds of completing the series than control practices (Table 2 and eFigure 5 in Supplement 2).

Heterogeneity of treatment effects analyses of HPV vaccine series initiation (Table 3) and completion (eTable 1 in Supplement 2) by patient and practice characteristics showed that heterogeneity for HPV vaccination improvement was seen only for series initiation and occurred primarily at pediatric practices (medical specialty interaction term F1,11 = 11.33, P = .006 for initiation and F1,11 = 5.42, P = .04 for completion) and private practices (practice type interaction term F1,13 = 33.63, P < .001 for initiation and F1,13 = 0.76, P = .40 for completion). However, analyses were somewhat limited by the low numbers of adolescents eligible for the vaccine in family medicine practices overall.

Heterogeneity of treatment effects analyses assessed HPV vaccination initiation (Table 4) and completion (eTable 2 in Supplement 2) by adolescent age, sex, and insurance. Increases in series initiation among both age categories were higher in intervention practices than controls. There was no differential treatment effect by sex (sex interaction term F1,13 = 0.01, P = .91 for initiation and F1,13 = 1.58, P = .23 for completion), but a differential treatment effect by patient insurance was seen for series initiation but not for completion (insurance interaction term F1,13 = 17.85, P < .001 for initiation and F1,13 = 2.47, P = .14 for completion). Series initiation increased substantially over time among patients with private insurance (aOR, 1.76; 95% CI, 1.55-1.99) but remained essentially unchanged among those with public insurance (aOR, 0.93; 95% CI, 0.76-1.14). Among the subset of practices with race (n = 6), and ethnicity (n = 2) data available, there were no differences in vaccination by the variables between study arms (eTable 3 and eTable 4 in Supplement 2).

Effect of MenACWY and Tdap Vaccination

Over time, there was a slight increase in MenACWY vaccination (0.4-PP increase to 62.8% for control and 2.1-PP increase to 55.6% for intervention) and a slight decrease in Tdap vaccination (3.4-PP decrease to 60.1% for control and 7.0-PP decrease to 50.1% for intervention). Neither comparison between study arms was significant (eTable 5 in Supplement 2).

Effect on Missed Opportunities for Vaccination

Overall, there was a significant reduction in missed opportunities for vaccination in intervention practices compared with controls (eFigure 6 in Supplement 2). This difference was significant for well-child care checkups (aOR, 0.61; 95% CI, 0.54-0.69) but not for sick visits (aOR, 0.87; 95% CI, 0.68-1.12).

Intervention Sustainability

Each intervention component was used by 26.0% to 90.0% of health care professionals over the 12-month study period. Of these, the communication techniques and fact sheets were reported as the most frequently used, with 72.2% to 90.0% and 51.5% to 84.4% of medical professionals reporting using them over the study period, respectively. Surveys done at the end of the intervention period demonstrated that 98.0% of health care professionals were likely to continue to use the fact sheets and that 91.0% of health care professionals were likely to continue to use the communication techniques.

Discussion

Implementation of a health care professional communication intervention to improve adolescent HPV vaccination resulted in a 9.5-PP increase in HPV vaccine series initiation compared with control practices, which was both clinically and statistically significant. The use of the intervention materials, particularly the communication techniques and fact sheets, was sustained over the 12-month intervention implementation period, and medical professionals intended to continue to use these components in the future. The intervention also mitigated decreases over time in HPV vaccine series completion.

To our knowledge, there are 2 other intervention studies that have focused on health care professional communication for improving HPV vaccination. Brewer and colleagues33 tested the effect of “announcements” (similar to the presumptive communication style) vs “conversations” (similar to the participatory communication style) on HPV vaccination levels in 29 practices. At the 6-month assessment, announcement practices had a 5.4-PP increase in HPV vaccine series initiation and no changes in receipt of Tdap, MenACWY, or HPV vaccine series completion compared with controls. In a smaller study, Perkins et al34 implemented a multicomponent intervention that included practice coaching every 4 to 6 weeks, HPV education for medical professionals (including “basic motivational interviewing principles,” assessment, and feedback on practices’ HPV vaccination rates compared with others in their region), and Maintenance of Certification part IV incentives. The HPV vaccination rates were higher in the intervention practice during the active study phase than controls but were sustained only for male participants when assessed 6 months later. Placing our results in this context, it seems that, while a presumptive/announcement approach to the initial HPV vaccine conversation can increase HPV vaccine initiation, there is added benefit to using additional components in our intervention, namely, the motivational interviewing training and customized HPV fact sheets. Inclusion of these items could explain why our intervention had a greater effect than that of the study by Brewer and colleagues,33 positively affected series completion, and seemed to have a sustained effect over a longer period compared with the study by Perkins et al.34

Our intervention appeared less effective in public compared with private practices. This result is opposite of what might be expected given that national data consistently demonstrate increased adolescent HPV vaccine uptake among populations typically served by public clinics.28,35-38 When examined at the practice level, 2 of the public practices in the intervention arm increased HPV vaccine series initiation levels (by 7.8% and 13.0% over time), whereas the third one decreased by 3.5%. One explanation for this could be that the third practice, which was substantially larger than the other 2, had a large number of trainees, and not all health care professionals were able to fully participate in the intervention training sessions and study meetings, effectively diluting any effect the intervention may have had in this practice.

Improvements in HPV vaccination among intervention practices occurred primarily at well-child care visits. While there was some decrease in missed opportunities for vaccination at sick visits in intervention practices, this decrease was small, and most vaccines were still provided during routine wellness examinations. Anecdotal reports from health care professionals in our study indicate that lack of time and prioritization of other health issues make vaccination at sick visits difficult, a finding that is supported by several other studies.15,39-41 While there are examples of clinical systems that have successfully implemented routine adolescent HPV vaccination at sick visits,17 this practice is not widespread. Given that adolescents see medical professionals for sick visits more commonly than for preventive visits,42,43 finding mechanisms to improve vaccination at sick visits is a clear research priority.

Limitations

This trial had some limitations. First, the most important limitation of our study was that we could not directly examine at an individual patient level the effect of specific intervention components on HPV vaccine uptake. Health care professionals reported quarterly on which intervention components they used during the previous month, rather than after each patient visit. The intervention was developed to be adaptable based on each practice’s and medical professional’s needs, resulting in variability between and within practices of which intervention components were used and under what circumstances. Based on health care professionals’ reports, it appears that the communication training and fact sheets were the most used and useful intervention components. Further research is needed to understand if the other components are needed. Second, an additional limitation is that having an immunization champion and the 2-hour planning meetings with practices, while not part of the intervention per se, could have also had an effect on vaccination rates. This effect was not specifically assessed. Third, we could only assess the vaccination status of the patients who had a clinic visit during the study period. Our intervention was not designed to affect patients who are not seen for care. Fourth, our intervention focused on a single geographic area and may not be generalizable. Fifth, although baseline assessments of HPV vaccination status among those aged 11 to 12 years was identical between arms, baseline vaccination status among those aged 11 to 17 years was slightly lower among intervention practices than control practices, which could have influenced the degree to which the intervention increased vaccination rates. Sixth, the overall influence of the intervention was modest, and all practices’ vaccination levels remained well below the national goal of 80% coverage.

Conclusions

In this cluster randomized clinical trial of a health care professional HPV vaccine communication intervention, there were substantial and sustained increases in HPV vaccine series initiation in intervention practices compared with controls over time. Medical professionals used some tool kit components more than others and planned to continue to use them in the future. Future research will need to examine if similar effects on vaccination rates can be achieved through more generalizable dissemination methods, such as via the internet or through the public health department network.

Back to top
Article Information

Accepted for Publication: January 2, 2018.

Corresponding Author: Amanda F. Dempsey, MD, PhD, MPH, Division of General Pediatrics, Department of Pediatrics, University of Colorado Denver, 13199 E Montview Blvd, Ste 300, Aurora, CO 80045 (amanda.dempsey@ucdenver.edu).

Published Online: March 5, 2018. doi:10.1001/jamapediatrics.2018.0016

Author Contributions: Dr Dempsey had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Dempsey, Lockhart, Garrett, Fisher, Dickinson, O’Leary.

Acquisition, analysis, or interpretation of data: Dempsey, Pyrzanowski, Lockhart, Barnard, Campagna, Dickinson, O’Leary.

Drafting of the manuscript: Dempsey, Garrett, O’Leary.

Critical revision of the manuscript for important intellectual content: Dempsey, Pyrzanowski, Lockhart, Barnard, Campagna, Fisher, Dickinson, O’Leary.

Statistical analysis: Campagna, Dickinson.

Obtained funding: Dempsey, O’Leary.

Administrative, technical, or material support: Dempsey, Pyrzanowski, Lockhart, Barnard, Garrett, O’Leary.

Study supervision: Dempsey, Pyrzanowski, Lockhart, O’Leary.

Conflict of Interest Disclosures: Dr Dempsey reported serving on advisory boards for Merck, Pfizer, and Sanofi Pasteur and reported being a consultant to Pfizer. Dr Dempsey reported that she does not receive any research funding from these agencies and that they had no role in the research described herein.

Funding/Support: This project was funded by grant U01P000801 from the Centers for Disease Control and Prevention.

Role of the Funder/Sponsor: The funding source was not involved in the design and conduct of the study or in the collection, management, or analysis of data. The funding source was involved in the interpretation of the data; preparation, review, and approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

References
1.
Viens  LJ, Henley  SJ, Watson  M,  et al.  Human papillomavirus–associated cancers: United States, 2008-2012.  MMWR Morb Mortal Wkly Rep. 2016;65(26):661-666.PubMedGoogle ScholarCrossref
2.
Markowitz  LE, Dunne  EF, Saraiya  M, Lawson  HW, Chesson  H, Unger  ER; Centers for Disease Control and Prevention (CDC); Advisory Committee on Immunization Practices (ACIP).  Quadrivalent Human Papillomavirus Vaccine: Recommendations of the Advisory Committee on Immunization Practices (ACIP).  MMWR Recomm Rep. 2007;56(RR-2):1-24.PubMedGoogle Scholar
3.
US Food and Drug Administration. Approved products: Gardasil. https://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM094042. Last updated January 9, 2018. Accessed December 1, 2017.
4.
US Food and Drug Administration. Gardasil package insert. http://www.fda.gov/downloads/BiologicsBloodVaccines/Vaccines/ApprovedProducts/UCM111263.pdf. Revised April 2015. Accessed November 28, 2017.
5.
Centers for Disease Control and Prevention (CDC).  FDA licensure of quadrivalent human papillomavirus vaccine (HPV4, Gardasil) for use in males and guidance from the Advisory Committee on Immunization Practices (ACIP).  MMWR Morb Mortal Wkly Rep. 2010;59(20):630-632.PubMedGoogle Scholar
6.
Walker  TY, Elam-Evans  LD, Singleton  JA,  et al.  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2016.  MMWR Morb Mortal Wkly Rep. 2017;66(33):874-882.PubMedGoogle ScholarCrossref
7.
Keim-Malpass  J, Mitchell  EM, DeGuzman  PB, Stoler  MH, Kennedy  C.  Legislative activity related to the human papillomavirus (HPV) vaccine in the United States (2006-2015): a need for evidence-based policy.  Risk Manag Healthc Policy. 2017;10:29-32.PubMedGoogle ScholarCrossref
8.
President’s Cancer Panel.  Accelerating HPV Vaccine Uptake: Urgency for Action to Prevent Cancer: A Report to the President of the United States from the President’s Cancer Panel. Bethesda, MD: National Cancer Institute; 2014.
9.
Lau  M, Lin  H, Flores  G.  Factors associated with human papillomavirus vaccine–series initiation and healthcare provider recommendation in US adolescent females: 2007 National Survey of Children’s Health.  Vaccine. 2012;30(20):3112-3118.PubMedGoogle ScholarCrossref
10.
Rahman  M, Laz  TH, McGrath  CJ, Berenson  AB.  Provider recommendation mediates the relationship between parental human papillomavirus (HPV) vaccine awareness and HPV vaccine initiation and completion among 13- to 17-year-old U.S. adolescent children.  Clin Pediatr (Phila). 2015;54(4):371-375.PubMedGoogle ScholarCrossref
11.
Vadaparampil  ST, Malo  TL, Sutton  SK,  et al.  Missing the target for routine human papillomavirus vaccination: consistent and strong physician recommendations are lacking for 11- to 12-year-old males.  Cancer Epidemiol Biomarkers Prev. 2016;25(10):1435-1446.PubMedGoogle ScholarCrossref
12.
Vadaparampil  ST, Malo  TL, Kahn  JA,  et al.  Physicians’ human papillomavirus vaccine recommendations, 2009 and 2011.  Am J Prev Med. 2014;46(1):80-84.PubMedGoogle ScholarCrossref
13.
Stokley  S, Jeyarajah  J, Yankey  D,  et al; Immunization Services Division, National Center for Immunization and Respiratory Diseases, CDC; Centers for Disease Control and Prevention (CDC).  Human papillomavirus vaccination coverage among adolescents, 2007-2013, and postlicensure vaccine safety monitoring, 2006-2014: United States.  MMWR Morb Mortal Wkly Rep. 2014;63(29):620-624.PubMedGoogle Scholar
14.
Gilkey  MB, Malo  TL, Shah  PD, Hall  ME, Brewer  NT.  Quality of physician communication about human papillomavirus vaccine: findings from a national survey.  Cancer Epidemiol Biomarkers Prev. 2015;24(11):1673-1679.PubMedGoogle ScholarCrossref
15.
Gilkey  MB, Moss  JL, Coyne-Beasley  T, Hall  ME, Shah  PD, Brewer  NT.  Physician communication about adolescent vaccination: how is human papillomavirus vaccine different?  Prev Med. 2015;77:181-185.PubMedGoogle ScholarCrossref
16.
Weinstein  N, Sandman  PM. The precaution adoption process model and its application. In: DiClemente  R, Crosby  RA, Kegler  MC, eds.  Emerging Theories in Health Promotion Research and Practice. San Francisco, CA: Jossey-Bass; 2002:16-39.
17.
Farmar  AM, Love-Osborne  K, Chichester  K, Breslin  K, Bronkan  K, Hambidge  SJ.  Achieving high adolescent HPV vaccination coverage.  Pediatrics. 2016;138(5):e20152653.PubMedGoogle ScholarCrossref
18.
Dickinson  LM, Beaty  B, Fox  C,  et al.  Pragmatic cluster randomized trials using covariate constrained randomization: a method for practice-based research networks (PBRNs).  J Am Board Fam Med. 2015;28(5):663-672.PubMedGoogle ScholarCrossref
19.
Glynn  RJ, Brookhart  MA, Stedman  M, Avorn  J, Solomon  DH.  Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers.  Med Care. 2007;45(10)(suppl 2):S38-S43.PubMedGoogle ScholarCrossref
20.
Allison  MA, Hurley  LP, Markowitz  L,  et al.  Primary care physicians’ perspectives about HPV vaccine.  Pediatrics. 2016;137(2):e20152488.PubMedGoogle ScholarCrossref
21.
Oster  NV, McPhillips-Tangum  CA, Averhoff  F, Howell  K.  Barriers to adolescent immunization: a survey of family physicians and pediatricians.  J Am Board Fam Pract. 2005;18(1):13-19.PubMedGoogle ScholarCrossref
22.
Raab  GM, Butcher  I.  Balance in cluster randomized trials.  Stat Med. 2001;20(3):351-365.PubMedGoogle ScholarCrossref
23.
Opel  DJ, Heritage  J, Taylor  JA,  et al.  The architecture of provider-parent vaccine discussions at health supervision visits.  Pediatrics. 2013;132(6):1037-1046.PubMedGoogle ScholarCrossref
24.
Kenward  MG, Lesaffre  E, Molenberghs  G.  An application of maximum likelihood and generalized estimating equations to the analysis of ordinal data from a longitudinal study with cases missing at random.  Biometrics. 1994;50(4):945-953.PubMedGoogle ScholarCrossref
25.
Hedeker  D, Gibbons  RD.  MIXREG: a computer program for mixed-effects regression analysis with autocorrelated errors.  Comput Methods Programs Biomed. 1996;49(3):229-252.PubMedGoogle ScholarCrossref
26.
Murray  DM, Varnell  SP, Blitstein  JL.  Design and analysis of group-randomized trials: a review of recent methodological developments.  Am J Public Health. 2004;94(3):423-432.PubMedGoogle ScholarCrossref
27.
Varnell  SP, Murray  DM, Janega  JB, Blitstein  JL.  Design and analysis of group-randomized trials: a review of recent practices.  Am J Public Health. 2004;94(3):393-399.PubMedGoogle ScholarCrossref
28.
Reagan-Steiner  S, Yankey  D, Jeyarajah  J,  et al.  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2015.  MMWR Morb Mortal Wkly Rep. 2016;65(33):850-858.PubMedGoogle ScholarCrossref
29.
Dempsey  A, Cohn  L, Dalton  V, Ruffin  M.  Patient and clinic factors associated with adolescent human papillomavirus vaccine utilization within a university-based health system.  Vaccine. 2010;28(4):989-995.PubMedGoogle ScholarCrossref
30.
Snijders  T, Bosker  RJ.  Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. Thousand Oaks, CA: SAGE Publications; 1999.
31.
Fu  LY, Zook  K, Gingold  J,  et al.  Frequent vaccination missed opportunities at primary care encounters contribute to underimmunization.  J Pediatr. 2015;166(2):412-417.PubMedGoogle ScholarCrossref
32.
Kepka  D, Spigarelli  MG, Warner  EL, Yoneoka  Y, McConnell  N, Balch  A.  Statewide analysis of missed opportunities for human papillomavirus vaccination using vaccine registry data.  Papillomavirus Res. 2016;2:128-132.PubMedGoogle ScholarCrossref
33.
Brewer  NT, Hall  ME, Malo  TL, Gilkey  MB, Quinn  B, Lathren  C.  Announcements versus conversations to improve HPV vaccination coverage: a randomized trial.  Pediatrics. 2017;139(1):e20161764.PubMedGoogle ScholarCrossref
34.
Perkins  RB, Zisblatt  L, Legler  A, Trucks  E, Hanchate  A, Gorin  SS.  Effectiveness of a provider-focused intervention to improve HPV vaccination rates in boys and girls.  Vaccine. 2015;33(9):1223-1229.PubMedGoogle ScholarCrossref
35.
Centers for Disease Control and Prevention (CDC).  National and state vaccination coverage among adolescents aged 13-17 years: United States, 2011.  MMWR Morb Mortal Wkly Rep. 2012;61(34):671-677.PubMedGoogle Scholar
36.
Centers for Disease Control and Prevention (CDC).  National and state vaccination coverage among adolescents aged 13-17 years: United States, 2012.  MMWR Morb Mortal Wkly Rep. 2013;62(34):685-693.PubMedGoogle Scholar
37.
Elam-Evans  LD, Yankey  D, Jeyarajah  J,  et al; Immunization Services Division, National Center for Immunization and Respiratory Diseases; Centers for Disease Control and Prevention (CDC).  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2013.  MMWR Morb Mortal Wkly Rep. 2014;63(29):625-633.PubMedGoogle Scholar
38.
Reagan-Steiner  S, Yankey  D, Jeyarajah  J,  et al.  National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years: United States, 2014.  MMWR Morb Mortal Wkly Rep. 2015;64(29):784-792.PubMedGoogle ScholarCrossref
39.
Gowda  C, Schaffer  SE, Dombkowski  KJ, Dempsey  AF.  Understanding attitudes toward adolescent vaccination and the decision-making dynamic among adolescents, parents and providers.  BMC Public Health. 2012;12:509.PubMedGoogle ScholarCrossref
40.
McSherry  LA, Dombrowski  SU, Francis  JJ,  et al; ATHENS Group.  “It’s a can of worms”: understanding primary care practitioners’ behaviours in relation to HPV using the Theoretical Domains Framework.  Implement Sci. 2012;7:73.PubMedGoogle ScholarCrossref
41.
Dempsey  AF, Abraham  LM, Dalton  V, Ruffin  M.  Understanding the reasons why mothers do or do not have their adolescent daughters vaccinated against human papillomavirus.  Ann Epidemiol. 2009;19(8):531-538.PubMedGoogle ScholarCrossref
42.
Dempsey  AF, Freed  GL.  Health care utilization by adolescents on Medicaid: implications for delivering vaccines.  Pediatrics. 2010;125(1):43-49.PubMedGoogle ScholarCrossref
43.
Rand  CM, Shone  LP, Albertin  C, Auinger  P, Klein  JD, Szilagyi  PG.  National health care visit patterns of adolescents: implications for delivery of new adolescent vaccines.  Arch Pediatr Adolesc Med. 2007;161(3):252-259.PubMedGoogle ScholarCrossref
×