[Skip to Navigation]
Sign In
Table 1.  Sociodemographic Characteristics, Health-Related Behaviors, and Sexual History, Stratified by Sex
Sociodemographic Characteristics, Health-Related Behaviors, and Sexual History, Stratified by Sex
Table 2.  Oral HPV Types Among 910 Indigenous Adults in South Australia
Oral HPV Types Among 910 Indigenous Adults in South Australia
Table 3.  Bivariate and Multivariable Associations With Prevalence of All Oral HPV Types, Stratified by Sex
Bivariate and Multivariable Associations With Prevalence of All Oral HPV Types, Stratified by Sex
Table 4.  Bivariate and Multivariable Associations With Prevalence of Oral HPV Types 13 and 32, Stratified by Sex
Bivariate and Multivariable Associations With Prevalence of Oral HPV Types 13 and 32, Stratified by Sex
Table 5.  Bivariate and Multivariable Associations With Prevalence of Oral HPV Types 16 and 18, Stratified by Sex
Bivariate and Multivariable Associations With Prevalence of Oral HPV Types 16 and 18, Stratified by Sex
1.
Lechner  M, Breeze  CE, O’Mahony  JF, Masterson  L.  Early detection of HPV-associated oropharyngeal cancer.   Lancet. 2019;393(10186):2123. doi:10.1016/S0140-6736(19)30227-2 PubMedGoogle ScholarCrossref
2.
D’Souza  G, Kreimer  AR, Viscidi  R,  et al.  Case-control study of human papillomavirus and oropharyngeal cancer.   N Engl J Med. 2007;356(19):1944-1956. doi:10.1056/NEJMoa065497 PubMedGoogle ScholarCrossref
3.
Hong  A, Lee  CS, Jones  D,  et al.  Rising prevalence of human papillomavirus-related oropharyngeal cancer in Australia over the last 2 decades.   Head Neck. 2016;38(5):743-750. doi:10.1002/hed.23942 PubMedGoogle ScholarCrossref
4.
Plummer  M, de Martel  C, Vignat  J, Ferlay  J, Bray  F, Franceschi  S.  Global burden of cancers attributable to infections in 2012: a synthetic analysis.   Lancet Glob Health. 2016;4(9):e609-e616. doi:10.1016/S2214-109X(16)30143-7 PubMedGoogle ScholarCrossref
5.
Castellsagué  X, Alemany  L, Quer  M,  et al; ICO International HPV in Head and Neck Cancer Study Group.  HPV involvement in head and neck cancers: comprehensive assessment of biomarkers in 3680 patients.   J Natl Cancer Inst. 2016;108(6):djv403. doi:10.1093/jnci/djv403 PubMedGoogle Scholar
6.
Yakin  M, Seo  B, Hussaini  H, Rich  A, Hunter  K.  Human papillomavirus and oral and oropharyngeal carcinoma: the essentials.   Aust Dent J. 2019;64(1):11-18. doi:10.1111/adj.12652 PubMedGoogle ScholarCrossref
7.
National Cancer Institute. Annual report to the nation on the status of cancer. Accessed September 30, 2019. https://seer.cancer.gov/report_to_nation/
8.
Office for National Statistics. Cancer registration statistics, England: first release, 2016. Accessed September 30, 2019. https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/cancerregistrationstatisticsengland/2016
9.
Henke  RP, Guèrin-Reverchon  I, Milde-Langosch  K, Koppang  HS, Löning  T.  In situ detection of human papillomavirus types 13 and 32 in focal epithelial hyperplasia of the oral mucosa.   J Oral Pathol Med. 1989;18(7):419-421. doi:10.1111/j.1600-0714.1989.tb01575.x PubMedGoogle ScholarCrossref
10.
Jayasooriya  PR, Abeyratne  S, Ranasinghe  AW, Tilakaratne  WM.  Focal epithelial hyperplasia (Heck’s disease): report of two cases with PCR detection of human papillomavirus DNA.   Oral Dis. 2004;10(4):240-243. doi:10.1111/j.1601-0825.2004.01012.x PubMedGoogle ScholarCrossref
11.
Bassioukas  K, Danielides  V, Georgiou  I, Photos  E, Zagorianakou  P, Skevas  A.  Oral focal epithelial hyperplasia.   Eur J Dermatol. 2000;10(5):395-397.PubMedGoogle Scholar
12.
Ozden  B, Gunduz  K, Gunhan  O, Ozden  FO.  A case report of focal epithelial hyperplasia (Heck’s disease) with PCR detection of human papillomavirus.   J Maxillofac Oral Surg. 2011;10(4):357-360. doi:10.1007/s12663-011-0184-2 PubMedGoogle ScholarCrossref
13.
Archard  HO, Heck  JW, Stanley  HR.  Focal epithelial hyperplasia: an unusual oral mucosal lesion found in Indian children.   Oral Surg Oral Med Oral Pathol. 1965;20:201-212. doi:10.1016/0030-4220(65)90192-1 PubMedGoogle ScholarCrossref
14.
González  LV, Gaviria  AM, Sanclemente  G,  et al.  Clinical, histopathological and virological findings in patients with focal epithelial hyperplasia from Colombia.   Int J Dermatol. 2005;44(4):274-279. doi:10.1111/j.1365-4632.2005.02321.x PubMedGoogle ScholarCrossref
15.
Wu  JSA, Florian  MC, Rodrigues  DA, Tomimori  J.  Skin diseases in indigenous population: retrospective epidemiological study at Xingu Indigenous Park (XIP) and review of the literature.   Int J Dermatol. 2017;56(12):1414-1420. doi:10.1111/ijd.13716 PubMedGoogle ScholarCrossref
16.
Eversole  LR.  Clinical Outline of Oral Pathology: Diagnosis and Treatment. People’s Medical Publishing House; 2011.
17.
Australian Institute of Health and Welfare.  Australia’s Health, 2018. Australian Institute of Health and Welfare; 2018.
18.
Cancer Australia.  Aboriginal and Torres Strait Islander Cancer Statistics. Australian Government; 2018.
19.
Banham  D, Roder  D, Keefe  D,  et al; CanDAD Aboriginal Community Reference Group and other CanDAD investigators.  Disparities in cancer stage at diagnosis and survival of Aboriginal and non–Aboriginal South Australians.   Cancer Epidemiol. 2017;48:131-139. doi:10.1016/j.canep.2017.04.013 PubMedGoogle ScholarCrossref
20.
Australian Institute of Health and Welfare.  Cancer in Aboriginal and Torres Strait Islander people of Australia. Australian Institute of Health and Welfare; 2018.
21.
Brotherton  JM, Winch  KL, Chappell  G,  et al.  HPV vaccination coverage and course completion rates for Indigenous Australian adolescents, 2015.   Med J Aust. 2019;211(1):31-36. doi:10.5694/mja2.50221 PubMedGoogle ScholarCrossref
22.
Garland  SM, Brotherton  JM, Condon  JR,  et al; WHINURS study group.  Human papillomavirus prevalence among Indigenous and non-Indigenous Australian women prior to a national HPV vaccination program.   BMC Med. 2011;9:104. doi:10.1186/1741-7015-9-104 PubMedGoogle ScholarCrossref
23.
Wood  ZC, Bain  CJ, Smith  DD, Whiteman  DC, Antonsson  A.  Oral human papillomavirus infection incidence and clearance: a systematic review of the literature.   J Gen Virol. 2017;98(4):519-526. doi:10.1099/jgv.0.000727 PubMedGoogle ScholarCrossref
24.
Antonsson  A, Cornford  M, Perry  S, Davis  M, Dunne  MP, Whiteman  DC.  Prevalence and risk factors for oral HPV infection in young Australians.   PLoS One. 2014;9(3):e91761. doi:10.1371/journal.pone.0091761 PubMedGoogle Scholar
25.
Jamieson  L, Garvey  G, Hedges  J,  et al.  Human papillomavirus and oropharyngeal cancer among Indigenous Australians: protocol for a prevalence study of oral-related human papillomavirus and cost-effectiveness of prevention.   JMIR Res Protoc. 2018;7(6):e10503. doi:10.2196/10503 PubMedGoogle Scholar
26.
Australian Institute of Health and Welfare.  Housing Circumstances of Indigenous Households: Tenure and Overcrowding. Australian Institute of Health and Welfare; 2014.
27.
de Roda Husman  AM, Walboomers  JMM, Hopman  E,  et al.  HPV prevalence in cytomorphologically normal cervical scrapes of pregnant women as determined by PCR: the age-related pattern.   J Med Virol. 1995;46(2):97-102. doi:10.1002/jmv.1890460203 PubMedGoogle ScholarCrossref
28.
Manos  MM, Ting  Y, Wright  DK, Lewis  AJ, Broker  TR.  The use of polymerase chain reaction amplification for the detection of genital human papillomaviruses.   Cancer Cell. 1989;7:209-214.Google Scholar
29.
García-Corona  C, Vega-Memije  E, Mosqueda-Taylor  A,  et al.  Association of HLA-DR4 (DRB1*0404) with human papillomavirus infection in patients with focal epithelial hyperplasia.   Arch Dermatol. 2004;140(10):1227-1231. doi:10.1001/archderm.140.10.1227 PubMedGoogle ScholarCrossref
30.
National Center for Biotechnology Information.  BLAST: basic local alignment search tool. Accessed April 17, 2020. https://blast.ncbi.nlm.nih.gov/Blast.cgi
31.
Rissel  C, Badcock  PB, Smith  AM,  et al.  Heterosexual experience and recent heterosexual encounters among Australian adults: the Second Australian Study of Health and Relationships.   Sex Health. 2014;11(5):416-426. doi:10.1071/SH14105 PubMedGoogle ScholarCrossref
32.
Wierzbicka  M, Berkhof  JH, Dikkers  FG.  Prophylactic human papilloma virus vaccination in head and neck: indications and future perspectives.   Curr Opin Otolaryngol Head Neck Surg. 2019;27(2):85-90. doi:10.1097/MOO.0000000000000525 PubMedGoogle ScholarCrossref
33.
Mazza  D, Petrovic  K, Chakraborty  S.  HPV vaccination of adult women: an audit of Australian general practitioners.   Aust N Z J Obstet Gynaecol. 2012;52(6):528-533. doi:10.1111/ajo.12002 PubMedGoogle ScholarCrossref
34.
Bennett  LK, Hinshaw  M.  Heck’s disease: diagnosis and susceptibility.   Pediatr Dermatol. 2009;26(1):87-89. doi:10.1111/j.1525-1470.2008.00830.x PubMedGoogle ScholarCrossref
35.
Syrjänen  S.  Oral manifestations of human papillomavirus infections.   Eur J Oral Sci. 2018;126(suppl 1):49-66. doi:10.1111/eos.12538 PubMedGoogle ScholarCrossref
36.
Kreimer  AR, Pierce Campbell  CM, Lin  HY,  et al.  Incidence and clearance of oral human papillomavirus infection in men: the HIM cohort study.   Lancet. 2013;382(9895):877-887. doi:10.1016/S0140-6736(13)60809-0 PubMedGoogle ScholarCrossref
37.
Rounds  T, Harvey  J.  Enrollment challenges: recruiting men to weight loss interventions.   Am J Mens Health. 2019;13(1):1557988319832120. doi:10.1177/1557988319832120 PubMedGoogle Scholar
38.
Buston  K.  Recruiting, retaining and engaging men in social interventions: lessons for implementation focusing on a prison-based parenting intervention for young incarcerated fathers.   Child Care Pract. 2018;24(2):164-180. doi:10.1080/13575279.2017.1420034 PubMedGoogle ScholarCrossref
39.
Isaacs  A, Pepper  H, Pyett  P, Gruis  H, Waples-Crowe  P, Oakley Browne  M.  What you do is important but how you do it is more important.   Qual Res J. 2011;11:51-61. doi:10.3316/QRJ1101051 Google ScholarCrossref
Original Investigation
Infectious Diseases
June 8, 2020

Prevalence of Oral Human Papillomavirus Infection Among Australian Indigenous Adults

Author Affiliations
  • 1Australian Research Centre for Population Oral Health, University of Adelaide, Adelaide, Australia
  • 2QIMR Berghofer Medical Research Institute, Royal Brisbane Hospital, Brisbane, Australia
  • 3Menzies School of Health Research, Spring Hill, Australia
  • 4Cancer Council New South Wales, Sydney, Australia
  • 5Sydney Medical School, University of Sydney, Sydney, Australia
  • 6Adelaide Dental School, University of Adelaide, Australia
  • 7Menzies Health Institute Queensland, Griffith University and Faculty of Dentistry, Queensland, Australia
  • 8Oral and Craniofacial Sciences, King’s College, London, United Kingdom
  • 9National Centre for Epidemiology and Population Health, ANU College of Health and Medicine, Australian National University, Australian Capital Territory, Australia
  • 10Strategic Partnerships, Aboriginal Health Division Women's and Children's Health Network, Adelaide, Adelaide, Australia
  • 11Aboriginal Health Council of South Australia, Adelaide, Australia
  • 12Wardliparingga Aboriginal Research Unit, South Australian Health and Medical Research Institute, Adelaide, Australia
  • 13University of South Australia School of Health Sciences, Adelaide, Australia
JAMA Netw Open. 2020;3(6):e204951. doi:10.1001/jamanetworkopen.2020.4951
Key Points español 中文 (chinese)

Question  What is the prevalence of oral human papillomavirus (HPV) infection among Indigenous Australians, a group at risk of oropharyngeal squamous cell carcinoma?

Findings  This cross-sectional study examined 910 Indigenous Australians for HPV infection with a particular focus on high-risk HPV types. Thirty-five percent of study participants had an oral HPV infection, 15 times the incidence reported in a study of young Australians and 5 times that reported in a systematic review from other countries.

Meaning  The findings of this study indicate that Indigenous Australians may be at higher risk of developing HPV-related oral cancer, which suggests that increased HPV vaccination coverage among this vulnerable population may be beneficial.

Abstract

Importance  Human papillomavirus (HPV) infection is associated with oropharyngeal squamous cell carcinoma. International estimates suggest overall oral HPV prevalence is 7.5%, with prevalence of oral HPV types 16 and 18 being 1.6%; prior Australian estimates suggest oral HPV prevalence is 2.3%, with HPV-16 and HPV-18 being 1.3%.

Objectives  To estimate the prevalence of oral HPV infection among Indigenous Australians and to report the prevalence of factors associated with high-risk HPV types (ie, HPV-16 and HPV-18) and HPV types linked with Heck disease (ie, HPV-13 and HPV-32).

Design, Setting, and Participants  This cross-sectional study analyzed HPV screening results from saliva samples collected from 1011 Indigenous Australians between February 2018 and January 2019. Data were analyzed from May 2018 to May 2019. Recruitment occurred through Aboriginal Community Controlled Health Organisations in South Australia. Eligibility included identifying as Indigenous, residing in South Australia, and being aged 18 years or older.

Main Outcomes and Measures  Saliva samples were collected, with microbial DNA for genotyping extracted. Sociodemographic parameters, health-related behaviors, and sexual history data were collected. Analyses were stratified by sex as well as by HPV types 13 and 32 (Heck disease) and 16 and 18 (high risk of oropharyngeal squamous cell carcinoma). Multivariable analyses were conducted to obtain adjusted odds ratios (ORs).

Results  Data were obtained for 910 participants (median [interquartile range] age, 37 [27-51] years); 595 participants (65%) were female and 572 (63%) resided in nonmetropolitan locations. In all, 321 saliva samples (35.3%; 95% CI, 32.2%-38.4%) were positive for oral HPV (106 [33.7%] men; 215 [36.1%] women). The highest prevalence was found for HPV types 13 and 32 (207 [22.7%] total; 60 [19.0%] men; 147 [24.7%] women) followed by HPV types 16 and 18 (30 [3.3%] total; 9 [2.9%] men; 21 [3.5%] women). After multivariable analysis, risk factors associated with HPV types 13 and 32 included nonmetropolitan residential status (OR, 2.06; 95% CI, 1.10-3.88) and not having had a tonsillectomy (OR, 2.74; 95% CI, 1.05-7.16). Among women, having obtained a high school education or less was associated with lower odds of HPV-16 and HPV-18 infection (OR, 0.16; 95% CI, 0.03-0.97).

Conclusions and Relevance  Prevalence of oral HPV infection in a large sample of Indigenous Australians was high, with one-third testing positive. The most prevalent HPV types were those associated with Heck disease. The prevalence of HPV types associated with oropharyngeal squamous cell carcinoma exceeded both Australian and international population-level estimates.

Introduction

Oropharyngeal squamous cell carcinoma (OPSCC) associated with human papillomavirus (HPV) disproportionately affects men and has one of the most rapidly increasing incidences of any cancer in high-income countries.1 The increased incidence is particularly noted among younger cohorts with minimal exposure to smoking and alcohol, the risk factors most commonly associated with OPSCC; the increased HPV incidence may be attributable to oral exposure to infected anogenital sites with changing sexual behaviors.2,3 Globally, the proportion of OPSCC attributable to HPV has been estimated as 23% to 31%; however, this varies by setting, particularly regarding exposure to HPV, tobacco, and alcohol.4,5 In Australia, Hong et al3 reported a more than 3-fold increase in the percentage of HPV-positive OPSCC in the last 2 decades, from 20% to 63%; HPV-16 is the most common type in HPV-positive OPSCC, although HPV-18 also plays a role.6 The 2 types together account for 85% of HPV-positive OPSCC (83% HPV-16; 2% HPV-18).5 In 2012, the incidence of OPSCC in men overtook that of cervical cancer in women in the United States,7 and similar findings were observed in the United Kingdom in 2016.8 Australian OPSCC incidence trends are in line with other high-income countries.3 While these countries have also experienced reduced rates of cervical cancer due to successful screening initiatives, the increase in OPSCC remains notable.

Focal epithelial hyperplasia, or Heck disease, is a relatively benign and rare condition caused by oral HPV types 13 or 32.9-12 It was first identified among a Navajo population in the United States13 and has since been reported among other indigenous groups throughout the world.14,15 Heck disease is characterized by multiple white or pink papules that occur diffusely throughout the oral cavity, with morphology that can present as slightly pale, smooth, or roughened surfaces. Although the papules will spontaneously regress without treatment over time, some patients opt for excisional biopsy for functional or cosmetic purposes.16

Aboriginal and Torres Strait Islanders (hereafter respectfully termed Indigenous) are the first peoples of Australia, having resided in the country for more than 50 000 years. Contemporary Indigenous Australians represent 3.3% of the total Australian population.17 They are overrepresented in almost all cancer statistics, including cancers of the head and neck.18 In an analysis of cancer registry data, Banham et al19 reported that in comparison with the general population of Australia, Indigenous individuals were 10 years younger at diagnosis, to be residents of geographically remote locations, and to have primary cancer sites of the head and neck, lung, liver, and cervix. In 2009 to 2013, Indigenous Australians were 1.9 times more likely to be diagnosed with head and neck cancer than non–Indigenous Australians, and were 3.4 times more likely to die.20 Risk of cancer death was associated with advanced stage at first observation, with more Indigenous than non-Indigenous individuals having distant metastases at diagnosis. Although HPV vaccine coverage across Indigenous adolescents in Australia is high, course completion is generally lower.21 Evidence suggests that while HPV infection in other anatomical sites is similar to non–Indigenous Australians, Indigenous Australians experience a higher prevalence of risk factors and other HPV genotypes.22

In a systematic review of 9 studies that collected oral HPV data from 3762 cancer-free, HIV-negative individuals from the United States, Brazil, Mexico, and Finland, Wood et al23 reported that 7.5 % (95% CI, 6.7%-8.4%) had an oral infection with any HPV type at baseline. In a study involving 307 Australian university students (aged 18-35 years), 7 students (6 men and 1 woman; 2.3%; 95% CI, 0.6%-3.9%) tested positive for oral HPV infection. Those positive for oral HPV were more likely to have received oral sex from more partners in their lifetime.24

Given the high risk of Indigenous Australians having both oral HPV infection and OPSCC and the potential benefits of HPV vaccination, the aims of this study were as follows: (1) to estimate the prevalence of oral HPV infection among Indigenous Australians; (2) to identify risk factors associated with Heck disease HPV types (HPV-13 and HPV-32); and (3) to identify risk factors associated with OPSCC-related HPV types (HPV-16 and HPV-18). We hypothesized that levels of any oral HPV infection, oral HPV infection associated with Heck disease (HPV-13 and HPV-32), and oral HPV infection associated with OPSCC (HPV-16 and HPV-18) among an Indigenous adult population would be higher than overall population estimates. We additionally hypothesized that risk factors for oral HPV infection would include male sex, social disadvantage, tobacco use, and early and frequent sexual activity.

Methods
Study Design and Participants

We used a large convenience sample (n = 1011) of adults aged 18 years or older who identified as being Indigenous in the Australian state of South Australia. Data were collected between February 2018 and January 2019 as part of a broader study investigating HPV and OPSCC among Indigenous Australians25 and were analyzed from October 2018 to July 2019. The study was governed by an Indigenous Reference Group, with data collected by trained Indigenous research officers. Participants were primarily recruited through Aboriginal Community Controlled Health Organisations, which were key stakeholders in the study. After having the study explained and signing informed consent forms, participants were asked to complete a questionnaire (with assistance from study staff if requested) that contained information on sociodemographic characteristics, health-related behaviors including tobacco and alcohol use, and sexual history. Participants then provided a saliva sample through spitting and dribbling that was collected in a commercially available kit (DNA Genotek Inc), from which microbial DNA was extracted for genotyping.

Ethical approval was received from the University of Adelaide Human Research Ethics Committee and the Aboriginal Health Council of South Australia’s Human Research Ethics Committee. This study follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies.

Self-reported Data

Sociodemographic characteristics included age, sex, geographic location, education, income, ownership of a means-tested government health care card, number of people in house the previous night, and car ownership. Health-related behaviors surveyed included tobacco, alcohol, nonprescription tobacco substitute (ie, vaporizer or e-cigarette), recreational drug use, having ever been diagnosed with an HPV infection (self-reported), HPV vaccination status (self-reported) and having had a tonsillectomy. Sexual behaviors across lifetime included number of people passionately kissed, having ever given oral sex, having ever received oral sex, having experienced sexual intercourse, and current relationship status. Age was dichotomized based on a median split. Geographic location was defined as metropolitan (ie, residing in Adelaide, South Australia’s capital city) and nonmetropolitan (residing elsewhere in the state). Highest educational attainment was categorized as high school or less and trade, TAFE, or university (TAFE stands for technical and further education and provides training for vocational occupations). Income was defined as job or Centrelink or other (Centrelink is the government agency responsible for means-tested welfare payments). Similarly, enrollment in government-administered health care card is means tested, and enables access to services such as publicly funded dental care. Number of people in house the previous night was used as a proxy measure of socioeconomic status (as used in studies conducted by the Australian Institute of Health and Welfare26) and was categorized according to whether the total number of people spending the previous night in the participant’s residence was 4 or more or less than 4. Tobacco smoking status was defined as currently smoking tobacco, formerly smoked, and never smoked, while response categories for alcohol consumption included daily, weekly, monthly, or never.

Laboratory Analysis

A viral kit for DNA extraction (Promega Maxwell) was used. β-globin polymerase chain reaction (PCR) with the primers PCO3 and PCO4 was carried out on all samples to ensure the presence of human DNA and that no PCR-inhibiting agents were present.27 All samples were analyzed with a nested PCR system (MY09/11) and GP5+/6+ that detects most mucosal HPV types and all high-risk HPV types that have oncogenic potential in mucosal tissue.28,29 All HPV-positive DNA samples were sequenced to confirm viral DNA sequences. For sequencing, HPV-positive PCR products were purified with a PCR purification kit in a magnetic 96-ring SPRIplate (Agencourt Biosciences). Sequencing reactions were performed containing the purified PCR products together with GP+ primer and BigDye Terminator. Sequence reactions were purified with a dye-terminator removal kit (Agencourt Biosciences) in a magnetic 96-ring SPRIplate. Direct sequencing was conducted, and sequence reactions were analyzed with an automated DNA sequencer (Applied Biosystems). The DNA sequences were compared with available sequences in GenBank through the National Center for Biotechnology Information BLASTn suite server.30 Participants with β-globin–positive saliva samples were included in the data analysis. (β-globin is a DNA integrity check; any samples with negative β-globin were invalid.)

Statistical Analysis

Basic descriptive analyses were conducted to ascertain frequencies of all HPV types detected and associations with sociodemographic data, health-related behaviors, and sexual history characteristics. Bivariate and multivariable logistic regression analyses were then conducted to identify risk factors for infection with HPV types associated with OPSCC (HPV-16 and HPV-18) and with Heck disease (HPV-13 and HPV-32). All data were stratified by sex. Differences were denoted to be statistically significant when 95% CIs did not overlap, and the χ2 P value in 2-tailed tests was less than .05. Odds ratios (ORs) with 95% CIs were calculated in the multivariable analyses. SAS version 9.4 (SAS Institute) was used for all analyses.

Results

A total of 910 β-globin–positive saliva samples were collected from Indigenous residents of South Australia aged 18 years or older. Among participants, the median (interquartile range [IQR]) age was 37 (27-51) years, with a median (IQR) age of 37 (27-49) years for men and 38 (27-52) years for women. More than half (51.6%; 95% CI, 48.4%-54.9%) of participants were aged 37 years or older (Table 1). Two-thirds (65.4%; 95% CI, 62.3%-68.5%) were women and 63% resided in nonmetropolitan locations (63.0%; 95% CI, 59.8%-66.1%). Overall, 68.0% (95% CI, 65.0%-71.1%) of participants reported a highest level of eductional attainment as high school or less, and more than three-quarters (76.0%; 95% CI, 73.2%-78.8%) received their income through Centrelink. More than one-third (36.7%; 95% CI, 33.5%-40.0%) of participants had 4 or more people in their house the previous night. Fifty-five percent (55.2%; 95% CI, 52.0%-58.5%) of participants owned their own car. Nearly 60% (58.2%; 95% CI, 54.9%-61.5%) currently smoked tobacco, and approximately 12% (11.6%; 95% CI, 9.5%-13.8%) reported currently smoking nonprescription tobacco substitutes. Nearly one-quarter (24.1%; 95% CI, 21.2%-26.9%) of participants consumed alcohol on a weekly basis, while approximately 21.1% (95% CI, 18.4%-23.7%) reported currently using recreational drugs. Two percent (2.0%; 95% CI, 1.1%-2.9%) of participants reported having ever had an HPV infection, but 17.1% (95% CI, 14.6%-19.6%) did not know. Approximately 8% (8.3%; 95% CI, 6.5%-10.1%) had been vaccinated against HPV (bearing in mind most participants would not have been eligible for free public HPV vaccination), although 34.1% (95% CI, 31.0%-37.2%) did not know their vaccination status. Approximately 13% (12.9%; 95% CI, 10.7%-15.1%) reported having had a tonsillectomy. Nearly two-thirds (65.4%; 95% CI, 62.1%-68.6%) of participants reported having passionately kissed 4 or more people, 64.6% (95% CI, 61.3%-67.9%) had given oral sex, and 64.7% (95% CI, 61.4%-68.0%) had received oral sex. Almost all (94.8%; 95% CI, 93.2%-96.3%) participants reported having had sexual intercourse, with 40% commencing sexual activity before the age of 16 years, with 64.0% of participants (95% CI, 60.5%-67.4%) reporting 4 or more sexual partners over their lifetime. Nearly all (93.6%; 95% CI, 92.0%-95.3%) participants reported sexual encounters with people predominantly of the opposite sex. More than half (50.9%; 95% CI, 47.4%-54.3%) were in current, stable long-term relationships.

More than one-third (35.3%) were positive for an oral HPV infection (Table 2); 33.7% of men and 36.1% of women. By far the most prevalent HPV types were those associated with Heck disease (HPV-13 and HPV-32) (22.8%; 19.1% in men and 24.7% in women). The next most prevalent HPV types were those associated with OPSCC (HPV-16 and HPV-18) (3.3%; 2.9% in men and 3.5% in women). A total of 38 HPV types were found, with the number of participants per type ranging from 1 to 119. There were no participants with multiple oral HPV types.

In bivariate analysis, we found an association between having any oral HPV type (Table 3) and receiving income through Centrelink compared with a job (prevalence, 37.5% [95% CI, 34.8%-41.1%] vs 28.7% [95% CI, 22.7%-34.7%]). In bivariate analysis, we found an association between having an HPV type associated with Heck disease and residing in a nonmetropolitan location compared with a metropolitan location (prevalence, 28.5% [95% CI, 24.8%-32.3%] vs 13.1% [9.5%-16.7%]), not owning a car compared with owning a car (prevalence, 28.0% [95% CI, 23.6%-32.4%] vs 18.5% [95% CI, 15.1%-21.9%]), having not had a tonsillectomy compared with having had a tonsillectomy (prevalence, 24.1% [95% CI, 20.9%-27.2%] vs 14.0% [95% CI, 7.6%-20.4%]), having never given oral sex compared with having given oral sex (prevalence, 27% [95% CI, 22.3%-32.6%] vs 18.7% [95% CI, 15.3%-22.0%]), having never received oral sex compared with having received oral sex (prevalence, 28.6% [95% CI, 23.3%-33.9%] vs 18.0% [95% CI, 14.6%-21.3%]), and having fewer than 4 sexual partners over a lifetime compared with having 4 or more sexual partners (prevalence, 28.1% [95% CI, 22.8%-33.5%] vs 18.4% [95% CI, 14.9%-21.8%]) (Table 4). In bivariate analysis, residing in a metropolitan location was associated with HPV types 16 and 18 compared with residing in a nonmetropolitan location (prevalence, 5.7% [95% CI, 3.1%-8.1%] vs 1.9% [95% CI, 0.8%-3.1%]) (Table 5). In multivariable analyses, the odds of oral HPV-13 or HPV-32 infection was over 2 times higher among those residing in a nonmetropolitan location compared with participants with metropolitan residence (OR, 2.06 [95% CI, 1.10-3.88]) and for participants who had not had a tonsillectomy compared with those who had received the procedure (OR, 2.74; 95% CI, 1.05-7.16) (Table 4). In multivariable analysis, the risk of HPV-16 or HPV-18 infection persisted among women with trade, TAFE, or university education (4.5%; 95% CI, 2.1%-6.9%) (Table 5).

Discussion

Consistent with our hypothesis, the prevalence of any oral HPV type, HPV types associated with Heck disease (HPV-13 and HPV-32), and HPV types associated with OPSCC (HPV-16 and HPV-18) among Indigenous Australians appeared to be higher than those reported both in other Australian studies and in populations from other countries. The prevalence of oral HPV in the current study was 15.3 times that reported in a study of young non–Indigenous Australians24 and 4.7 times that reported by Antonsson et al23 in a systematic review involving 9 studies from other countries.

The prevalence of HPV-13 or HPV-32 was 0 in the Australian study. The systematic review reported no prevalence estimates for HPV-13 or HPV-32, but it is unclear if that is because the prevalence was 0 or if these estimates were not analyzed. The prevalence of HPV types associated with OPSCC in the current study was 2.5 times that reported in the Australian study and 2.1 times that reported in the review of studies from other countries.

Our additional hypothesis that risk factors would include male sex, social disadvantage, tobacco use, and early and high levels of sexual activity proved only partially true. Indicators of social disadvantage were associated with Heck disease, but so were low rates of sexual activity. There were no apparent risk factors for HPV types associated with OPSCC aside from residing in a metropolitan location. The prevalence of HPV among those having given oral sex (64.6%) was much lower in the current study than in the second Australian Study of Health and Relationships (77.0%).31

The high levels of oral carriage of HPV in our study are concerning, particularly the high prevalence of HPV types associated with OPSCC. It is particularly interesting that the rates were higher among women, across both younger and older demographic characteristics. The findings speak to an urgent need to ensure high HPV vaccination coverage in Indigenous adolescents (although this does not prevent infection with HPV-13 and HPV-32), particularly given the evidence of HPV vaccine efficacy in decreasing the subsequent prevalence of oral HPV infection.32 Further research could assess the effectiveness and cost-effectiveness of immunization of those in older age groups (ie, those aged 20 years or older who are no longer eligible for free vaccination). Efforts to extend the benefits provided by vaccination would need to take into account the lower vaccine effectiveness among those already exposed to HPV and the long latent period between a causal HPV infection and invasive disease. Both the effectiveness of vaccination against persistent oral HPV infection at older ages and its cost-effectiveness are yet to be demonstrated. Australian adolescents aged up to 19 years can receive 2 doses of the HPV vaccine free of charge as part of the National HPV Vaccination Program, and HPV vaccination is also available in Australia for women aged 20 to 45 years and men aged 20 to 26 years but is not reimbursed in the public program. Evidence has shown that the level of elective uptake in women not eligible for vaccination through the publicly funded program in Australia is low (ie, 11%).33

The rates of oral carriage of HPV types associated with Heck disease in our study were higher than what has been presented in the literature to date. For example, in the Australian study among university students, there were no HPV-13 or HPV-32 types identified.24 Although factors that determine susceptibility for Heck disease are unclear, genetic susceptibility—especially concerning the human lymphocytic antigen (HLA)-DR4(DRB1*0404) allele (an allele occurring frequently in Indigenous populations of the Americas but with no documented reports among Indigenous Australians)—is thought to play a major role in vulnerability to HPV-13 and HPV-32.29 It is reassuring that HPV types associated with Heck disease are considered low risk (ie, they are not found in cancers), with spontaneous regression of clinical lesions during a mean of 18 months.34 It is perhaps unsurprising that associations with HPV-13 and HPV-32 in our study included low sexual activity, given that Heck disease is not associated with high sexual activity.34 The hypothesized mode of transition is horizontal (mouth-to-mouth), commencing early in infancy via the mother.35

The critical issue with high-risk oral HPV infections regarding OPSCC (or any HPV-related head and neck cancer) is persistent oral HPV infection. In a large cohort study of incidence and clearance of oral HPV among men who did not have HIV or anogenital cancer, Kreimer et al36 reported that during the first 12 months of follow-up, 4.4% of men acquired an incident oral HPV infection, with 1.7% of this 4.4% being an oncogenic HPV type and 0.6% of the 4.4% being HPV-16. Acquisition of oral oncogenic HPV was significantly associated with tobacco smoking and being single and was similar across included countries, age groups, and reported sexual behaviors. The median duration of infection was 6.9 months for any oral HPV, 6.3 months for oncogenic HPV, and 7.3 months for HPV-16. Eight of the 18 incident oral HPV-16 infections (44.4%) persisted for 6 or more months. The authors concluded that newly acquired oral oncogenic HPV infections in healthy men were rare and that most cleared within 1 year. The incidence, clearance, or persistence of oral HPV infections among Indigenous Australians remains unknown and is the subject of continuing research.

Limitations

This study has limitations. It did not include clinical dental examinations, which would have revealed any physical manifestations of both Heck disease and early-stage OPSCC. The study was not representative, with almost two-thirds of participants being women. Oral carriage of HPV is usually higher among men, and because only 35% of participants in our study were men, our findings may underestimate the true prevalence of oral HPV in the Indigenous population. The difficulties in recruiting men to health-related studies is widely documented,37,38 including in the Indigenous Australian context.39 We found no association between sex and HPV infection in our sample. Conversely, given that we found that higher HPV prevalence was associated with living in nonmetropolitan areas and that people living in these areas were overrepresented in our sample compared with the Indigenous population in South Australia, this may have overestimated prevalence. In direct comparisons with other studies, this study did not age match.

Conclusions

In this study, the overall prevalence of HPV detected in oral fluid in a large convenience sample of Indigenous Australians was high, with one-third demonstrating carriage on a single occasion. The most prevalent HPV types were those associated with Heck disease (HPV-13 and HPV-32). The next most prevalent were types most strongly associated with OPSCC (HPV-16 and HPV-18). Prevalence of these types appeared to exceed both Australian and international population-level estimates.

Back to top
Article Information

Accepted for Publication: February 6, 2020.

Published: June 8, 2020. doi:10.1001/jamanetworkopen.2020.4951

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 Jamieson LM et al. JAMA Network Open.

Corresponding Author: Lisa M. Jamieson, PhD, University of Adelaide, 57 North Terrace, Adelaide 5005, Australia (lisa.jamieson@adelaide.edu.au).

Author Contributions: Dr Jamieson had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Jamieson, Garvey, Logan, Johnson, Hedges, Dunbar, Leane, Hill, Brown, Canfell.

Acquisition, analysis, or interpretation of data: Jamieson, Antonsson, Ju, Smith, Logan, Johnson, Sethi, Dunbar, Roder, De Souza, Canfell.

Drafting of the manuscript: Jamieson, Ju, Hedges, Hill.

Critical revision of the manuscript for important intellectual content: Jamieson, Antonsson, Garvey, Ju, Smith, Logan, Johnson, Sethi, Dunbar, Leane, Brown, Roder, De Souza, Canfell.

Statistical analysis: Ju, Sethi.

Obtained funding: Jamieson, Garvey, Logan, Dunbar, Leane, Canfell.

Administrative, technical, or material support: Logan, Johnson, Sethi, Hill, Brown, Roder, De Souza.

Supervision: Jamieson, Antonsson, Johnson, Canfell.

Cultural advice and support: Leane.

Conflict of Interest Disclosures: Dr Canfell reported being co–principal investigator of an investigator-initiated trial of primary human papillomavirus screening in Australia (Compass) that is conducted and funded by the Victorian Cytology Service, a government-funded health promotion charity that has received funding contributions from Roche Molecular Systems and Ventana Inc. Neither Dr Canfell nor her institution on her behalf have received funding from industry for this or any other project. Dr Garvey reported receiving salary support from the National Health and Medical Research Council (NHMRC) during the conduct of the study. Dr Smith reported receiving salary support from NHMRC and grants from Cancer Institute New South Wales during the conduct of the study. Dr Logan reported salary support from NHMRC during the conduct of the study. Dr Johnson reported salary support from NHMRC during the conduct of the study. Dr Canfell reported salary support from NHMRC Australia during the conduct of the study, and grants from Roche Molecular Systems outside the submitted work. No other disclosures were reported.

Funding/Support: This study was funded by the Australia’s NHMRC under grant APP1120215.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: We thank the study participants, Indigenous Reference Group, staff who collected data, and key participating Aboriginal Community Controlled Health Organisations.

Additional Information: The original data, including all clinical and epidemiological data, used in this work will be made available upon request. Requests should be directed to the corresponding author.

References
1.
Lechner  M, Breeze  CE, O’Mahony  JF, Masterson  L.  Early detection of HPV-associated oropharyngeal cancer.   Lancet. 2019;393(10186):2123. doi:10.1016/S0140-6736(19)30227-2 PubMedGoogle ScholarCrossref
2.
D’Souza  G, Kreimer  AR, Viscidi  R,  et al.  Case-control study of human papillomavirus and oropharyngeal cancer.   N Engl J Med. 2007;356(19):1944-1956. doi:10.1056/NEJMoa065497 PubMedGoogle ScholarCrossref
3.
Hong  A, Lee  CS, Jones  D,  et al.  Rising prevalence of human papillomavirus-related oropharyngeal cancer in Australia over the last 2 decades.   Head Neck. 2016;38(5):743-750. doi:10.1002/hed.23942 PubMedGoogle ScholarCrossref
4.
Plummer  M, de Martel  C, Vignat  J, Ferlay  J, Bray  F, Franceschi  S.  Global burden of cancers attributable to infections in 2012: a synthetic analysis.   Lancet Glob Health. 2016;4(9):e609-e616. doi:10.1016/S2214-109X(16)30143-7 PubMedGoogle ScholarCrossref
5.
Castellsagué  X, Alemany  L, Quer  M,  et al; ICO International HPV in Head and Neck Cancer Study Group.  HPV involvement in head and neck cancers: comprehensive assessment of biomarkers in 3680 patients.   J Natl Cancer Inst. 2016;108(6):djv403. doi:10.1093/jnci/djv403 PubMedGoogle Scholar
6.
Yakin  M, Seo  B, Hussaini  H, Rich  A, Hunter  K.  Human papillomavirus and oral and oropharyngeal carcinoma: the essentials.   Aust Dent J. 2019;64(1):11-18. doi:10.1111/adj.12652 PubMedGoogle ScholarCrossref
7.
National Cancer Institute. Annual report to the nation on the status of cancer. Accessed September 30, 2019. https://seer.cancer.gov/report_to_nation/
8.
Office for National Statistics. Cancer registration statistics, England: first release, 2016. Accessed September 30, 2019. https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/cancerregistrationstatisticsengland/2016
9.
Henke  RP, Guèrin-Reverchon  I, Milde-Langosch  K, Koppang  HS, Löning  T.  In situ detection of human papillomavirus types 13 and 32 in focal epithelial hyperplasia of the oral mucosa.   J Oral Pathol Med. 1989;18(7):419-421. doi:10.1111/j.1600-0714.1989.tb01575.x PubMedGoogle ScholarCrossref
10.
Jayasooriya  PR, Abeyratne  S, Ranasinghe  AW, Tilakaratne  WM.  Focal epithelial hyperplasia (Heck’s disease): report of two cases with PCR detection of human papillomavirus DNA.   Oral Dis. 2004;10(4):240-243. doi:10.1111/j.1601-0825.2004.01012.x PubMedGoogle ScholarCrossref
11.
Bassioukas  K, Danielides  V, Georgiou  I, Photos  E, Zagorianakou  P, Skevas  A.  Oral focal epithelial hyperplasia.   Eur J Dermatol. 2000;10(5):395-397.PubMedGoogle Scholar
12.
Ozden  B, Gunduz  K, Gunhan  O, Ozden  FO.  A case report of focal epithelial hyperplasia (Heck’s disease) with PCR detection of human papillomavirus.   J Maxillofac Oral Surg. 2011;10(4):357-360. doi:10.1007/s12663-011-0184-2 PubMedGoogle ScholarCrossref
13.
Archard  HO, Heck  JW, Stanley  HR.  Focal epithelial hyperplasia: an unusual oral mucosal lesion found in Indian children.   Oral Surg Oral Med Oral Pathol. 1965;20:201-212. doi:10.1016/0030-4220(65)90192-1 PubMedGoogle ScholarCrossref
14.
González  LV, Gaviria  AM, Sanclemente  G,  et al.  Clinical, histopathological and virological findings in patients with focal epithelial hyperplasia from Colombia.   Int J Dermatol. 2005;44(4):274-279. doi:10.1111/j.1365-4632.2005.02321.x PubMedGoogle ScholarCrossref
15.
Wu  JSA, Florian  MC, Rodrigues  DA, Tomimori  J.  Skin diseases in indigenous population: retrospective epidemiological study at Xingu Indigenous Park (XIP) and review of the literature.   Int J Dermatol. 2017;56(12):1414-1420. doi:10.1111/ijd.13716 PubMedGoogle ScholarCrossref
16.
Eversole  LR.  Clinical Outline of Oral Pathology: Diagnosis and Treatment. People’s Medical Publishing House; 2011.
17.
Australian Institute of Health and Welfare.  Australia’s Health, 2018. Australian Institute of Health and Welfare; 2018.
18.
Cancer Australia.  Aboriginal and Torres Strait Islander Cancer Statistics. Australian Government; 2018.
19.
Banham  D, Roder  D, Keefe  D,  et al; CanDAD Aboriginal Community Reference Group and other CanDAD investigators.  Disparities in cancer stage at diagnosis and survival of Aboriginal and non–Aboriginal South Australians.   Cancer Epidemiol. 2017;48:131-139. doi:10.1016/j.canep.2017.04.013 PubMedGoogle ScholarCrossref
20.
Australian Institute of Health and Welfare.  Cancer in Aboriginal and Torres Strait Islander people of Australia. Australian Institute of Health and Welfare; 2018.
21.
Brotherton  JM, Winch  KL, Chappell  G,  et al.  HPV vaccination coverage and course completion rates for Indigenous Australian adolescents, 2015.   Med J Aust. 2019;211(1):31-36. doi:10.5694/mja2.50221 PubMedGoogle ScholarCrossref
22.
Garland  SM, Brotherton  JM, Condon  JR,  et al; WHINURS study group.  Human papillomavirus prevalence among Indigenous and non-Indigenous Australian women prior to a national HPV vaccination program.   BMC Med. 2011;9:104. doi:10.1186/1741-7015-9-104 PubMedGoogle ScholarCrossref
23.
Wood  ZC, Bain  CJ, Smith  DD, Whiteman  DC, Antonsson  A.  Oral human papillomavirus infection incidence and clearance: a systematic review of the literature.   J Gen Virol. 2017;98(4):519-526. doi:10.1099/jgv.0.000727 PubMedGoogle ScholarCrossref
24.
Antonsson  A, Cornford  M, Perry  S, Davis  M, Dunne  MP, Whiteman  DC.  Prevalence and risk factors for oral HPV infection in young Australians.   PLoS One. 2014;9(3):e91761. doi:10.1371/journal.pone.0091761 PubMedGoogle Scholar
25.
Jamieson  L, Garvey  G, Hedges  J,  et al.  Human papillomavirus and oropharyngeal cancer among Indigenous Australians: protocol for a prevalence study of oral-related human papillomavirus and cost-effectiveness of prevention.   JMIR Res Protoc. 2018;7(6):e10503. doi:10.2196/10503 PubMedGoogle Scholar
26.
Australian Institute of Health and Welfare.  Housing Circumstances of Indigenous Households: Tenure and Overcrowding. Australian Institute of Health and Welfare; 2014.
27.
de Roda Husman  AM, Walboomers  JMM, Hopman  E,  et al.  HPV prevalence in cytomorphologically normal cervical scrapes of pregnant women as determined by PCR: the age-related pattern.   J Med Virol. 1995;46(2):97-102. doi:10.1002/jmv.1890460203 PubMedGoogle ScholarCrossref
28.
Manos  MM, Ting  Y, Wright  DK, Lewis  AJ, Broker  TR.  The use of polymerase chain reaction amplification for the detection of genital human papillomaviruses.   Cancer Cell. 1989;7:209-214.Google Scholar
29.
García-Corona  C, Vega-Memije  E, Mosqueda-Taylor  A,  et al.  Association of HLA-DR4 (DRB1*0404) with human papillomavirus infection in patients with focal epithelial hyperplasia.   Arch Dermatol. 2004;140(10):1227-1231. doi:10.1001/archderm.140.10.1227 PubMedGoogle ScholarCrossref
30.
National Center for Biotechnology Information.  BLAST: basic local alignment search tool. Accessed April 17, 2020. https://blast.ncbi.nlm.nih.gov/Blast.cgi
31.
Rissel  C, Badcock  PB, Smith  AM,  et al.  Heterosexual experience and recent heterosexual encounters among Australian adults: the Second Australian Study of Health and Relationships.   Sex Health. 2014;11(5):416-426. doi:10.1071/SH14105 PubMedGoogle ScholarCrossref
32.
Wierzbicka  M, Berkhof  JH, Dikkers  FG.  Prophylactic human papilloma virus vaccination in head and neck: indications and future perspectives.   Curr Opin Otolaryngol Head Neck Surg. 2019;27(2):85-90. doi:10.1097/MOO.0000000000000525 PubMedGoogle ScholarCrossref
33.
Mazza  D, Petrovic  K, Chakraborty  S.  HPV vaccination of adult women: an audit of Australian general practitioners.   Aust N Z J Obstet Gynaecol. 2012;52(6):528-533. doi:10.1111/ajo.12002 PubMedGoogle ScholarCrossref
34.
Bennett  LK, Hinshaw  M.  Heck’s disease: diagnosis and susceptibility.   Pediatr Dermatol. 2009;26(1):87-89. doi:10.1111/j.1525-1470.2008.00830.x PubMedGoogle ScholarCrossref
35.
Syrjänen  S.  Oral manifestations of human papillomavirus infections.   Eur J Oral Sci. 2018;126(suppl 1):49-66. doi:10.1111/eos.12538 PubMedGoogle ScholarCrossref
36.
Kreimer  AR, Pierce Campbell  CM, Lin  HY,  et al.  Incidence and clearance of oral human papillomavirus infection in men: the HIM cohort study.   Lancet. 2013;382(9895):877-887. doi:10.1016/S0140-6736(13)60809-0 PubMedGoogle ScholarCrossref
37.
Rounds  T, Harvey  J.  Enrollment challenges: recruiting men to weight loss interventions.   Am J Mens Health. 2019;13(1):1557988319832120. doi:10.1177/1557988319832120 PubMedGoogle Scholar
38.
Buston  K.  Recruiting, retaining and engaging men in social interventions: lessons for implementation focusing on a prison-based parenting intervention for young incarcerated fathers.   Child Care Pract. 2018;24(2):164-180. doi:10.1080/13575279.2017.1420034 PubMedGoogle ScholarCrossref
39.
Isaacs  A, Pepper  H, Pyett  P, Gruis  H, Waples-Crowe  P, Oakley Browne  M.  What you do is important but how you do it is more important.   Qual Res J. 2011;11:51-61. doi:10.3316/QRJ1101051 Google ScholarCrossref
×