[Skip to Content]
[Skip to Content Landing]
Figure 1.
CONSORT Diagram
CONSORT Diagram

We identified 63 789 women who were between the ages of 30 and 64 years, had not had a hysterectomy, had a primary care clinician at Kaiser Permanente Washington (KPWA), and had been continuously enrolled in the health system for at least 3 years and 5 months. Of these women, 17 256 (27.1%) had not had a Papanicolaou test within 3 years and 5 months. Postrandomization exclusions were applied owing to delays in data warehouse update. The total included in the intervention group analysis (9960) equals the total included in the round 1 analysis group (8120) plus the total included in the round 2 analysis group (1631) plus the total included in the round 3 analysis group (209). Similarly, the total included in the control group analysis (9891) equals the total included in the round 1 analysis group (8111) plus the total included in the round 2 analysis group (1585) plus the total included in the round 3 analysis group (195).

Figure 2.
Diagram of Cervical Cancer Screening, Diagnosis, and Treatment Outcomes
Diagram of Cervical Cancer Screening, Diagnosis, and Treatment Outcomes

CIN2+ indicates positive results for cervical intraepithelial neoplasia grade 2 or higher; HPV, human papillomavirus; hrHPV, high-risk HPV; Unsat, unsatisfactory.

aSelf-sampling other hrHPV+ only or unsatisfactory results required in-clinic follow-up to complete screening uptake.

bOf these 20 women, 15 went directly to colposcopy and 5 went to Papanicolaou test or cotest before colposcopy.

cThese 5 women completed screening uptake by receiving colposcopy in the 6 month screening window.

dThis 1 woman did not complete screening uptake because colposcopy was completed outside the 6-month screening window.

eIncludes 2 women with home kit other hrHPV+ and Papanicolaou test with unknown result. Because unknown Papanicolaou test results were coded as normal for analysis, the final result is considered other hrHPV+ and Papanicolaou test negative. One of these women received a colposcopy.

fFollow-up recommendations were per national guidelines.22

Figure 3.
Cumulative Incidence of Screening Uptake by Allocation and Screening Modality Within the Intervention Group
Cumulative Incidence of Screening Uptake by Allocation and Screening Modality Within the Intervention Group
Table 1.  
Baseline Characteristics of Intervention and Control Group Participants
Baseline Characteristics of Intervention and Control Group Participants
Table 2.  
Screening, Diagnosis, and Treatment Outcomes for the Intervention vs Control Group
Screening, Diagnosis, and Treatment Outcomes for the Intervention vs Control Group
1.
Sabatino  SA, White  MC, Thompson  TD, Klabunde  CN; Centers for Disease Control and Prevention (CDC).  Cancer screening test use—United States, 2013.  MMWR Morb Mortal Wkly Rep. 2015;64(17):464-468.PubMedGoogle Scholar
2.
National Committee for Quality Assurance. Cervical cancer screening (CCS). https://www.ncqa.org/hedis/measures/cervical-cancer-screening/. Published 2018. Accessed April 1, 2019.
3.
White  A, Thompson  TD, White  MC,  et al.  Cancer screening test use—United States, 2015.  MMWR Morb Mortal Wkly Rep. 2017;66(8):201-206. doi:10.15585/mmwr.mm6608a1PubMedGoogle ScholarCrossref
4.
Oscarsson  MG, Benzein  EG, Wijma  BE.  Reasons for non-attendance at cervical screening as reported by non-attendees in Sweden.  J Psychosom Obstet Gynaecol. 2008;29(1):23-31. doi:10.1080/01674820701504619PubMedGoogle ScholarCrossref
5.
Glasgow  RE, Whitlock  EP, Valanis  BG, Vogt  TM.  Barriers to mammography and Pap smear screening among women who recently had neither, one or both types of screening.  Ann Behav Med. 2000;22(3):223-228. doi:10.1007/BF02895117PubMedGoogle ScholarCrossref
6.
Eaker  S, Adami  HO, Sparén  P.  Reasons women do not attend screening for cervical cancer: a population-based study in Sweden.  Prev Med. 2001;32(6):482-491. doi:10.1006/pmed.2001.0844PubMedGoogle ScholarCrossref
7.
Waller  J, Bartoszek  M, Marlow  L, Wardle  J.  Barriers to cervical cancer screening attendance in England: a population-based survey.  J Med Screen. 2009;16(4):199-204. doi:10.1258/jms.2009.009073PubMedGoogle ScholarCrossref
8.
Goins  KV, Zapka  JG, Geiger  AM,  et al.  Implementation of systems strategies for breast and cervical cancer screening services in health maintenance organizations.  Am J Manag Care. 2003;9(11):745-755.PubMedGoogle Scholar
9.
Leyden  WA, Manos  MM, Geiger  AM,  et al.  Cervical cancer in women with comprehensive health care access: attributable factors in the screening process.  J Natl Cancer Inst. 2005;97(9):675-683. doi:10.1093/jnci/dji115PubMedGoogle ScholarCrossref
10.
Saslow  D, Solomon  D, Lawson  HW,  et al; American Cancer Society; American Society for Colposcopy and Cervical Pathology; American Society for Clinical Pathology.  American Cancer Society, American Society for Colposcopy and Cervical Pathology, and American Society for Clinical Pathology screening guidelines for the prevention and early detection of cervical cancer.  Am J Clin Pathol. 2012;137(4):516-542. doi:10.1309/AJCPTGD94EVRSJCGPubMedGoogle ScholarCrossref
11.
Centers for Disease Control and Prevention. Cancers associated with human papillomavirus, United States— 2012–2016: US Cancer Statistics data brief no 10. https://www.cdc.gov/cancer/uscs/pdf/USCS-DataBrief-No10-August2019-h.pdf. Accessed September 25, 2019.
12.
Kinney  W, Sung  HY, Kearney  KA, Miller  M, Sawaya  G, Hiatt  RA.  Missed opportunities for cervical cancer screening of HMO members developing invasive cervical cancer (ICC).  Gynecol Oncol. 1998;71(3):428-430. doi:10.1006/gyno.1998.5135PubMedGoogle ScholarCrossref
13.
Janerich  DT, Hadjimichael  O, Schwartz  PE,  et al.  The screening histories of women with invasive cervical cancer, Connecticut.  Am J Public Health. 1995;85(6):791-794. doi:10.2105/AJPH.85.6.791PubMedGoogle ScholarCrossref
14.
Curry  SJ, Krist  AH, Owens  DK,  et al; US Preventive Services Task Force.  Screening for cervical cancer: US Preventive Services Task Force Recommendation Statement.  JAMA. 2018;320(7):674-686. doi:10.1001/jama.2018.10897PubMedGoogle ScholarCrossref
15.
Arbyn  M, Smith  SB, Temin  S, Sultana  F, Castle  P; Collaboration on Self-Sampling and HPV Testing.  Detecting cervical precancer and reaching underscreened women by using HPV testing on self samples: updated meta-analyses.  BMJ. 2018;363:k4823. doi:10.1136/bmj.k4823PubMedGoogle ScholarCrossref
16.
Polman  NJ, Ebisch  RMF, Heideman  DAM,  et al.  Performance of human papillomavirus testing on self-collected versus clinician-collected samples for the detection of cervical intraepithelial neoplasia of grade 2 or worse: a randomised, paired screen-positive, non-inferiority trial.  Lancet Oncol. 2019;20(2):229-238. doi:10.1016/S1470-2045(18)30763-0PubMedGoogle ScholarCrossref
17.
Arbyn  M, Castle  PE.  Offering self-sampling kits for HPV testing to reach women who do not attend in the regular cervical cancer screening program.  Cancer Epidemiol Biomarkers Prev. 2015;24(5):769-772. doi:10.1158/1055-9965.EPI-14-1417PubMedGoogle ScholarCrossref
18.
Smith  M, Lew  JB, Simms  K, Canfell  K.  Impact of HPV sample self-collection for underscreened women in the renewed cervical screening program.  Med J Aust. 2016;204(5):1941e-7. doi:10.5694/mja15.00912PubMedGoogle Scholar
19.
Winer  RL, Tiro  JA, Miglioretti  DL,  et al.  Rationale and design of the HOME trial: a pragmatic randomized controlled trial of home-based human papillomavirus (HPV) self-sampling for increasing cervical cancer screening uptake and effectiveness in a U.S. healthcare system.  Contemp Clin Trials. 2018;64:77-87. doi:10.1016/j.cct.2017.11.004PubMedGoogle ScholarCrossref
20.
Loudon  K, Treweek  S, Sullivan  F, Donnan  P, Thorpe  KE, Zwarenstein  M.  The PRECIS-2 tool: designing trials that are fit for purpose.  BMJ. 2015;350:h2147. doi:10.1136/bmj.h2147PubMedGoogle ScholarCrossref
21.
Moyer  VA; US Preventive Services Task Force.  Screening for cervical cancer: U.S. Preventive Services Task Force recommendation statement.  Ann Intern Med. 2012;156(12):880-891, W312. doi:10.7326/0003-4819-156-12-201206190-00424PubMedGoogle ScholarCrossref
22.
Massad  LS, Einstein  MH, Huh  WK,  et al; 2012 ASCCP Consensus Guidelines Conference.  2012 Updated consensus guidelines for the management of abnormal cervical cancer screening tests and cancer precursors.  J Low Genit Tract Dis. 2013;17(5)(suppl 1):S1-S27. doi:10.1097/LGT.0b013e318287d329PubMedGoogle ScholarCrossref
23.
Huh  WK, Ault  KA, Chelmow  D,  et al.  Use of primary high-risk human papillomavirus testing for cervical cancer screening: interim clinical guidance.  Gynecol Oncol. 2015;136(2):178-182. doi:10.1016/j.ygyno.2014.12.022PubMedGoogle ScholarCrossref
24.
Deyo  RA, Cherkin  DC, Ciol  MA.  Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.  J Clin Epidemiol. 1992;45(6):613-619. doi:10.1016/0895-4356(92)90133-8PubMedGoogle ScholarCrossref
25.
Bais  AG, van Kemenade  FJ, Berkhof  J,  et al.  Human papillomavirus testing on self-sampled cervicovaginal brushes: an effective alternative to protect nonresponders in cervical screening programs.  Int J Cancer. 2007;120(7):1505-1510. doi:10.1002/ijc.22484PubMedGoogle ScholarCrossref
26.
Gök  M, Heideman  DA, van Kemenade  FJ,  et al.  HPV testing on self collected cervicovaginal lavage specimens as screening method for women who do not attend cervical screening: cohort study.  BMJ. 2010;340:c1040. doi:10.1136/bmj.c1040PubMedGoogle ScholarCrossref
27.
Virtanen  A, Nieminen  P, Luostarinen  T, Anttila  A.  Self-sample HPV tests as an intervention for nonattendees of cervical cancer screening in Finland: a randomized trial.  Cancer Epidemiol Biomarkers Prev. 2011;20(9):1960-1969. doi:10.1158/1055-9965.EPI-11-0307PubMedGoogle ScholarCrossref
28.
Gyllensten  U, Sanner  K, Gustavsson  I, Lindell  M, Wikström  I, Wilander  E.  Short-time repeat high-risk HPV testing by self-sampling for screening of cervical cancer.  Br J Cancer. 2011;105(5):694-697. doi:10.1038/bjc.2011.277PubMedGoogle ScholarCrossref
29.
Castle  PE, Fetterman  B, Poitras  N, Lorey  T, Shaber  R, Kinney  W.  Five-year experience of human papillomavirus DNA and Papanicolaou test cotesting.  Obstet Gynecol. 2009;113(3):595-600. doi:10.1097/AOG.0b013e3181996ffaPubMedGoogle ScholarCrossref
30.
Castle  PE, Wacholder  S, Sherman  ME,  et al.  Absolute risk of a subsequent abnormal pap among oncogenic human papillomavirus DNA-positive, cytologically negative women.  Cancer. 2002;95(10):2145-2151. doi:10.1002/cncr.10927PubMedGoogle ScholarCrossref
31.
Sultana  F, English  DR, Simpson  JA,  et al.  Home-based HPV self-sampling improves participation by never-screened and under-screened women: results from a large randomized trial (iPap) in Australia.  Int J Cancer. 2016;139(2):281-290. doi:10.1002/ijc.30031PubMedGoogle ScholarCrossref
32.
Castle  P, Feldman  S, Perkins  RB.  The next generation of cervical cancer screening: should guidelines focus on best practices for the future or current screening capacity?  J Low Genit Tract Dis. 2018;22(2):91-96. doi:10.1097/LGT.0000000000000378PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Obstetrics and Gynecology
    November 6, 2019

    Effect of Mailed Human Papillomavirus Test Kits vs Usual Care Reminders on Cervical Cancer Screening Uptake, Precancer Detection, and Treatment: A Randomized Clinical Trial

    Author Affiliations
    • 1Department of Epidemiology, University of Washington, Seattle
    • 2Kaiser Permanente Washington Health Research Institute, Seattle
    • 3Department of Clinical Sciences, University of Texas Southwestern Medical Center, Dallas
    • 4Division of Biostatistics, Department of Public Health Sciences, University of California, Davis
    • 5Kaiser Permanente Washington, Renton
    JAMA Netw Open. 2019;2(11):e1914729. doi:10.1001/jamanetworkopen.2019.14729
    Key Points español 中文 (chinese)

    Question  Do mailed human papillomavirus self-sampling kits increase detection and treatment of cervical precancers and screening uptake vs usual care (reminders for in-clinic screening)?

    Findings  This randomized clinical trial included 19 851 women; 26% were screened after receiving a human papillomavirus kit vs 17% with usual care, a significant difference. There was no statistically significant difference in the number of cases of precancers detected or treated.

    Meaning  This study indicates that mailing human papillomavirus kits to underscreened women can increase cervical cancer screening, and implementation efforts should strategize how to further increase kit uptake and follow-up of positive results to maximize detection and treatment of precancers in women at high risk.

    Abstract

    Importance  In the United States, more than 50% of cervical cancers are diagnosed in underscreened women. Cervical cancer screening guidelines now include primary human papillomavirus (HPV) testing as a recommended strategy. Home-based HPV self-sampling is a viable option for increasing screening compliance and effectiveness; however, US data are needed to inform health care system implementation.

    Objective  To evaluate effectiveness of mailed HPV self-sampling kits vs usual care reminders for in-clinic screening to increase detection and treatment of cervical intraepithelial neoplasia grade 2 or worse (CIN2+) and uptake of cervical cancer screening.

    Design, Setting, and Participants  Randomized clinical trial conducted in Kaiser Permanente Washington, a US integrated health care delivery system. Women aged 30 to 64 years with health plan enrollment for 3 years and 5 months or more, a primary care clinician, no Papanicolaou test within 3 years and 5 months, and no hysterectomy were identified through electronic medical records and enrolled from February 25, 2014, to August 29, 2016, with follow-up through February 26, 2018.

    Interventions  The control group received usual care (annual patient reminders and ad hoc outreach from primary care clinics). The intervention group received usual care plus a mailed HPV self-sampling kit.

    Main Outcomes and Measures  Two primary outcomes were (1) CIN2+ detection within 6 months of screening and (2) treatment within 6 months of CIN2+ detection. Screening uptake within 6 months of randomization was a secondary outcome.

    Results  A total of 19 851 women (mean [SD] age, 50.1 [9.5] years) were included, with 9960 randomized to the intervention group and 9891 randomized to the control group. All women randomized were included in analysis. In the intervention group, 12 participants with CIN2+ were detected compared with 8 in the control group (relative risk, 1.49; 95% CI, 0.61-3.64) and 12 cases were treated vs 7 in the control group (relative risk, 1.70; 95% CI, 0.67-4.32). Screening uptake was higher in the intervention group (2618 participants [26.3%] vs 1719 participants [17.4%]; relative risk, 1.51; 95% CI, 1.43-1.60).

    Conclusions and Relevance  Mailing HPV kits to underscreened women increased screening uptake compared with usual care alone, with no significant differences in precancer detection or treatment. Results support the feasibility of mailing HPV kits to women who are overdue for screening as an outreach strategy to increase screening uptake in US health care systems. Efforts to increase kit uptake and follow-up of positive results are warranted to maximize detection and treatment of CIN2+.

    Trial Registration  ClinicalTrials.gov identifier: NCT02005510

    Introduction

    In the United States, 25% of women delay or forego recommended cervical cancer screening.1-3 Well-documented barriers include lack of time or transportation, difficulties finding childcare or taking time off work, fear of pelvic examinations, and prior negative experiences with screening.4-9 Reducing underscreening is a key prevention priority,10 as more than 50% of the 12 000 cervical cancers diagnosed annually are in underscreened women.9,11-13

    In 2018, the US Preventive Services Task Force released updated cervical cancer screening guidelines14 that include 3 recommended options for women aged 30 to 65 years: Papanicolaou testing alone, Papanicolaou and human papillomavirus (HPV) cotesting, and primary HPV-only screening (a new strategy). With primary HPV screening, home-based screening is an emerging option because HPV tests (unlike Papanicolaou tests) can be performed on clinician- or self-collected samples with comparable sensitivity.15,16 Furthermore, population-based randomized clinical trials in countries with organized screening programs (ie, centrally designed and managed, with standardized screening invitations and follow-up for a specific target population) demonstrated that mailing HPV self-sampling kits to underscreened women increased participation compared with invitations for clinic-based screening15 and diagnostic follow-up compliance after HPV-positive self-sampling results was high,15 yielding increased detection of cervical intraepithelial neoplasia grade 2 or higher (CIN2+).15 Consequently, Australia and the Netherlands—the first countries to implement primary HPV screening—have included HPV self-sampling options for underscreened women.17,18

    Data from settings without organized screening programs—including opportunistic and/or health system–based screening programs in the United States—are needed to evaluate whether home-based HPV self-sampling increases screening participation and effectiveness. The Home-Based Options to Make Cervical Cancer Screening Easy (HOME) trial19 compared a programmatic strategy of mailed HPV self-sampling kits with usual care (outreach via patient reminders to attend in-clinic screening) for increasing detection and treatment of cervical precancers and uptake of cervical cancer screening. The HOME study is the first US trial, to our knowledge, to evaluate effectiveness of HPV self-sampling as a screening method by assessing the entire cervical cancer prevention process (diagnostic follow-up and treatment) and measuring precancer outcomes. To our knowledge, it is also the first HPV self-sampling trial in any setting to evaluate treated precancer cases as an outcome.

    Methods

    The HOME study was a parallel, investigator-blinded, randomized clinical trial comparing 2 programmatic strategies for improving screening effectiveness and uptake in women who were not adherent to routine Papanicolaou screening. The trial was fully embedded within Kaiser Permanente Washington (KPWA) (see the article by Winer et al19 for details about pragmatic aspects according to Pragmatic-Explanatory Continuum Indicator Summary–2 [PRECIS-2] criteria20) and was approved by KPWA and University of Washington institutional review boards. The trial followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline (Figure 1). The trial protocol is available in Supplement 1.

    We identified potentially eligible women using electronic medical record (EMR) data; all eligible women were enrolled under a waiver of informed consent. No incentives were offered for participation. The institutional review boards determined that enrolling women under a waiver of informed consent was minimal risk compared with the scientific benefit of reducing participation bias. Women became eligible 5 months after receiving an annual preventive care reminder letter indicating they were due or overdue for their Papanicolaou screening (ie, no prior screening or most recent screening >3 years prior) to allow women to achieve screening uptake without additional intervention and to ensure that the 6-month time frame for assessing screening uptake ended before the next year’s preventive care reminder letter was sent. Eligible women (1) were aged 30 to 64 years; (2) had not had a hysterectomy; (3) had a KPWA primary care clinician; (4) had been continuously enrolled for 3 years and 5 months or longer; and (5) had not had a Papanicolaou test within 3 years and 5 months. (Although screening guidelines recommend extending the routine screening interval from 3 to 5 years with Papanicolaou and HPV cotesting,21 none of the women would have been on a 5-year screening interval because cotesting was rarely performed at KPWA prior to August 2013.) Women who previously indicated they did not want to be contacted for research studies, had a pregnancy-related procedure or diagnosis code within 3 months or less, or had an “interpreter needed” flag (kit materials were in English) in the EMR were excluded.

    Participants were randomly allocated 1:1 to the intervention or control group between February 25, 2014, and August 29, 2016. To meet our sample size goals during the study timeline, 1 year after randomization, control group participants were reassessed for eligibility and rerandomization.

    Women in the control group received usual care outreach protocols to attend Papanicolaou screening. Outreach includes the birthday letter reminder, clinician-targeted automatic alerts for overdue women that persist until a Papanicolaou test is ordered or the alert is overridden, and primary care team communication (varies across clinics). Women randomized to usual care did not receive the mailed HPV kit or contact from the study team.

    Women in the intervention group received usual care plus a mailed HPV self-sampling kit with a prepaid return envelope addressed to the KPWA clinical laboratory. Mailings included an invitation letter, research information sheet, and educational materials on how to self-collect and return a sample. Because HPV self-screening is not standard of care in the United States, the letter advised women to receive routine Papanicolaou testing regardless of whether they chose to complete HPV self-sampling. Women were informed that participation was voluntary and provided with a telephone number to call with questions or to opt out of having their individual-level medical record data used for research. To mirror KPWA’s prevention outreach protocols, if the kit was not returned within 3 weeks, study staff conducted up to 3 reminder calls. Samples were tested with the US Food and Drug Administration–approved Cobas 4800 HPV Test (Roche Diagnostics). Results were documented in the EMR and women’s primary care teams managed results communication and follow-up care. Women with negative or unsatisfactory results were advised to attend in-clinic Papanicolaou testing or cotesting. Women positive for high-risk HPV types other than HPV-16 or HPV-18 were advised to receive in-clinic cotesting. Women positive for HPV-16 or HPV-18 were recommended for immediate colposcopy, per the American Society for Colposcopy and Cervical Pathology management guidelines.22 Standardized protocols were developed by clinical and system-level collaborators to educate primary care teams on recommended follow-up (published previously19).

    We used programmatic extraction and manual EMR review to identify 2 primary outcomes: histologically diagnosed CIN2+ and treated CIN2+. The 2 primary outcomes were selected based on clinical relevance. We did not adjust for multiple comparisons or have an order or gatekeeping strategy for testing given that these 2 outcomes are strongly correlated with each other (CIN2+ treatment cannot occur without a CIN2+ diagnosis), and it was highly unlikely that only 1 would be significant. To count as an outcome, CIN2+ had to be preceded by an abnormal screening result within 6 months after randomization or rerandomization. Diagnosed CIN2+ was captured within 6 months after an abnormal screening result, and treated CIN2+ was captured within 6 months after a CIN2+ diagnosis. With these windows, each woman was followed up for a maximum of 18 months after randomization.

    Secondary outcomes included screening uptake and abnormal screening results, obtained from the EMR. Uptake was captured within 6 months after randomization or rerandomization and defined as (1) receiving a Papanicolaou screening or cotest; (2) self-sampling HPV-16 or HPV-18 positive (regardless of any subsequent in-clinic follow-up, because recommended management is to proceed directly to diagnostic evaluation23); (3) self-sampling HPV negative (regardless of any subsequent in-clinic follow-up, based on the expectation that programs implementing HPV self-sampling would not require in-clinic follow-up of negative results); or (4) self-sampling HPV positive for types other than HPV-16 or HPV-18 only or unsatisfactory and receiving a Papanicolaou screening or cotest. Abnormal screening results were captured within 6 months after randomization and defined as a result warranting repeated testing, surveillance, or immediate colposcopy before returning to routine screening. In addition to the prespecified secondary outcomes, we also measured screening initiation, defined as either receiving a Papanicolaou screening or cotest or self-sampling (without the requirement to follow up in-clinic after self-sampling HPV-positive for types other than HPV-16 or HPV-18 only or unsatisfactory).

    Participant characteristics at randomization (age, race, ethnicity, health plan enrollment duration, time since last Papanicolaou test, census block median household income, travel time to primary care clinic, body mass index, tobacco use, and Charlson Comorbidity Index score24) were derived from EMR data. Outcome data were available for all participants, including women who did not return a kit or opted out of medical record review (details of data access and aggregation for these groups were published previously19).

    The HOME trial was powered on the primary outcomes of diagnosed and treated CIN2+ assuming a 2-sided α of .05 and used assumptions based on the literature25-30 and 2012 preliminary data.19 We estimated we would identify approximately 17 600 eligible women over 2.5 years and would have 85% power to detect between-group differences in proportions with diagnosed CIN2+ (0.188% vs 0.036% [relative risk {RR}, 5.2] in the intervention and control groups, respectively) and 81% power for treated CIN2+ (0.170% vs 0.033% [RR, 5.2] in the intervention and control groups, respectively). We estimated 100% power to detect differences in abnormal screening results (1.09% vs 0.44% [RR, 2.5] in the intervention and control groups, respectively) and screening uptake (35.9% vs 8.8% [RR, 4.1] in the intervention and control groups, respectively).

    Statistical Analysis

    We analyzed data using the intention-to-treat principle. Denominators for each group included all women randomized, minus the small number of women identified as ineligible after randomization (Figure 1). We described the distribution of participant characteristics by randomization group to identify any imbalances warranting adjustment in regression models. Outcome proportions in the intervention group were compared with the usual care group and RRs estimated using log-binomial regression. Absolute risk differences were estimated using binomial regression with an identity link. Robust variance estimates were used to account for within-participant correlation due to rerandomized participants contributing more than 1 observation period. Statistical significance was defined as a 2-sided P < .05.

    As a prespecified exploratory analysis, positive predictive value of an abnormal screening test for detecting CIN2+ was estimated within each randomization group. Two different denominators were evaluated: women with abnormal screening results warranting colposcopic referral and women receiving colposcopy after an abnormal result.

    In a post hoc analysis, we used Kaplan-Meier methods and log-rank tests to compare time from randomization to screening uptake in the control vs intervention group. Time to screening uptake was also compared among the control group vs intervention group women who returned HPV kits vs intervention group women with in-clinic screening only. Analyses were conducted using SAS statistical software version 9.4 (SAS Institute).

    Results

    From February 25, 2014, to August 29, 2016, 16 590 women were randomized to the intervention group (n = 8283) or the control group (n = 8307) (Figure 1). From March 23, 2015, to August 29, 2016, 3231 women who were initially randomized to the control group and still eligible 1 year after randomization were rerandomized; between March 21, 2016, and August 29, 2016, 409 additional control group women were rerandomized a second time. Each rerandomization was treated as a distinct observation in the analyses. Therefore, the total number randomized was 20 230; 379 were retroactively excluded, and the remaining 19 851 (mean [SD] age, 50.1 [9.5] years) were included in the intention-to-treat analysis (9960 in the intervention group, 9891 in the control group). Baseline characteristics were similar between groups (Table 1).

    Primary Outcomes

    Twelve patients with CIN2+ (0.12%) were detected in the intervention group (including 2 cases in women who returned a kit) vs 8 (0.08%) in the control group (Figure 2 and Table 2). The RR for CIN2+ detection was 1.49 (95% CI, 0.61-3.64) in the intervention vs control group. All women with CIN2+ (0.12%) were treated in the intervention group vs 7 in the control group (0.07%); the RR for CIN2+ treatment was 1.70 (95% CI, 0.67-4.32) in the intervention vs control group.

    Prespecified Secondary Outcomes

    Within the intervention group, 2618 women (26.3%) completed screening uptake vs 1719 (17.4%) in the control group. The RR for screening uptake was 1.51 (95% CI, 1.43-1.60) in the intervention vs the control group; absolute difference was 8.9% (95% CI, 7.8%-10.0%) (Figure 2 and Table 2). In the intervention group, 225 women (2.3%) received abnormal screening results vs 114 (1.2%) in the control group; abnormal screening RR was 1.96 (95% CI, 1.57-2.45) in the intervention vs control group.

    Within the intervention group, 2646 women (26.6%) initiated screening; 1206 women (12.1%) initiated their screening episode by returning an HPV kit and 1440 women (14.5%) went directly to in-clinic screening (Figure 2). Among the 1206 women who returned a kit, 1098 attained screening uptake based on their HPV kit results alone, including 1064 (88.2%) with HPV-negative results and 34 (2.8%) with HPV-16– or HPV-18–positive results (Figure 2; eFigure in Supplement 2). The remaining 108 patients (9.0%) had kit results that were positive only for types other than HPV-16 or HPV-18 (8.5%) or unsatisfactory (0.5%) and required in-clinic follow-up to complete screening uptake; 80 of these 108 women (74.1%) attained screening uptake by attending in-clinic follow-up and 28 (25.9%) did not (Figure 2).

    Additional Analyses

    Four women returned nonviable HPV kits, which were not tested (including 2 received beyond maximum specimen stability of 21 days after the collection date and 2 with visible stool in the collection tube). All 4 were notified and offered a replacement kit; 3 returned a second kit within 6 months of randomization, received HPV-negative results, and were counted in the screening uptake numerator. Additionally, 47 of the 1440 intervention group women (3.3%) who went directly to in-clinic screening subsequently returned an HPV kit; these HPV kit results did not affect primary or secondary outcomes.

    Time to screening uptake was shorter in the intervention group vs the control group (log-rank P < .001) (Figure 3). Restricting to women who completed screening uptake, the median (interquartile range) number of days to screening uptake was 80 (43-129) for women in the control group and 46 (29-93) for women in the intervention group. Results for intervention group women who went directly to in-clinic screening (median [interquartile range], 78 [37-126] days) were similar to results for women in the control group, whereas time to screening uptake was shorter among women who returned an HPV kit (median [interquartile range], 37 [22-49] days). In all, 27.5% of kits were returned within 21 days (before reminder call initiation).

    The number of abnormal screening results warranting colposcopic referral was similar in the intervention group (82 referrals, representing 36.4% of abnormal screening results) vs the control group (42 referrals, representing 36.8% of abnormal screening results) (Figure 2). Among those with results warranting colposcopy, 55 (67.1%) in the intervention group received colposcopy vs 35 (83.3%) in the control group. Within the intervention group, the proportion of abnormal results warranting colposcopy was similar in women who returned kits (38 of 106 [35.8%]) vs women who went directly to in-clinic screening (44 of 119 [37.0%]), and the number correctly receiving colposcopy was 24 of 38 (63.2%) in the kit returner group vs 31 of 44 (70.5%) in the direct to in-clinic screening group. Additional subgroup details, including colposcopies received against study recommendations or American Society for Colposcopy and Cervical Pathology management guidelines, are depicted in Figure 2.

    Within the intervention group, the CIN2+ positive predictive value was 14.6% (95% CI, 7.0%-22.3%) (12 of 82) among women with abnormal screening results warranting colposcopy and 21.8% (95% CI, 10.9%-32.7%) (12 of 55) when restricting the denominator to women who received colposcopy. The positive predictive value for CIN2+ was 19.0% (95% CI, 7.2%-30.9%) (8 of 42) in women in the control group warranting colposcopy and 22.9% (95% CI, 9.0%-36.8%) (8 of 35) restricting to those who received colposcopy.

    No unexpected and 5 expected19 (2 light bleeding; 3 discomfort) adverse events were self-reported.

    Discussion

    In this randomized clinical trial, there were no significant differences in CIN2+ detection or treatment among underscreened women who received a mailed HPV self-sampling kit compared with usual care alone. However, mailing HPV kits significantly increased screening uptake and reduced time to screening compared with usual care.

    We are unaware of other HPV self-sampling trials that evaluated treated CIN2+ cases as an outcome. However, many international trials have evaluated the effect of mailing HPV kits to underscreened women on CIN2+ detection, with mixed results.15 High-risk HPV–positive self-sampling triage strategies vary across settings and influence CIN2+ detection rates.15 Detection of CIN2+ was higher comparing HPV self-sampling with control groups in trials in which women were referred directly to colposcopy (pooled RR including 6 trials, 3.0; 95% CI, 1.9-4.8) compared with trials that triaged women to additional screening (usually cytology) (pooled RR including 7 trials, 1.8; 95% CI, 0.8-4.0).15 One Australian trial triaged HPV-16 and HPV-18–positive results directly to colposcopy and other high-risk-positive results to Papanicolaou testing (similar to this trial); 76% received appropriate follow-up after an HPV-positive self-sampling result, and no significant between-group differences in CIN2+ detection were reported (although the trial was not powered on this outcome).15,31 Although our trial was powered on CIN2+ detection and treatment outcomes, power calculations were based on assumptions of higher kit uptake (30%), lower control group screening uptake (9%), and higher follow-up adherence (particularly for HPV-positive kit results [89%]) than was observed.19 Additionally, HPV positivity (11.3%) was higher than estimated (7.5%), but comparable to that reported in the most recent meta-analysis (pooled positivity, 11%; 95% CI, 10%-12%).15

    Among the intervention women who self-sampled, most with nonnegative results attended in-clinic follow-up (70%); however, only 59% with HPV-16 or HPV-18–positive results (highest risk for CIN2+) attended recommended colposcopy, and 74% with other high-risk HPV–positive or unsatisfactory results attended in-clinic follow-up. Our study protocol deployed several implementation strategies to encourage appropriate follow-up of abnormal results, including (1) specific guidance for clinicians in electronic test results, (2) monitoring communications between the clinician and patients of the recommended follow-up plan, and (3) sending electronic messages to clinicians when follow-up plans were not communicated. These strategies were also part of trial safety monitoring. Cervical cancer screening guidelines in the United States are complex and dynamic; for health care systems to successfully incorporate new screening modalities such as HPV self-sampling, additional resources to support screening, diagnosis, and treatment completion may be warranted.32

    In a meta-analysis of 19 trials in countries with organized screening programs, 25% of underscreened women were screened (HPV kit or Papanicolaou test) after receiving an unsolicited HPV self-sampling kit in the mail,15 similar to the 26% uptake in the intervention group. However, in this trial, slightly more women who screened in the intervention group chose in-clinic screening (14%) over self-sampling (12%), whereas in the meta-analysis, more women who were mailed a kit chose self-sampling (19%) over in-clinic screening (6%).15 The absolute (9%) and relative (1.5) increases observed in screening uptake in the mailed HPV kit vs usual care alone group in our trial were within the lower range of estimates from international trials. The pooled absolute difference in the meta-analysis was 13% (95% CI, 10%-15%), and the pooled RR was 2.3 (95% CI, 1.9-2.9).15 Further investigation of US clinician and patient knowledge and attitudes are needed to understand why uptake of mailed HPV kit screening was lower than that observed internationally.

    The HOME trial was strengthened by the pragmatic design, with direct integration of the intervention into existing clinical protocols. Data from EMRs were used to identify and randomize women who would be eligible for the intervention if it were adopted into clinical practice, thereby enhancing generalizability of the results. Results from the post hoc time-to-screen analysis add context to the findings; the rapid time to screening uptake associated with mailed kits is an additional benefit from a health system resource perspective.

    Limitations

    This study has several limitations. A key difference between HOME and similar international trials was the need to consider the impact of the HPV self-sampling intervention on Healthcare Effectiveness Data and Information Set quality metrics.2 During the trial, only completed Papanicolaou tests counted toward the Healthcare Effectiveness Data and Information Set cervical cancer screening outcome. This directly affected the trial design in 2 important ways. First, to allow enough time for the delivery system to activate overdue women, it was necessary to delay mailing HPV kits for 5 months after the most recent annual preventive services reminder letter. Second, kit materials recommended all women attend Papanicolaou screening, even with HPV-negative self-sampling results, which may have negatively affected kit uptake. Furthermore, mailing kits as part of a research study may have had either a positive or negative effect on kit uptake. Results may have been different if (as in most international studies) HPV kits had been mailed to underscreened women as part of standard care and with the reassurance that negative results would not require in-clinic follow-up. An additional limitation was that kit materials were developed in English only, necessitating the exclusion of non–English-speaking women.

    Conclusions

    Mailing HPV kits to underscreened women increased screening uptake compared with usual care alone, with no significant differences in clinical outcomes. Nearly three-quarters of women who received the intervention remained underscreened, and follow-up among self-testing HPV-positive women has room for improvement. Health care systems in the United States considering implementing primary HPV screening and outreach strategies with HPV self-sampling should focus on approaches to increase kit uptake and follow-up of positive results. Additional research efforts should consider a variety of strategies to increase cervical cancer screening.

    Back to top
    Article Information

    Accepted for Publication: September 17, 2019.

    Published: November 6, 2019. doi:10.1001/jamanetworkopen.2019.14729

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Winer RL et al. JAMA Network Open.

    Corresponding Author: Rachel L. Winer, PhD, MPH, University of Washington HPV Research Group, 325 Ninth Ave, PO Box 359933, Seattle, WA 98104 (rlw@uw.edu).

    Author Contributions: Drs Winer and Buist had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Winer, Tiro, Miglioretti, Thayer, Buist.

    Acquisition, analysis, or interpretation of data: Winer, Lin, Miglioretti, Beatty, Gao, Kimbel, Buist.

    Drafting of the manuscript: Winer.

    Critical revision of the manuscript for important intellectual content: All authors.

    Statistical analysis: Lin, Miglioretti, Gao.

    Obtained funding: Winer, Tiro, Buist.

    Administrative, technical, or material support: Lin, Beatty, Kimbel, Thayer, Buist.

    Supervision: Winer, Buist.

    Conflict of Interest Disclosures: Dr Winer reported receiving grants from the National Institutes of Health and Centers for Disease Control and Prevention outside the submitted work. Dr Tiro reported receiving grants from the National Institutes of Health, Lyda Hill Philanthropies, the Agency for Healthcare Research and Quality, and the Cancer Prevention & Research Institute of Texas outside the submitted work; as well as membership on the American Cancer Society's National HPV Vaccination Roundtable and attendance at St Jude's HPV Vaccine Strategy Workshop. No other disclosures were reported.

    Funding/Support: This work was supported by the National Cancer Institute of the National Institutes of Health (grant R01CA168598).

    Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Meeting Presentation: This study was presented in part at EUROGIN 2017; October 9, 2017; Amsterdam, the Netherlands; the 42nd annual American Society of Preventive Oncology conference; March 12, 2018; New York, New York; the 32nd International Papillomavirus Conference; October 4, 2018; Sydney, Australia; and the 2019 International Cancer Screening Network Conference; June 4, 2019; Rotterdam, the Netherlands.

    Data Sharing Statement: See Supplement 3.

    References
    1.
    Sabatino  SA, White  MC, Thompson  TD, Klabunde  CN; Centers for Disease Control and Prevention (CDC).  Cancer screening test use—United States, 2013.  MMWR Morb Mortal Wkly Rep. 2015;64(17):464-468.PubMedGoogle Scholar
    2.
    National Committee for Quality Assurance. Cervical cancer screening (CCS). https://www.ncqa.org/hedis/measures/cervical-cancer-screening/. Published 2018. Accessed April 1, 2019.
    3.
    White  A, Thompson  TD, White  MC,  et al.  Cancer screening test use—United States, 2015.  MMWR Morb Mortal Wkly Rep. 2017;66(8):201-206. doi:10.15585/mmwr.mm6608a1PubMedGoogle ScholarCrossref
    4.
    Oscarsson  MG, Benzein  EG, Wijma  BE.  Reasons for non-attendance at cervical screening as reported by non-attendees in Sweden.  J Psychosom Obstet Gynaecol. 2008;29(1):23-31. doi:10.1080/01674820701504619PubMedGoogle ScholarCrossref
    5.
    Glasgow  RE, Whitlock  EP, Valanis  BG, Vogt  TM.  Barriers to mammography and Pap smear screening among women who recently had neither, one or both types of screening.  Ann Behav Med. 2000;22(3):223-228. doi:10.1007/BF02895117PubMedGoogle ScholarCrossref
    6.
    Eaker  S, Adami  HO, Sparén  P.  Reasons women do not attend screening for cervical cancer: a population-based study in Sweden.  Prev Med. 2001;32(6):482-491. doi:10.1006/pmed.2001.0844PubMedGoogle ScholarCrossref
    7.
    Waller  J, Bartoszek  M, Marlow  L, Wardle  J.  Barriers to cervical cancer screening attendance in England: a population-based survey.  J Med Screen. 2009;16(4):199-204. doi:10.1258/jms.2009.009073PubMedGoogle ScholarCrossref
    8.
    Goins  KV, Zapka  JG, Geiger  AM,  et al.  Implementation of systems strategies for breast and cervical cancer screening services in health maintenance organizations.  Am J Manag Care. 2003;9(11):745-755.PubMedGoogle Scholar
    9.
    Leyden  WA, Manos  MM, Geiger  AM,  et al.  Cervical cancer in women with comprehensive health care access: attributable factors in the screening process.  J Natl Cancer Inst. 2005;97(9):675-683. doi:10.1093/jnci/dji115PubMedGoogle ScholarCrossref
    10.
    Saslow  D, Solomon  D, Lawson  HW,  et al; American Cancer Society; American Society for Colposcopy and Cervical Pathology; American Society for Clinical Pathology.  American Cancer Society, American Society for Colposcopy and Cervical Pathology, and American Society for Clinical Pathology screening guidelines for the prevention and early detection of cervical cancer.  Am J Clin Pathol. 2012;137(4):516-542. doi:10.1309/AJCPTGD94EVRSJCGPubMedGoogle ScholarCrossref
    11.
    Centers for Disease Control and Prevention. Cancers associated with human papillomavirus, United States— 2012–2016: US Cancer Statistics data brief no 10. https://www.cdc.gov/cancer/uscs/pdf/USCS-DataBrief-No10-August2019-h.pdf. Accessed September 25, 2019.
    12.
    Kinney  W, Sung  HY, Kearney  KA, Miller  M, Sawaya  G, Hiatt  RA.  Missed opportunities for cervical cancer screening of HMO members developing invasive cervical cancer (ICC).  Gynecol Oncol. 1998;71(3):428-430. doi:10.1006/gyno.1998.5135PubMedGoogle ScholarCrossref
    13.
    Janerich  DT, Hadjimichael  O, Schwartz  PE,  et al.  The screening histories of women with invasive cervical cancer, Connecticut.  Am J Public Health. 1995;85(6):791-794. doi:10.2105/AJPH.85.6.791PubMedGoogle ScholarCrossref
    14.
    Curry  SJ, Krist  AH, Owens  DK,  et al; US Preventive Services Task Force.  Screening for cervical cancer: US Preventive Services Task Force Recommendation Statement.  JAMA. 2018;320(7):674-686. doi:10.1001/jama.2018.10897PubMedGoogle ScholarCrossref
    15.
    Arbyn  M, Smith  SB, Temin  S, Sultana  F, Castle  P; Collaboration on Self-Sampling and HPV Testing.  Detecting cervical precancer and reaching underscreened women by using HPV testing on self samples: updated meta-analyses.  BMJ. 2018;363:k4823. doi:10.1136/bmj.k4823PubMedGoogle ScholarCrossref
    16.
    Polman  NJ, Ebisch  RMF, Heideman  DAM,  et al.  Performance of human papillomavirus testing on self-collected versus clinician-collected samples for the detection of cervical intraepithelial neoplasia of grade 2 or worse: a randomised, paired screen-positive, non-inferiority trial.  Lancet Oncol. 2019;20(2):229-238. doi:10.1016/S1470-2045(18)30763-0PubMedGoogle ScholarCrossref
    17.
    Arbyn  M, Castle  PE.  Offering self-sampling kits for HPV testing to reach women who do not attend in the regular cervical cancer screening program.  Cancer Epidemiol Biomarkers Prev. 2015;24(5):769-772. doi:10.1158/1055-9965.EPI-14-1417PubMedGoogle ScholarCrossref
    18.
    Smith  M, Lew  JB, Simms  K, Canfell  K.  Impact of HPV sample self-collection for underscreened women in the renewed cervical screening program.  Med J Aust. 2016;204(5):1941e-7. doi:10.5694/mja15.00912PubMedGoogle Scholar
    19.
    Winer  RL, Tiro  JA, Miglioretti  DL,  et al.  Rationale and design of the HOME trial: a pragmatic randomized controlled trial of home-based human papillomavirus (HPV) self-sampling for increasing cervical cancer screening uptake and effectiveness in a U.S. healthcare system.  Contemp Clin Trials. 2018;64:77-87. doi:10.1016/j.cct.2017.11.004PubMedGoogle ScholarCrossref
    20.
    Loudon  K, Treweek  S, Sullivan  F, Donnan  P, Thorpe  KE, Zwarenstein  M.  The PRECIS-2 tool: designing trials that are fit for purpose.  BMJ. 2015;350:h2147. doi:10.1136/bmj.h2147PubMedGoogle ScholarCrossref
    21.
    Moyer  VA; US Preventive Services Task Force.  Screening for cervical cancer: U.S. Preventive Services Task Force recommendation statement.  Ann Intern Med. 2012;156(12):880-891, W312. doi:10.7326/0003-4819-156-12-201206190-00424PubMedGoogle ScholarCrossref
    22.
    Massad  LS, Einstein  MH, Huh  WK,  et al; 2012 ASCCP Consensus Guidelines Conference.  2012 Updated consensus guidelines for the management of abnormal cervical cancer screening tests and cancer precursors.  J Low Genit Tract Dis. 2013;17(5)(suppl 1):S1-S27. doi:10.1097/LGT.0b013e318287d329PubMedGoogle ScholarCrossref
    23.
    Huh  WK, Ault  KA, Chelmow  D,  et al.  Use of primary high-risk human papillomavirus testing for cervical cancer screening: interim clinical guidance.  Gynecol Oncol. 2015;136(2):178-182. doi:10.1016/j.ygyno.2014.12.022PubMedGoogle ScholarCrossref
    24.
    Deyo  RA, Cherkin  DC, Ciol  MA.  Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases.  J Clin Epidemiol. 1992;45(6):613-619. doi:10.1016/0895-4356(92)90133-8PubMedGoogle ScholarCrossref
    25.
    Bais  AG, van Kemenade  FJ, Berkhof  J,  et al.  Human papillomavirus testing on self-sampled cervicovaginal brushes: an effective alternative to protect nonresponders in cervical screening programs.  Int J Cancer. 2007;120(7):1505-1510. doi:10.1002/ijc.22484PubMedGoogle ScholarCrossref
    26.
    Gök  M, Heideman  DA, van Kemenade  FJ,  et al.  HPV testing on self collected cervicovaginal lavage specimens as screening method for women who do not attend cervical screening: cohort study.  BMJ. 2010;340:c1040. doi:10.1136/bmj.c1040PubMedGoogle ScholarCrossref
    27.
    Virtanen  A, Nieminen  P, Luostarinen  T, Anttila  A.  Self-sample HPV tests as an intervention for nonattendees of cervical cancer screening in Finland: a randomized trial.  Cancer Epidemiol Biomarkers Prev. 2011;20(9):1960-1969. doi:10.1158/1055-9965.EPI-11-0307PubMedGoogle ScholarCrossref
    28.
    Gyllensten  U, Sanner  K, Gustavsson  I, Lindell  M, Wikström  I, Wilander  E.  Short-time repeat high-risk HPV testing by self-sampling for screening of cervical cancer.  Br J Cancer. 2011;105(5):694-697. doi:10.1038/bjc.2011.277PubMedGoogle ScholarCrossref
    29.
    Castle  PE, Fetterman  B, Poitras  N, Lorey  T, Shaber  R, Kinney  W.  Five-year experience of human papillomavirus DNA and Papanicolaou test cotesting.  Obstet Gynecol. 2009;113(3):595-600. doi:10.1097/AOG.0b013e3181996ffaPubMedGoogle ScholarCrossref
    30.
    Castle  PE, Wacholder  S, Sherman  ME,  et al.  Absolute risk of a subsequent abnormal pap among oncogenic human papillomavirus DNA-positive, cytologically negative women.  Cancer. 2002;95(10):2145-2151. doi:10.1002/cncr.10927PubMedGoogle ScholarCrossref
    31.
    Sultana  F, English  DR, Simpson  JA,  et al.  Home-based HPV self-sampling improves participation by never-screened and under-screened women: results from a large randomized trial (iPap) in Australia.  Int J Cancer. 2016;139(2):281-290. doi:10.1002/ijc.30031PubMedGoogle ScholarCrossref
    32.
    Castle  P, Feldman  S, Perkins  RB.  The next generation of cervical cancer screening: should guidelines focus on best practices for the future or current screening capacity?  J Low Genit Tract Dis. 2018;22(2):91-96. doi:10.1097/LGT.0000000000000378PubMedGoogle ScholarCrossref
    ×