Efficacy of Smartphone Applications for Smoking Cessation: A Randomized Clinical Trial | Lifestyle Behaviors | JAMA Internal Medicine | JAMA Network
[Skip to Navigation]
Figure.  CONSORT Diagram for iCanQuit Trial
CONSORT Diagram for iCanQuit Trial

IP indicates internet protocol; PIN, personal identification number.

aTo increase enrollment of racial/ethnic minorities and men, some nonminorities and women who were otherwise eligible for study enrollment were randomly selected to be excluded.

Table 1.  Major Differences Between iCanQuit and QuitGuide
Major Differences Between iCanQuit and QuitGuide
Table 2.  Baseline Demographics and Smoking Behavior
Baseline Demographics and Smoking Behavior
Table 3.  Smoking Cessation Outcomes by Follow-up Time Pointa
Smoking Cessation Outcomes by Follow-up Time Pointa
Table 4.  Treatment Engagement and Satisfactiona
Treatment Engagement and Satisfactiona
1.
GBD 2015 Risk Factors Collaborators.  Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015.   Lancet. 2016;388(10053):1659-1724. doi:10.1016/S0140-6736(16)31679-8 PubMedGoogle ScholarCrossref
2.
GBD 2015 Tobacco Collaborators.  Smoking prevalence and attributable disease burden in 195 countries and territories, 1990-2015: a systematic analysis from the Global Burden of Disease Study 2015.   Lancet. 2017;389(10082):1885-1906. doi:10.1016/S0140-6736(17)30819-X PubMedGoogle ScholarCrossref
3.
Husten  CG.  A call for ACTTION: increasing access to tobacco-use treatment in our nation.   Am J Prev Med. 2010;38(3)(suppl):S414-S417. doi:10.1016/j.amepre.2009.12.006 PubMedGoogle ScholarCrossref
4.
Whittaker  R, McRobbie  H, Bullen  C, Rodgers  A, Gu  Y, Dobson  R.  Mobile phone text messaging and app-based interventions for smoking cessation.   Cochrane Database Syst Rev. 2019;10:CD006611. doi:10.1002/14651858.CD006611.pub5 PubMedGoogle Scholar
5.
Pew Research Center. Mobile fact sheet. Published June 12, 2019. Accessed August 14, 2020. https://www.pewresearch.org/internet/fact-sheet/mobile/
6.
Fiore  MC, Jaén  CR, Baker  TB,  et al  Treating Tobacco Use and Dependence: 2008 Update: Clinical Practice Guideline. Rockville, MD: US Department of Health and Human Services, Public Health Service, 2008.
7.
Hayes  SC, Levin  ME, Plumb-Vilardaga  J, Villatte  JL, Pistorello  J.  Acceptance and commitment therapy and contextual behavioral science: examining the progress of a distinctive model of behavioral and cognitive therapy.   Behav Ther. 2013;44(2):180-198. doi:10.1016/j.beth.2009.08.002 PubMedGoogle ScholarCrossref
8.
Hernández-López  M, Luciano  MC, Bricker  JB, Roales-Nieto  JG, Montesinos  F.  Acceptance and commitment therapy for smoking cessation: a preliminary study of its effectiveness in comparison with cognitive behavioral therapy.   Psychol Addict Behav. 2009;23(4):723-730. doi:10.1037/a0017632 PubMedGoogle ScholarCrossref
9.
Bricker  J, Wyszynski  C, Comstock  B, Heffner  JL.  Pilot randomized controlled trial of web-based acceptance and commitment therapy for smoking cessation.   Nicotine Tob Res. 2013;15(10):1756-1764. doi:10.1093/ntr/ntt056 PubMedGoogle ScholarCrossref
10.
Bricker  JB, Mull  KE, McClure  JB, Watson  NL, Heffner  JL.  Improving quit rates of web-delivered interventions for smoking cessation: full-scale randomized trial of WebQuit.org versus Smokefree.gov.   Addiction. 2018;113(5):914-923. doi:10.1111/add.14127PubMedGoogle ScholarCrossref
11.
Bricker  JB, Mull  KE, Kientz  JA,  et al.  Randomized, controlled pilot trial of a smartphone app for smoking cessation using acceptance and commitment therapy.   Drug Alcohol Depend. 2014;143:87-94. doi:10.1016/j.drugalcdep.2014.07.006 PubMedGoogle ScholarCrossref
12.
Koçak  ND, Eren  A, Boğa  S,  et al.  Relapse rate and factors related to relapse in a 1-year follow-up of subjects participating in a smoking cessation program.   Respir Care. 2015;60(12):1796-1803. doi:10.4187/respcare.03883 PubMedGoogle ScholarCrossref
13.
Killeen  PR.  Markov model of smoking cessation.   Proc Natl Acad Sci U S A. 2011;108(suppl 3):15549-15556. doi:10.1073/pnas.1011277108 PubMedGoogle ScholarCrossref
14.
Herd  N, Borland  R.  The natural history of quitting smoking: findings from the International Tobacco Control (ITC) Four Country Survey.   Addiction. 2009;104(12):2075-2087. doi:10.1111/j.1360-0443.2009.02731.x PubMedGoogle ScholarCrossref
15.
Ferguson  J, Bauld  L, Chesterman  J, Judge  K.  The English smoking treatment services: one-year outcomes.   Addiction. 2005;100(suppl 2):59-69. doi:10.1111/j.1360-0443.2005.01028.x PubMedGoogle ScholarCrossref
16.
Radloff  LS.  The CES-D scale: a self-report depression scale for research in the general population.   Appl Psychol Meas. 1977;1(3):385-401. doi:10.1177/014662167700100306 Google ScholarCrossref
17.
Apple App Store. iCanQuit: experimental arm. Accessed April 7, 2020. https://apps.apple.com/us/app/icanquit/id1205729317?app=itunes&ign-mpt=uo%3D4
18.
Google Play Store. iCanQuit: experimental arm. Accessed April 7, 2020. https://play.google.com/store/apps/details?id=org.fredhutch.icanquitr
19.
Apple App Store. iCanQuit: comparison arm. Accessed April 7, 2020. https://apps.apple.com/us/app/icanquit/id1205729312?app=itunes&ign-mpt=uo%3D4
20.
Google Play Store. iCanQuit: comparison arm. Accessed April 7, 2020. https://play.google.com/store/apps/details?id=org.fredhutch.icanquitc
21.
Shields  PG, Herbst  RS, Arenberg  D,  et al.  Smoking cessation, version 1.2016, NCCN Clinical Practice Guidelines in Oncology.   J Natl Compr Canc Netw. 2016;14(11):1430-1468. doi:10.6004/jnccn.2016.0152 PubMedGoogle ScholarCrossref
22.
Roy  M, Dum  M, Sobell  LC,  et al.  Comparison of the Quick Drinking Screen and the alcohol Timeline Followback with outpatient alcohol abusers.   Subst Use Misuse. 2008;43(14):2116-2123. doi:10.1080/10826080802347586 PubMedGoogle ScholarCrossref
23.
Heatherton  TF, Kozlowski  LT, Frecker  RC, Fagerström  KO.  The Fagerström Test for Nicotine Dependence: a revision of the Fagerström Tolerance Questionnaire.   Br J Addict. 1991;86(9):1119-1127. doi:10.1111/j.1360-0443.1991.tb01879.x PubMedGoogle ScholarCrossref
24.
Hedeker  D, Mermelstein  RJ, Demirtas  H.  Analysis of binary outcomes with missing data: missing = smoking, last observation carried forward, and a little multiple imputation.   Addiction. 2007;102(10):1564-1573. doi:10.1111/j.1360-0443.2007.01946.x PubMedGoogle ScholarCrossref
25.
Nelson  DB, Partin  MR, Fu  SS, Joseph  AM, An  LC.  Why assigning ongoing tobacco use is not necessarily a conservative approach to handling missing tobacco cessation outcomes.   Nicotine Tob Res. 2009;11(1):77-83. doi:10.1093/ntr/ntn013 PubMedGoogle ScholarCrossref
26.
Blankers  M, Smit  ES, van der Pol  P, de Vries  H, Hoving  C, van Laar  M.  The missing=smoking assumption: a fallacy in internet-based smoking cessation trials?   Nicotine Tob Res. 2016;18(1):25-33. doi:10.1093/ntr/ntv055PubMedGoogle Scholar
27.
Kernan  WN, Viscoli  CM, Makuch  RW, Brass  LM, Horwitz  RI.  Stratified randomization for clinical trials.   J Clin Epidemiol. 1999;52(1):19-26. doi:10.1016/S0895-4356(98)00138-3 PubMedGoogle ScholarCrossref
28.
The R Project for Statistical Computing. R: a language and environment for statistical computing. Accessed January 20, 2020. https://www.R-project.org/
29.
Venables  WN, Ripley  BD.  Modern Applied Statistics with S. 4th ed. Springer; 2002. doi:10.1007/978-0-387-21706-2
30.
van Buuren  S, Groothuis-Oudshoorn  K.  mice: multivariate imputation by chained equations in R.   J Stat Softw. 2011;45(3):1-67.Google Scholar
31.
Watson  NL, Mull  KE, Heffner  JL, McClure  JB, Bricker  JB.  Participant recruitment and retention in remote eHealth intervention trials: methods and lessons learned from a large randomized controlled trial of two web-based smoking interventions.   J Med Internet Res. 2018;20(8):e10351. doi:10.2196/10351 PubMedGoogle Scholar
32.
Cha  S, Ganz  O, Cohn  AM, Ehlke  SJ, Graham  AL.  Feasibility of biochemical verification in a web-based smoking cessation study.   Addict Behav. 2017;73:204-208. doi:10.1016/j.addbeh.2017.05.020 PubMedGoogle ScholarCrossref
33.
Herbec  A, Brown  J, Shahab  L, West  R.  Lessons learned from unsuccessful use of personal carbon monoxide monitors to remotely assess abstinence in a pragmatic trial of a smartphone stop smoking app—a secondary analysis.   Addict Behav Rep. 2018;9:100122. doi:10.1016/j.abrep.2018.07.003 PubMedGoogle Scholar
34.
Thrul  J, Meacham  MC, Ramo  DE.  A novel and remote biochemical verification method of smoking abstinence: predictors of participant compliance.   Tob Prev Cessat. 2018;4:20. doi:10.18332/tpc/90649 PubMedGoogle ScholarCrossref
35.
Garrison  KA, Pal  P, O’Malley  SS,  et al.  Craving to quit: a randomized controlled trial of smartphone app-based mindfulness training for smoking cessation.   Nicotine Tob Res. 2020;22(3):324-331. doi:10.1093/ntr/nty126 PubMedGoogle ScholarCrossref
36.
van der Aalst  CM, de Koning  HJ.  Biochemical verification of the self-reported smoking status of screened male smokers of the Dutch-Belgian randomized controlled lung cancer screening trial.   Lung Cancer. 2016;94:96-101. doi:10.1016/j.lungcan.2016.02.001 PubMedGoogle ScholarCrossref
37.
Wong  SLS, Shields  M, Leatherdale  S, Malaison  E, Hammond  D.  Assessment of validity of self-reported smoking status.   Health Rep. 2012;23(1):47-53.PubMedGoogle Scholar
38.
Benowitz  NL, Jacob  P  III, Ahijevych  K,  et al; SRNT Subcommittee on Biochemical Verification.  Biochemical verification of tobacco use and cessation.   Nicotine Tob Res. 2002;4(2):149-159. doi:10.1080/14622200210123581 PubMedGoogle ScholarCrossref
39.
Benowitz  NL, Bernert  JT, Foulds  J,  et al.  Biochemical verification of tobacco use and abstinence: 2019 Update.   Nicotine Tob Res. 2020;22(7):1086-1097. doi:10.1093/ntr/ntz132PubMedGoogle ScholarCrossref
40.
Graham  JW.  Missing data analysis: making it work in the real world.   Annu Rev Psychol. 2009;60:549-576. doi:10.1146/annurev.psych.58.110405.085530 PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    September 21, 2020

    Efficacy of Smartphone Applications for Smoking Cessation: A Randomized Clinical Trial

    Author Affiliations
    • 1Fred Hutchinson Cancer Research Center, Division of Public Health Sciences, Seattle, Washington
    • 2Department of Psychology, University of Washington, Seattle
    JAMA Intern Med. 2020;180(11):1472-1480. doi:10.1001/jamainternmed.2020.4055
    Key Points

    Question  Is a smartphone application based on acceptance and commitment therapy (ACT) efficacious for smoking cessation?

    Findings  In this 2-group stratified, double-blind, individually randomized clinical trial of 2415 adult smokers with a 12-month follow-up and high retention, participants assigned to the smartphone application based on ACT had 1.49 times higher odds of quitting smoking compared with the participants assigned to the smartphone application based on US clinical practice guidelines.

    Meaning  Compared with a US clinical practice guidelines–based application that teaches avoidance of smoking triggers, an ACT-based application that teaches acceptance of smoking triggers was more efficacious for quitting smoking.

    Abstract

    Importance  Smoking is a leading cause of premature death globally. Smartphone applications for smoking cessation are ubiquitous and address barriers to accessing traditional treatments, yet there is limited evidence for their efficacy.

    Objective  To determine the efficacy of a smartphone application for smoking cessation based on acceptance and commitment therapy (ACT) vs a National Cancer Institute smoking cessation application based on US clinical practice guidelines (USCPG).

    Design, Setting, and Participants  A 2-group, stratified, double-blind, individually randomized clinical trial was conducted from May 27, 2017, to September 28, 2018, among 2415 adult cigarette smokers (n = 1214 for the ACT-based smoking cessation application group and n = 1201 for the USCPG-based smoking cessation application group) with 3-, 6-, and 12-month postrandomization follow-up. The study was prespecified in the trial protocol. Follow-up data collection started on August 26, 2017, and ended at the last randomized participant’s 12-month follow-up survey on December 23, 2019. Data were analyzed from February 25 to April 3, 2020. The primary analysis was performed on a complete-case basis, with intent-to-treat missing as smoking and multiple imputation sensitivity analyses.

    Interventions  iCanQuit, an ACT-based smoking cessation application, which taught acceptance of smoking triggers, and the National Cancer Institute QuitGuide, a USCPG-based smoking cessation application, which taught avoidance of smoking triggers.

    Main Outcomes and Measures  The primary outcome was self-reported 30-day point prevalence abstinence (PPA) at 12 months after randomization. Secondary outcomes were 7-day PPA at 12 months after randomization, prolonged abstinence, 30-day and 7-day PPA at 3 and 6 months after randomization, missing data imputed with multiple imputation or coded as smoking, and cessation of all tobacco products (including e-cigarettes) at 12 months after randomization.

    Results  Participants were 2415 adult cigarette smokers (1700 women [70.4%]; 1666 White individuals [69.0%] and 868 racial/ethnic minorities [35.9%]; mean [SD] age at enrollment, 38.2 [10.9] years) from all 50 US states. The 3-month follow-up data retention rate was 86.7% (2093), the 6-month retention rate was 88.4% (2136), and the 12-month retention rate was 87.2% (2107). For the primary outcome of 30-day PPA at the 12-month follow-up, iCanQuit participants had 1.49 times higher odds of quitting smoking compared with QuitGuide participants (28.2% [293 of 1040] vs 21.1% [225 of 1067]; odds ratio [OR], 1.49; 95% CI, 1.22-1.83; P < .001). Effect sizes were very similar and statistically significant for 7-day PPA at the 12-month follow-up (OR, 1.35; 95% CI, 1.12-1.63; P = .002), prolonged abstinence at the 12-month follow-up (OR, 2.00; 95% CI, 1.45-2.76; P < .001), abstinence from all tobacco products (including e-cigarettes) at the 12-month follow-up (OR, 1.60; 95% CI, 1.28-1.99; P < .001), 30-day PPA at 3-month follow-up (OR, 2.20; 95% CI, 1.68-2.89; P < .001), 30-day PPA at 6-month follow-up (OR, 2.03; 95% CI, 1.63-2.54; P < .001), 7-day PPA at 3-month follow-up (OR, 2.04; 95% CI, 1.64-2.54; P < .001), and 7-day PPA at 6-month follow-up (OR, 1.73; 95% CI, 1.42-2.10; P < .001).

    Conclusions and Relevance  This trial provides evidence that, compared with a USCPG-based smartphone application, an ACT-based smartphone application was more efficacious for quitting cigarette smoking and thus can be an impactful treatment option.

    Trial Registration  ClinicalTrials.gov Identifier: NCT02724462

    Introduction

    Cigarette smoking is a leading cause of early death and disability1 and accounts for more than 1 in 10 deaths worldwide.2 Barriers to accessing smoking cessation treatments include low reimbursement for clinicians and low demand for in-person treatment.3 Since 2012, smartphone applications for smoking cessation have been addressing access barriers by serving as digital interventions with high population-level reach.4 There are now approximately 490 English-language smoking cessation applications, which have been downloaded an estimated total of 33 million times, according to an April 2020 analysis by SensorTower.com of all English-language cigarette smoking cessation applications on the Google Play and Apple App stores downloaded to smartphone devices (R. Nelson, SensorTower.com, personal communication, April 15, 2020). In the United States, the reach of smoking cessation applications has been aided by the fact that, as of 2019, 81% of all adults owned smartphones—up from 35% in 2011.5

    Despite their ubiquity, there is limited evidence for the efficacy of smartphone applications for smoking cessation, to our knowledge. A 2019 Cochrane review included only 5 randomized trials testing the efficacy of smoking cessation smartphone applications, all of which were compared with lower-intensity cessation interventions (ie, lower-intensity application or nonapplication with minimal support).4 These applications, which were based mainly on the US Clinical Practice Guidelines (USCPG),6 had modest abstinence rates at the 6-month follow-up (eg, self-reported rates ranged from 4% to 18%4). Overall, there was no evidence that smartphone applications improved the likelihood of smoking cessation (relative risk, 1.00; 95% CI, 0.66-1.52; I2 = 59%; 3079 participants). The Cochrane review called for rigorous randomized trials of smartphone applications for smoking cessation, and we see room for substantial improvement in the abstinence rates achieved with the use of these applications.

    One smoking cessation treatment model that has promise when delivered as a smartphone application is acceptance and commitment therapy (ACT).7 Acceptance and commitment therapy teaches skills for allowing urges to smoke to pass without smoking, which is conceptually distinct from USCPG-based standard approaches that teach avoidance of urges.6 Acceptance and commitment therapy motivates smokers to quit by appealing to their values, whereas the USCPG-based approaches motivate by using reason and logic.6 Acceptance and commitment therapy was promising for smoking cessation across a variety of delivery modalities, including a pilot randomized trial comparing an ACT-based smartphone application with the National Cancer Institute’s (NCI’s) smartphone application (QuitGuide) that followed the USCPG.8-11 Therefore, the purpose of the present study was to conduct a full-scale randomized clinical trial to determine the efficacy of a smartphone application for smoking cessation (iCanQuit) based on ACT, compared with an NCI smoking cessation application based on the USCPG (QuitGuide).

    Methods
    Study Design

    The design was a blinded, parallel, 2-group randomized clinical trial comparing iCanQuit with QuitGuide. Participants were recruited online, were randomized, and completed follow-up surveys at 3, 6, and 12 months after randomization. The 12-month primary end point accounted for the high relapse rates that commonly occur by 12 months.12-15 On the basis of the 2-month abstinence rates observed in a pilot trial11 and relapse rates occurring between 2 and 12 months after randomization,12-15 the study was 80% powered for a 2-tailed significant difference between an 11.0% iCanQuit quit rate and a 7.0% QuitGuide quit rate with a sample size of 1622. However, we set the target recruitment to 2500 participants for later exploratory analyses. The study was prespecified in the trial protocol. Details on the trial protocol are available in Supplement 1. All study activities were approved by the Fred Hutchinson Cancer Research Center Institutional Review Board. Participants provided consent online by clicking an “I accept” button option on the online consent form. Results are reported according to the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.

    Procedures
    Participants and Enrollment

    From May 27, 2017, to September 28, 2018, we recruited smokers nationally via Facebook ads (1943 of 2415 [80.5%]), a survey sampling company (336 of 2415 [13.9%]), search engine results (65 of 2415 [2.7%]), and referral from friends and family (71 of 2415 [2.9%]). Participants could have more than a single recruitment source; a few participants reported multiple sources (eg, search engine results and Facebook ads). The Facebook ad cost per click was $0.55, cost per randomized participant was $13.60, and total impressions were 5 962 400. Eligibility criteria included the following: age 18 years or older; 5 or more cigarettes smoked per day for the past year; wants to quit smoking within the next 30 days; if concurrently using any other tobacco products (eg, e-cigarettes), wants to quit using them within the next 30 days; has an interest in learning skills to quit smoking; willing to be randomly assigned to either condition; resides in the United States; has daily access to their own iPhone or Android smartphone; knows how to download smartphone applications; willing and able to read in English; has never used QuitGuide and is not currently using other smoking cessation treatment; has never participated in our prior studies; no household members already enrolled; are willing to complete outcome surveys, and can provide contact information for themselves and 2 relatives. Some advertisements were targeted to racial/ethnic minorities and men, and enrollment was limited to no more than 70% White participants and no more than 70% women, to ensure racial/ethnic minority and male representation.

    Participants completed an encrypted, web-based screening survey and were notified of their eligibility via email. They then clicked on a secured emailed link to the study website, where they provided consent and completed the baseline survey. At each enrollment step, the study was presented as a comparison of 2 smartphone applications for smoking cessation.

    Because enrollment occurred online, additional actions were taken to ensure that enrollees were eligible, including CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) authentication, review of IP (internet protocol) addresses for duplicates or non-US origin, and review of survey logs for suspicious response times (<90 seconds to complete the screening or <10 minutes to complete the baseline survey). In suspicious cases, participants were contacted by staff. If participants’ information could not be confirmed (n = 68), they were not enrolled.

    Randomization, Follow-up, Blinding, and Contamination

    After completing the baseline survey, participants were randomly assigned in a 1:1 manner to either iCanQuit or QuitGuide using randomly permuted blocks of size 2, 4, and 6, stratified by daily smoking frequency (≤20 vs ≥21 cigarettes per day), educational level (high school or less vs some college or more), race/ethnicity (minority race/ethnicity vs non-Hispanic White), and results of depression screening (Center for Epidemiological Studies Depression Scale16 score ≤15 vs ≥16). Random assignments were concealed from participants throughout the trial. The random allocation sequence was generated by a database manager and implemented automatically by the study website. Neither research staff nor study participants had access to upcoming randomized study group assignments. In both groups, participants could access their interventions from the moment of randomization and beyond (ie, after the end of the 12-month follow-up period). Follow-up data collection started on August 26, 2017, and ended at the last randomized participant’s 12-month follow-up survey on December 23, 2019.

    For blinding, each application was branded as “iCanQuit” and did not mention either ACT or QuitGuide. Contamination between applications was avoided with a unique username and password provided only to the individual user and by having an eligibility criterion of not having other household members participating in the study.

    Interventions
    iCanQuit

    iCanQuit (version 1.2.1; released 201717,18) teaches ACT skills for coping with smoking urges, staying motivated, and preventing relapse. After setting up a personalized quit plan in which users can learn about US Food and Drug Administration–approved cessation medications that they can obtain on their own, users are taken to the home screen, where they can progress through 8 levels of the intervention content, receive on-demand help in coping with smoking urges, track the daily number of cigarettes smoked, and track how many urges they let pass without smoking. The program is self-paced, and content is unlocked in a sequential manner. For the first 4 levels, exercises are unlocked immediately after the prior exercise is complete. For the last 4 levels, the next level will not unlock until users record 7 consecutive smoke-free days. If a participant lapses (eg, records having smoked a cigarette), the program encourages (but does not require) them to set a new quit date and return to the first 4 levels for preparation (eAppendix in Supplement 2). iCanQuit is a research application created for this randomized clinical trial, and its content is not yet available to the public.

    QuitGuide

    The iCanQuit application was compared with NCI’s QuitGuide application (version 1.2.2; released 2014),19,20 which, with the NCI’s permission, we posted on the Google Play and Apple stores in a blinded format branded as “iCanQuit.” We selected QuitGuide for comparison for the following reasons: (1) it follows the USCPG6; (2) it is a smartphone application, and thus avoids confounding treatment content with treatment delivery modality; (3) its content is based directly on the NCI’s smokefree.gov website, a well-established digital intervention21; and (4) it is nonproprietary and freely available to the public, providing maximal transparency and replicability.

    QuitGuide contained 4 sections of content. “Thinking about quitting” focuses on motivations to quit by encouraging users to think of reasons for quitting and providing information on the general health consequences of smoking and quitting. “Preparing to Quit” helps users develop a quit plan; helps users identify smoking behaviors, triggers, and reasons for being smoke-free; helps users identify social support for quitting; and provides information on US Food and Drug Administration–approved medications for quitting smoking. “Quitting” teaches skills for avoiding cravings to smoke, such as finding replacement behaviors (eg, chewing on carrot sticks) and staying busy. “Staying Quit” presents tips, motivations, and actions to stay smoke-free and skills for coping with slips via fighting cravings and trying to be positive. See the Box for major similarities and Table 1 for differences between the 2 applications.

    Box Section Ref ID
    Box.

    Major Similarities Between iCanQuit and QuitGuide

    • Education and skills for preparing to quit smoking

    • Education and skills for preventing relapse after quitting, including self-compassion, learning, and starting again

    • Intention formation, including setting a specific, actionable plan for quitting smoking that includes setting a quit date

    • Education on US Food and Drug Administration–approved medications for smoking cessation

    • Skills for coping with cravings to smoke

    • Education on common triggers to smoke and barriers to cessation, nicotine withdrawal reactions, and how to seek support for smoking cessation

    • Presented as a step-by-step guide with content at sixth-grade or less reading level

    Measures

    At baseline, participants reported on demographic characteristics, depression (Center for Epidemiological Studies Depression Scale16), alcohol use (Quick Drinking Screen22), nicotine dependence (Fagerström Test for Nicotine Dependence23), and smoking in their social environment (eg, number of adults at home who smoke). The primary outcome was self-reported complete-case 30-day point prevalence abstinence (PPA; ie, no smoking at all in the past 30 days) at the 12-month follow-up (eAppendix in Supplement 2). Secondary outcomes were 7-day PPA at 12 months after randomization, prolonged abstinence, 30-day and 7-day PPA at 3 and 6 months after randomization, missing data imputed with multiple imputation or coded as smokers, and cessation of all tobacco products (including e-cigarettes) at 12 months after randomization (eAppendix in Supplement 2).

    Objective measures of application engagement were collected for 12 months after randomization. The number of times a participant opened their assigned application, minutes spent per session of use, and number of unique days of use were calculated from data automatically logged by Google Analytics. Treatment satisfaction outcomes were the extent to which participants were satisfied with the assigned application, the assigned application was useful for quitting, and participants would recommend assigned application to a friend (eAppendix in Supplement 2).

    Statistical Analysis

    Data were analyzed from February 25 to April 3, 2020. The primary analysis was performed on a complete-case basis, with intent-to-treat missing as smoking and multiple imputation sensitivity analyses. Primary and secondary outcomes are described above. The missing  =  smoking imputation was a secondary outcome because it may be biased, including a bias in favor of the group with lower attrition.24-26 The small differential attrition at 6- and 12-month follow-up (ie, 3% difference; Figure) had a low risk of bias for the primary, complete-case analysis,4 and the multiple imputation provided a further test of the sensitivity of this primary analysis (eAppendix in Supplement 2). We used logistic regression models for the cessation outcome as well as secondary binary outcomes associated with cessation and treatment satisfaction. Negative binomial models were used to assess differences between treatment groups for zero-inflated count outcomes (eg, number of application openings), whereas generalized linear models were used for continuous outcomes. We adjusted for all 4 stratification variables used in randomization to avoid losing power and obtaining incorrect 95% CIs.27 We also adjusted for baseline number of alcoholic drinks per day to reduce the potential for confounding, as this variable was slightly different between groups (P = .07) and was associated with the primary cessation outcome (P = .01). All statistical tests were 2-sided, with results deemed statistically significant at P < .05, and analyses were completed using R, version 3.6.1,28 library “MASS”29 for negative binomial regression, and library “mice”30 for multiple imputation.

    Results

    A total of 12 881 individuals were screened and 2503 participants were randomly assigned to a smoking cessation application (1254 to iCanQuit and 1249 to QuitGuide). Owing to a technical error in our automated enrollment system, 40 participants in the iCanQuit group and 48 participants in the QuitGuide group were excluded after randomization because they were determined to be ineligible (eg, same household). Thus, the full analyzable sample was 2415 (1214 in the iCanQuit group and 1201 in the QuitGuide group). The follow-up data retention rates were 86.7% (2093 of 2415) overall at 3 months (iCanQuit, 85.9% [1043 of 1214] vs QuitGuide, 87.4% [1050 of 1201]; P = .20), 88.4% (2136 of 2415) overall at 6 months (iCanQuit, 87.1% [1058 of 1214] vs QuitGuide, 89.8% [1078 of 1201]; P = .05), and 87.2% (2107 of 2415) overall at 12 months (iCanQuit, 85.7% [1040 of 1214] vs QuitGuide, 88.8% [1067 of 1201]; P = .02]) (Figure).

    Mean (SD) age at enrollment was 38.2 (10.9) years (Table 2). Participants included 1700 women (70.4%), 1666 White individuals (69.0%), and 868 racial/ethnic minorities (35.9%). A total of 995 participants (41.2%) had a high school education or less. Regarding smoking, 2009 participants (83.2%) had smoked for 10 years or more and 1803 (74.7%) smoked more than one-half pack (≥11 cigarettes) per day. There were no statistically significant differences between the 2 groups on any baseline variable. Participants were from all 50 US states (eFigure in Supplement 2).

    Smoking Cessation

    For the primary outcome of 30-day PPA at the 12-month follow-up, iCanQuit participants had 1.49 times higher odds of quitting smoking compared with QuitGuide participants (28.2% [293 of 1040] vs 21.1% [225 of 1067]; odds ratio [OR], 1.49; 95% CI, 1.22-1.83; P < .001); these results were similar when all 2503 randomized participants were included (28.5% [306 of 1074] vs 21.0% [234 of 1113]; OR, 1.50; 95% CI, 1.23-1.83; P < .001). Effect sizes were similar and all were statistically significant for 7-day PPA at the 12-month follow-up (OR, 1.35; 95% CI, 1.12-1.63; P = .002), prolonged abstinence at the 12-month follow-up (OR, 2.00; 95% CI, 1.45-2.76; P < .001), abstinence from all tobacco products (including e-cigarettes) at the 12-month follow-up (OR, 1.60; 95% CI, 1.28-1.99; P < .001), 30-day PPA at 3-month follow-up (OR, 2.20; 95% CI, 1.68-2.89; P < .001), 30-day PPA at 6-month follow-up (OR, 2.03; 95% CI, 1.63-2.54; P < .001), 7-day PPA at 3-month follow-up (OR, 2.04; 95% CI, 1.64-2.54; P < .001), and 7-day PPA at 6-month follow-up (OR, 1.73; 95% CI, 1.42-2.10; P < .001) (Table 3). Effect sizes were also similar and all were statistically significant when missing data were imputed with multiple imputation or coded as smokers (eTable in Supplement 2).

    Use and Satisfaction

    As shown in Table 4, compared with participants using QuitGuide, iCanQuit participants had a higher mean (SD) number of times the application was opened (37.5 [88.4] vs 9.9 [50.0]; P < .001), mean (SD) minutes spent per session (3.9 [5.3] vs 2.6 [2.6] minutes; P < .001), and mean (SD) number of unique days of use (24.3 [50.2] vs 7.1 [15.8] days; P < .001). Compared with participants using QuitGuide, iCanQuit participants reported higher overall satisfaction (865 of 977 [88.5%] vs 773 of 1002 [77.1%]; P < .001), found it more useful for quitting (805 of 1005 [80.1%] vs 739 of 1033 [71.5%]; P < .001), and were more likely to recommend it to a friend (840 of 1011 [83.1%] vs 724 of 1024 [70.7%]; P < .001).

    Discussion

    The present study determined the efficacy of a smartphone application for smoking cessation (iCanQuit) based on ACT compared with an NCI smoking cessation application (QuitGuide) based on the USCPG. For the primary outcome of 30-day PPA at the 12-month follow-up, iCanQuit participants were 1.49 times more likely to quit smoking compared with QuitGuide participants (28.2% abstinent vs 21.1% abstinent). Effect sizes were similar and statistically significant for all secondary outcomes.

    The current study advances the evidence base for smartphone applications for smoking cessation. Prior randomized clinical trials in a 2019 Cochrane review ranged in sample size from 49 to 1599 and had a weighted mean 55.3% final outcome data retention rate.4 By contrast, the current trial is, to our knowledge, now the largest to date, had a substantially higher retention rate (ie, 87.2% vs 55.3%), and had twice the follow-up length (ie, 12 vs 6 months). The self-reported 6-month abstinence rates of individuals using smartphone applications included in the Cochrane review ranged from 4% to 18%,4 which is within the range of the abstinence rates observed for the QuitGuide application. That participants using iCanQuit had substantially higher odds of quitting than those using QuitGuide suggests that iCanQuit is an advance compared with a smartphone intervention that followed the USCPG. Future mediational process research should examine theoretical processes as well as specific features listed in the Box and Table 1 to understand why iCanQuit was the more efficacious intervention.

    Strengths and Limitations

    This study has multiple strengths, including a large sample size and 12-month follow-up. Notably, the 87.2% 12-month outcome retention rate contributes to confidence in the study findings. Our group’s methods for obtaining high retention rates are described elsewhere.31 The broad demographic characteristics of the sample from all 50 US states increased confidence in the generalizability of the study findings and overcame a key limitation of the prior trials, which tended to include less diverse and more educated samples.4

    This study also has some limitations. First, remote biochemical data collection for the cessation outcome data was not conducted. We elected not to do so, as there are 3 major methodological problems with remote biochemical data collection: high attrition, problems with identifying the person providing the sample, and the high cost relative to the likely low percentage of falsifying from a high reach–low intensity intervention.32-35 Although there is evidence of high levels of agreement between self-reported and biochemically validated smoking status,36,37 the external validity of the self-reported smoking status in this trial is not known. However, given the double-blinding of the intervention, we see no compelling reason why the false reporting rate would be higher in one intervention group vs the other group; thus, there is no strong rationale for a bias in the ORs. Owing to low demand characteristics for false reporting, the Society for Research on Nicotine and Tobacco Subcommittee on Biochemical Verification recommended that biochemical confirmation be considered unnecessary in population-based studies with no face-to-face contact and studies in which data are optimally collected through the internet, telephone, or mail.38,39 Second, there was a small differential attrition at the 6- and 12-month follow-up that somewhat biased the imputation of missing data as smoking abstinence rates in favor of QuitGuide. Although iCanQuit abstinence rates were still statistically significantly higher than those of QuitGuide in the analysis imputing missing as smoking despite this bias, we deem the complete-case and multiple imputation analyses to be more reliable.24,25 Finally, owing to a technical error in the Google Analytics system, the full 12 months of application use data were available for only the first 1467 participants. Because this error occurred independently of the participants or the interventions, the resulting missing data are an ignorable threat to the validity of the current analysis.40

    Conclusions

    This trial provides evidence that, compared with a USCPG-based smartphone application, an ACT-based smartphone application was more efficacious for quitting cigarette smoking. iCanQuit can be an impactful treatment option; based on the main result, for every 100 000 smokers reached with iCanQuit, 28 000 would quit smoking.

    Back to top
    Article Information

    Accepted for Publication: July 7, 2020.

    Corresponding Author: Jonathan B. Bricker, PhD, Fred Hutchinson Cancer Research Center, Division of Public Health Sciences, 1100 Fairview Ave North, PO Box 19024, M3-B232, Seattle, WA 98109 (jbricker@fredhutch.org).

    Published Online: September 21, 2020. doi:10.1001/jamainternmed.2020.4055

    Author Contributions: Dr Bricker and Ms Mull had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: Bricker, Heffner.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: Bricker, Watson, Mull, Sullivan.

    Critical revision of the manuscript for important intellectual content: Bricker, Watson, Mull, Heffner.

    Statistical analysis: Mull.

    Obtained funding: Bricker.

    Administrative, technical, or material support: Bricker, Watson, Sullivan, Heffner.

    Supervision: Bricker.

    Conflict of Interest Disclosures: Dr Bricker reported receiving grants from the National Cancer Institute during the conduct of the study; serving on the scientific advisory board for and receiving personal fees from Chrono Therapeutics outside the submitted work; and reported that the Fred Hutchinson Cancer Research Center has applied for a US patent that pertains to the content of the iCanQuit application. 2Morrow, Inc, a Kirkland, Washington–based software company, has licensed this technology from the Fred Hutchinson Cancer Research Center. Dr Bricker had no personal financial relationships with this patent application, the licensing agreement, or 2Morrow, Inc. Ms Mull reported receiving grants from the National Institutes of Health/National Cancer Institute during the conduct of the study. Dr Heffner reported receiving nonfinancial support from Pfizer outside the submitted work. None of the authors has a financial relationship with the iCanQuit application and thus will not receive any compensation when it becomes publicly available. No other disclosures were reported.

    Funding/Support: This study was funded by grant R01CA192849 from the National Cancer Institute (Dr Bricker).

    Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Data Sharing Statement: See Supplement 3.

    Additional Contributions: We gratefully acknowledge the contributions of the entire study staff, most notably Eric Meier, MS, BS, and Eric Strand, BSc, Datatope LLC, and Carolyn Ehret, MS, RDN, CD, and Alanna Boynton, MS, RD, Fred Hutchinson Cancer Research Center, as well as the design services of Ayogo Inc and the development services of Moby Inc. Meier and Strand were compensated for their contributions. We are very appreciative of every study participant.

    References
    1.
    GBD 2015 Risk Factors Collaborators.  Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015.   Lancet. 2016;388(10053):1659-1724. doi:10.1016/S0140-6736(16)31679-8 PubMedGoogle ScholarCrossref
    2.
    GBD 2015 Tobacco Collaborators.  Smoking prevalence and attributable disease burden in 195 countries and territories, 1990-2015: a systematic analysis from the Global Burden of Disease Study 2015.   Lancet. 2017;389(10082):1885-1906. doi:10.1016/S0140-6736(17)30819-X PubMedGoogle ScholarCrossref
    3.
    Husten  CG.  A call for ACTTION: increasing access to tobacco-use treatment in our nation.   Am J Prev Med. 2010;38(3)(suppl):S414-S417. doi:10.1016/j.amepre.2009.12.006 PubMedGoogle ScholarCrossref
    4.
    Whittaker  R, McRobbie  H, Bullen  C, Rodgers  A, Gu  Y, Dobson  R.  Mobile phone text messaging and app-based interventions for smoking cessation.   Cochrane Database Syst Rev. 2019;10:CD006611. doi:10.1002/14651858.CD006611.pub5 PubMedGoogle Scholar
    5.
    Pew Research Center. Mobile fact sheet. Published June 12, 2019. Accessed August 14, 2020. https://www.pewresearch.org/internet/fact-sheet/mobile/
    6.
    Fiore  MC, Jaén  CR, Baker  TB,  et al  Treating Tobacco Use and Dependence: 2008 Update: Clinical Practice Guideline. Rockville, MD: US Department of Health and Human Services, Public Health Service, 2008.
    7.
    Hayes  SC, Levin  ME, Plumb-Vilardaga  J, Villatte  JL, Pistorello  J.  Acceptance and commitment therapy and contextual behavioral science: examining the progress of a distinctive model of behavioral and cognitive therapy.   Behav Ther. 2013;44(2):180-198. doi:10.1016/j.beth.2009.08.002 PubMedGoogle ScholarCrossref
    8.
    Hernández-López  M, Luciano  MC, Bricker  JB, Roales-Nieto  JG, Montesinos  F.  Acceptance and commitment therapy for smoking cessation: a preliminary study of its effectiveness in comparison with cognitive behavioral therapy.   Psychol Addict Behav. 2009;23(4):723-730. doi:10.1037/a0017632 PubMedGoogle ScholarCrossref
    9.
    Bricker  J, Wyszynski  C, Comstock  B, Heffner  JL.  Pilot randomized controlled trial of web-based acceptance and commitment therapy for smoking cessation.   Nicotine Tob Res. 2013;15(10):1756-1764. doi:10.1093/ntr/ntt056 PubMedGoogle ScholarCrossref
    10.
    Bricker  JB, Mull  KE, McClure  JB, Watson  NL, Heffner  JL.  Improving quit rates of web-delivered interventions for smoking cessation: full-scale randomized trial of WebQuit.org versus Smokefree.gov.   Addiction. 2018;113(5):914-923. doi:10.1111/add.14127PubMedGoogle ScholarCrossref
    11.
    Bricker  JB, Mull  KE, Kientz  JA,  et al.  Randomized, controlled pilot trial of a smartphone app for smoking cessation using acceptance and commitment therapy.   Drug Alcohol Depend. 2014;143:87-94. doi:10.1016/j.drugalcdep.2014.07.006 PubMedGoogle ScholarCrossref
    12.
    Koçak  ND, Eren  A, Boğa  S,  et al.  Relapse rate and factors related to relapse in a 1-year follow-up of subjects participating in a smoking cessation program.   Respir Care. 2015;60(12):1796-1803. doi:10.4187/respcare.03883 PubMedGoogle ScholarCrossref
    13.
    Killeen  PR.  Markov model of smoking cessation.   Proc Natl Acad Sci U S A. 2011;108(suppl 3):15549-15556. doi:10.1073/pnas.1011277108 PubMedGoogle ScholarCrossref
    14.
    Herd  N, Borland  R.  The natural history of quitting smoking: findings from the International Tobacco Control (ITC) Four Country Survey.   Addiction. 2009;104(12):2075-2087. doi:10.1111/j.1360-0443.2009.02731.x PubMedGoogle ScholarCrossref
    15.
    Ferguson  J, Bauld  L, Chesterman  J, Judge  K.  The English smoking treatment services: one-year outcomes.   Addiction. 2005;100(suppl 2):59-69. doi:10.1111/j.1360-0443.2005.01028.x PubMedGoogle ScholarCrossref
    16.
    Radloff  LS.  The CES-D scale: a self-report depression scale for research in the general population.   Appl Psychol Meas. 1977;1(3):385-401. doi:10.1177/014662167700100306 Google ScholarCrossref
    17.
    Apple App Store. iCanQuit: experimental arm. Accessed April 7, 2020. https://apps.apple.com/us/app/icanquit/id1205729317?app=itunes&ign-mpt=uo%3D4
    18.
    Google Play Store. iCanQuit: experimental arm. Accessed April 7, 2020. https://play.google.com/store/apps/details?id=org.fredhutch.icanquitr
    19.
    Apple App Store. iCanQuit: comparison arm. Accessed April 7, 2020. https://apps.apple.com/us/app/icanquit/id1205729312?app=itunes&ign-mpt=uo%3D4
    20.
    Google Play Store. iCanQuit: comparison arm. Accessed April 7, 2020. https://play.google.com/store/apps/details?id=org.fredhutch.icanquitc
    21.
    Shields  PG, Herbst  RS, Arenberg  D,  et al.  Smoking cessation, version 1.2016, NCCN Clinical Practice Guidelines in Oncology.   J Natl Compr Canc Netw. 2016;14(11):1430-1468. doi:10.6004/jnccn.2016.0152 PubMedGoogle ScholarCrossref
    22.
    Roy  M, Dum  M, Sobell  LC,  et al.  Comparison of the Quick Drinking Screen and the alcohol Timeline Followback with outpatient alcohol abusers.   Subst Use Misuse. 2008;43(14):2116-2123. doi:10.1080/10826080802347586 PubMedGoogle ScholarCrossref
    23.
    Heatherton  TF, Kozlowski  LT, Frecker  RC, Fagerström  KO.  The Fagerström Test for Nicotine Dependence: a revision of the Fagerström Tolerance Questionnaire.   Br J Addict. 1991;86(9):1119-1127. doi:10.1111/j.1360-0443.1991.tb01879.x PubMedGoogle ScholarCrossref
    24.
    Hedeker  D, Mermelstein  RJ, Demirtas  H.  Analysis of binary outcomes with missing data: missing = smoking, last observation carried forward, and a little multiple imputation.   Addiction. 2007;102(10):1564-1573. doi:10.1111/j.1360-0443.2007.01946.x PubMedGoogle ScholarCrossref
    25.
    Nelson  DB, Partin  MR, Fu  SS, Joseph  AM, An  LC.  Why assigning ongoing tobacco use is not necessarily a conservative approach to handling missing tobacco cessation outcomes.   Nicotine Tob Res. 2009;11(1):77-83. doi:10.1093/ntr/ntn013 PubMedGoogle ScholarCrossref
    26.
    Blankers  M, Smit  ES, van der Pol  P, de Vries  H, Hoving  C, van Laar  M.  The missing=smoking assumption: a fallacy in internet-based smoking cessation trials?   Nicotine Tob Res. 2016;18(1):25-33. doi:10.1093/ntr/ntv055PubMedGoogle Scholar
    27.
    Kernan  WN, Viscoli  CM, Makuch  RW, Brass  LM, Horwitz  RI.  Stratified randomization for clinical trials.   J Clin Epidemiol. 1999;52(1):19-26. doi:10.1016/S0895-4356(98)00138-3 PubMedGoogle ScholarCrossref
    28.
    The R Project for Statistical Computing. R: a language and environment for statistical computing. Accessed January 20, 2020. https://www.R-project.org/
    29.
    Venables  WN, Ripley  BD.  Modern Applied Statistics with S. 4th ed. Springer; 2002. doi:10.1007/978-0-387-21706-2
    30.
    van Buuren  S, Groothuis-Oudshoorn  K.  mice: multivariate imputation by chained equations in R.   J Stat Softw. 2011;45(3):1-67.Google Scholar
    31.
    Watson  NL, Mull  KE, Heffner  JL, McClure  JB, Bricker  JB.  Participant recruitment and retention in remote eHealth intervention trials: methods and lessons learned from a large randomized controlled trial of two web-based smoking interventions.   J Med Internet Res. 2018;20(8):e10351. doi:10.2196/10351 PubMedGoogle Scholar
    32.
    Cha  S, Ganz  O, Cohn  AM, Ehlke  SJ, Graham  AL.  Feasibility of biochemical verification in a web-based smoking cessation study.   Addict Behav. 2017;73:204-208. doi:10.1016/j.addbeh.2017.05.020 PubMedGoogle ScholarCrossref
    33.
    Herbec  A, Brown  J, Shahab  L, West  R.  Lessons learned from unsuccessful use of personal carbon monoxide monitors to remotely assess abstinence in a pragmatic trial of a smartphone stop smoking app—a secondary analysis.   Addict Behav Rep. 2018;9:100122. doi:10.1016/j.abrep.2018.07.003 PubMedGoogle Scholar
    34.
    Thrul  J, Meacham  MC, Ramo  DE.  A novel and remote biochemical verification method of smoking abstinence: predictors of participant compliance.   Tob Prev Cessat. 2018;4:20. doi:10.18332/tpc/90649 PubMedGoogle ScholarCrossref
    35.
    Garrison  KA, Pal  P, O’Malley  SS,  et al.  Craving to quit: a randomized controlled trial of smartphone app-based mindfulness training for smoking cessation.   Nicotine Tob Res. 2020;22(3):324-331. doi:10.1093/ntr/nty126 PubMedGoogle ScholarCrossref
    36.
    van der Aalst  CM, de Koning  HJ.  Biochemical verification of the self-reported smoking status of screened male smokers of the Dutch-Belgian randomized controlled lung cancer screening trial.   Lung Cancer. 2016;94:96-101. doi:10.1016/j.lungcan.2016.02.001 PubMedGoogle ScholarCrossref
    37.
    Wong  SLS, Shields  M, Leatherdale  S, Malaison  E, Hammond  D.  Assessment of validity of self-reported smoking status.   Health Rep. 2012;23(1):47-53.PubMedGoogle Scholar
    38.
    Benowitz  NL, Jacob  P  III, Ahijevych  K,  et al; SRNT Subcommittee on Biochemical Verification.  Biochemical verification of tobacco use and cessation.   Nicotine Tob Res. 2002;4(2):149-159. doi:10.1080/14622200210123581 PubMedGoogle ScholarCrossref
    39.
    Benowitz  NL, Bernert  JT, Foulds  J,  et al.  Biochemical verification of tobacco use and abstinence: 2019 Update.   Nicotine Tob Res. 2020;22(7):1086-1097. doi:10.1093/ntr/ntz132PubMedGoogle ScholarCrossref
    40.
    Graham  JW.  Missing data analysis: making it work in the real world.   Annu Rev Psychol. 2009;60:549-576. doi:10.1146/annurev.psych.58.110405.085530 PubMedGoogle ScholarCrossref
    ×