[Skip to Content]
Sign In
Individual Sign In
Create an Account
Institutional Sign In
OpenAthens Shibboleth
[Skip to Content Landing]
Figure.
Receiver Operating Characteristic Curves Among Nodules First Seen in the Prevalence Round of Screening
Receiver Operating Characteristic Curves Among Nodules First Seen in the Prevalence Round of Screening

AUC indicates area under the curve; Mayo, model developed by the Mayo Clinic; PanCan 1b, Pan-Canadian Early Detection of Lung Cancer Study (PanCan) model using a parsimonious approach including spiculation; PanCan 2b, PanCan model using a comprehensive approach including spiculation; PanCan MeanDiam, PanCan 2b model replacing nodule diameter with mean diameter (calculated as [largest nodule diameter + perpendicular diameter]/2); PanCan Vol, PanCan-2b model replacing nodule diameter with volume; PKUPH, model developed by the Peking University People’s Hospital; UKLS, UK Lung Cancer Screening trial model; VA, model developed by the US Department of Veterans Affairs.

Table 1.  
Nodule Characteristics by Malignancy Status and Screening Round
Nodule Characteristics by Malignancy Status and Screening Round
Table 2.  
Participant Characteristics at Randomization
Participant Characteristics at Randomization
Table 3.  
Observed Absolute Risk for Nodule Malignancy vs Predicted Model Estimates by Nodule Size During Prevalence Screeninga
Observed Absolute Risk for Nodule Malignancy vs Predicted Model Estimates by Nodule Size During Prevalence Screeninga
Table 4.  
Classification Performance at Predicted Risk Thresholds in LUSI During Prevalence Screenings and Other Screening Studies
Classification Performance at Predicted Risk Thresholds in LUSI During Prevalence Screenings and Other Screening Studies
1.
Aberle  DR, Adams  AM, Berg  CD,  et al; National Lung Screening Trial Research Team.  Reduced lung-cancer mortality with low-dose computed tomographic screening.  N Engl J Med. 2011;365(5):395-409. doi:10.1056/NEJMoa1102873PubMedGoogle ScholarCrossref
2.
International Association for the Study of Lung Cancer.  Nelson study shows CT screening for nodule volume management reduces lung cancer mortality by 26 percent in men [press release]. IASLC 19th World Conference on Lung Cancer. https://wclc2018.iaslc.org/media/2018%20WCLC%20Press%20Program%20Press%20Release%20De%20Koning%209.25%20FINAL%20.pdf. Accessed January 8, 2020.
3.
Paci  E, Puliti  D, Lopes Pegna  A,  et al; the ITALUNG Working Group.  Mortality, survival and incidence rates in the ITALUNG randomised lung cancer screening trial.  Thorax. 2017;72(9):825-831. doi:10.1136/thoraxjnl-2016-209825PubMedGoogle ScholarCrossref
4.
Pastorino  U, Silva  M, Sestini  S,  et al.  Prolonged lung cancer screening reduced 10-year mortality in the MILD trial: new confirmation of lung cancer screening efficacy.  Ann Oncol. 2019;30(7):1162-1169. doi:10.1093/annonc/mdz117PubMedGoogle ScholarCrossref
5.
Becker  N, Motsch  E, Trotter  A,  et al.  Lung cancer mortality reduction by LDCT screening: results from the randomized German LUSI trial [published online June 4, 2019].  Int J Cancer. doi:10.1002/ijc.32486PubMedGoogle Scholar
6.
MacMahon  H, Naidich  DP, Goo  JM,  et al.  Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017.  Radiology. 2017;284(1):228-243. doi:10.1148/radiol.2017161659PubMedGoogle ScholarCrossref
7.
Wood  DE, Kazerooni  E, Baum  SL,  et al; National comprehension cancer network.  Lung cancer screening, version 1.2015: featured updates to the NCCN guidelines.  J Natl Compr Canc Netw. 2015;13(1):23-34. doi:10.6004/jnccn.2015.0006PubMedGoogle ScholarCrossref
8.
Kanne  JP, Jensen  LE, Mohammed  TL,  et al; Expert Panel on Thoracic Imaging.  ACR appropriateness Criteria radiographically detected solitary pulmonary nodule.  J Thorac Imaging. 2013;28(1):W1-3. doi:10.1097/RTI.0b013e31827657c8PubMedGoogle ScholarCrossref
9.
American College of Radiology.  Lung CT Screening Reporting & Data System. https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Lung-Rads. Accessed January 8, 2020.
10.
Baldwin  DR, Callister  ME; Guideline Development Group.  The British Thoracic Society guidelines on the investigation and management of pulmonary nodules.  Thorax. 2015;70(8):794-798. doi:10.1136/thoraxjnl-2015-207221PubMedGoogle ScholarCrossref
11.
Swensen  SJ, Silverstein  MD, Ilstrup  DM, Schleck  CD, Edell  ES.  The probability of malignancy in solitary pulmonary nodules: application to small radiologically indeterminate nodules.  Arch Intern Med. 1997;157(8):849-855. doi:10.1001/archinte.1997.00440290031002PubMedGoogle ScholarCrossref
12.
Gould  MK, Ananth  L, Barnett  PG; Veterans Affairs SNAP Cooperative Study Group.  A clinical model to estimate the pretest probability of lung cancer in patients with solitary pulmonary nodules.  Chest. 2007;131(2):383-388. doi:10.1378/chest.06-1261PubMedGoogle ScholarCrossref
13.
Li  Y, Chen  KZ, Wang  J.  Development and validation of a clinical prediction model to estimate the probability of malignancy in solitary pulmonary nodules in Chinese people.  Clin Lung Cancer. 2011;12(5):313-319. doi:10.1016/j.cllc.2011.06.005PubMedGoogle ScholarCrossref
14.
Nair  VS, Sundaram  V, Desai  M, Gould  MK.  Accuracy of models to identify lung nodule cancer risk in the National Lung Screening Trial.  Am J Respir Crit Care Med. 2018;197(9):1220-1223. doi:10.1164/rccm.201708-1632LEPubMedGoogle ScholarCrossref
15.
McWilliams  A, Tammemagi  MC, Mayo  JR,  et al.  Probability of cancer in pulmonary nodules detected on first screening CT.  N Engl J Med. 2013;369(10):910-919. doi:10.1056/NEJMoa1214726PubMedGoogle ScholarCrossref
16.
Tammemägi  M, Ritchie  AJ, Atkar-Khattra  S,  et al; Pan-Canadian Early Detection of Lung Cancer Study Group.  Predicting malignancy risk of screen-detected lung nodules-mean diameter or volume.  J Thorac Oncol. 2019;14(2):203-211. doi:10.1016/j.jtho.2018.10.006PubMedGoogle ScholarCrossref
17.
Marcus  MW, Duffy  SW, Devaraj  A,  et al.  Probability of cancer in lung nodules using sequential volumetric screening up to 12 months: the UKLS trial.  Thorax. 2019;74(8):761-767. doi:10.1136/thoraxjnl-2018-212263PubMedGoogle ScholarCrossref
18.
White  CS, Dharaiya  E, Campbell  E, Boroczky  L.  The Vancouver Lung Cancer Risk Prediction model: assessment by using a subset of the National Lung Screening Trial cohort.  Radiology. 2017;283(1):264-272. doi:10.1148/radiol.2016152627PubMedGoogle ScholarCrossref
19.
Winkler Wille  MM, van Riel  SJ, Saghir  Z,  et al.  Predictive accuracy of the PanCan lung cancer risk prediction model: external validation based on CT from the Danish Lung Cancer Screening Trial.  Eur Radiol. 2015;25(10):3093-3099. doi:10.1007/s00330-015-3689-0PubMedGoogle ScholarCrossref
20.
Becker  N, Motsch  E, Gross  ML,  et al.  Randomized study on early detection of lung cancer with MSCT in Germany: results of the first 3 years of follow-up after randomization.  J Thorac Oncol. 2015;10(6):890-896. doi:10.1097/JTO.0000000000000530PubMedGoogle ScholarCrossref
21.
Jobst  BJ, Weinheimer  O, Trauth  M,  et al.  Effect of smoking cessation on quantitative computed tomography in smokers at risk in a lung cancer screening population.  Eur Radiol. 2018;28(2):807-815. doi:10.1007/s00330-017-5030-6PubMedGoogle ScholarCrossref
22.
Jiang  J.  Linear and Generalized Linear Mixed Models and Their Applications. New York, NY: Springer-Verlag; 2007.
23.
Pan  W.  Akaike’s information criterion in generalized estimating equations.  Biometrics. 2001;57(1):120-125. doi:10.1111/j.0006-341X.2001.00120.xPubMedGoogle ScholarCrossref
24.
Obuchowski  NA.  Nonparametric analysis of clustered ROC curve data.  Biometrics. 1997;53(2):567-578. doi:10.2307/2533958PubMedGoogle ScholarCrossref
25.
Smith  PJ, Hadgu  A.  Sensitivity and specificity for correlated observations.  Stat Med. 1992;11(11):1503-1509. doi:10.1002/sim.4780111108PubMedGoogle ScholarCrossref
26.
Coughlin  SS, Trock  B, Criqui  MH, Pickle  LW, Browner  D, Tefft  MC.  The logistic modeling of sensitivity, specificity, and predictive value of a diagnostic test.  J Clin Epidemiol. 1992;45(1):1-7. doi:10.1016/0895-4356(92)90180-UPubMedGoogle ScholarCrossref
27.
Genders  TS, Spronk  S, Stijnen  T, Steyerberg  EW, Lesaffre  E, Hunink  MG.  Methods for calculating sensitivity and specificity of clustered data: a tutorial.  Radiology. 2012;265(3):910-916. doi:10.1148/radiol.12120509PubMedGoogle ScholarCrossref
28.
Lemeshow  S, Hosmer  DW  Jr.  A review of goodness of fit statistics for use in the development of logistic regression models.  Am J Epidemiol. 1982;115(1):92-106. doi:10.1093/oxfordjournals.aje.a113284PubMedGoogle ScholarCrossref
29.
Rufibach  K.  Use of Brier score to assess binary predictions.  J Clin Epidemiol. 2010;63(8):938-939. doi:10.1016/j.jclinepi.2009.11.009PubMedGoogle ScholarCrossref
30.
Horeweg  N, van Rosmalen  J, Heuvelmans  MA,  et al.  Lung cancer probability in patients with CT-detected pulmonary nodules: a prespecified analysis of data from the NELSON trial of low-dose CT screening.  Lancet Oncol. 2014;15(12):1332-1341. doi:10.1016/S1470-2045(14)70389-4PubMedGoogle ScholarCrossref
31.
Han  D, Heuvelmans  MA, Vliegenthart  R,  et al.  Influence of lung nodule margin on volume- and diameter-based reader variability in CT lung cancer screening.  Br J Radiol. 2018;91(1090):20170405. doi:10.1259/bjr.20170405PubMedGoogle Scholar
32.
Schultz  EM, Sanders  GD, Trotter  PR,  et al.  Validation of two models to estimate the probability of malignancy in patients with solitary pulmonary nodules.  Thorax. 2008;63(4):335-341. doi:10.1136/thx.2007.084731PubMedGoogle ScholarCrossref
33.
Isbell  JM, Deppen  S, Putnam  JB  Jr,  et al.  Existing general population models inaccurately predict lung cancer risk in patients referred for surgical evaluation.  Ann Thorac Surg. 2011;91(1):227-233. doi:10.1016/j.athoracsur.2010.08.054PubMedGoogle ScholarCrossref
34.
Xiao  F, Liu  D, Guo  Y,  et al.  Novel and convenient method to evaluate the character of solitary pulmonary nodule-comparison of three mathematical prediction models and further stratification of risk factors.  PLoS One. 2013;8(10):e78271. doi:10.1371/journal.pone.0078271PubMedGoogle Scholar
35.
Deppen  SA, Blume  JD, Aldrich  MC,  et al.  Predicting lung cancer prior to surgical resection in patients with lung nodules.  J Thorac Oncol. 2014;9(10):1477-1484. doi:10.1097/JTO.0000000000000287PubMedGoogle ScholarCrossref
36.
Al-Ameri  A, Malhotra  P, Thygesen  H,  et al.  Risk of malignancy in pulmonary nodules: a validation study of four prediction models.  Lung Cancer. 2015;89(1):27-30. doi:10.1016/j.lungcan.2015.03.018PubMedGoogle ScholarCrossref
37.
Massion  PP, Healey  GF, Peek  LJ,  et al.  Autoantibody signature enhances the positive predictive power of computed tomography and nodule-based risk models for detection of lung cancer.  J Thorac Oncol. 2017;12(3):578-584. doi:10.1016/j.jtho.2016.08.143PubMedGoogle ScholarCrossref
38.
Chung  K, Mets  OM, Gerke  PK,  et al.  Brock malignancy risk calculator for pulmonary nodules: validation outside a lung cancer screening population.  Thorax. 2018;73(9):857-863. doi:10.1136/thoraxjnl-2017-211372PubMedGoogle ScholarCrossref
39.
Zhao  H, Marshall  HM, Yang  IA,  et al.  Screen-detected subsolid pulmonary nodules: long-term follow-up and application of the PanCan lung cancer risk prediction model.  Br J Radiol. 2016;89(1060):20160016. doi:10.1259/bjr.20160016PubMedGoogle Scholar
40.
Marshall  HM, Zhao  H, Bowman  RV,  et al.  The effect of different radiological models on diagnostic accuracy and lung cancer screening performance.  Thorax. 2017;72(12):1147-1150. doi:10.1136/thoraxjnl-2016-209624PubMedGoogle ScholarCrossref
41.
O’Keeffe  LM, Taylor  G, Huxley  RR, Mitchell  P, Woodward  M, Peters  SAE.  Smoking as a risk factor for lung cancer in women and men: a systematic review and meta-analysis.  BMJ Open. 2018;8(10):e021611. doi:10.1136/bmjopen-2018-021611PubMedGoogle Scholar
42.
Pinsky  PF, Church  TR, Izmirlian  G, Kramer  BS.  The National Lung Screening Trial: results stratified by demographics, smoking history, and lung cancer histology.  Cancer. 2013;119(22):3976-3983. doi:10.1002/cncr.28326PubMedGoogle ScholarCrossref
43.
Pinsky  PF, Gierada  DS, Nath  PH, Munden  R.  Lung cancer risk associated with new solid nodules in the National Lung Screening Trial.  AJR Am J Roentgenol. 2017;209(5):1009-1014. doi:10.2214/AJR.17.18252PubMedGoogle ScholarCrossref
44.
Huang  P, Lin  CT, Li  Y,  et al.  Prediction of lung cancer risk at follow-up screening with low-dose CT: a training and validation study of a deep learning method.  Lancet Digital Health. 2019;1(7):e353-e362. doi:10.1016/S2589-7500(19)30159-1Google ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Oncology
    February 14, 2020

    Evaluation of Prediction Models for Identifying Malignancy in Pulmonary Nodules Detected via Low-Dose Computed Tomography

    Author Affiliations
    • 1Division of Cancer Epidemiology, German Cancer Research Center, Heidelberg, Germany
    • 2Translational Lung Research Center Heidelberg, German Center for Lung Research, Heidelberg, Germany
    • 3Division of Radiology, German Cancer Research Center, Heidelberg, Germany
    • 4Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, Heidelberg, Germany
    • 5Department of Diagnostic and Interventional Radiology with Nuclear Medicine, Thoraxklinik-Heidelberg GmbH, Heidelberg, Germany
    JAMA Netw Open. 2020;3(2):e1921221. doi:10.1001/jamanetworkopen.2019.21221
    Key Points español 中文 (chinese)

    Question  Are nodule malignancy prediction models developed in screening contexts superior to those developed in clinical contexts in terms of discrimination and calibration?

    Findings  In this diagnostic study of 1159 participants from the German Lung Cancer Screening Intervention trial, models originally developed using data from the Pan-Canadian Early Detection of Lung Cancer Study showed excellent discrimination and calibration compared with models developed in clinical contexts when applied to prevalence screening low-dose computed tomography images.

    Meaning  These findings confirmed the good performance and transferability of the Pan-Canadian Early Detection of Lung Cancer Study models in population screening settings, especially for nodules detected on prevalence screen.

    Abstract

    Importance  Malignancy prediction models based on participant-related characteristics and imaging parameters from low-dose computed tomography (CT) may improve decision-making regarding nodule management and diagnosis in lung cancer screening.

    Objective  To externally validate 5 malignancy prediction models that were developed in screening settings, compared with 3 models that were developed in clinical settings, in terms of discrimination and absolute risk calibration among participants in the German Lung Cancer Screening Intervention trial.

    Design, Setting, and Participants  In this population-based diagnostic study, malignancy probabilities were estimated by applying 8 prediction models to data from 1159 participants in the intervention arm of the Lung Cancer Screening Intervention trial, a randomized clinical trial conducted from October 23, 2007, to April 30, 2016, with ongoing follow-up. This analysis considers end points up to 1 year after individuals’ last screening visit. Inclusion criteria for participants were at least 1 noncalcified pulmonary nodule detected on any of 5 annual screening visits, receiving a lung cancer diagnosis within the active screening phase of the Lung Cancer Screening Intervention trial, and an unequivocal identification of the malignant nodules. Data analysis was performed from February 1, 2019, through December 5, 2019.

    Interventions  Five annual rounds of low-dose multislice CT.

    Main Outcomes and Measures  Discrimination ability and calibration of malignancy probabilities estimated by 5 models developed in data from screening studies (4 Pan-Canadian Early Detection of Lung Cancer Study [PanCan] models using a parsimonious approach including nodule spiculation [PanCan-1b] or a comprehensive approach including nodule spiculation [PanCan-2b], and PanCan-2b replacing the nodule diameter variable with mean diameter [PanCan-MD] or volume [PanCan-VOL], as well as a model developed by the UK Lung Cancer Screening trial) and 3 models developed in clinical settings (US Department of Veterans Affairs, Mayo Clinic, and Peking University People’s Hospital).

    Results  A total of 1159 participants (median [range] age, 57.63 [50.34-71.89] years; 763 [65.8%] men) with 3903 pulmonary nodules were included in this study. For nodules detected in the prevalence round of CT, the PanCan models showed excellent discrimination (PanCan-1b: area under the curve [AUC], 0.93 [95% CI, 0.87-0.99]; PanCan-2b: AUC, 0.94 [95% CI, 0.89-0.99]; PanCan-MD: AUC, 0.94 [95% CI, 0.91-0.98]; PanCan-VOL: AUC, 0.94 [95% CI, 0.90-0.98]), and all of the screening models except PanCan-MD and PanCan-VOL showed acceptable calibration (PanCan-1b: Spiegelhalter z = −1.081; P = .28; PanCan-2b: Spiegelhalter z = 0.436; P = .67; PanCan-MD: Spiegelhalter z = 3.888; P < .001; PanCan-VOL: Spiegelhalter z = 1.978; P = .05; UK Lung Cancer Screening trial: Spiegelhalter z = −1.076; P = .28), whereas the other models showed worse discrimination and calibration, from an AUC of 0.58 (95% CI, 0.46-0.70) for the UK Lung Cancer Screening trial model to an AUC of 0.89 (95% CI, 0.82-0.97) for the Mayo Clinic model.

    Conclusions and Relevance  This diagnostic study found that PanCan models showed excellent discrimination and calibration in prevalence screenings, confirming their ability to improve nodule management in screening settings, although calibration to nodules detected in follow-up scans should be improved. The models developed by the Mayo Clinic, Peking University People’s Hospital, Department of Veterans Affairs, and UK Lung Cancer Screening Trial did not perform as well.

    Introduction

    The US National Lung Cancer Screening Trial (NLST)1 and other randomized clinical trials in Europe2-5 have shown that low-dose (LD) computed tomography (CT) is a viable screening tool for reducing lung cancer mortality among long-term smokers. However, screening entails harms in the form of false-positive screening test referrals and follow-up diagnostics. The efficiency of cancer screening using CT depends in part on optimized criteria for reporting and managing lung nodules; therefore, radiologic and oncologic expert societies have developed guidelines6-10 for using nodule characteristics as predictors of malignancy and as decision-making indicators to guide further diagnostic work.

    For nodules detected by CT for which there is an absence of information on nodule growth, several research groups have developed statistical models to determine the likelihood of malignancy based on radiologic features and patient-related attributes, such as age, sex, smoking history, presence of emphysema or other respiratory diseases, and personal or family history of cancer. Initial models were based on data from clinical settings, such as at the Mayo Clinic,11 US Department of Veterans Affairs (VA) clinics,12 or Peking University People’s Hospital (PKUPH).13 However, these models were fitted to data from patients with high pretest probability of malignancy, focused on larger, mostly solid and often solitary nodules detected incidentally or in symptomatic individuals, and may overestimate malignancy risk for nodules detected in screening contexts.14

    Malignancy prediction models were first developed in a screening setting at Brock University, Toronto, Canada, using data from prevalence screenings in the Pan-Canadian Early Detection of Lung Cancer Study (PanCan) LDCT screening trial.15 Initial PanCan models15 (sometimes also referred to as Brock models) were generated using either a parsimonious (hereafter, PanCan-1) or a comprehensive (hereafter, PanCan-2) approach for variable selection without (PanCan-1a and PanCan-2a) or with (PanCan-1b and PanCan-2b) nodule spiculation among the radiologic variables. The more comprehensive model including spiculation, PanCan-2b, was recently updated by replacing nodule diameter with multidimensional measurements: mean diameter (PanCan-MD) or volume (PanCan-VOL).16 Independently of PanCan, a new model was developed recently in the context of the UK Lung Cancer Screening (UKLS) trial17 using nodule volume.

    The PanCan models have been externally validated for discrimination in the NLST16,18 and Danish Lung Cancer Screening Trial,19 but they have not been externally validated for calibration. The recent UKLS model17 has not been externally validated, to our knowledge. Using data from the German Lung Cancer Screening Intervention (LUSI) trial,5,20 we evaluated the earlier and more recent versions of the PanCan models, as well as the UKLS model, in terms of discrimination ability, calibration, and operational performance (eg, sensitivity, specificity, and positive predictive values). For comparison, we also present findings for the Mayo Clinic, VA, and PKUPH models, as well as for pulmonary nodules first observed in follow-up (ie, incidence) screenings.

    Methods
    Study Design and Participants

    The LUSI trial5,20 is a screening trial among adults aged 50 to 69 years with a history of heavy smoking (defined as ≥25 years of smoking ≥15 cigarettes per day or ≥30 years smoking ≥10 cigarettes per day and ≤10 years since smoking cessation) randomized into a screening intervention group, which included an LD multislice CT scan at time of randomization and 4 annual follow-up screening examinations, and a control group with no intervention. Participants were recruited as a random sample from population registries in Heidelberg, Germany, and surrounding areas and were assigned to the CT screening group or the control group. All participants provided written informed consent. The LUSI trial was approved by the ethics committee of the University of Heidelberg and the German Federal Office for Radiation Protection. This analysis of LUSI trial data is covered under the original ethical board approval per regulations of the ethics committee of the University of Heidelberg. This study followed the Standards for Reporting of Diagnostic Accuracy (STARD) reporting guideline for diagnostic studies.

    Image Acquisition and Reading

    Computed tomography scans were obtained using a Toshiba 16-row scanner from October 23, 2007, to December 31, 2009, or a Siemens 128-row scanner from March 18, 2010, until the end of the screening phase of the LUSI trial on May 25, 2016. We used Median computer-aided detection software (Median Technologies) to read CT images, and nodule measures were derived, with largest diameter given in millimeters, perpendicular diameter given in millimeters, and volume given in millimeters-cubed. Perifissural nodules with oval or triangular shape and/or smooth delineations were excluded under the assumption that they were lymphatic nodules. Further data collected for each CT scan included nodule identifier, location, type (ie, solid or subsolid), shape and border characteristics (eg, presence of spiculation, translucence), and calcification. Presence of emphysema was determined on CT images using a densitometry method.21

    Nodule Evaluation and Management

    All CT images were evaluated by 2 trained chest radiologists (including S.D.), and decisions were made based on nodule size and growth (in cases of nodules that had previously been recorded) (eTable 1 in the Supplement). Participants with a lung cancer diagnosis made later than 1 year after the last CT scan or for whom the malignant nodule(s) could not be identified in earlier CT images were excluded.

    Spirometry Data

    In all study participants, spirometry was performed using MasterScreen IOS (VIASYS Healthcare) to determine 1-second forced expiratory volume (FEV1) and forced vital capacity (FVC), and FEV1 to FVC ratios were calculated from the largest FEV1 and FVC values recorded in any 1 of 2 repeated assessments. Detailed descriptions of imaging, nodule, and lung function assessments are given in the eAppendix in the Supplement.

    Statistical Analysis

    All nodules detected during screening were analyzed using data from the LDCT image in which they were first seen. For participant-related characteristics, we used the Mann-Whitney U test for differences in continuous data and the χ2 test or Fisher exact test for categorical data, as appropriate. For differences in nodule-specific characteristics, we used mixed-effects logistic regression with participant as the random effect.22 We applied 8 models from 6 published studies to our data to calculate the probability of nodule malignancy (eTable 2 in the Supplement): 4 PanCan models (PanCan-1b15: parsimonious model including spiculation; PanCan-2b15: full model including spiculation; PanCan-MD16: PanCan-2b with the mean of the largest and perpendicular diameters as nodule size; and PanCan-VOL16: PanCan-2b with nodule volume as nodule size), the recently developed UKLS model17; and the models developed in VA clincs,12 the Mayo Clinic,11 and PKUPH clinics.13

    We explored associations of radiologic parameters and participant-related characteristics with risk of nodule malignancy by fitting multivariable logistic regression models via generalized estimating equations using prevalence and incidence screening rounds combined and accounting for the correlation structure of multiple pulmonary nodules per individual on the LUSI data. Model selection was based on the quasi–Akaike Information Criterion23 (eAppendix in the Supplement).

    For each selected model, the ability to discriminate nodules by malignancy status was evaluated through cluster-adjusted receiver operating characteristic curves and the corresponding area underneath the curve (AUC).24 Given the correlation structure of nodules within participants, sensitivity, specificity, and positive and negative predictive values were estimated using generalized estimating equations.25-27 For comparison, we also estimated the discrimination capacity of models fitted on LUSI databased on a larger set of variables using cluster bootstrapping (B = 1000).24 Model calibration was assessed by examining observed vs predicted nodule malignancy rates by category of nodule size (<5 mm, 5 to <8 mm, 8 to 10 mm, or >10 mm; as used in the LUSI CT evaluation algorithm) as well as by deciles of predicted risk. The Hosmer-Lemeshow goodness of fit test28 was used to examine the fit between predicted and observed malignancy probabilities across deciles of predicted risk, and Brier scores (BS) and Spiegelhalter z test were used to assess the overall deviance of risk predictions estimated by models vs observed rates.29 Hosmer-Lemeshow goodness of fit and χ2 for independence P values were 1-sided, and all other P values were 2-sided. Statistical significance was set at .05.

    All analyses were performed using R statistical software for version 3.5.1 (R Project for Statistical Computing), with packages gee, Hmisc, lme4, MuMIn, rms, pROC, and ROCR, as well as the clusteredROC function. Data analysis was performed from February 1, 2019 to December 5, 2019.

    Results
    Study Participants and Pulmonary Nodule Findings

    During 5 screening rounds, 1182 participants in the LUSI LDCT arm showed at least 1 noncalcified pulmonary nodule. A total of 62 participants were diagnosed with lung cancer up to 12 months after their last LDCT screening participation: 56 cancers detected via screening and 6 interval cancers.5 For 54 of 56 cancers detected via screening, malignancy was linked to 1 (51 participants) or more (3 participants) nodules. For 2 cancers detected via screening without nodules, the decision of referral to further diagnostic work was based on other pulmonary abnormalities. For 2 of 6 interval cases, no nodules were observed on LDCT screenings; for the remaining 4 cases with nonsuspicious nodules, no information was available to unequivocally link any of these to malignancy. Nodules with benign-appearing calcification and individuals with lung cancer whose lung tumors could not be linked back to a specific nodule were excluded from statistical analyses.

    After removing participants with a lung cancer diagnosis made later than 1 year after the last CT scan or for whom the malignant nodule(s) could not be identified in earlier CT images, data was available for 3903 pulmonary nodules from 1159 individuals (median [range] age, 57.63 [50.34-71.89] years; 763 [65.8%] men) (eFigure 1 in the Supplement). Detailed nodule counts by diameter, type (solid vs subsolid), malignancy status, and screening round of first observation (prevalence or incidence) are presented in eTable 3 in the Supplement. Of 3903 nodules observed, 2883 nodules (73.9%; 32 of these [1.1%] identified as malignant) were first seen during the prevalence screening, whereas 1020 nodules (26.1%; 31 of these [3.0%] identified as malignant), were first observed in 1 of 4 incidence screenings. Irrespective of screening round, more than 70% of malignant nodules (prevalence round: 25 of 32 nodules [78.1%]; incidence rounds: 25 of 31 nodules [80.6%]) had a diameter of 8 mm or greater. For these nodules 8 mm or greater, the malignancy rate was higher among nodules first observed in the prevalence round (12.8% [95% CI, 8.6%-18.5%]) than among those first observed in the incidence round (8.6% [95% CI, 5.7%-12.5%]); conversely, mean malignancy rate was higher for nodules smaller than 8 mm first observed in incidence rounds (0.8% [95% CI, 0.3%-1.9%]) than those first observed in the prevalence round (0.3% [95% CI, 0.1%- 0.6%]).

    Irrespective of screening round of first detection, compared with benign nodules, malignant nodules were significantly more often located in the left upper lobe (561 nodules [14.6%] vs 23 nodules [36.5%]; P < .001), were larger in terms of diameter (median [range], 4.8 [2.2-65.6] mm vs 11.6 [3.5-107.7] mm; P < .001) and volume (median [range], 41.4 [5.0-28947] mm3 vs 357.1 [19.3-17466] mm3; P < .001) and more often had spiculated borders (193 nodules [5.0%] vs 23 nodules [36.5%]; P < .001) (Table 1). Compared with patients with benign nodules, patients with malignant nodules were statistically significantly more likely to be older (median [range] age, 57.38 [50.34-71.89] years vs 59.88 [51.90-69.98] years; P < .001), have emphysema (474 patients [42.9%] vs 35 patients [64.8%]; P = .002), and have lower FEV1 (median [range], 2.88 [0.66-6.11] L vs 2.66 [1.26-4.24] L; P = .009); whereas sex, smoking status at randomization, smoking duration and intensity, self-reported history of extrathoracic cancer, years since smoking cessation, presence of asthma or bronchitis, and FVC showed no significant association with malignancy diagnosis in the LUSI data (Table 2). Data on family history of cancer were not available in our data set.

    Evaluation of Predictive Models

    For nodules detected at the participants’ first screening (prevalence round), all PanCan models showed high discrimination accuracy. Among the PanCan variants, the comprehensive PanCan-2b model achieved only marginally better discrimination (AUC, 0.94 [95% CI, 0.89-0.99]) than the parsimonious PanCan-1b model (AUC, 0.93 [95% CI, 0.87-0.99]), and there was no improvement in discrimination with the use of volume in the PanCan-VOL model (AUC, 0.90 [95% CI, 0.90-0.98]) or 2-dimensional perpendicular mean diameter in the PanCan-MD model (AUC, 0.94 [95% CI, 0.91-0.98]). The discrimination by UKLS was poor (AUC, 0.58 [95% CI, 0.46-0.70]). A reduced UKLS model (eTable 2 in the Supplement) that ignored variables with definitions or prevalence markedly differing from LUSI performed better in our data (prevalence round: AUC, 0.79 [95% CI, 0.68-0.89]; incidence rounds: AUC, 0.60 [95% CI, 0.49-0.70]) compared with the original version. For comparison, the models originally developed in clinical settings showed moderately good discrimination, (VA : AUC, 0.84 [95% CI, 0.76-0.92]; Mayo Clinic: AUC, 0.89 [95% CI, 0.82-0.97]; PKUPH: AUC, 0.87 [95% CI, 0.79-0.995]) (Figure).

    Compared with the prevalence round, all models (except for UKLS) showed useful, although reduced, discrimination when applied to nodules first noticed in any of the incidence rounds (PanCan-1b: AUC, 0.93 [95% CI, 0.87-0.99]; PanCan-2b: AUC, 0.94 [95% CI, 0.89-0.99]; PanCan-MD: AUC, 0.94 [95% CI, 0.91-0.98]; PanCan-VOL: AUC, 0.94 [95% CI, 0.90-0.98]; UKLS: AUC, 0.58 [95% CI, 0.46-0.70]; VA: AUC, 0.84 [95% CI, 0.76-0.92]; Mayo Clinic: AUC, 0.89 [95% CI, 0.82-0.97]; PKUPH: AUC, 0.87 [95% CI, 0.79-0.97]) (eFigure 2 in the Supplement). This difference may be explained by reduced variability in nodule size (Table 1; eTable 3 in the Supplement).

    Regarding model calibration, for nodules detected at the prevalence screen, visual inspection of predicted absolute probabilities of malignancy and observed malignancy rates within categories defined by nodule size suggested best agreement for PanCan-1b (Table 3; eTable 4 in the Supplement). Additional comparisons within deciles of the predicted probability scores (eTable 5 and eTable 6 in the Supplement) combined with Hosmer-Lemeshow (HL) tests for deviance between predicted and observed rates showed acceptable calibration for PanCan-1b (HL = 7.71; P = .56), PanCan-2b (HL = 7.23; P = .61), and PanCan-VOL (HL = 10.89; P = .28) but not for PanCan-MD (HL = 30.53; P < .001) or UKLS (HL = 158.99; P < .001) (eTable 7 in the Supplement). Alternatively, using BSs and Spiegelhalter z tests, acceptable calibration was found for PanCan-1b (BS = 0.009; Spiegelhalter z = −1.081; P = .28), PanCan-2b (BS = 0.009; Spiegelhalter z = 0.436; P = .67), and UKLS (BS = 0.012; Spiegelhalter z = −1.076; P = .28) but not for PanCan-VOL (BS = 0.009; Spiegelhalter z = 1.978; P = .05) or PanCan-MD (BS = 0.009, Spiegelhalter z = 3.888; P < .001) (eTable 7 in the Supplement). Malignancy probabilities estimated by models developed in clinical contexts were all strongly overestimated (VA: Spiegelhalter z = −25.24; P < .001; Mayo Clinic: Spiegelhalter z = −12.63; P < .001; PKUPH: Spiegelhalter z = −19.35; P < .001) (Table 3) and were poorly calibrated according to all tests performed (VA: BS = 0.063; HL = 774.38; P < .001; Mayo Clinic: BS = 0.014; HL = 162.32; P < .001; PKUPH: BS = 0.094; HL = 1119.9; P < .001). None of the models showed acceptable calibration on nodules first observed in incidence screens.

    Table 4 shows estimated sensitivity, specificity, and positive and negative predictive values for malignancy at 2%, 5% or 10% model probability thresholds for nodules observed in the prevalence screening for the PanCan and UKLS models. PanCan-1b yielded highest sensitivities at 2% risk threshold (0.81 [95% CI, 0.67-0.94]), and 10% risk threshold (0.52 [95% CI, 0.32-0.71]), but lower specificities (2%: 0.90 [95% CI, 0.89-0.92]; 10%: 0.99 [95% CI, 0.98-0.99]) than the other models. The highest specificities at the 2% and 10% risk thresholds were observed for PanCan-MD (2%: 0.96 [95% CI, 0.95-0.97]; 10%: 0.99 [95% CI, 0.99-1.00]) but were accompanied by lower sensitivities (2%: 0.72 [95% CI, 0.56-0.86]; 10%: 0.43 [95% CI, 0.25-0.61]). The lowest positive predictive values at 2% or 5% risk thresholds were for PanCan-1b (0.08 [95% CI, 0.05-0.12]; 5%: 0.19 [95% CI, 0.11-0.27]), while the highest were for PanCan-MD (2%: 0.17 [95% CI, 0.10-0.24]; 5%: 0.33 [95% CI, 0.20-0.45]). On our data, PanCan-1b showed very similar estimates for sensitivity and specificity compared with those from the original PanCan study data.15 However, the recent PanCan-VOL and PanCan-MD updates in PanCan showed higher sensitivity and lower specificity in our study but similar PPV.16 External validation of PanCan-2b in the NLST also showed higher sensitivity, lower specificity, and comparable PPV18 compared with estimates in LUSI (Table 4). The UKLS model showed inferior sensitivity (2%: 0.25 [95% CI, 0.11-0.39]; 5%: 0.20 [95% CI, 0.07-0.33]; 10%: 0.14 [95% CI, 0.01-0.27]) and specificity (2%: 0.83 [95% CI, 0.81-0.85]; 5%: 0.92 [95% CI, 0.91-0.94]; 10%: 0.97 [95% CI, 0.96-0.98]) compared with PanCan models.

    Logistic regression models fitted via generalized estimating equations on LUSI data combined with model selection via backward elimination based on the quasi–Akaike Information Criterion resulted in a model retaining age (β coefficient = 0.06 [95% CI, 0.01 to 0.11]; P = .02), years since quitting smoking (β coefficient = 0.83 [95% CI, −0.15 to 1.82]; P = .10), bronchitis (β coefficient = −1.23 [95% CI, −2.29 to −0.17]; P = .03), nodule mean diameter (β coefficient = 0.14 [95% CI, 0.09 to 0.19]; P < .001), nodule location (β coefficient = 1.23 [95% CI, 0.35 to 2.11]; P = .01), and spiculation (β coefficient = 1.72 [95% CI, 1.02 to 2.42]; P < .001) as significant variables (eTable 8 in the Supplement), whereas sex, self-reported history of extrathoracic cancer, smoking duration, emphysema (based on CT results), FVC, nodule type, and nodule count per scan showed no further association with malignancy. This final model yielded discrimination AUC of 0.90 (95% CI, 0.83-0.93)24 (bootstrap AUC, 0.88 [95% CI, 0.84-0.92]) for nodules detected in the prevalence round and 0.81 (95% CI, 0.71-0.90) (bootstrap AUC, 0.81 [95% CI, 0.73-0.87]) for nodules first detected in the incidence round.

    Discussion

    Using data of the German LUSI trial, this diagnostic study found that 4 variants of the PanCan prediction model, developed in context of the Canadian Early Detection of Lung Cancer Study, each provided good discrimination between malignant and nonmalignant pulmonary nodules observed at individuals’ first (prevalence) LDCT screening, whereas more moderate discrimination was observed for 3 models originally developed on the basis of patient data from the Mayo Clinic and VA and PKUPH clinics. By contrast, discrimination for the model recently developed in context of the UKLS was poor.

    To our knowledge, these analyses are the first to provide external validation of the recent UKLS model17 and a complete external validation for PanCan-MD and PanCan-VOL models.16 Our analyses showed no clear superiority for the PanCan-VOL or PanCan-MD models over the older PanCan model versions, confirming observations in PanCan16 but not those from Horeweg et al.30 In LUSI, diameters were retrieved by software, thus based on a 3-dimensional estimation, which possibly led to a more accurate estimation of nodule diameter compared with manual diameter measurements used in other studies.31 Overall, our observations are in line with those from other validation studies, in context of incidentally or symptomatically detected pulmonary nodules13,32-38 or in LDCT screening settings,14,17-19,39,40 which mostly have also shown discrimination indices of AUCs of approximately 0.80 and higher, and similar ranking by discrimination capacity when comparing performance of the PanCan, VA, Mayo Clinic, and PKUPH models.

    Aside from good discrimination, PanCan-1b, PanCan-2b, and PanCan-VOL, but not PanCan-MD or UKLS, also showed acceptable calibration of estimated absolute malignancy probabilities when applied to data from the prevalence screening. In the original study leading to development of the initial PanCan models, acceptable calibration was reported on validation on data from a chemoprevention trial by the British Columbia Cancer Agency,15 and for PanCan-1b and PanCan-2b in NLST data14,18 and a pilot LDCT screening trial in Queensland, Australia.39,40 For the more recent PanCan-MD and PanCan-VOL models, by contrast, Tammemägi et al16 reported lower calibration accuracy by such tests in NLST data. The PanCan-1b model showed similar sensitivity, specificity, and positive predictive values at selected (2%, 5%, or 10%) probability thresholds in LUSI and PanCan data,15 while among LUSI, PanCan, and NLST data, these estimates were somewhat more variable for PanCan-2b, PanCan-VOL, and PanCan-MD.16,18 For the models developed in clinical settings, we found overestimated malignancy risk estimates, which were also observed for the VA and Mayo Clinic models on NLST data.14 None of the models (including the PanCan variants) showed acceptable calibration when applied to nodules first observed in incidence screens.

    Depending on the context (ie, clinical or screening) in which they were developed, malignancy prediction models include different sets of predictor variables. While age is an established factor for lung cancer risk, it was not retained as predictor in most PanCan models (PanCan-1b, PanCan-MD, or PanCan-VOL) or in those fitted in LUSI. However, in the UKLS model, age appeared together with smoking duration. With regard to smoking, while lifetime smoking status (ever vs never) is not relevant in LUSI or other screening contexts, including UKLS and PanCan, as screening generally is targeted to long-term smokers, our data and the analyses for the development of UKLS model17 show that in multivariable models that include age and detailed nodule characteristics, additional smoking-related information may improve malignancy prediction in combination with CT imaging, even among individuals at high risk who are eligible for lung cancer screening.

    Sex, associated with increased risk of malignancy for women based on the PanCan and UKLS models, showed no predictive value in our data and was found to be associated with lower malignancy risk in the Danish Lung Cancer Screening Trial.19 The female-to-male sex ratio for lung cancer incidence varies across populations and by age, in relation to variable prevalence of and changing trends in smoking habits among women,41 which could explain population differences in the predictive value of sex for nodule malignancy. Also, among patients with lung cancer, women more often have adenocarcinomas as opposed to other histologic tumor subtypes.5,42 Adenocarcinomas develop from slowly growing nodules with longer lead time until clinical relevance, which makes a benefit from lung cancer screening in terms of mortality reduction more likely.2,5,42 Thus, it may be relevant to refine malignancy prediction by sex and tailor models to specific screening populations.

    The PanCan and UKLS models, which were fitted on data from a first prevalence screening, may be less applicable to new nodules in follow-up screenings.16 In NLST, the malignancy risk of nodules 4 to 6 mm or larger or 6 to 8 mm or larger was higher than that of nodules found at the baseline screening, and malignancies associated with new nodules were significantly less likely to be adenocarcinomas.43 We could not confirm these observations in LUSI data because of small numbers (eTable 3 in the Supplement).

    Strengths and Limitations

    Our study has some strengths, including that (with only 2 exceptions), malignancy was linked to specific nodules, as opposed to NLST in which malignancy was assumed to be always linked the largest nodule observed.14,16,18 Also, our study is the first, to our knowledge, to compare and document differences in the performance of malignancy risk predictions for models applied to nodules detected in prevalence or incidence screenings and for models originally developed in screening vs clinical settings.

    Our study also has some limitations. One limitation is the missing information in LUSI for several risk factors, including family history of lung or extrathoracic cancer and possible differences in CT-based assessment of emphysema as compared with PanCan.

    A limitation of all models examined in this analysis is that they do not incorporate measures of longitudinal nodule growth as a predictor of malignancy. For nodules of intermediate size, current diagnostic criteria for determining malignancy include early recall examinations for the determination of volume doubling time of nodules observed at an individual’s first screening or a direct assessment of longitudinal growth for nodules already noted at earlier visits when screening participants return for annual follow-up screenings. Detection algorithms are now being developed that integrate relevant nodule and nonnodule features on repeated CT screening examinations over time to predict the presence of lung cancer.30,44

    Variables difficult to standardize across populations, such as lung disease diagnosis and nodule type categories, may hinder model transferability. In fact, a reduced UKLS model (eTable 2 in the Supplement) that ignored variables with definitions or prevalence markedly differing from LUSI performed better in our data compared with the original version. A more general limitation not only of LUSI but also other studies, including PanCan and UKLS, is the relatively small study size, with 2028 participants at baseline screenings in UKLS, 2029 participants at baseline screenings in LUSI, and 2537 participants at baseline screenings in PanCan, and limited numbers of malignant nodules observed. Accuracy of variable selection and model calibration may be improved in data from larger studies (eg, based on pooled data worldwide).

    Conclusions

    Our findings suggest that the PanCan models have good discrimination and confirmatory evidence for calibration accuracy of predicted malignancy risks when applied to nodules observed in an individual’s first (prevalence) screening examination, suggesting that such models may become useful tools for optimizing nodule management in population screening settings. Estimates of current models seem most applicable to nodules detected on a first screening examination, the specific context in which PanCan and UKLS were developed. For individuals screened at regular intervals, as in organized screening programs, models may be further developed to incorporate estimates of nodule volume doubling times, determined on early recall follow-up CT or determined directly for nodules already noted on an earlier visit when individuals return for annual incidence screenings.44

    Back to top
    Article Information

    Accepted for Publication: December 16, 2019.

    Published: February 14, 2020. doi:10.1001/jamanetworkopen.2019.21221

    Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2020 González Maldonado S et al. JAMA Network Open.

    Corresponding Author: Rudolf Kaaks, PhD, Division of Cancer Epidemiology, German Cancer Research Centre, Im Neuenheimer Feld 581, Heidelberg 69120, Germany (r.kaaks@dkfz.de).

    Author Contributions: Dr Kaaks had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    Concept and design: González Maldonado, Delorme, Kauczor, Heussel, Kaaks.

    Acquisition, analysis, or interpretation of data: All authors.

    Drafting of the manuscript: González Maldonado, Delorme, Kaaks.

    Critical revision of the manuscript for important intellectual content: González Maldonado, Delorme, Hüsing, Motsch, Kauczor, Heussel.

    Statistical analysis: González Maldonado, Hüsing, Kaaks.

    Obtained funding: Kauczor.

    Administrative, technical, or material support: González Maldonado, Delorme, Motsch, Kauczor, Heussel, Kaaks.

    Supervision: Delorme, Motsch, Kauczor, Heussel, Kaaks.

    Conflict of Interest Disclosures: Dr Delorme reported receiving grants from German Research Foundation (DFG) and Dietmar Hopp Foundation during the conduct of the study and serving as a member of the German Radiation Protection Commission and as the chair of its Committee on Radiation Protection in Medicine. Dr Kauczor reported receiving grants from DFG and Dietmar Hopp Foundation during the conduct of the study; grants, personal fees, and nonfinancial support from Philips; grants and personal fees from Siemens; non-financial support from Bayer; and personal fees from Boehringer Ingelheim, Merck Sharp & Dohme, and AstraZeneca outside the submitted work. Dr Heussel reported serving as a member of the German Center for Lung Research, and founding member of the Working Team in Infections in Immunocompromized Hosts of the German Society of Hematology/Oncology, faculty member of European Society of Thoracic Radiology, European Respiratory Society, and member in European Imaging Biomarkers Alliance; owning stock in GlaxoSmithKline and a patent for a method and device for representing the microstructure of the lungs (IPC8 Class: AA61B5055FI; PAN: 20080208038), and serving as a consultant or receiving personal fees from the European Conference on Infections in Leukemia, European Congress of Clinical Microbiology and Infectious Diseases, European Organization for Research and Treatment of Cancer Mycose Study Group, Schering-Plough, Pfizer, Basilea Pharmaceutica, Boehringer Ingelheim, Novartis, Roche Holding, Astellas Pharma, Gilead Sciences, Merck Sharpe & Dohme, Eli Lilly and Company, InterMune, and Fresenius Medical Care; receiving research funding from Siemens, Pfizer, MeVis Medical Solutions, Boehringer Ingelheim, and the German Center for Lung Research; receiving lecture fees from Gilead Sciences, Essex Pharma, Schering-Plough, AstraZeneca, Eli Lilly and Company, Roche, MSD, Pfizer, Bracco, Meda, InterMune, Chiesi Farmaceutici, Siemens, Covidien, Pierre Fabre, Boehringer Ingelheim, Grifols, Novartis, Basilea, and Bayer; serving on committees for the Chest Working Group of the German Roentgen Society, National Guidelines for Bronchial Carcinoma, Mesothelioma, Chronic Obstructive Pulmonary Disease, Screening for Bronchial Carcinoma, Computed Tomography and Magnetic Resonance Imaging of the Chest, Pneumonia, Hospital-Acquired Pneumonia; serving as editor of Medizinische Klinik, Intensivmedizin und Notfallmedizin at Springer publishing. No other disclosures were reported.

    Funding/Support: The study was funded by the Dietmar Hopp-Foundation together with the German Research Foundation (BE 2486/2-1) from 2007 to 2010, and by the German Research Foundation (BE 2486/2-2) from 2010 to 2013. Dr Hüsing was supported by grants from the German Center for Lung Research.

    Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    Additional Contributions: From 2007 to 2011, Marie-Louise Gross, MD, carried out recruitment with initial patient information and randomization. Kirsten Lenner-Fertig, BSc, analyzed blood samples and organized freezer storage. Andrea Albrecht, BSc, and Ulrike Beckhaus, BSc, mailed the annual questionnaires, performed the scanning of the filled-in questionnaires and data entry into the database, and kept telephone contact in case of doubtful answers or missing feedback. The low-dose computed tomography scans were performed by Jessica Engelhardt, BSc, and Martina Jochim, BSc. Jan Tremper, MD; Monica Eichinger, MD; Daiva Elzbieta Optazaite, MD; Michael Puderbach, MD; and Mark Wielpütz, MD, provided radiologic evaluations of the low-dose computed tomography images. They were not compensated for their contributions outside of their normal salaries.

    References
    1.
    Aberle  DR, Adams  AM, Berg  CD,  et al; National Lung Screening Trial Research Team.  Reduced lung-cancer mortality with low-dose computed tomographic screening.  N Engl J Med. 2011;365(5):395-409. doi:10.1056/NEJMoa1102873PubMedGoogle ScholarCrossref
    2.
    International Association for the Study of Lung Cancer.  Nelson study shows CT screening for nodule volume management reduces lung cancer mortality by 26 percent in men [press release]. IASLC 19th World Conference on Lung Cancer. https://wclc2018.iaslc.org/media/2018%20WCLC%20Press%20Program%20Press%20Release%20De%20Koning%209.25%20FINAL%20.pdf. Accessed January 8, 2020.
    3.
    Paci  E, Puliti  D, Lopes Pegna  A,  et al; the ITALUNG Working Group.  Mortality, survival and incidence rates in the ITALUNG randomised lung cancer screening trial.  Thorax. 2017;72(9):825-831. doi:10.1136/thoraxjnl-2016-209825PubMedGoogle ScholarCrossref
    4.
    Pastorino  U, Silva  M, Sestini  S,  et al.  Prolonged lung cancer screening reduced 10-year mortality in the MILD trial: new confirmation of lung cancer screening efficacy.  Ann Oncol. 2019;30(7):1162-1169. doi:10.1093/annonc/mdz117PubMedGoogle ScholarCrossref
    5.
    Becker  N, Motsch  E, Trotter  A,  et al.  Lung cancer mortality reduction by LDCT screening: results from the randomized German LUSI trial [published online June 4, 2019].  Int J Cancer. doi:10.1002/ijc.32486PubMedGoogle Scholar
    6.
    MacMahon  H, Naidich  DP, Goo  JM,  et al.  Guidelines for management of incidental pulmonary nodules detected on CT images: from the Fleischner Society 2017.  Radiology. 2017;284(1):228-243. doi:10.1148/radiol.2017161659PubMedGoogle ScholarCrossref
    7.
    Wood  DE, Kazerooni  E, Baum  SL,  et al; National comprehension cancer network.  Lung cancer screening, version 1.2015: featured updates to the NCCN guidelines.  J Natl Compr Canc Netw. 2015;13(1):23-34. doi:10.6004/jnccn.2015.0006PubMedGoogle ScholarCrossref
    8.
    Kanne  JP, Jensen  LE, Mohammed  TL,  et al; Expert Panel on Thoracic Imaging.  ACR appropriateness Criteria radiographically detected solitary pulmonary nodule.  J Thorac Imaging. 2013;28(1):W1-3. doi:10.1097/RTI.0b013e31827657c8PubMedGoogle ScholarCrossref
    9.
    American College of Radiology.  Lung CT Screening Reporting & Data System. https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Lung-Rads. Accessed January 8, 2020.
    10.
    Baldwin  DR, Callister  ME; Guideline Development Group.  The British Thoracic Society guidelines on the investigation and management of pulmonary nodules.  Thorax. 2015;70(8):794-798. doi:10.1136/thoraxjnl-2015-207221PubMedGoogle ScholarCrossref
    11.
    Swensen  SJ, Silverstein  MD, Ilstrup  DM, Schleck  CD, Edell  ES.  The probability of malignancy in solitary pulmonary nodules: application to small radiologically indeterminate nodules.  Arch Intern Med. 1997;157(8):849-855. doi:10.1001/archinte.1997.00440290031002PubMedGoogle ScholarCrossref
    12.
    Gould  MK, Ananth  L, Barnett  PG; Veterans Affairs SNAP Cooperative Study Group.  A clinical model to estimate the pretest probability of lung cancer in patients with solitary pulmonary nodules.  Chest. 2007;131(2):383-388. doi:10.1378/chest.06-1261PubMedGoogle ScholarCrossref
    13.
    Li  Y, Chen  KZ, Wang  J.  Development and validation of a clinical prediction model to estimate the probability of malignancy in solitary pulmonary nodules in Chinese people.  Clin Lung Cancer. 2011;12(5):313-319. doi:10.1016/j.cllc.2011.06.005PubMedGoogle ScholarCrossref
    14.
    Nair  VS, Sundaram  V, Desai  M, Gould  MK.  Accuracy of models to identify lung nodule cancer risk in the National Lung Screening Trial.  Am J Respir Crit Care Med. 2018;197(9):1220-1223. doi:10.1164/rccm.201708-1632LEPubMedGoogle ScholarCrossref
    15.
    McWilliams  A, Tammemagi  MC, Mayo  JR,  et al.  Probability of cancer in pulmonary nodules detected on first screening CT.  N Engl J Med. 2013;369(10):910-919. doi:10.1056/NEJMoa1214726PubMedGoogle ScholarCrossref
    16.
    Tammemägi  M, Ritchie  AJ, Atkar-Khattra  S,  et al; Pan-Canadian Early Detection of Lung Cancer Study Group.  Predicting malignancy risk of screen-detected lung nodules-mean diameter or volume.  J Thorac Oncol. 2019;14(2):203-211. doi:10.1016/j.jtho.2018.10.006PubMedGoogle ScholarCrossref
    17.
    Marcus  MW, Duffy  SW, Devaraj  A,  et al.  Probability of cancer in lung nodules using sequential volumetric screening up to 12 months: the UKLS trial.  Thorax. 2019;74(8):761-767. doi:10.1136/thoraxjnl-2018-212263PubMedGoogle ScholarCrossref
    18.
    White  CS, Dharaiya  E, Campbell  E, Boroczky  L.  The Vancouver Lung Cancer Risk Prediction model: assessment by using a subset of the National Lung Screening Trial cohort.  Radiology. 2017;283(1):264-272. doi:10.1148/radiol.2016152627PubMedGoogle ScholarCrossref
    19.
    Winkler Wille  MM, van Riel  SJ, Saghir  Z,  et al.  Predictive accuracy of the PanCan lung cancer risk prediction model: external validation based on CT from the Danish Lung Cancer Screening Trial.  Eur Radiol. 2015;25(10):3093-3099. doi:10.1007/s00330-015-3689-0PubMedGoogle ScholarCrossref
    20.
    Becker  N, Motsch  E, Gross  ML,  et al.  Randomized study on early detection of lung cancer with MSCT in Germany: results of the first 3 years of follow-up after randomization.  J Thorac Oncol. 2015;10(6):890-896. doi:10.1097/JTO.0000000000000530PubMedGoogle ScholarCrossref
    21.
    Jobst  BJ, Weinheimer  O, Trauth  M,  et al.  Effect of smoking cessation on quantitative computed tomography in smokers at risk in a lung cancer screening population.  Eur Radiol. 2018;28(2):807-815. doi:10.1007/s00330-017-5030-6PubMedGoogle ScholarCrossref
    22.
    Jiang  J.  Linear and Generalized Linear Mixed Models and Their Applications. New York, NY: Springer-Verlag; 2007.
    23.
    Pan  W.  Akaike’s information criterion in generalized estimating equations.  Biometrics. 2001;57(1):120-125. doi:10.1111/j.0006-341X.2001.00120.xPubMedGoogle ScholarCrossref
    24.
    Obuchowski  NA.  Nonparametric analysis of clustered ROC curve data.  Biometrics. 1997;53(2):567-578. doi:10.2307/2533958PubMedGoogle ScholarCrossref
    25.
    Smith  PJ, Hadgu  A.  Sensitivity and specificity for correlated observations.  Stat Med. 1992;11(11):1503-1509. doi:10.1002/sim.4780111108PubMedGoogle ScholarCrossref
    26.
    Coughlin  SS, Trock  B, Criqui  MH, Pickle  LW, Browner  D, Tefft  MC.  The logistic modeling of sensitivity, specificity, and predictive value of a diagnostic test.  J Clin Epidemiol. 1992;45(1):1-7. doi:10.1016/0895-4356(92)90180-UPubMedGoogle ScholarCrossref
    27.
    Genders  TS, Spronk  S, Stijnen  T, Steyerberg  EW, Lesaffre  E, Hunink  MG.  Methods for calculating sensitivity and specificity of clustered data: a tutorial.  Radiology. 2012;265(3):910-916. doi:10.1148/radiol.12120509PubMedGoogle ScholarCrossref
    28.
    Lemeshow  S, Hosmer  DW  Jr.  A review of goodness of fit statistics for use in the development of logistic regression models.  Am J Epidemiol. 1982;115(1):92-106. doi:10.1093/oxfordjournals.aje.a113284PubMedGoogle ScholarCrossref
    29.
    Rufibach  K.  Use of Brier score to assess binary predictions.  J Clin Epidemiol. 2010;63(8):938-939. doi:10.1016/j.jclinepi.2009.11.009PubMedGoogle ScholarCrossref
    30.
    Horeweg  N, van Rosmalen  J, Heuvelmans  MA,  et al.  Lung cancer probability in patients with CT-detected pulmonary nodules: a prespecified analysis of data from the NELSON trial of low-dose CT screening.  Lancet Oncol. 2014;15(12):1332-1341. doi:10.1016/S1470-2045(14)70389-4PubMedGoogle ScholarCrossref
    31.
    Han  D, Heuvelmans  MA, Vliegenthart  R,  et al.  Influence of lung nodule margin on volume- and diameter-based reader variability in CT lung cancer screening.  Br J Radiol. 2018;91(1090):20170405. doi:10.1259/bjr.20170405PubMedGoogle Scholar
    32.
    Schultz  EM, Sanders  GD, Trotter  PR,  et al.  Validation of two models to estimate the probability of malignancy in patients with solitary pulmonary nodules.  Thorax. 2008;63(4):335-341. doi:10.1136/thx.2007.084731PubMedGoogle ScholarCrossref
    33.
    Isbell  JM, Deppen  S, Putnam  JB  Jr,  et al.  Existing general population models inaccurately predict lung cancer risk in patients referred for surgical evaluation.  Ann Thorac Surg. 2011;91(1):227-233. doi:10.1016/j.athoracsur.2010.08.054PubMedGoogle ScholarCrossref
    34.
    Xiao  F, Liu  D, Guo  Y,  et al.  Novel and convenient method to evaluate the character of solitary pulmonary nodule-comparison of three mathematical prediction models and further stratification of risk factors.  PLoS One. 2013;8(10):e78271. doi:10.1371/journal.pone.0078271PubMedGoogle Scholar
    35.
    Deppen  SA, Blume  JD, Aldrich  MC,  et al.  Predicting lung cancer prior to surgical resection in patients with lung nodules.  J Thorac Oncol. 2014;9(10):1477-1484. doi:10.1097/JTO.0000000000000287PubMedGoogle ScholarCrossref
    36.
    Al-Ameri  A, Malhotra  P, Thygesen  H,  et al.  Risk of malignancy in pulmonary nodules: a validation study of four prediction models.  Lung Cancer. 2015;89(1):27-30. doi:10.1016/j.lungcan.2015.03.018PubMedGoogle ScholarCrossref
    37.
    Massion  PP, Healey  GF, Peek  LJ,  et al.  Autoantibody signature enhances the positive predictive power of computed tomography and nodule-based risk models for detection of lung cancer.  J Thorac Oncol. 2017;12(3):578-584. doi:10.1016/j.jtho.2016.08.143PubMedGoogle ScholarCrossref
    38.
    Chung  K, Mets  OM, Gerke  PK,  et al.  Brock malignancy risk calculator for pulmonary nodules: validation outside a lung cancer screening population.  Thorax. 2018;73(9):857-863. doi:10.1136/thoraxjnl-2017-211372PubMedGoogle ScholarCrossref
    39.
    Zhao  H, Marshall  HM, Yang  IA,  et al.  Screen-detected subsolid pulmonary nodules: long-term follow-up and application of the PanCan lung cancer risk prediction model.  Br J Radiol. 2016;89(1060):20160016. doi:10.1259/bjr.20160016PubMedGoogle Scholar
    40.
    Marshall  HM, Zhao  H, Bowman  RV,  et al.  The effect of different radiological models on diagnostic accuracy and lung cancer screening performance.  Thorax. 2017;72(12):1147-1150. doi:10.1136/thoraxjnl-2016-209624PubMedGoogle ScholarCrossref
    41.
    O’Keeffe  LM, Taylor  G, Huxley  RR, Mitchell  P, Woodward  M, Peters  SAE.  Smoking as a risk factor for lung cancer in women and men: a systematic review and meta-analysis.  BMJ Open. 2018;8(10):e021611. doi:10.1136/bmjopen-2018-021611PubMedGoogle Scholar
    42.
    Pinsky  PF, Church  TR, Izmirlian  G, Kramer  BS.  The National Lung Screening Trial: results stratified by demographics, smoking history, and lung cancer histology.  Cancer. 2013;119(22):3976-3983. doi:10.1002/cncr.28326PubMedGoogle ScholarCrossref
    43.
    Pinsky  PF, Gierada  DS, Nath  PH, Munden  R.  Lung cancer risk associated with new solid nodules in the National Lung Screening Trial.  AJR Am J Roentgenol. 2017;209(5):1009-1014. doi:10.2214/AJR.17.18252PubMedGoogle ScholarCrossref
    44.
    Huang  P, Lin  CT, Li  Y,  et al.  Prediction of lung cancer risk at follow-up screening with low-dose CT: a training and validation study of a deep learning method.  Lancet Digital Health. 2019;1(7):e353-e362. doi:10.1016/S2589-7500(19)30159-1Google ScholarCrossref
    ×