Validation of a Deep Learning Algorithm for the Detection of Malignant Pulmonary Nodules in Chest Radiographs | Lung Cancer | JAMA Network Open | JAMA Network
[Skip to Navigation]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address 35.153.100.128. Please contact the publisher to request reinstatement.
1.
de Koning  HJ, van der Aalst  CM, de Jong  PA,  et al.  Reduced lung-cancer mortality with volume CT screening in a randomized trial.   N Engl J Med. 2020;382(6):503-513. doi:10.1056/NEJMoa1911793 PubMedGoogle Scholar
2.
Aberle DR, Adams AM, Berg CD; National Lung Screening Trial Research Team.  Reduced lung-cancer mortality with low-dose computed tomographic screening.   N Engl J Med. 2011;365(5):395-409. doi:10.1056/NEJMoa1102873 PubMedGoogle Scholar
3.
Aberle  DR, DeMello  S, Berg  CD,  et al; National Lung Screening Trial Research Team.  Results of the two incidence screenings in the National Lung Screening Trial.   N Engl J Med. 2013;369(10):920-931. doi:10.1056/NEJMoa1208962 PubMedGoogle Scholar
4.
De Koning  H, Van Der Aalst  C, Ten Haaf  K, Oudkerk  M.  PL02.05 Effects of volume CT lung cancer screening: mortality results of the NELSON randomised-controlled population based trial.   J Thorac Oncol. 2018;13(10):S185. doi:10.1016/j.jtho.2018.08.012 Google Scholar
5.
Church  TR, Black  WC, Aberle  DR,  et al; National Lung Screening Trial Research Team.  Results of initial low-dose computed tomographic screening for lung cancer.   N Engl J Med. 2013;368(21):1980-1991. doi:10.1056/NEJMoa1209120 PubMedGoogle Scholar
6.
de Hoop  B, Schaefer-Prokop  C, Gietema  HA,  et al.  Screening for lung cancer with digital chest radiography: sensitivity and number of secondary work-up CT examinations.   Radiology. 2010;255(2):629-637. doi:10.1148/radiol.09091308 PubMedGoogle Scholar
7.
Bach  PB, Mirkin  JN, Oliver  TK,  et al.  Benefits and harms of CT screening for lung cancer: a systematic review.   JAMA. 2012;307(22):2418-2429. doi:10.1001/jama.2012.5521 PubMedGoogle Scholar
8.
Ahmad  U, Detterbeck  FC.  Current status of lung cancer screening.   Semin Thorac Cardiovasc Surg. 2012;24(1):27-36. doi:10.1053/j.semtcvs.2012.01.014 PubMedGoogle Scholar
9.
Nam  JG, Park  S, Hwang  EJ,  et al.  Development and validation of deep learning–based automatic detection algorithm for malignant pulmonary nodules on chest radiographs.   Radiology. 2019;290(1):218-228. doi:10.1148/radiol.2018180237 PubMedGoogle Scholar
10.
Gavelli  G, Giampalma  E.  Sensitivity and specificity of chest X-ray screening for lung cancer: review article.   Cancer. 2000;89(11 Suppl):2453-2456. doi:10.1002/1097-0142(20001201)89:11+<2453::AID-CNCR21>3.0.CO;2-M PubMedGoogle Scholar
11.
Muhm  JR, Miller  WE, Fontana  RS, Sanderson  DR, Uhlenhopp  MA.  Lung cancer detected during a screening program using four-month chest radiographs.   Radiology. 1983;148(3):609-615. doi:10.1148/radiology.148.3.6308709 PubMedGoogle Scholar
12.
Austin  JH, Romney  BM, Goldsmith  LS.  Missed bronchogenic carcinoma: radiographic findings in 27 patients with a potentially resectable lesion evident in retrospect.   Radiology. 1992;182(1):115-122. doi:10.1148/radiology.182.1.1727272 PubMedGoogle Scholar
13.
Schalekamp  S, van Ginneken  B, Koedam  E,  et al.  Computer-aided detection improves detection of pulmonary nodules in chest radiographs beyond the support by bone-suppressed images.   Radiology. 2014;272(1):252-261. doi:10.1148/radiol.14131315 PubMedGoogle Scholar
14.
Quadrelli  S, Lyons  G, Colt  H, Chimondeguy  D, Buero  A.  Clinical characteristics and prognosis of incidentally detected lung cancers.   Int J Surg Oncol. 2015;2015:287604. doi:10.1155/2015/287604 PubMedGoogle Scholar
15.
Ausawalaithong  W, Thirach  A, Marukatat  S, Wilaiprasitporn  T. Automatic lung cancer prediction from chest x-ray images using the deep learning approach. In: 2018 11th Biomedical Engineering International Conference (BMEICON). Institute of Electrical and Electronics Engineers; 2018:1-5. doi:10.1109/BMEiCON.2018.8609997
16.
Tataru  C, Yi  D, Shenoyas  A, Ma  A. Deep learning for abnormality detection in chest x-ray images. Stanford University; 2017. Accessed September 4, 2020. http://cs231n.stanford.edu/reports/2017/pdfs/527.pdf
17.
Gang  P, Zhen  W, Zeng  W,  et al. Dimensionality reduction in deep learning for chest X-ray analysis of lung cancer. In: 2018 10th International Conference on Advanced Computational Intelligence (ICACI). Institute of Electrical and Electronics Engineers; 2018:878-883. doi:10.1109/ICACI.2018.8377579
18.
Hwang  EJ, Nam  JG, Lim  WH,  et al.  Deep learning for chest radiograph diagnosis in the emergency department.   Radiology. 2019;293(3):573-580. doi:10.1148/radiol.2019191225 PubMedGoogle Scholar
19.
Rajpurkar  P, Irvin  J, Ball  RL,  et al.  Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists.   PLoS Med. 2018;15(11):e1002686. doi:10.1371/journal.pmed.1002686 PubMedGoogle Scholar
20.
Cicero  M, Bilbily  A, Colak  E,  et al.  Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs.   Invest Radiol. 2017;52(5):281-287. doi:10.1097/RLI.0000000000000341 PubMedGoogle Scholar
21.
Sim  Y, Chung  MJ, Kotter  E,  et al.  Deep convolutional neural network–based software improves radiologist detection of malignant lung nodules on chest radiographs.   Radiology. 2020;294(1):199-209. doi:10.1148/radiol.2019182465PubMedGoogle Scholar
22.
Aberle  DR, Berg  CD, Black  WC,  et al; National Lung Screening Trial Research Team.  The National Lung Screening Trial: overview and study design.   Radiology. 2011;258(1):243-253. doi:10.1148/radiol.10091808 PubMedGoogle Scholar
23.
Patz  EF  Jr, Greco  E, Gatsonis  C, Pinsky  P, Kramer  BS, Aberle  DR.  Lung cancer incidence and mortality in National Lung Screening Trial participants who underwent low-dose CT prevalence screening: a retrospective cohort analysis of a randomised, multicentre, diagnostic screening trial.   Lancet Oncol. 2016;17(5):590-599. doi:10.1016/S1470-2045(15)00621-X PubMedGoogle Scholar
24.
NLST nodule detection. GitHub. Accessed March 29, 2020. https://github.com/lunit-io/nlst-nodule-detection
25.
He  K, Zhang  X, Ren  S, Sun  J. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers; 2016:770-778. doi:10.1109/CVPR.2016.90
26.
Oquab  M, Bottou  L, Laptev  I, Sivic  J. Is object localization for free? weakly-supervised learning with convolutional neural networks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers; 2015:685-694. doi:10.1109/CVPR.2015.7298668
27.
Cubuk  ED, Zoph  B, Mane  D, Vasudevan  V, Le  QV. AutoAugment: learning augmentation strategies from data. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Institute of Electrical and Electronics Engineers; 2019:113-123. doi:10.1109/CVPR.2019.00020
28.
Hwang  EJ, Park  S, Jin  K-N,  et al; DLAD Development and Evaluation Group.  Development and validation of a deep learning–based automated detection algorithm for major thoracic diseases on chest radiographs.   JAMA Netw Open. 2019;2(3):e191095. doi:10.1001/jamanetworkopen.2019.1095 PubMedGoogle Scholar
29.
Leisenring  W, Alonzo  T, Pepe  MS.  Comparisons of predictive values of binary medical diagnostic tests for paired designs.   Biometrics. 2000;56(2):345-351. doi:10.1111/j.0006-341X.2000.00345.x PubMedGoogle Scholar
30.
Majkowska  A, Mittal  S, Steiner  DF,  et al.  Chest radiograph interpretation with deep learning models: assessment with radiologist-adjudicated reference standards and population-adjusted evaluation.   Radiology. 2020;294(2):421-431. doi:10.1148/radiol.2019191293PubMedGoogle Scholar
31.
Purandare  NC, Rangarajan  V.  Imaging of lung cancer: implications on staging and management.   Indian J Radiol Imaging. 2015;25(2):109-120. doi:10.4103/0971-3026.155831 PubMedGoogle Scholar
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    Original Investigation
    Imaging
    September 24, 2020

    Validation of a Deep Learning Algorithm for the Detection of Malignant Pulmonary Nodules in Chest Radiographs

    Author Affiliations
    • 1Lunit, Seoul, South Korea
    • 2Division of Thoracic Imaging, Department of Radiology, Massachusetts General Hospital, Boston
    • 3Harvard Medical School, Boston, Massachusetts
    JAMA Netw Open. 2020;3(9):e2017135. doi:10.1001/jamanetworkopen.2020.17135
    Key Points

    Question  Does an artificial intelligence algorithm trained to detect pulmonary nodules improve lung cancer detection on chest radiographs?

    Findings  In this diagnostic study of data from 5485 participants in the National Lung Screening Trial, the sensitivity and specificity of an artificial intelligence algorithm for nodule detection were 86% and 85%, respectively. When the same artificial intelligence algorithm was applied for cancer detection, the sensitivity was 94%, specificity 83%, positive predictive value 3%, and negative predictive value was 100% for the detection of malignant pulmonary nodules.

    Meaning  The study findings suggest that an artificial intelligence algorithm trained to detect pulmonary nodules can help to improve lung cancer detection on chest radiographs.

    Abstract

    Importance  The improvement of pulmonary nodule detection, which is a challenging task when using chest radiographs, may help to elevate the role of chest radiographs for the diagnosis of lung cancer.

    Objective  To assess the performance of a deep learning–based nodule detection algorithm for the detection of lung cancer on chest radiographs from participants in the National Lung Screening Trial (NLST).

    Design, Setting, and Participants  This diagnostic study used data from participants in the NLST ro assess the performance of a deep learning–based artificial intelligence (AI) algorithm for the detection of pulmonary nodules and lung cancer on chest radiographs using separate training (in-house) and validation (NLST) data sets. Baseline (T0) posteroanterior chest radiographs from 5485 participants (full T0 data set) were used to assess lung cancer detection performance, and a subset of 577 of these images (nodule data set) were used to assess nodule detection performance. Participants aged 55 to 74 years who currently or formerly (ie, quit within the past 15 years) smoked cigarettes for 30 pack-years or more were enrolled in the NLST at 23 US centers between August 2002 and April 2004. Information on lung cancer diagnoses was collected through December 31, 2009. Analyses were performed between August 20, 2019, and February 14, 2020.

    Exposures  Abnormality scores produced by the AI algorithm.

    Main Outcomes and Measures  The performance of an AI algorithm for the detection of lung nodules and lung cancer on radiographs, with lung cancer incidence and mortality as primary end points.

    Results  A total of 5485 participants (mean [SD] age, 61.7 [5.0] years; 3030 men [55.2%]) were included, with a median follow-up duration of 6.5 years (interquartile range, 6.1-6.9 years). For the nodule data set, the sensitivity and specificity of the AI algorithm for the detection of pulmonary nodules were 86.2% (95% CI, 77.8%-94.6%) and 85.0% (95% CI, 81.9%-88.1%), respectively. For the detection of all cancers, the sensitivity was 75.0% (95% CI, 62.8%-87.2%), the specificity was 83.3% (95% CI, 82.3%-84.3%), the positive predictive value was 3.8% (95% CI, 2.6%-5.0%), and the negative predictive value was 99.8% (95% CI, 99.6%-99.9%). For the detection of malignant pulmonary nodules in all images of the full T0 data set, the sensitivity was 94.1% (95% CI, 86.2%-100.0%), the specificity was 83.3% (95% CI, 82.3%-84.3%), the positive predictive value was 3.4% (95% CI, 2.2%-4.5%), and the negative predictive value was 100.0% (95% CI, 99.9%-100.0%). In digital radiographs of the nodule data set, the AI algorithm had higher sensitivity (96.0% [95% CI, 88.3%-100.0%] vs 88.0% [95% CI, 75.3%-100.0%]; P = .32) and higher specificity (93.2% [95% CI, 89.9%-96.5%] vs 82.8% [95% CI, 77.8%-87.8%]; P = .001) for nodule detection compared with the NLST radiologists. For malignant pulmonary nodule detection on digital radiographs of the full T0 data set, the sensitivity of the AI algorithm was higher (100.0% [95% CI, 100.0%-100.0%] vs 94.1% [95% CI, 82.9%-100.0%]; P = .32) compared with the NLST radiologists, and the specificity (90.9% [95% CI, 89.6%-92.1%] vs 91.0% [95% CI, 89.7%-92.2%]; P = .91), positive predictive value (8.2% [95% CI, 4.4%-11.9%] vs 7.8% [95% CI, 4.1%-11.5%]; P = .65), and negative predictive value (100.0% [95% CI, 100.0%-100.0%] vs 99.9% [95% CI, 99.8%-100.0%]; P = .32) were similar to those of NLST radiologists.

    Conclusions and Relevance  In this study, the AI algorithm performed better than NLST radiologists for the detection of pulmonary nodules on digital radiographs. When used as a second reader, the AI algorithm may help to detect lung cancer.

    ×