Reproducibility of abnormality detection on chest radiographs using convolutional neural network in paired radiographs obtained within a short-term interval

We evaluated the reproducibility of computer-aided detections (CADs) with a convolutional neural network (CNN) on chest radiographs (CXRs) of abnormal pulmonary patterns in patients, acquired within a short-term interval. Anonymized CXRs (n = 9792) obtained from 2010 to 2016 and comprising five types of disease patterns, including the nodule (N), consolidation (C), interstitial opacity (IO), pleural effusion (PLE), and pneumothorax (PN), were included. The number of normal and abnormal CXRs was 6068 and 3724, respectively. The number of CXRs (region of interests, ROIs) of N, C, IO, PLE, and PN was 944 (1092), 550 (721), 280 (538), 1361 (1661), and 589 (622), respectively. CXRs were randomly allocated to training, tuning, and test sets in 70:10:20 ratios. Two thoracic radiologists labeled and delineated the ROIs of each disease pattern. The CAD system was developed using eDenseYOLO. For the reproducibility evaluation of developed CAD, paired CXRs of various diseases (N = 121, C = 28, IO = 12, PLE = 67, and PN = 20), acquired within a short-term interval from the test sets without any changes confirmed by thoracic radiologists, were used to evaluate CAD reproducibility. Percent positive agreement (PPAs) and Chamberlain’s percent positive agreement (CPPAs) were used to evaluate CAD reproducibility. The figure of merit (FOM) of five classes based on eDenseYOLO showed N-0.72 (0.68–0.75), C-0.41 (0.33–0.43), IO-0.97 (0.96–0.98), PLE-0.94 (0.92–95), and PN-0.87 (0.76–0.93). The PPAs of the five disease patterns including N, C, IO, PLE, and PN were 83.39%, 74.14%, 95.12%, 96.84%, and 84.58%, respectively, whereas the values of CPPAs were 71.70%, 59.13%, 91.16%, 93.91%, and 74.17%, respectively. The reproducibility of abnormal pulmonary patterns from CXRs, based on deep learning-based CAD, showed different results; this is important for assessing the reproducible performance of CAD in clinical settings.