Observer Performance Evaluation of a Deep Learning Model for Multilabel Classification of Active Tuberculosis Lung Zone-Wise Manifestations

Background Chest X-rays (CXRs) are widely used for cost-effective screening of active pulmonary tuberculosis despite their limitations in sensitivity and specificity when interpreted by clinicians or radiologists. To address this issue, computer-aided detection (CAD) algorithms, particularly deep learning architectures based on convolution, have been developed to automate the analysis of radiography imaging. Deep learning algorithms have shown promise in accurately classifying lung abnormalities using chest X-ray images. In this study, we utilized the EfficientNet B4 model, which was pre-trained on ImageNet with 380x380 input dimensions, using its weights for transfer learning, and was modified with a series of components including global average pooling, batch normalization, dropout, and a classifier with 12 image-wise and 44 segment-wise lung zone evaluation classes using sigmoid activation. Objectives Assess the clinical usefulness of our previously created EfficientNet B4 model in identifying lung zone-specific abnormalities related to active tuberculosis through an observer performance test involving a skilled clinician operating in tuberculosis-specific environments. Methods The ground truth was established by a radiologist who examined all sample CXRs to identify lung zone-wise abnormalities. An expert clinician working in tuberculosis-specific settings independently reviewed the same CXR with blinded access to the ground truth. Simultaneously, the CXRs were classified using the EfficientNet B4 model. The clinician's assessments were then compared with the model's predictions, and the agreement between the two was measured using the kappa coefficient, evaluating the model's performance in classifying active tuberculosis manifestations across lung zones. Results The results show a strong agreement (Kappa ≥0.81) seen for lung zone-wise abnormalities of pneumothorax, mediastinal shift, emphysema, fibrosis, calcifications, pleural effusion, and cavity. Substantial agreement (Kappa = 0.61-0.80) for cavity, mediastinal shift, volume loss, and collapsed lungs. The Kappa score for lung zone-wise abnormalities is moderate (0.41-0.60) for 39% of cases. In image-wise agreement, the EfficientNet B4 model's performance ranges from moderate to almost perfect across categories, while in lung zone-wise agreement, it varies from fair to almost perfect. The results show strong agreement between the EfficientNet B4 model and the human reader in detecting lung zone-wise and image-wise manifestations. Conclusion The clinical utility of the EfficientNet B4 models to detect the abnormalities can aid clinicians in primary care settings for screening and triaging tuberculosis where resources are constrained or overburdened.