Attention-Based Deep Learning System for Classification of Breast Lesions—Multimodal, Weakly Supervised Approach

SIMPLE SUMMARY: Breast cancer affects millions of women worldwide. We aim to provide radiologists with automatic support for mammography review. We propose deploying deep learning models with Multiple Instance Learning algorithms for breast cancer diagnosis (cancer versus non-cancer classification) based on digital mammography images, taking advantage of data annotated at an image level without indicating where the lesion is. We employed algorithms to analyse original, high-resolution images with minimal reduction in size. We used graphical suggestions of attentional maps to verify the correctness of the algorithm and to indicate the areas of the images where cancer lesions are. Finally, we performed comparative and validation studies on external datasets differing in the number of images, pixel intensity levels, and subtypes of existing lesions, showing high accuracy and potential for generalisability of the algorithms. ABSTRACT: Breast cancer is the most frequent female cancer, with a considerable disease burden and high mortality. Early diagnosis with screening mammography might be facilitated by automated systems supported by deep learning artificial intelligence. We propose a model based on a weakly supervised Clustering-constrained Attention Multiple Instance Learning (CLAM) classifier able to train under data scarcity effectively. We used a private dataset with 1174 non-cancer and 794 cancer images labelled at the image level with pathological ground truth confirmation. We used feature extractors (ResNet-18, ResNet-34, ResNet-50 and EfficientNet-B0) pre-trained on ImageNet. The best results were achieved with multimodal-view classification using both CC and MLO images simultaneously, resized by half, with a patch size of 224 px and an overlap of 0.25. It resulted in AUC-ROC = 0.896 ± 0.017, F1-score 81.8 ± 3.2, accuracy 81.6 ± 3.2, precision 82.4 ± 3.3, and recall 81.6 ± 3.2. Evaluation with the Chinese Mammography Database, with 5-fold cross-validation, patient-wise breakdowns, and transfer learning, resulted in AUC-ROC 0.848 ± 0.015, F1-score 78.6 ± 2.0, accuracy 78.4 ± 1.9, precision 78.8 ± 2.0, and recall 78.4 ± 1.9. The CLAM algorithm’s attentional maps indicate the features most relevant to the algorithm in the images. Our approach was more effective than in many other studies, allowing for some explainability and identifying erroneous predictions based on the wrong premises.