Multi-center study on predicting breast cancer lymph node status from core needle biopsy specimens using multi-modal and multi-instance deep learning

The objective of our study is to develop a deep learning model based on clinicopathological data and digital pathological image of core needle biopsy specimens for predicting breast cancer lymph node metastasis. We collected 3701 patients from the Fourth Hospital of Hebei Medical University and 190 patients from four medical centers in Hebei Province. Integrating clinicopathological data and image features build multi-modal and multi-instance (MMMI) deep learning model to obtain the final prediction. For predicting with or without lymph node metastasis, the AUC was 0.770, 0.709, 0.809 based on the clinicopathological features, WSI and MMMI, respectively. For predicting four classification of lymph node status (no metastasis, isolated tumor cells (ITCs), micrometastasis, and macrometastasis), the prediction based on clinicopathological features, WSI and MMMI were compared. The AUC for no metastasis was 0.770, 0.709, 0.809, respectively; ITCs were 0.619, 0.531, 0.634, respectively; micrometastasis were 0.636, 0.617, 0.691, respectively; and macrometastasis were 0.748, 0.691, 0.758, respectively. The MMMI model achieved the highest prediction accuracy. For prediction of different molecular types of breast cancer, MMMI demonstrated a better prediction accuracy for any type of lymph node status, especially in the molecular type of triple negative breast cancer (TNBC). In the external validation sets, MMMI also showed better prediction accuracy in the four classification, with AUC of 0.725, 0.757, 0.525, and 0.708, respectively. Finally, we developed a breast cancer lymph node metastasis prediction model based on a MMMI model. Through all cases tests, the results showed that the overall prediction ability was high.