Vessel Delineation Using U-Net: A Sparse Labeled Deep Learning Approach for Semantic Segmentation of Histological Images

SIMPLE SUMMARY: In our study, we aimed to create an accurate segmentation algorithm of blood vessels within histologically stained tumor tissue using deep learning. Blood vessels are crucial for supplying nutrients to tumor cells, and accurately identifying them is essential for understanding tumor development and designing effective treatments. We conducted a comprehensive investigation by comparing various deep learning architectural methods. Additionally, we reduced the time spent for data annotation and introduced a sparse labeling technique, by which only a limited amount of data was labeled for training the model. We showed that U-Net with a combination of attention gates and residual links yielded the highest precision and accuracy compared to other tested architectures. This demonstrates that our approach, even with sparse labeling, can effectively identify blood vessels and provide accurate segmentation within tumor tissue. These findings are promising for improving our understanding of tumor vasculature and potentially contributing to improved treatment strategies. ABSTRACT: Semantic segmentation is an important imaging analysis method enabling the identification of tissue structures. Histological image segmentation is particularly challenging, having large structural information while providing only limited training data. Additionally, labeling these structures to generate training data is time consuming. Here, we demonstrate the feasibility of a semantic segmentation using U-Net with a novel sparse labeling technique. The basic U-Net architecture was extended by attention gates, residual and recurrent links, and dropout regularization. To overcome the high class imbalance, which is intrinsic to histological data, under- and oversampling and data augmentation were used. In an ablation study, various architectures were evaluated, and the best performing model was identified. This model contains attention gates, residual links, and a dropout regularization of 0.125. The segmented images show accurate delineations of the vascular structures (with a precision of 0.9088 and an AUC-ROC score of 0.9717), and the segmentation algorithm is robust to images containing staining variations and damaged tissue. These results demonstrate the feasibility of sparse labeling in combination with the modified U-Net architecture.