Multimodal investigation of bladder cancer data based on computed tomography, whole slide imaging, and transcriptomics

BACKGROUND: Multimodal analysis has shown great potential in the diagnosis and management of cancer. This study aimed to determine the multimodal data associations between radiological, pathologic, and molecular characteristics in bladder cancer. METHODS: A retrospective study of computed tomography (CT), pathologic slice, and RNA sequencing data from 127 consecutive adult patients in China who underwent bladder surgery and were pathologically diagnosed with bladder cancer was conducted. A total of 200 radiological and 1,029 pathologic features were extracted by radiomics and pathomics. Multimodal associations analysis and structural equation modeling were used to measure the cross-modal associations and structural relationships between CT and pathologic slice. A convolutional neural network was constructed for molecular subtyping based on multimodal imaging features. Class activation maps were used to examine the feature contribution in model decision-making. Cox regression and Kaplan-Meier survival analysis were used to explore the relevance of multimodal features to the prognosis of patients with bladder cancer. RESULTS: A total of 77 densely associated blocks of feature pairs were identified between CT and whole slide images. The largest cross-modal associated block reflected the tumor-grade properties. A significant relation was found between pathological features and molecular subtypes (β=0.396; P<0.001). High-grade bladder cancer showed heterogeneity of significance across different scales and higher disorders at the microscopic level. The fused radiological and pathologic features achieved higher accuracy (area under the curve: 0.89; 95% CI: 0.75–1.0) than the unimodal method. Thirteen prognosis-related features from CT and whole slide images were identified. CONCLUSIONS: Our work demonstrated the associations between CT, pathologic slices, and molecular signatures, and the potential to use multimodal data analysis in related clinical applications. Multimodal data analysis showed the potential of cross-inference of modal data and had higher diagnostic accuracy than the unimodal method.