Attention mechanism based multi-sequence MRI fusion improves prediction of response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer

BACKGROUND: Accurate prediction of response to neoadjuvant chemoradiotherapy (nCRT) is very important for treatment plan decision in locally advanced rectal cancer (LARC). The aim of this study was to investigate whether self-attention mechanism based multi-sequence fusion strategy applied to multiparametric magnetic resonance imaging (MRI) based deep learning or hand-crafted radiomics model construction can improve prediction of response to nCRT in LARC. METHODS: This retrospective analysis enrolled 422 consecutive patients with LARC who received nCRT before surgery at two hospitals. All patients underwent multiparametric MRI scans with three imaging sequences. Tumor regression grade (TRG) was used to assess the response of nCRT based on the resected specimen. Patients were separated into 2 groups: poor responders (TRG 2, 3) versus good responders (TRG 0, 1). A self-attention mechanism, namely channel attention, was applied to fuse the three sequence information for deep learning and radiomics models construction. For comparison, other two models without channel attention were also constructed. All models were developed in the same hospital and validated in the other hospital. RESULTS: The deep learning model with channel attention mechanism achieved area under the curves (AUCs) of 0.898 in the internal validation cohort and 0.873 in the external validation cohort, which was the best performed model in all cohorts. More importantly, both the deep learning and radiomics model that applied channel attention mechanism performed better than those without channel attention mechanism. CONCLUSIONS: The self-attention mechanism based multi-sequence fusion strategy can improve prediction of response to nCRT in LARC. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s13014-023-02352-y.