Multitask Learning with Convolutional Neural Networks and Vision Transformers Can Improve Outcome Prediction for Head and Neck Cancer Patients

SIMPLE SUMMARY: Increasing treatment efficacy for head and neck cancer requires the utilization of patient-specific biomarkers to personalize therapy. Deep neural networks show promise for extracting prognostic biomarkers from medical imaging data to predict loco-regional tumor recurrence or disease progression. However, training these networks can be challenging due to limited available data, potentially affecting prediction quality. To address these challenges, we investigated the effectiveness of multiple multitask learning strategies, where two distinct outcome tasks and an auxiliary tumor segmentation objective were simultaneously optimized. This approach aimed to enhance the training process, leading to better parameter configurations and improved predictive performance. Our analysis, conducted on two multicentric datasets using convolutional neural networks and vision transformers, indicated performance benefits of outcome models trained using multitask strategies over models trained solely on a single outcome task. ABSTRACT: Neural-network-based outcome predictions may enable further treatment personalization of patients with head and neck cancer. The development of neural networks can prove challenging when a limited number of cases is available. Therefore, we investigated whether multitask learning strategies, implemented through the simultaneous optimization of two distinct outcome objectives (multi-outcome) and combined with a tumor segmentation task, can lead to improved performance of convolutional neural networks (CNNs) and vision transformers (ViTs). Model training was conducted on two distinct multicenter datasets for the endpoints loco-regional control (LRC) and progression-free survival (PFS), respectively. The first dataset consisted of pre-treatment computed tomography (CT) imaging for 290 patients and the second dataset contained combined positron emission tomography (PET)/CT data of 224 patients. Discriminative performance was assessed by the concordance index (C-index). Risk stratification was evaluated using log-rank tests. Across both datasets, CNN and ViT model ensembles achieved similar results. Multitask approaches showed favorable performance in most investigations. Multi-outcome CNN models trained with segmentation loss were identified as the optimal strategy across cohorts. On the PET/CT dataset, an ensemble of multi-outcome CNNs trained with segmentation loss achieved the best discrimination (C-index: 0.29, 95% confidence interval (CI): 0.22–0.36) and successfully stratified patients into groups with low and high risk of disease progression ([Formula: see text]). On the CT dataset, ensembles of multi-outcome CNNs and of single-outcome ViTs trained with segmentation loss performed best (C-index: 0.26 and 0.26, CI: 0.18–0.34 and 0.18–0.35, respectively), both with significant risk stratification for LRC in independent validation ([Formula: see text] and [Formula: see text]). Further validation of the developed multitask-learning models is planned based on a prospective validation study, which has recently completed recruitment.