A novel molecular signature identifies mixed subtypes in renal cell carcinoma with poor prognosis and independent response to immunotherapy

BACKGROUND: Renal cell carcinoma (RCC) is a heterogeneous disease comprising histologically defined subtypes. For therapy selection, precise subtype identification and individualized prognosis are mandatory, but currently limited. Our aim was to refine subtyping and outcome prediction across main subtypes, assuming that a tumor is composed of molecular features present in distinct pathological subtypes. METHODS: Individual RCC samples were modeled as linear combination of the main subtypes (clear cell (ccRCC), papillary (pRCC), chromophobe (chRCC)) using computational gene expression deconvolution. The new molecular subtyping was compared with histological classification of RCC using the Cancer Genome Atlas (TCGA) cohort (n = 864; ccRCC: 512; pRCC: 287; chRCC: 65) as well as 92 independent histopathologically well-characterized RCC. Predicted continuous subtypes were correlated to cancer-specific survival (CSS) in the TCGA cohort and validated in 242 independent RCC. Association with treatment-related progression-free survival (PFS) was studied in the JAVELIN Renal 101 (n = 726) and IMmotion151 trials (n = 823). CSS and PFS were analyzed using the Kaplan–Meier and Cox regression analysis. RESULTS: One hundred seventy-four signature genes enabled reference-free molecular classification of individual RCC. We unambiguously assign tumors to either ccRCC, pRCC, or chRCC and uncover molecularly heterogeneous tumors (e.g., with ccRCC and pRCC features), which are at risk of worse outcome. Assigned proportions of molecular subtype-features significantly correlated with CSS (ccRCC (P = 4.1E − 10), pRCC (P = 6.5E − 10), chRCC (P = 8.6E − 06)) in TCGA. Translation into a numerical RCC-R(isk) score enabled prognosis in TCGA (P = 9.5E − 11). Survival modeling based on the RCC-R score compared to pathological categories was significantly improved (P = 3.6E − 11). The RCC-R score was validated in univariate (P = 3.2E − 05; HR = 3.02, 95% CI: 1.8–5.08) and multivariate analyses including clinicopathological factors (P = 0.018; HR = 2.14, 95% CI: 1.14–4.04). Heterogeneous PD-L1-positive RCC determined by molecular subtyping showed increased PFS with checkpoint inhibition versus sunitinib in the JAVELIN Renal 101 (P = 3.3E − 04; HR = 0.52, 95% CI: 0.36 − 0.75) and IMmotion151 trials (P = 0.047; HR = 0.69, 95% CI: 0.48 − 1). The prediction of PFS significantly benefits from classification into heterogeneous and unambiguous subtypes in both cohorts (P = 0.013 and P = 0.032). CONCLUSION: Switching from categorical to continuous subtype classification across most frequent RCC subtypes enables outcome prediction and fosters personalized treatment strategies. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s13073-022-01105-y.