Mutational signatures reveal ternary relationships between homologous recombination repair, APOBEC, and mismatch repair in gynecological cancers

BACKGROUND: Revealing the impacts of endogenous and exogenous mutagenesis processes is essential for understanding the etiology of somatic genomic alterations and designing precise prognostication and treatment strategies for cancer. DNA repair deficiency is one of the main sources of endogenous mutagenesis and is increasingly recognized as a target for cancer therapeutics. The role and prevalence of mechanisms that underly different forms of DNA repair deficiencies and their interactions remain to be elucidated in gynecological malignancies. METHODS: We analyzed 1231 exomes and 268 whole-genomes from three major gynecological malignancies including uterine corpus endometrial carcinoma (UCEC) as well as ovarian and cervical cancers. We also analyzed data from 134 related cell lines. We extracted and compared de novo and refitted mutational signature profiles using complementary and confirmatory approaches and performed interaction analysis to detect co-occurring and mutually exclusive signatures. RESULTS: We found an inverse relationship between homologous recombination deficiency (HRd) and mismatch repair deficiency (MMRd). Moreover, APOBEC co-occurred with HRd but was mutually exclusive with MMRd. UCEC tumors were dominated by MMRd, yet a subset of them manifested the HRd and APOBEC signatures. Conversely, ovarian tumors were dominated by HRd, while a subset represented MMRd and APOBEC. In contrast to both, cervical tumors were dominated by APOBEC with a small subsets showing the POLE, HRd, and MMRd signatures. Although the type, prevalence, and heterogeneity of mutational signatures varied across the tumor types, the patterns of co-occurrence and exclusivity were consistently observed in all. Notably, mutational signatures in gynecological tumor cell lines reflected those detected in primary tumors. CONCLUSIONS: Taken together, these analyses indicate that application of mutation signature analysis not only advances our understanding of mutational processes and their interactions, but also it has the potential to stratify patients that could benefit from treatments available for tumors harboring distinct mutational signatures and to improve clinical decision-making for gynecological malignancies. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12967-022-03259-0.