Adaptive dynamics of unstable cancer populations: The canonical equation

In most instances of tumour development, genetic instability plays a role in allowing cancer cell populations to respond to selection barriers, such as physical constraints or immune responses, and rapidly adapt to an always changing environment. Modelling instability is a nontrivial task, since by definition evolving instability leads to changes in the underlying landscape. In this article, we explore mathematically a simple version of unstable tumour progression using the formalism of adaptive dynamics (AD) where selection and mutation are explicitly coupled. Using a set of basic fitness landscapes, the so‐called canonical equation for the evolution of genetic instability on a minimal scenario associated with a population of unstable cells is derived. We obtain explicit expressions for the evolution of mutation probabilities, and the implications of the model on further experimental studies and potential mutagenic therapies are discussed.