Phenotypic heterogeneity in modeling cancer evolution

The unwelcome evolution of malignancy during cancer progression emerges through a selection process in a complex heterogeneous population structure. In the present work, we investigate evolutionary dynamics in a phenotypically heterogeneous population of stem cells (SCs) and their associated progenitors. The fate of a malignant mutation is determined not only by overall stem cell and non-stem cell growth rates but also differentiation and dedifferentiation rates. We investigate the effect of such a complex population structure on the evolution of malignant mutations. We derive exactly calculated results for the fixation probability of a mutant arising in each of the subpopulations. The exactly calculated results are in almost perfect agreement with the numerical simulations. Moreover, a condition for evolutionary advantage of a mutant cell versus the wild type population is given in the present study. We also show that microenvironment-induced plasticity in invading mutants leads to more aggressive mutants with higher fixation probability. Our model predicts that decreasing polarity between stem and non-stem cells’ turnover would raise the survivability of non-plastic mutants; while it would suppress the development of malignancy for plastic mutants. The derived results are novel and general with potential applications in nature; we discuss our model in the context of colorectal/intestinal cancer (at the epithelium). However, the model clearly needs to be validated through appropriate experimental data. This novel mathematical framework can be applied more generally to a variety of problems concerning selection in heterogeneous populations, in other contexts such as population genetics, and ecology.