Effects of niche overlap on coexistence, fixation and invasion in a population of two interacting species

Synergistic and antagonistic interactions in multi-species populations—such as resource sharing and competition—result in remarkably diverse behaviours in populations of interacting cells, such as in soil or human microbiomes, or clonal competition in cancer. The degree of inter- and intra-specific interaction can often be quantified through the notion of an ecological ‘niche’. Typically, weakly interacting species that occupy largely distinct niches result in stable mixed populations, while strong interactions and competition for the same niche result in rapid extinctions of some species and fixations of others. We investigate the transition of a deterministically stable mixed population to a stochasticity-induced fixation as a function of the niche overlap between the two species. We also investigate the effect of the niche overlap on the population stability with respect to external invasions. Our results have important implications for a number of experimental systems.