Evolutionarily stable strategy analysis and its links to demography and genetics through invasion fitness

Evolutionarily stable strategy (ESS) analysis pioneered by Maynard Smith and Price took off in part because it often does not require explicit assumptions about the genetics and demography of a population in contrast to population genetic models. Though this simplicity is useful, it obscures the degree to which ESS analysis applies to populations with more realistic genetics and demography: for example, how does ESS analysis handle complexities such as kin selection, group selection and variable environments when phenotypes are affected by multiple genes? In this paper, I review the history of the ESS concept and show how early uncertainty about the method lead to important mathematical theory linking ESS analysis to general population genetic models. I use this theory to emphasize the link between ESS analysis and the concept of invasion fitness. I give examples of how invasion fitness can measure kin selection, group selection and the evolution of linked modifier genes in response to variable environments. The ESSs in these examples depend crucially on demographic and genetic parameters, which highlights how ESS analysis will continue to be an important tool in understanding evolutionary patterns as new models address the increasing abundance of genetic and long-term demographic data in natural populations. This article is part of the theme issue ‘Half a century of evolutionary games: a synthesis of theory, application and future directions’.