Evolutionarily stable strategies in stable and periodically fluctuating populations: The Rosenzweig–MacArthur predator–prey model

An evolutionarily stable strategy (ESS) is an evolutionary strategy that, if adapted by a population, cannot be invaded by any deviating (mutant) strategy. The concept of ESS has been extensively studied and widely applied in ecology and evolutionary biology [M. Smith, On Evolution (1972)] but typically on the assumption that the system is ecologically stable. With reference to a Rosenzweig–MacArthur predator–prey model [M. Rosenzweig, R. MacArthur, Am. Nat. 97, 209–223 (1963)], we derive the mathematical conditions for the existence of an ESS when the ecological dynamics have asymptotically stable limit points as well as limit cycles. By extending the framework of Reed and Stenseth [J. Reed, N. C. Stenseth, J. Theoret. Biol. 108, 491–508 (1984)], we find that ESSs occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. These functions are identified and shown to have a similar form for both stable and fluctuating populations. We illustrate these results with a concrete example.