Reproductive skew, fighting costs and winner–loser effects in social dominance evolution

1. Social hierarchies are often found in group‐living animals and can be formed through pairwise aggressive interactions. The dominance rank can influence reproductive success (RS) with a skew towards high‐ranking individuals. 2. Using game theory, we investigate how the opportunity for differently ranked individuals to achieve RS influences the costs of hierarchy formation and the strength of winner and loser effects. 3. In our model, individuals adjust their aggressive and submissive behaviour towards others through reinforcement learning. The learning is based on rewards and penalties, which depend on relative fighting ability. From individual‐based simulations, we determine evolutionary equilibria of traits such as learning rates. We examine situations that differ in the extent of monopolisation of contested RS by dominants and in the proportion of total RS that is contested. 4. The model implements two kinds of fighting costs: a decrease in effective fighting ability from damage (loss of condition) and a risk of mortality that increases with the total accumulated damage. Either of these costs can limit the amount of fighting. 5. We find that individuals form stable dominance hierarchies, with a positive correlation between dominance position and fighting ability. The accumulated costs differ between dominance positions, with the highest costs paid by low or intermediately ranked individuals. Costs tend to be higher in high‐skew situations. 6. We identify a ‘stay‐in, opt‐out’ syndrome, comprising a range from weaker (stay‐in) to stronger (opt‐out) winner–loser effects. We interpret the opt‐out phenotype to be favoured by selection on lower ranked individuals to opt out of contests over social dominance, because it is more pronounced when more of the total RS is uncontested. 7. We discuss our results in relation to field and experimental observations and argue that there is a need for empirical investigation of the behaviour and reproductive success of lower ranked individuals.