The maintenance of polygenic sex determination depends on the dominance of fitness effects which are predictive of the role of sexual antagonism

In species with polygenic sex determination (PSD), multiple male- and female-determining loci on different proto-sex chromosomes segregate as polymorphisms within populations. The extent to which these polymorphisms are at stable equilibria is not yet resolved. Previous work demonstrated that PSD is most likely to be maintained as a stable polymorphism when the proto-sex chromosomes have opposite (sexually antagonistic) fitness effects in males and females. However, these models usually consider PSD systems with only two proto-sex chromosomes, or they do not broadly consider the dominance of the alleles under selection. To address these shortcomings, I used forward population genetic simulations to identify selection pressures that can maintain PSD under different dominance scenarios in a system with more than two proto-sex chromosomes (modeled after the house fly). I found that overdominant fitness effects of male-determining proto-Y chromosomes are more likely to maintain PSD than dominant, recessive, or additive fitness effects. The overdominant fitness effects that maintain PSD tend to have proto-Y chromosomes with sexually antagonistic effects (male-beneficial and female-detrimental). In contrast, dominant fitness effects that maintain PSD tend to have sexually antagonistic multi-chromosomal genotypes, but the individual proto-sex chromosomes do not have sexually antagonistic effects. These results demonstrate that sexual antagonism can be an emergent property of the multi-chromosome genotype without individual sexually antagonistic chromosomes. My results further illustrate how the dominance of fitness effects has consequences for both the likelihood that PSD will be maintained as well as the role sexually antagonistic selection is expected to play in maintaining the polymorphism.