Assortative mating and self-fertilization differ in their contributions to reinforcement, cascade speciation, and diversification

Cascade speciation and reinforcement can evolve rapidly when traits are pleiotropic and act as both signal/cue in nonrandom mating. Here, we examine the contribution of two key traits—assortative mating and self-fertilization—to reinforcement and (by extension) cascade speciation. First, using a population genetic model of reinforcement we find that both assortative mating and self-fertilization can make independent contributions to increased reproductive isolation, consistent with reinforcement. Self-fertilization primarily evolves due to its 2-fold transmission advantage when inbreeding depression (d) is lower (d < 0.45) but evolves as a function of the cost of hybridization under higher inbreeding depression (0.45 < d < 0.48). When both traits can evolve simultaneously, increased self-fertilization often prohibits the evolution of assortative mating. We infer that, under specific conditions, mating system transitions are more likely to lead to increased reproductive isolation and initiate cascade speciation, than assortative mating. Based on the results of our simulations, we hypothesized that transitions to self-fertilization could contribute to clade-wide diversification if reinforcement or cascade speciation is common. We tested this hypothesis with comparative data from two different groups. Consistent with our hypothesis, there was a trend towards uniparental reproduction being associated with increased diversification rate in the Nematode phylum. For the plant genus Mimulus, however, self-fertilization was associated with reduced diversification. Reinforcement driving speciation via transitions to self-fertilization might be short lived or unsustainable across macroevolutionary scales in some systems (some plants), but not others (such as nematodes), potentially due to differences in susceptibility to inbreeding depression and/or the ability to transition between reproductive modes.