In search of the Goldilocks zone for hybrid speciation

Hybridization has recently gained considerable interest both as a unique opportunity for observing speciation mechanisms and as a potential engine for speciation. The latter remains a controversial topic. It was recently hypothesized that the reciprocal sorting of genetic incompatibilities from parental species could result in hybrid speciation, when the hybrid population maintains a mixed combination of the parental incompatibilities that prevents further gene exchange with both parental populations. However, the specifics of the purging/sorting process of multiple incompatibilities have not been examined theoretically. We here investigate the allele-frequency dynamics of an isolated hybrid population that results from a single hybridization event. Using models of two or four loci, we investigate the fate of one or two genetic incompatibilities of the Dobzhansky-Muller type (DMIs). We study how various parameters affect both the sorting/purging of the DMIs and the probability of observing hybrid speciation by reciprocal sorting. We find that the probability of hybrid speciation is strongly dependent on the linkage architecture (i.e. the order and recombination rate between loci along chromosomes), the population size of the hybrid population, and the initial relative contributions of the parental populations to the hybrid population. We identify a Goldilocks zone for specific linkage architectures and intermediate recombination rates, in which hybrid speciation becomes highly probable. Whereas an equal contribution of parental populations to the hybrid population maximizes the hybrid speciation probability in the Goldilocks zone, other linkage architectures yield unintuitive asymmetric maxima. We provide an explanation for this pattern, and discuss our results both with respect to the best conditions for observing hybrid speciation in nature and their implications regarding patterns of introgression in hybrid zones.