Population size as a major determinant of mating system and population genetic differentiation in a narrow endemic chasmophyte

BACKGROUND: Mating system is one of the major determinants of intra- and interspecific genetic structure, but may vary within and between plant populations. Our study model included all known populations of Moehringia tommasinii (Caryophyllaceae), a narrow endemic plant inhabiting rock crevices in the northwestern Adriatic, and some populations of co-occurring and widespread M. muscosa, an ecologically divergent relative with an overlapping flowering period. We performed reciprocal crosses within and between taxa and used molecular markers to assess the extent of gene flow within and between populations and taxa. Using coefficient of inbreeding, population size, seed weight, pollen-to-ovule ratio, and flower display size, we also looked for evidence of a selfing syndrome. RESULTS: A surprisingly high variation in mating systems was observed among populations of M. tommasinii. These populations exhibited genetic structuring, with their size positively correlated with both seed weight and pollen production. Although a selfing syndrome could not be confirmed as the majority of selfing resulted from allogamous treatments, the occurrence of selfing was notable. In the presence of M. muscosa, at a site where both species coexist closely, a distinct pattern of fruit production was observed in M. tommasinii following various pollination treatments. Molecular and morphometric data provided evidence of hybridization followed by local extinction at this site. CONCLUSIONS: Population size proved to be the most important factor affecting the mating system in genetically structured populations of M. tommasinii. Lighter seeds and lower pollen production observed in populations with pronounced selfing do not provide enough evidence for the selfing syndrome. Detected gene flow between M. tommasinii and the sympatric M. muscosa suggested weak reproductive barriers between the taxa, which could pose a conservation problems for the former species. Hybridization leading to local extinction may also resulted in floral polymorphism and disruption of mating patterns of M. tommasinii. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12870-023-04384-8.