Large-scale pedigree analysis highlights rapidly mutating Y-chromosomal short tandem repeats for differentiating patrilineal relatives and predicting their degrees of consanguinity

Rapidly mutating Y-chromosomal short tandem repeats (RM Y-STRs) were suggested for differentiating patrilineally related men as relevant in forensic genetics, anthropological genetics, and genetic genealogy. Empirical data are available for closely related males, while differentiation rates for more distant relatives are scarce. Available RM Y-STR mutation rate estimates are typically based on father–son pair data, while pedigree-based studies for efficient analysis requiring less samples are rare. Here, we present a large-scale pedigree analysis in 9379 pairs of men separated by 1–34 meioses on 30 Y-STRs with increased mutation rates including all known RM Y-STRs (RMplex). For comparison, part of the samples were genotyped at 25 standard Y-STRs mostly with moderate mutation rates (Yfiler Plus). For 43 of the 49 Y-STRs analyzed, pedigree-based mutation rates were similar to previous father–son based estimates, while for six markers significant differences were observed. Male relative differentiation rates from the 30 RMplex Y-STRs were 43%, 84%, 96%, 99%, and 100% for relatives separated by one, four, six, nine, and twelve meioses, respectively, which largely exceeded rates obtained by 25 standard Y-STRs. Machine learning based models for predicting the degree of patrilineal consanguinity yielded accurate and reasonably precise predictions when using RM Y-STRs. Fully matching haplotypes resulted in a 95% confidence interval of 1–6 meioses with RMplex compared to 1–25 with Yfiler Plus. Our comprehensive pedigree study demonstrates the value of RM Y-STRs for differentiating male relatives of various types, in many cases achieving individual identification, thereby overcoming the largest limitation of forensic Y-chromosome analysis. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s00439-022-02493-2.