Patterns of genetic variation and the role of selection in HTR1A and HTR1B in macaques (Macaca)

BACKGROUND: Research has increasingly highlighted the role of serotonin in behavior. However, few researchers have examined serotonin in an evolutionary context, although such research could provide insight into the evolution of important behaviors. The genus Macaca represents a useful model to address this, as this genus shows a wide range of behavioral variation. In addition, many genetic features of the macaque serotonin system are similar to those of humans, and as common models in biomedical research, knowledge of the genetic variation and evolution of serotonin functioning in macaques are particularly relevant for studies of human evolution. Here, we examine the role of selection in the macaque serotonin system by comparing patterns of genetic variation for two genes that code for two types of serotonin receptors – HTR1A and HTR1B – across five species of macaques. RESULTS: The pattern of variation is significantly different for HTR1A compared to HTR1B. Specifically, there is an increase in between-species variation compared to within-species variation for HTR1A. Phylogenetic analyses indicate that portions of HTR1A show an elevated level of nonsynonymous substitutions. Together these analyses are indicative of positive selection acting on HTR1A, but not HTR1B. Furthermore, the haplotype network for HTR1A is inconsistent with the species tree, potentially due to both deep coalescence and selection. CONCLUSIONS: The results of this study indicate distinct evolutionary histories for HTR1A and HTR1B, with HTR1A showing evidence of selection and a high level of divergence among species, a factor which may have an impact on biomedical research that uses these species as models. The wide genetic variation of HTR1A may also explain some of the species differences in behavior, although further studies on the phenotypic effect of the sequenced polymorphisms are needed to confirm this. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s12863-014-0116-5) contains supplementary material, which is available to authorized users.