Collective interaction effects associated with mammalian behavioral traits reveal genetic factors connecting fear and hemostasis

BACKGROUND: Investigation of the genetic architectures that influence the behavioral traits of animals can provide important insights into human neuropsychiatric phenotypes. These traits, however, are often highly polygenic, with individual loci contributing only small effects to the overall association. The polygenicity makes it challenging to explain, for example, the widely observed comorbidity between stress and cardiac disease. METHODS: We present an algorithm for inferring the collective association of a large number of interacting gene variants with a quantitative trait. Using simulated data, we demonstrate that by taking into account the non-uniform distribution of genotypes within a cohort, we can achieve greater power than regression-based methods for high-dimensional inference. RESULTS: We analyzed genome-wide data sets of outbred mice and pet dogs, and found neurobiological pathways whose associations with behavioral traits arose primarily from interaction effects: γ-carboxylated coagulation factors and downstream neuronal signaling were highly associated with conditioned fear, consistent with our previous finding in human post-traumatic stress disorder (PTSD) data. Prepulse inhibition in mice was associated with serotonin transporter and platelet homeostasis, and noise-induced fear in dogs with hemostasis. CONCLUSIONS: Our findings suggest a novel explanation for the observed comorbidity between PTSD/anxiety and cardiovascular diseases: key coagulation factors modulating hemostasis also regulate synaptic plasticity affecting the learning and extinction of fear. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1186/s12888-018-1753-4) contains supplementary material, which is available to authorized users.