Orthology and paralogy constraints: satisfiability and consistency

BACKGROUND: A variety of methods based on sequence similarity, reconciliation, synteny or functional characteristics, can be used to infer orthology and paralogy relations between genes of a given gene family [Formula: see text]. But is a given set [Formula: see text] of orthology/paralogy constraints possible, i.e., can they simultaneously co-exist in an evolutionary history for [Formula: see text]? While previous studies have focused on full sets of constraints, here we consider the general case where [Formula: see text] does not necessarily involve a constraint for each pair of genes. The problem is subdivided in two parts: (1) Is [Formula: see text] satisfiable, i.e. can we find an event-labeled gene tree G inducing [Formula: see text]? (2) Is there such a G which is consistent, i.e., such that all displayed triplet phylogenies are included in a species tree? RESULTS: Previous results on the Graph sandwich problem can be used to answer to (1), and we provide polynomial-time algorithms for satisfiability and consistency with a given species tree. We also describe a new polynomial-time algorithm for the case of consistency with an unknown species tree and full knowledge of pairwise orthology/paralogy relationships, as well as a branch-and-bound algorithm in the case when unknown relations are present. We show that our algorithms can be used in combination with ProteinOrtho, a sequence similarity-based orthology detection tool, to extract a set of robust orthology/paralogy relationships.