Beta-PSMC: uncovering more detailed population history using beta distribution

BACKGROUND: Inferring the demographic history of a population is essential in population genetic studies. Though the inference methods based on the sequentially Markov coalescent can present the population history in detail, these methods assume that the population size remains unchanged in each time interval during discretizing the hidden state in the hidden Markov model. Therefore, these methods fail to uncover the detailed population history in each time interval. RESULTS: We present a new method called Beta-PSMC, which introduces the probability density function of a beta distribution with a broad variety of shapes into the Pairwise Sequentially Markovian Coalescent (PSMC) model to refine the population history in each discretized time interval in place of the assumption that the population size is unchanged. Using simulation, we demonstrate that Beta-PSMC can uncover more detailed population history, and improve the accuracy and resolution of the recent population history inference. We also apply Beta-PSMC to infer the population history of Adélie penguin and find that the fluctuation in population size is contrary to the temperature change 15–27 thousand years ago. CONCLUSIONS: Beta-PSMC extends PSMC by allowing more detailed fluctuation of population size in each discretized time interval with the probability density function of beta distribution and will serve as a useful tool for population genetics. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12864-022-09021-6.