Cargando…

LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals

Markerless 3D pose estimation has become an indispensable tool for kinematic studies of laboratory animals. Most current methods recover 3D pose by multi-view triangulation of deep network-based 2D pose estimates. However, triangulation requires multiple, synchronized cameras and elaborate calibrati...

Descripción completa

Detalles Bibliográficos
Autores principales: Gosztolai, Adam, Günel, Semih, Ríos, Victor Lobato, Abrate, Marco Pietro, Morales, Daniel, Rhodin, Helge, Fua, Pascal, Ramdya, Pavan
Formato: Online Artículo Texto
Lenguaje:English
Publicado: 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7611544/
https://www.ncbi.nlm.nih.gov/pubmed/34354294
http://dx.doi.org/10.1038/s41592-021-01226-z
_version_ 1783605282966863872
author Gosztolai, Adam
Günel, Semih
Ríos, Victor Lobato
Abrate, Marco Pietro
Morales, Daniel
Rhodin, Helge
Fua, Pascal
Ramdya, Pavan
author_facet Gosztolai, Adam
Günel, Semih
Ríos, Victor Lobato
Abrate, Marco Pietro
Morales, Daniel
Rhodin, Helge
Fua, Pascal
Ramdya, Pavan
author_sort Gosztolai, Adam
collection PubMed
description Markerless 3D pose estimation has become an indispensable tool for kinematic studies of laboratory animals. Most current methods recover 3D pose by multi-view triangulation of deep network-based 2D pose estimates. However, triangulation requires multiple, synchronized cameras and elaborate calibration protocols that hinder its widespread adoption in laboratory studies. Here we describe LiftPose3D, a deep network-based method that overcomes these barriers by reconstructing 3D poses from a single 2D camera view. We illustrate LiftPose3D’s versatility by applying it to multiple experimental systems using flies, mice, rats, and macaque monkeys and in circumstances where 3D triangulation is impractical or impossible. Our framework achieves accurate lifting for stereotyped and non-stereotyped behaviors from different camera angles. Thus, LiftPose3D permits high-quality 3D pose estimation in the absence of complex camera arrays, tedious calibration procedures, and despite occluded body parts in freely behaving animals.
format Online
Article
Text
id pubmed-7611544
institution National Center for Biotechnology Information
language English
publishDate 2021
record_format MEDLINE/PubMed
spelling pubmed-76115442021-08-20 LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals Gosztolai, Adam Günel, Semih Ríos, Victor Lobato Abrate, Marco Pietro Morales, Daniel Rhodin, Helge Fua, Pascal Ramdya, Pavan Nat Methods Article Markerless 3D pose estimation has become an indispensable tool for kinematic studies of laboratory animals. Most current methods recover 3D pose by multi-view triangulation of deep network-based 2D pose estimates. However, triangulation requires multiple, synchronized cameras and elaborate calibration protocols that hinder its widespread adoption in laboratory studies. Here we describe LiftPose3D, a deep network-based method that overcomes these barriers by reconstructing 3D poses from a single 2D camera view. We illustrate LiftPose3D’s versatility by applying it to multiple experimental systems using flies, mice, rats, and macaque monkeys and in circumstances where 3D triangulation is impractical or impossible. Our framework achieves accurate lifting for stereotyped and non-stereotyped behaviors from different camera angles. Thus, LiftPose3D permits high-quality 3D pose estimation in the absence of complex camera arrays, tedious calibration procedures, and despite occluded body parts in freely behaving animals. 2021-08-01 2021-08-05 /pmc/articles/PMC7611544/ /pubmed/34354294 http://dx.doi.org/10.1038/s41592-021-01226-z Text en https://www.springernature.com/gp/open-research/policies/accepted-manuscript-termsUsers may view, print, copy, and download text and data-mine the content in such documents, for the purposes of academic research, subject always to the full Conditions of use: https://www.springernature.com/gp/open-research/policies/accepted-manuscript-terms
spellingShingle Article
Gosztolai, Adam
Günel, Semih
Ríos, Victor Lobato
Abrate, Marco Pietro
Morales, Daniel
Rhodin, Helge
Fua, Pascal
Ramdya, Pavan
LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals
title LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals
title_full LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals
title_fullStr LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals
title_full_unstemmed LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals
title_short LiftPose3D, a deep learning-based approach for transforming 2D to 3D pose in laboratory animals
title_sort liftpose3d, a deep learning-based approach for transforming 2d to 3d pose in laboratory animals
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7611544/
https://www.ncbi.nlm.nih.gov/pubmed/34354294
http://dx.doi.org/10.1038/s41592-021-01226-z
work_keys_str_mv AT gosztolaiadam liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals
AT gunelsemih liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals
AT riosvictorlobato liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals
AT abratemarcopietro liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals
AT moralesdaniel liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals
AT rhodinhelge liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals
AT fuapascal liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals
AT ramdyapavan liftpose3dadeeplearningbasedapproachfortransforming2dto3dposeinlaboratoryanimals