Cargando…
STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network
In recent years, human motion prediction has become an active research topic in computer vision. However, owing to the complexity and stochastic nature of human motion, it remains a challenging problem. In previous works, human motion prediction has always been treated as a typical inter-sequence pr...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Springer Nature Singapore
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9338210/ https://www.ncbi.nlm.nih.gov/pubmed/35904666 http://dx.doi.org/10.1186/s42492-022-00112-5 |
_version_ | 1784759918679556096 |
---|---|
author | Chen, Lujing Liu, Rui Yang, Xin Zhou, Dongsheng Zhang, Qiang Wei, Xiaopeng |
author_facet | Chen, Lujing Liu, Rui Yang, Xin Zhou, Dongsheng Zhang, Qiang Wei, Xiaopeng |
author_sort | Chen, Lujing |
collection | PubMed |
description | In recent years, human motion prediction has become an active research topic in computer vision. However, owing to the complexity and stochastic nature of human motion, it remains a challenging problem. In previous works, human motion prediction has always been treated as a typical inter-sequence problem, and most works have aimed to capture the temporal dependence between successive frames. However, although these approaches focused on the effects of the temporal dimension, they rarely considered the correlation between different joints in space. Thus, the spatio-temporal coupling of human joints is considered, to propose a novel spatio-temporal network based on a transformer and a gragh convolutional network (GCN) (STTG-Net). The temporal transformer is used to capture the global temporal dependencies, and the spatial GCN module is used to establish local spatial correlations between the joints for each frame. To overcome the problems of error accumulation and discontinuity in the motion prediction, a revision method based on fusion strategy is also proposed, in which the current prediction frame is fused with the previous frame. The experimental results show that the proposed prediction method has less prediction error and the prediction motion is smoother than previous prediction methods. The effectiveness of the proposed method is also demonstrated comparing it with the state-of-the-art method on the Human3.6 M dataset. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s42492-022-00112-5. |
format | Online Article Text |
id | pubmed-9338210 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Springer Nature Singapore |
record_format | MEDLINE/PubMed |
spelling | pubmed-93382102022-07-31 STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network Chen, Lujing Liu, Rui Yang, Xin Zhou, Dongsheng Zhang, Qiang Wei, Xiaopeng Vis Comput Ind Biomed Art Original Article In recent years, human motion prediction has become an active research topic in computer vision. However, owing to the complexity and stochastic nature of human motion, it remains a challenging problem. In previous works, human motion prediction has always been treated as a typical inter-sequence problem, and most works have aimed to capture the temporal dependence between successive frames. However, although these approaches focused on the effects of the temporal dimension, they rarely considered the correlation between different joints in space. Thus, the spatio-temporal coupling of human joints is considered, to propose a novel spatio-temporal network based on a transformer and a gragh convolutional network (GCN) (STTG-Net). The temporal transformer is used to capture the global temporal dependencies, and the spatial GCN module is used to establish local spatial correlations between the joints for each frame. To overcome the problems of error accumulation and discontinuity in the motion prediction, a revision method based on fusion strategy is also proposed, in which the current prediction frame is fused with the previous frame. The experimental results show that the proposed prediction method has less prediction error and the prediction motion is smoother than previous prediction methods. The effectiveness of the proposed method is also demonstrated comparing it with the state-of-the-art method on the Human3.6 M dataset. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s42492-022-00112-5. Springer Nature Singapore 2022-07-29 /pmc/articles/PMC9338210/ /pubmed/35904666 http://dx.doi.org/10.1186/s42492-022-00112-5 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Original Article Chen, Lujing Liu, Rui Yang, Xin Zhou, Dongsheng Zhang, Qiang Wei, Xiaopeng STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network |
title | STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network |
title_full | STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network |
title_fullStr | STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network |
title_full_unstemmed | STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network |
title_short | STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network |
title_sort | sttg-net: a spatio-temporal network for human motion prediction based on transformer and graph convolution network |
topic | Original Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9338210/ https://www.ncbi.nlm.nih.gov/pubmed/35904666 http://dx.doi.org/10.1186/s42492-022-00112-5 |
work_keys_str_mv | AT chenlujing sttgnetaspatiotemporalnetworkforhumanmotionpredictionbasedontransformerandgraphconvolutionnetwork AT liurui sttgnetaspatiotemporalnetworkforhumanmotionpredictionbasedontransformerandgraphconvolutionnetwork AT yangxin sttgnetaspatiotemporalnetworkforhumanmotionpredictionbasedontransformerandgraphconvolutionnetwork AT zhoudongsheng sttgnetaspatiotemporalnetworkforhumanmotionpredictionbasedontransformerandgraphconvolutionnetwork AT zhangqiang sttgnetaspatiotemporalnetworkforhumanmotionpredictionbasedontransformerandgraphconvolutionnetwork AT weixiaopeng sttgnetaspatiotemporalnetworkforhumanmotionpredictionbasedontransformerandgraphconvolutionnetwork |