Cargando…
Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum
Various action recognition approaches have recently been proposed with the aid of three-dimensional (3D) convolution and a multiple stream structure. However, existing methods are sensitive to background and optical flow noise, which prevents from learning the main object in a video frame. Furthermo...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7411841/ https://www.ncbi.nlm.nih.gov/pubmed/32668715 http://dx.doi.org/10.3390/s20143894 |
_version_ | 1783568471812997120 |
---|---|
author | Kim, Hyunwoo Park, Seokmok Park, Hyeokjin Paik, Joonki |
author_facet | Kim, Hyunwoo Park, Seokmok Park, Hyeokjin Paik, Joonki |
author_sort | Kim, Hyunwoo |
collection | PubMed |
description | Various action recognition approaches have recently been proposed with the aid of three-dimensional (3D) convolution and a multiple stream structure. However, existing methods are sensitive to background and optical flow noise, which prevents from learning the main object in a video frame. Furthermore, they cannot reflect the accuracy of each stream in the process of combining multiple streams. In this paper, we present a novel action recognition method that improves the existing method using optical flow and a multi-stream structure. The proposed method consists of two parts: (i) optical flow enhancement process using image segmentation and (ii) score fusion process by applying weighted sum of the accuracy. The enhancement process can help the network to efficiently analyze the flow information of the main object in the optical flow frame, thereby improving accuracy. A different accuracy of each stream can be reflected to the fused score while using the proposed score fusion method. We achieved an accuracy of 98.2% on UCF-101 and 82.4% on HMDB-51. The proposed method outperformed many state-of-the-art methods without changing the network structure and it is expected to be easily applied to other networks. |
format | Online Article Text |
id | pubmed-7411841 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-74118412020-08-25 Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum Kim, Hyunwoo Park, Seokmok Park, Hyeokjin Paik, Joonki Sensors (Basel) Letter Various action recognition approaches have recently been proposed with the aid of three-dimensional (3D) convolution and a multiple stream structure. However, existing methods are sensitive to background and optical flow noise, which prevents from learning the main object in a video frame. Furthermore, they cannot reflect the accuracy of each stream in the process of combining multiple streams. In this paper, we present a novel action recognition method that improves the existing method using optical flow and a multi-stream structure. The proposed method consists of two parts: (i) optical flow enhancement process using image segmentation and (ii) score fusion process by applying weighted sum of the accuracy. The enhancement process can help the network to efficiently analyze the flow information of the main object in the optical flow frame, thereby improving accuracy. A different accuracy of each stream can be reflected to the fused score while using the proposed score fusion method. We achieved an accuracy of 98.2% on UCF-101 and 82.4% on HMDB-51. The proposed method outperformed many state-of-the-art methods without changing the network structure and it is expected to be easily applied to other networks. MDPI 2020-07-13 /pmc/articles/PMC7411841/ /pubmed/32668715 http://dx.doi.org/10.3390/s20143894 Text en © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Letter Kim, Hyunwoo Park, Seokmok Park, Hyeokjin Paik, Joonki Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum |
title | Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum |
title_full | Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum |
title_fullStr | Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum |
title_full_unstemmed | Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum |
title_short | Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum |
title_sort | enhanced action recognition using multiple stream deep learning with optical flow and weighted sum |
topic | Letter |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7411841/ https://www.ncbi.nlm.nih.gov/pubmed/32668715 http://dx.doi.org/10.3390/s20143894 |
work_keys_str_mv | AT kimhyunwoo enhancedactionrecognitionusingmultiplestreamdeeplearningwithopticalflowandweightedsum AT parkseokmok enhancedactionrecognitionusingmultiplestreamdeeplearningwithopticalflowandweightedsum AT parkhyeokjin enhancedactionrecognitionusingmultiplestreamdeeplearningwithopticalflowandweightedsum AT paikjoonki enhancedactionrecognitionusingmultiplestreamdeeplearningwithopticalflowandweightedsum |