Cargando…

Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network

The advantages of an event camera, such as low power consumption, large dynamic range, and low data redundancy, enable it to shine in extreme environments where traditional image sensors are not competent, especially in high-speed moving target capture and extreme lighting conditions. Optical flow r...

Descripción completa

Detalles Bibliográficos
Autores principales: Zhang, Yisa, Lv, Hengyi, Zhao, Yuchen, Feng, Yang, Liu, Hailong, Bi, Guoling
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9867051/
https://www.ncbi.nlm.nih.gov/pubmed/36677264
http://dx.doi.org/10.3390/mi14010203
_version_ 1784876246208872448
author Zhang, Yisa
Lv, Hengyi
Zhao, Yuchen
Feng, Yang
Liu, Hailong
Bi, Guoling
author_facet Zhang, Yisa
Lv, Hengyi
Zhao, Yuchen
Feng, Yang
Liu, Hailong
Bi, Guoling
author_sort Zhang, Yisa
collection PubMed
description The advantages of an event camera, such as low power consumption, large dynamic range, and low data redundancy, enable it to shine in extreme environments where traditional image sensors are not competent, especially in high-speed moving target capture and extreme lighting conditions. Optical flow reflects the target’s movement information, and the target’s detailed movement can be obtained using the event camera’s optical flow information. However, the existing neural network methods for optical flow prediction of event cameras has the problems of extensive computation and high energy consumption in hardware implementation. The spike neural network has spatiotemporal coding characteristics, so it can be compatible with the spatiotemporal data of an event camera. Moreover, the sparse coding characteristic of the spike neural network makes it run with ultra-low power consumption on neuromorphic hardware. However, because of the algorithmic and training complexity, the spike neural network has not been applied in the prediction of the optical flow for the event camera. For this case, this paper proposes an end-to-end spike neural network to predict the optical flow of the discrete spatiotemporal data stream for the event camera. The network is trained with the spatio-temporal backpropagation method in a self-supervised way, which fully combines the spatiotemporal characteristics of the event camera while improving the network performance. Compared with the existing methods on the public dataset, the experimental results show that the method proposed in this paper is equivalent to the best existing methods in terms of optical flow prediction accuracy, and it can save 99% more power consumption than the existing algorithm, which is greatly beneficial to the hardware implementation of the event camera optical flow prediction., laying the groundwork for future low-power hardware implementation of optical flow prediction for event cameras.
format Online
Article
Text
id pubmed-9867051
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-98670512023-01-22 Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network Zhang, Yisa Lv, Hengyi Zhao, Yuchen Feng, Yang Liu, Hailong Bi, Guoling Micromachines (Basel) Article The advantages of an event camera, such as low power consumption, large dynamic range, and low data redundancy, enable it to shine in extreme environments where traditional image sensors are not competent, especially in high-speed moving target capture and extreme lighting conditions. Optical flow reflects the target’s movement information, and the target’s detailed movement can be obtained using the event camera’s optical flow information. However, the existing neural network methods for optical flow prediction of event cameras has the problems of extensive computation and high energy consumption in hardware implementation. The spike neural network has spatiotemporal coding characteristics, so it can be compatible with the spatiotemporal data of an event camera. Moreover, the sparse coding characteristic of the spike neural network makes it run with ultra-low power consumption on neuromorphic hardware. However, because of the algorithmic and training complexity, the spike neural network has not been applied in the prediction of the optical flow for the event camera. For this case, this paper proposes an end-to-end spike neural network to predict the optical flow of the discrete spatiotemporal data stream for the event camera. The network is trained with the spatio-temporal backpropagation method in a self-supervised way, which fully combines the spatiotemporal characteristics of the event camera while improving the network performance. Compared with the existing methods on the public dataset, the experimental results show that the method proposed in this paper is equivalent to the best existing methods in terms of optical flow prediction accuracy, and it can save 99% more power consumption than the existing algorithm, which is greatly beneficial to the hardware implementation of the event camera optical flow prediction., laying the groundwork for future low-power hardware implementation of optical flow prediction for event cameras. MDPI 2023-01-13 /pmc/articles/PMC9867051/ /pubmed/36677264 http://dx.doi.org/10.3390/mi14010203 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Zhang, Yisa
Lv, Hengyi
Zhao, Yuchen
Feng, Yang
Liu, Hailong
Bi, Guoling
Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network
title Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network
title_full Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network
title_fullStr Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network
title_full_unstemmed Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network
title_short Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network
title_sort event-based optical flow estimation with spatio-temporal backpropagation trained spiking neural network
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9867051/
https://www.ncbi.nlm.nih.gov/pubmed/36677264
http://dx.doi.org/10.3390/mi14010203
work_keys_str_mv AT zhangyisa eventbasedopticalflowestimationwithspatiotemporalbackpropagationtrainedspikingneuralnetwork
AT lvhengyi eventbasedopticalflowestimationwithspatiotemporalbackpropagationtrainedspikingneuralnetwork
AT zhaoyuchen eventbasedopticalflowestimationwithspatiotemporalbackpropagationtrainedspikingneuralnetwork
AT fengyang eventbasedopticalflowestimationwithspatiotemporalbackpropagationtrainedspikingneuralnetwork
AT liuhailong eventbasedopticalflowestimationwithspatiotemporalbackpropagationtrainedspikingneuralnetwork
AT biguoling eventbasedopticalflowestimationwithspatiotemporalbackpropagationtrainedspikingneuralnetwork