Cargando…

Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding

Recognising familiar places is a competence required in many engineering applications that interact with the real world such as robot navigation. Combining information from different sensory sources promotes robustness and accuracy of place recognition. However, mismatch in data registration, dimens...

Descripción completa

Detalles Bibliográficos
Autores principales: Pearson, Martin J., Dora, Shirin, Struckmeier, Oliver, Knowles, Thomas C., Mitchinson, Ben, Tiwari, Kshitij, Kyrki, Ville, Bohte, Sander, Pennartz, Cyriel M.A.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8710724/
https://www.ncbi.nlm.nih.gov/pubmed/34966789
http://dx.doi.org/10.3389/frobt.2021.732023
_version_ 1784623222325510144
author Pearson, Martin J.
Dora, Shirin
Struckmeier, Oliver
Knowles, Thomas C.
Mitchinson, Ben
Tiwari, Kshitij
Kyrki, Ville
Bohte, Sander
Pennartz, Cyriel M.A.
author_facet Pearson, Martin J.
Dora, Shirin
Struckmeier, Oliver
Knowles, Thomas C.
Mitchinson, Ben
Tiwari, Kshitij
Kyrki, Ville
Bohte, Sander
Pennartz, Cyriel M.A.
author_sort Pearson, Martin J.
collection PubMed
description Recognising familiar places is a competence required in many engineering applications that interact with the real world such as robot navigation. Combining information from different sensory sources promotes robustness and accuracy of place recognition. However, mismatch in data registration, dimensionality, and timing between modalities remain challenging problems in multisensory place recognition. Spurious data generated by sensor drop-out in multisensory environments is particularly problematic and often resolved through adhoc and brittle solutions. An effective approach to these problems is demonstrated by animals as they gracefully move through the world. Therefore, we take a neuro-ethological approach by adopting self-supervised representation learning based on a neuroscientific model of visual cortex known as predictive coding. We demonstrate how this parsimonious network algorithm which is trained using a local learning rule can be extended to combine visual and tactile sensory cues from a biomimetic robot as it naturally explores a visually aliased environment. The place recognition performance obtained using joint latent representations generated by the network is significantly better than contemporary representation learning techniques. Further, we see evidence of improved robustness at place recognition in face of unimodal sensor drop-out. The proposed multimodal deep predictive coding algorithm presented is also linearly extensible to accommodate more than two sensory modalities, thereby providing an intriguing example of the value of neuro-biologically plausible representation learning for multimodal navigation.
format Online
Article
Text
id pubmed-8710724
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-87107242021-12-28 Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding Pearson, Martin J. Dora, Shirin Struckmeier, Oliver Knowles, Thomas C. Mitchinson, Ben Tiwari, Kshitij Kyrki, Ville Bohte, Sander Pennartz, Cyriel M.A. Front Robot AI Robotics and AI Recognising familiar places is a competence required in many engineering applications that interact with the real world such as robot navigation. Combining information from different sensory sources promotes robustness and accuracy of place recognition. However, mismatch in data registration, dimensionality, and timing between modalities remain challenging problems in multisensory place recognition. Spurious data generated by sensor drop-out in multisensory environments is particularly problematic and often resolved through adhoc and brittle solutions. An effective approach to these problems is demonstrated by animals as they gracefully move through the world. Therefore, we take a neuro-ethological approach by adopting self-supervised representation learning based on a neuroscientific model of visual cortex known as predictive coding. We demonstrate how this parsimonious network algorithm which is trained using a local learning rule can be extended to combine visual and tactile sensory cues from a biomimetic robot as it naturally explores a visually aliased environment. The place recognition performance obtained using joint latent representations generated by the network is significantly better than contemporary representation learning techniques. Further, we see evidence of improved robustness at place recognition in face of unimodal sensor drop-out. The proposed multimodal deep predictive coding algorithm presented is also linearly extensible to accommodate more than two sensory modalities, thereby providing an intriguing example of the value of neuro-biologically plausible representation learning for multimodal navigation. Frontiers Media S.A. 2021-12-13 /pmc/articles/PMC8710724/ /pubmed/34966789 http://dx.doi.org/10.3389/frobt.2021.732023 Text en Copyright © 2021 Pearson, Dora, Struckmeier, Knowles, Mitchinson, Tiwari, Kyrki, Bohte and Pennartz. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Robotics and AI
Pearson, Martin J.
Dora, Shirin
Struckmeier, Oliver
Knowles, Thomas C.
Mitchinson, Ben
Tiwari, Kshitij
Kyrki, Ville
Bohte, Sander
Pennartz, Cyriel M.A.
Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding
title Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding
title_full Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding
title_fullStr Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding
title_full_unstemmed Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding
title_short Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding
title_sort multimodal representation learning for place recognition using deep hebbian predictive coding
topic Robotics and AI
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8710724/
https://www.ncbi.nlm.nih.gov/pubmed/34966789
http://dx.doi.org/10.3389/frobt.2021.732023
work_keys_str_mv AT pearsonmartinj multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT dorashirin multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT struckmeieroliver multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT knowlesthomasc multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT mitchinsonben multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT tiwarikshitij multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT kyrkiville multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT bohtesander multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding
AT pennartzcyrielma multimodalrepresentationlearningforplacerecognitionusingdeephebbianpredictivecoding