Cargando…

Multi-view classification with convolutional neural networks

Humans’ decision making process often relies on utilizing visual information from different views or perspectives. However, in machine-learning-based image classification we typically infer an object’s class from just a single image showing an object. Especially for challenging classification proble...

Descripción completa

Detalles Bibliográficos
Autores principales: Seeland, Marco, Mäder, Patrick
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7802953/
https://www.ncbi.nlm.nih.gov/pubmed/33434208
http://dx.doi.org/10.1371/journal.pone.0245230
_version_ 1783635847501840384
author Seeland, Marco
Mäder, Patrick
author_facet Seeland, Marco
Mäder, Patrick
author_sort Seeland, Marco
collection PubMed
description Humans’ decision making process often relies on utilizing visual information from different views or perspectives. However, in machine-learning-based image classification we typically infer an object’s class from just a single image showing an object. Especially for challenging classification problems, the visual information conveyed by a single image may be insufficient for an accurate decision. We propose a classification scheme that relies on fusing visual information captured through images depicting the same object from multiple perspectives. Convolutional neural networks are used to extract and encode visual features from the multiple views and we propose strategies for fusing these information. More specifically, we investigate the following three strategies: (1) fusing convolutional feature maps at differing network depths; (2) fusion of bottleneck latent representations prior to classification; and (3) score fusion. We systematically evaluate these strategies on three datasets from different domains. Our findings emphasize the benefit of integrating information fusion into the network rather than performing it by post-processing of classification scores. Furthermore, we demonstrate through a case study that already trained networks can be easily extended by the best fusion strategy, outperforming other approaches by large margin.
format Online
Article
Text
id pubmed-7802953
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-78029532021-01-22 Multi-view classification with convolutional neural networks Seeland, Marco Mäder, Patrick PLoS One Research Article Humans’ decision making process often relies on utilizing visual information from different views or perspectives. However, in machine-learning-based image classification we typically infer an object’s class from just a single image showing an object. Especially for challenging classification problems, the visual information conveyed by a single image may be insufficient for an accurate decision. We propose a classification scheme that relies on fusing visual information captured through images depicting the same object from multiple perspectives. Convolutional neural networks are used to extract and encode visual features from the multiple views and we propose strategies for fusing these information. More specifically, we investigate the following three strategies: (1) fusing convolutional feature maps at differing network depths; (2) fusion of bottleneck latent representations prior to classification; and (3) score fusion. We systematically evaluate these strategies on three datasets from different domains. Our findings emphasize the benefit of integrating information fusion into the network rather than performing it by post-processing of classification scores. Furthermore, we demonstrate through a case study that already trained networks can be easily extended by the best fusion strategy, outperforming other approaches by large margin. Public Library of Science 2021-01-12 /pmc/articles/PMC7802953/ /pubmed/33434208 http://dx.doi.org/10.1371/journal.pone.0245230 Text en © 2021 Seeland, Mäder http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Research Article
Seeland, Marco
Mäder, Patrick
Multi-view classification with convolutional neural networks
title Multi-view classification with convolutional neural networks
title_full Multi-view classification with convolutional neural networks
title_fullStr Multi-view classification with convolutional neural networks
title_full_unstemmed Multi-view classification with convolutional neural networks
title_short Multi-view classification with convolutional neural networks
title_sort multi-view classification with convolutional neural networks
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7802953/
https://www.ncbi.nlm.nih.gov/pubmed/33434208
http://dx.doi.org/10.1371/journal.pone.0245230
work_keys_str_mv AT seelandmarco multiviewclassificationwithconvolutionalneuralnetworks
AT maderpatrick multiviewclassificationwithconvolutionalneuralnetworks