Cargando…
Multi-view classification with convolutional neural networks
Humans’ decision making process often relies on utilizing visual information from different views or perspectives. However, in machine-learning-based image classification we typically infer an object’s class from just a single image showing an object. Especially for challenging classification proble...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7802953/ https://www.ncbi.nlm.nih.gov/pubmed/33434208 http://dx.doi.org/10.1371/journal.pone.0245230 |
_version_ | 1783635847501840384 |
---|---|
author | Seeland, Marco Mäder, Patrick |
author_facet | Seeland, Marco Mäder, Patrick |
author_sort | Seeland, Marco |
collection | PubMed |
description | Humans’ decision making process often relies on utilizing visual information from different views or perspectives. However, in machine-learning-based image classification we typically infer an object’s class from just a single image showing an object. Especially for challenging classification problems, the visual information conveyed by a single image may be insufficient for an accurate decision. We propose a classification scheme that relies on fusing visual information captured through images depicting the same object from multiple perspectives. Convolutional neural networks are used to extract and encode visual features from the multiple views and we propose strategies for fusing these information. More specifically, we investigate the following three strategies: (1) fusing convolutional feature maps at differing network depths; (2) fusion of bottleneck latent representations prior to classification; and (3) score fusion. We systematically evaluate these strategies on three datasets from different domains. Our findings emphasize the benefit of integrating information fusion into the network rather than performing it by post-processing of classification scores. Furthermore, we demonstrate through a case study that already trained networks can be easily extended by the best fusion strategy, outperforming other approaches by large margin. |
format | Online Article Text |
id | pubmed-7802953 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-78029532021-01-22 Multi-view classification with convolutional neural networks Seeland, Marco Mäder, Patrick PLoS One Research Article Humans’ decision making process often relies on utilizing visual information from different views or perspectives. However, in machine-learning-based image classification we typically infer an object’s class from just a single image showing an object. Especially for challenging classification problems, the visual information conveyed by a single image may be insufficient for an accurate decision. We propose a classification scheme that relies on fusing visual information captured through images depicting the same object from multiple perspectives. Convolutional neural networks are used to extract and encode visual features from the multiple views and we propose strategies for fusing these information. More specifically, we investigate the following three strategies: (1) fusing convolutional feature maps at differing network depths; (2) fusion of bottleneck latent representations prior to classification; and (3) score fusion. We systematically evaluate these strategies on three datasets from different domains. Our findings emphasize the benefit of integrating information fusion into the network rather than performing it by post-processing of classification scores. Furthermore, we demonstrate through a case study that already trained networks can be easily extended by the best fusion strategy, outperforming other approaches by large margin. Public Library of Science 2021-01-12 /pmc/articles/PMC7802953/ /pubmed/33434208 http://dx.doi.org/10.1371/journal.pone.0245230 Text en © 2021 Seeland, Mäder http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Seeland, Marco Mäder, Patrick Multi-view classification with convolutional neural networks |
title | Multi-view classification with convolutional neural networks |
title_full | Multi-view classification with convolutional neural networks |
title_fullStr | Multi-view classification with convolutional neural networks |
title_full_unstemmed | Multi-view classification with convolutional neural networks |
title_short | Multi-view classification with convolutional neural networks |
title_sort | multi-view classification with convolutional neural networks |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7802953/ https://www.ncbi.nlm.nih.gov/pubmed/33434208 http://dx.doi.org/10.1371/journal.pone.0245230 |
work_keys_str_mv | AT seelandmarco multiviewclassificationwithconvolutionalneuralnetworks AT maderpatrick multiviewclassificationwithconvolutionalneuralnetworks |