Cargando…
The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions
Biological learning systems are outstanding in their ability to learn from limited training data compared to the most successful learning machines, i.e., Deep Neural Networks (DNNs). What are the key aspects that underlie this data efficiency gap is an unresolved question at the core of biological a...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8842477/ https://www.ncbi.nlm.nih.gov/pubmed/35173595 http://dx.doi.org/10.3389/fncom.2022.760085 |
_version_ | 1784651058915573760 |
---|---|
author | D'Amario, Vanessa Srivastava, Sanjana Sasaki, Tomotake Boix, Xavier |
author_facet | D'Amario, Vanessa Srivastava, Sanjana Sasaki, Tomotake Boix, Xavier |
author_sort | D'Amario, Vanessa |
collection | PubMed |
description | Biological learning systems are outstanding in their ability to learn from limited training data compared to the most successful learning machines, i.e., Deep Neural Networks (DNNs). What are the key aspects that underlie this data efficiency gap is an unresolved question at the core of biological and artificial intelligence. We hypothesize that one important aspect is that biological systems rely on mechanisms such as foveations in order to reduce unnecessary input dimensions for the task at hand, e.g., background in object recognition, while state-of-the-art DNNs do not. Datasets to train DNNs often contain such unnecessary input dimensions, and these lead to more trainable parameters. Yet, it is not clear whether this affects the DNNs' data efficiency because DNNs are robust to increasing the number of parameters in the hidden layers, and it is uncertain whether this holds true for the input layer. In this paper, we investigate the impact of unnecessary input dimensions on the DNNs data efficiency, namely, the amount of examples needed to achieve certain generalization performance. Our results show that unnecessary input dimensions that are task-unrelated substantially degrade data efficiency. This highlights the need for mechanisms that remove task-unrelated dimensions, such as foveation for image classification, in order to enable data efficiency gains. |
format | Online Article Text |
id | pubmed-8842477 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-88424772022-02-15 The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions D'Amario, Vanessa Srivastava, Sanjana Sasaki, Tomotake Boix, Xavier Front Comput Neurosci Neuroscience Biological learning systems are outstanding in their ability to learn from limited training data compared to the most successful learning machines, i.e., Deep Neural Networks (DNNs). What are the key aspects that underlie this data efficiency gap is an unresolved question at the core of biological and artificial intelligence. We hypothesize that one important aspect is that biological systems rely on mechanisms such as foveations in order to reduce unnecessary input dimensions for the task at hand, e.g., background in object recognition, while state-of-the-art DNNs do not. Datasets to train DNNs often contain such unnecessary input dimensions, and these lead to more trainable parameters. Yet, it is not clear whether this affects the DNNs' data efficiency because DNNs are robust to increasing the number of parameters in the hidden layers, and it is uncertain whether this holds true for the input layer. In this paper, we investigate the impact of unnecessary input dimensions on the DNNs data efficiency, namely, the amount of examples needed to achieve certain generalization performance. Our results show that unnecessary input dimensions that are task-unrelated substantially degrade data efficiency. This highlights the need for mechanisms that remove task-unrelated dimensions, such as foveation for image classification, in order to enable data efficiency gains. Frontiers Media S.A. 2022-01-31 /pmc/articles/PMC8842477/ /pubmed/35173595 http://dx.doi.org/10.3389/fncom.2022.760085 Text en Copyright © 2022 D'Amario, Srivastava, Sasaki and Boix. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Neuroscience D'Amario, Vanessa Srivastava, Sanjana Sasaki, Tomotake Boix, Xavier The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions |
title | The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions |
title_full | The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions |
title_fullStr | The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions |
title_full_unstemmed | The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions |
title_short | The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions |
title_sort | data efficiency of deep learning is degraded by unnecessary input dimensions |
topic | Neuroscience |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8842477/ https://www.ncbi.nlm.nih.gov/pubmed/35173595 http://dx.doi.org/10.3389/fncom.2022.760085 |
work_keys_str_mv | AT damariovanessa thedataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions AT srivastavasanjana thedataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions AT sasakitomotake thedataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions AT boixxavier thedataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions AT damariovanessa dataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions AT srivastavasanjana dataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions AT sasakitomotake dataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions AT boixxavier dataefficiencyofdeeplearningisdegradedbyunnecessaryinputdimensions |