Complexity control by gradient descent in deep networks

Overparametrized deep networks predict well, despite the lack of an explicit complexity control during training, such as an explicit regularization term. For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normal...

Descripción completa

Detalles Bibliográficos
Autores principales: Poggio, Tomaso, Liao, Qianli, Banburski, Andrzej
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Nature Publishing Group UK 2020
Materias:
Article
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7039878/
https://www.ncbi.nlm.nih.gov/pubmed/32094327
http://dx.doi.org/10.1038/s41467-020-14663-9
_version_ 1783500871367131136
author Poggio, Tomaso
Liao, Qianli
Banburski, Andrzej
author_facet Poggio, Tomaso
Liao, Qianli
Banburski, Andrzej
author_sort Poggio, Tomaso
collection PubMed
description Overparametrized deep networks predict well, despite the lack of an explicit complexity control during training, such as an explicit regularization term. For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification.
format Online
Article
Text
id pubmed-7039878
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher Nature Publishing Group UK
record_format MEDLINE/PubMed
spelling pubmed-70398782020-03-04 Complexity control by gradient descent in deep networks Poggio, Tomaso Liao, Qianli Banburski, Andrzej Nat Commun Article Overparametrized deep networks predict well, despite the lack of an explicit complexity control during training, such as an explicit regularization term. For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification. Nature Publishing Group UK 2020-02-24 /pmc/articles/PMC7039878/ /pubmed/32094327 http://dx.doi.org/10.1038/s41467-020-14663-9 Text en © The Author(s) 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
spellingShingle Article
Poggio, Tomaso
Liao, Qianli
Banburski, Andrzej
Complexity control by gradient descent in deep networks
title Complexity control by gradient descent in deep networks
title_full Complexity control by gradient descent in deep networks
title_fullStr Complexity control by gradient descent in deep networks
title_full_unstemmed Complexity control by gradient descent in deep networks
title_short Complexity control by gradient descent in deep networks
title_sort complexity control by gradient descent in deep networks
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7039878/
https://www.ncbi.nlm.nih.gov/pubmed/32094327
http://dx.doi.org/10.1038/s41467-020-14663-9
work_keys_str_mv AT poggiotomaso complexitycontrolbygradientdescentindeepnetworks
AT liaoqianli complexitycontrolbygradientdescentindeepnetworks
AT banburskiandrzej complexitycontrolbygradientdescentindeepnetworks

Ejemplares similares