Cargando…

Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning

The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses, we introduce variational characterizations that naturally suggest a two-step scheme...

Descripción completa

Detalles Bibliográficos
Autores principales: García Trillos, Nicolas, Kaplan, Zachary, Sanz-Alonso, Daniel
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7515000/
https://www.ncbi.nlm.nih.gov/pubmed/33267225
http://dx.doi.org/10.3390/e21050511
_version_ 1783586717649862656
author García Trillos, Nicolas
Kaplan, Zachary
Sanz-Alonso, Daniel
author_facet García Trillos, Nicolas
Kaplan, Zachary
Sanz-Alonso, Daniel
author_sort García Trillos, Nicolas
collection PubMed
description The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses, we introduce variational characterizations that naturally suggest a two-step scheme for their optimization, based on the iterative shift of a probability density and the calculation of a best Gaussian approximation in Kullback–Leibler divergence. Disregarding approximation error in these two steps, the variational characterizations allow us to show a simple monotonicity result for training error along optimization iterates. The two-step optimization schemes for local entropy and heat regularized loss differ only over which argument of the Kullback–Leibler divergence is used to find the best Gaussian approximation. Local entropy corresponds to minimizing over the second argument, and the solution is given by moment matching. This allows replacing traditional backpropagation calculation of gradients by sampling algorithms, opening an avenue for gradient-free, parallelizable training of neural networks. However, our presentation also acknowledges the potential increase in computational cost of naive optimization of regularized costs, thus giving a less optimistic view than existing works of the gains facilitated by loss regularization.
format Online
Article
Text
id pubmed-7515000
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-75150002020-11-09 Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning García Trillos, Nicolas Kaplan, Zachary Sanz-Alonso, Daniel Entropy (Basel) Article The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses, we introduce variational characterizations that naturally suggest a two-step scheme for their optimization, based on the iterative shift of a probability density and the calculation of a best Gaussian approximation in Kullback–Leibler divergence. Disregarding approximation error in these two steps, the variational characterizations allow us to show a simple monotonicity result for training error along optimization iterates. The two-step optimization schemes for local entropy and heat regularized loss differ only over which argument of the Kullback–Leibler divergence is used to find the best Gaussian approximation. Local entropy corresponds to minimizing over the second argument, and the solution is given by moment matching. This allows replacing traditional backpropagation calculation of gradients by sampling algorithms, opening an avenue for gradient-free, parallelizable training of neural networks. However, our presentation also acknowledges the potential increase in computational cost of naive optimization of regularized costs, thus giving a less optimistic view than existing works of the gains facilitated by loss regularization. MDPI 2019-05-20 /pmc/articles/PMC7515000/ /pubmed/33267225 http://dx.doi.org/10.3390/e21050511 Text en © 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
García Trillos, Nicolas
Kaplan, Zachary
Sanz-Alonso, Daniel
Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning
title Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning
title_full Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning
title_fullStr Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning
title_full_unstemmed Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning
title_short Variational Characterizations of Local Entropy and Heat Regularization in Deep Learning
title_sort variational characterizations of local entropy and heat regularization in deep learning
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7515000/
https://www.ncbi.nlm.nih.gov/pubmed/33267225
http://dx.doi.org/10.3390/e21050511
work_keys_str_mv AT garciatrillosnicolas variationalcharacterizationsoflocalentropyandheatregularizationindeeplearning
AT kaplanzachary variationalcharacterizationsoflocalentropyandheatregularizationindeeplearning
AT sanzalonsodaniel variationalcharacterizationsoflocalentropyandheatregularizationindeeplearning