Cargando…
Stochastic Control for Bayesian Neural Network Training
In this paper, we propose to leverage the Bayesian uncertainty information encoded in parameter distributions to inform the learning procedure for Bayesian models. We derive a first principle stochastic differential equation for the training dynamics of the mean and uncertainty parameter in the vari...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9407447/ https://www.ncbi.nlm.nih.gov/pubmed/36010761 http://dx.doi.org/10.3390/e24081097 |
_version_ | 1784774366328782848 |
---|---|
author | Winkler, Ludwig Ojeda, César Opper, Manfred |
author_facet | Winkler, Ludwig Ojeda, César Opper, Manfred |
author_sort | Winkler, Ludwig |
collection | PubMed |
description | In this paper, we propose to leverage the Bayesian uncertainty information encoded in parameter distributions to inform the learning procedure for Bayesian models. We derive a first principle stochastic differential equation for the training dynamics of the mean and uncertainty parameter in the variational distributions. On the basis of the derived Bayesian stochastic differential equation, we apply the methodology of stochastic optimal control on the variational parameters to obtain individually controlled learning rates. We show that the resulting optimizer, StochControlSGD, is significantly more robust to large learning rates and can adaptively and individually control the learning rates of the variational parameters. The evolution of the control suggests separate and distinct dynamical behaviours in the training regimes for the mean and uncertainty parameters in Bayesian neural networks. |
format | Online Article Text |
id | pubmed-9407447 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-94074472022-08-26 Stochastic Control for Bayesian Neural Network Training Winkler, Ludwig Ojeda, César Opper, Manfred Entropy (Basel) Article In this paper, we propose to leverage the Bayesian uncertainty information encoded in parameter distributions to inform the learning procedure for Bayesian models. We derive a first principle stochastic differential equation for the training dynamics of the mean and uncertainty parameter in the variational distributions. On the basis of the derived Bayesian stochastic differential equation, we apply the methodology of stochastic optimal control on the variational parameters to obtain individually controlled learning rates. We show that the resulting optimizer, StochControlSGD, is significantly more robust to large learning rates and can adaptively and individually control the learning rates of the variational parameters. The evolution of the control suggests separate and distinct dynamical behaviours in the training regimes for the mean and uncertainty parameters in Bayesian neural networks. MDPI 2022-08-09 /pmc/articles/PMC9407447/ /pubmed/36010761 http://dx.doi.org/10.3390/e24081097 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Winkler, Ludwig Ojeda, César Opper, Manfred Stochastic Control for Bayesian Neural Network Training |
title | Stochastic Control for Bayesian Neural Network Training |
title_full | Stochastic Control for Bayesian Neural Network Training |
title_fullStr | Stochastic Control for Bayesian Neural Network Training |
title_full_unstemmed | Stochastic Control for Bayesian Neural Network Training |
title_short | Stochastic Control for Bayesian Neural Network Training |
title_sort | stochastic control for bayesian neural network training |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9407447/ https://www.ncbi.nlm.nih.gov/pubmed/36010761 http://dx.doi.org/10.3390/e24081097 |
work_keys_str_mv | AT winklerludwig stochasticcontrolforbayesianneuralnetworktraining AT ojedacesar stochasticcontrolforbayesianneuralnetworktraining AT oppermanfred stochasticcontrolforbayesianneuralnetworktraining |