Cargando…
Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks
In this work, we present a local intrinsic rule that we developed, dubbed IP, inspired by the Infomax rule. Like Infomax, this rule works by controlling the gain and bias of a neuron to regulate its rate of fire. We discuss the biological plausibility of the IP rule and compare it to batch normalisa...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7511202/ https://www.ncbi.nlm.nih.gov/pubmed/32966302 http://dx.doi.org/10.1371/journal.pone.0238454 |
_version_ | 1783585916420358144 |
---|---|
author | Shaw, Nolan Peter Jackson, Tyler Orchard, Jeff |
author_facet | Shaw, Nolan Peter Jackson, Tyler Orchard, Jeff |
author_sort | Shaw, Nolan Peter |
collection | PubMed |
description | In this work, we present a local intrinsic rule that we developed, dubbed IP, inspired by the Infomax rule. Like Infomax, this rule works by controlling the gain and bias of a neuron to regulate its rate of fire. We discuss the biological plausibility of the IP rule and compare it to batch normalisation. We demonstrate that the IP rule improves learning in deep networks, and provides networks with considerable robustness to increases in synaptic learning rates. We also sample the error gradients during learning and show that the IP rule substantially increases the size of the gradients over the course of learning. This suggests that the IP rule solves the vanishing gradient problem. Supplementary analysis is provided to derive the equilibrium solutions that the neuronal gain and bias converge to using our IP rule. An analysis demonstrates that the IP rule results in neuronal information potential similar to that of Infomax, when tested on a fixed input distribution. We also show that batch normalisation also improves information potential, suggesting that this may be a cause for the efficacy of batch normalisation—an open problem at the time of this writing. |
format | Online Article Text |
id | pubmed-7511202 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-75112022020-10-01 Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks Shaw, Nolan Peter Jackson, Tyler Orchard, Jeff PLoS One Research Article In this work, we present a local intrinsic rule that we developed, dubbed IP, inspired by the Infomax rule. Like Infomax, this rule works by controlling the gain and bias of a neuron to regulate its rate of fire. We discuss the biological plausibility of the IP rule and compare it to batch normalisation. We demonstrate that the IP rule improves learning in deep networks, and provides networks with considerable robustness to increases in synaptic learning rates. We also sample the error gradients during learning and show that the IP rule substantially increases the size of the gradients over the course of learning. This suggests that the IP rule solves the vanishing gradient problem. Supplementary analysis is provided to derive the equilibrium solutions that the neuronal gain and bias converge to using our IP rule. An analysis demonstrates that the IP rule results in neuronal information potential similar to that of Infomax, when tested on a fixed input distribution. We also show that batch normalisation also improves information potential, suggesting that this may be a cause for the efficacy of batch normalisation—an open problem at the time of this writing. Public Library of Science 2020-09-23 /pmc/articles/PMC7511202/ /pubmed/32966302 http://dx.doi.org/10.1371/journal.pone.0238454 Text en © 2020 Shaw et al http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Shaw, Nolan Peter Jackson, Tyler Orchard, Jeff Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks |
title | Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks |
title_full | Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks |
title_fullStr | Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks |
title_full_unstemmed | Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks |
title_short | Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks |
title_sort | biological batch normalisation: how intrinsic plasticity improves learning in deep neural networks |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7511202/ https://www.ncbi.nlm.nih.gov/pubmed/32966302 http://dx.doi.org/10.1371/journal.pone.0238454 |
work_keys_str_mv | AT shawnolanpeter biologicalbatchnormalisationhowintrinsicplasticityimproveslearningindeepneuralnetworks AT jacksontyler biologicalbatchnormalisationhowintrinsicplasticityimproveslearningindeepneuralnetworks AT orchardjeff biologicalbatchnormalisationhowintrinsicplasticityimproveslearningindeepneuralnetworks |