Cargando…
Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks
Equilibrium propagation is a learning framework that marks a step forward in the search for a biologically-plausible implementation of deep learning, and could be implemented efficiently in neuromorphic hardware. Previous applications of this framework to layered networks encountered a vanishing gra...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8165608/ https://www.ncbi.nlm.nih.gov/pubmed/34079446 http://dx.doi.org/10.3389/fncom.2021.627357 |
_version_ | 1783701358946287616 |
---|---|
author | Gammell, Jimmy Buckley, Sonia Nam, Sae Woo McCaughan, Adam N. |
author_facet | Gammell, Jimmy Buckley, Sonia Nam, Sae Woo McCaughan, Adam N. |
author_sort | Gammell, Jimmy |
collection | PubMed |
description | Equilibrium propagation is a learning framework that marks a step forward in the search for a biologically-plausible implementation of deep learning, and could be implemented efficiently in neuromorphic hardware. Previous applications of this framework to layered networks encountered a vanishing gradient problem that has not yet been solved in a simple, biologically-plausible way. In this paper, we demonstrate that the vanishing gradient problem can be mitigated by replacing some of a layered network's connections with random layer-skipping connections in a manner inspired by small-world networks. This approach would be convenient to implement in neuromorphic hardware, and is biologically-plausible. |
format | Online Article Text |
id | pubmed-8165608 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-81656082021-06-01 Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks Gammell, Jimmy Buckley, Sonia Nam, Sae Woo McCaughan, Adam N. Front Comput Neurosci Neuroscience Equilibrium propagation is a learning framework that marks a step forward in the search for a biologically-plausible implementation of deep learning, and could be implemented efficiently in neuromorphic hardware. Previous applications of this framework to layered networks encountered a vanishing gradient problem that has not yet been solved in a simple, biologically-plausible way. In this paper, we demonstrate that the vanishing gradient problem can be mitigated by replacing some of a layered network's connections with random layer-skipping connections in a manner inspired by small-world networks. This approach would be convenient to implement in neuromorphic hardware, and is biologically-plausible. Frontiers Media S.A. 2021-05-17 /pmc/articles/PMC8165608/ /pubmed/34079446 http://dx.doi.org/10.3389/fncom.2021.627357 Text en Copyright © 2021 Gammell, Buckley, Nam and McCaughan. The U.S. Government is authorized to reproduce and distribute reprints for governmental purpose notwithstanding any copyright annotation thereon. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Neuroscience Gammell, Jimmy Buckley, Sonia Nam, Sae Woo McCaughan, Adam N. Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks |
title | Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks |
title_full | Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks |
title_fullStr | Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks |
title_full_unstemmed | Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks |
title_short | Layer-Skipping Connections Improve the Effectiveness of Equilibrium Propagation on Layered Networks |
title_sort | layer-skipping connections improve the effectiveness of equilibrium propagation on layered networks |
topic | Neuroscience |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8165608/ https://www.ncbi.nlm.nih.gov/pubmed/34079446 http://dx.doi.org/10.3389/fncom.2021.627357 |
work_keys_str_mv | AT gammelljimmy layerskippingconnectionsimprovetheeffectivenessofequilibriumpropagationonlayerednetworks AT buckleysonia layerskippingconnectionsimprovetheeffectivenessofequilibriumpropagationonlayerednetworks AT namsaewoo layerskippingconnectionsimprovetheeffectivenessofequilibriumpropagationonlayerednetworks AT mccaughanadamn layerskippingconnectionsimprovetheeffectivenessofequilibriumpropagationonlayerednetworks |