Cargando…
HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) chan...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Cornell University
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10593070/ https://www.ncbi.nlm.nih.gov/pubmed/37873007 |
_version_ | 1785124388635410432 |
---|---|
author | Liu, Yuhan Helena Baratin, Aristide Cornford, Jonathan Mihalas, Stefan Shea-Brown, Eric Lajoie, Guillaume |
author_facet | Liu, Yuhan Helena Baratin, Aristide Cornford, Jonathan Mihalas, Stefan Shea-Brown, Eric Lajoie, Guillaume |
author_sort | Liu, Yuhan Helena |
collection | PubMed |
description | In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity generally has a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights — in particular their effective rank — influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting. |
format | Online Article Text |
id | pubmed-10593070 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Cornell University |
record_format | MEDLINE/PubMed |
spelling | pubmed-105930702023-10-24 HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS Liu, Yuhan Helena Baratin, Aristide Cornford, Jonathan Mihalas, Stefan Shea-Brown, Eric Lajoie, Guillaume ArXiv Article In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity generally has a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights — in particular their effective rank — influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting. Cornell University 2023-10-12 /pmc/articles/PMC10593070/ /pubmed/37873007 Text en https://creativecommons.org/licenses/by-nc-sa/4.0/This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (https://creativecommons.org/licenses/by-nc-sa/4.0/) , which allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms. |
spellingShingle | Article Liu, Yuhan Helena Baratin, Aristide Cornford, Jonathan Mihalas, Stefan Shea-Brown, Eric Lajoie, Guillaume HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS |
title | HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS |
title_full | HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS |
title_fullStr | HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS |
title_full_unstemmed | HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS |
title_short | HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS |
title_sort | how connectivity structure shapes rich and lazy learning in neural circuits |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10593070/ https://www.ncbi.nlm.nih.gov/pubmed/37873007 |
work_keys_str_mv | AT liuyuhanhelena howconnectivitystructureshapesrichandlazylearninginneuralcircuits AT baratinaristide howconnectivitystructureshapesrichandlazylearninginneuralcircuits AT cornfordjonathan howconnectivitystructureshapesrichandlazylearninginneuralcircuits AT mihalasstefan howconnectivitystructureshapesrichandlazylearninginneuralcircuits AT sheabrowneric howconnectivitystructureshapesrichandlazylearninginneuralcircuits AT lajoieguillaume howconnectivitystructureshapesrichandlazylearninginneuralcircuits |