Cargando…

HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS

In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) chan...

Descripción completa

Detalles Bibliográficos
Autores principales: Liu, Yuhan Helena, Baratin, Aristide, Cornford, Jonathan, Mihalas, Stefan, Shea-Brown, Eric, Lajoie, Guillaume
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Cornell University 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10593070/
https://www.ncbi.nlm.nih.gov/pubmed/37873007
_version_ 1785124388635410432
author Liu, Yuhan Helena
Baratin, Aristide
Cornford, Jonathan
Mihalas, Stefan
Shea-Brown, Eric
Lajoie, Guillaume
author_facet Liu, Yuhan Helena
Baratin, Aristide
Cornford, Jonathan
Mihalas, Stefan
Shea-Brown, Eric
Lajoie, Guillaume
author_sort Liu, Yuhan Helena
collection PubMed
description In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity generally has a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights — in particular their effective rank — influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting.
format Online
Article
Text
id pubmed-10593070
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Cornell University
record_format MEDLINE/PubMed
spelling pubmed-105930702023-10-24 HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS Liu, Yuhan Helena Baratin, Aristide Cornford, Jonathan Mihalas, Stefan Shea-Brown, Eric Lajoie, Guillaume ArXiv Article In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity generally has a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights — in particular their effective rank — influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting. Cornell University 2023-10-12 /pmc/articles/PMC10593070/ /pubmed/37873007 Text en https://creativecommons.org/licenses/by-nc-sa/4.0/This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (https://creativecommons.org/licenses/by-nc-sa/4.0/) , which allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.
spellingShingle Article
Liu, Yuhan Helena
Baratin, Aristide
Cornford, Jonathan
Mihalas, Stefan
Shea-Brown, Eric
Lajoie, Guillaume
HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
title HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
title_full HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
title_fullStr HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
title_full_unstemmed HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
title_short HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
title_sort how connectivity structure shapes rich and lazy learning in neural circuits
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10593070/
https://www.ncbi.nlm.nih.gov/pubmed/37873007
work_keys_str_mv AT liuyuhanhelena howconnectivitystructureshapesrichandlazylearninginneuralcircuits
AT baratinaristide howconnectivitystructureshapesrichandlazylearninginneuralcircuits
AT cornfordjonathan howconnectivitystructureshapesrichandlazylearninginneuralcircuits
AT mihalasstefan howconnectivitystructureshapesrichandlazylearninginneuralcircuits
AT sheabrowneric howconnectivitystructureshapesrichandlazylearninginneuralcircuits
AT lajoieguillaume howconnectivitystructureshapesrichandlazylearninginneuralcircuits