Cargando…

A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making

Deep learning models are powerful tools for representing the complex learning processes and decision-making strategies used by humans. Such neural network models make fewer assumptions about the underlying mechanisms thus providing experimental flexibility in terms of applicability. However, this co...

Descripción completa

Detalles Bibliográficos
Autores principales: Zhang, Lili, Vashisht, Himanshu, Totev, Andrey, Trinh, Nam, Ward, Tomas
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9453750/
https://www.ncbi.nlm.nih.gov/pubmed/36092038
http://dx.doi.org/10.3389/fpsyg.2022.943198
_version_ 1784785207945068544
author Zhang, Lili
Vashisht, Himanshu
Totev, Andrey
Trinh, Nam
Ward, Tomas
author_facet Zhang, Lili
Vashisht, Himanshu
Totev, Andrey
Trinh, Nam
Ward, Tomas
author_sort Zhang, Lili
collection PubMed
description Deep learning models are powerful tools for representing the complex learning processes and decision-making strategies used by humans. Such neural network models make fewer assumptions about the underlying mechanisms thus providing experimental flexibility in terms of applicability. However, this comes at the cost of involving a larger number of parameters requiring significantly more data for effective learning. This presents practical challenges given that most cognitive experiments involve relatively small numbers of subjects. Laboratory collaborations are a natural way to increase overall dataset size. However, data sharing barriers between laboratories as necessitated by data protection regulations encourage the search for alternative methods to enable collaborative data science. Distributed learning, especially federated learning (FL), which supports the preservation of data privacy, is a promising method for addressing this issue. To verify the reliability and feasibility of applying FL to train neural networks models used in the characterization of decision making, we conducted experiments on a real-world, many-labs data pool including experiment data-sets from ten independent studies. The performance of single models trained on single laboratory data-sets was poor. This unsurprising finding supports the need for laboratory collaboration to train more reliable models. To that end we evaluated four collaborative approaches. The first approach represents conventional centralized learning (CL-based) and is the optimal approach but requires complete sharing of data which we wish to avoid. The results however establish a benchmark for the other three approaches, federated learning (FL-based), incremental learning (IL-based), and cyclic incremental learning (CIL-based). We evaluate these approaches in terms of prediction accuracy and capacity to characterize human decision-making strategies. The FL-based model achieves performance most comparable to that of the CL-based model. This indicates that FL has value in scaling data science methods to data collected in computational modeling contexts when data sharing is not convenient, practical or permissible.
format Online
Article
Text
id pubmed-9453750
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-94537502022-09-09 A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making Zhang, Lili Vashisht, Himanshu Totev, Andrey Trinh, Nam Ward, Tomas Front Psychol Psychology Deep learning models are powerful tools for representing the complex learning processes and decision-making strategies used by humans. Such neural network models make fewer assumptions about the underlying mechanisms thus providing experimental flexibility in terms of applicability. However, this comes at the cost of involving a larger number of parameters requiring significantly more data for effective learning. This presents practical challenges given that most cognitive experiments involve relatively small numbers of subjects. Laboratory collaborations are a natural way to increase overall dataset size. However, data sharing barriers between laboratories as necessitated by data protection regulations encourage the search for alternative methods to enable collaborative data science. Distributed learning, especially federated learning (FL), which supports the preservation of data privacy, is a promising method for addressing this issue. To verify the reliability and feasibility of applying FL to train neural networks models used in the characterization of decision making, we conducted experiments on a real-world, many-labs data pool including experiment data-sets from ten independent studies. The performance of single models trained on single laboratory data-sets was poor. This unsurprising finding supports the need for laboratory collaboration to train more reliable models. To that end we evaluated four collaborative approaches. The first approach represents conventional centralized learning (CL-based) and is the optimal approach but requires complete sharing of data which we wish to avoid. The results however establish a benchmark for the other three approaches, federated learning (FL-based), incremental learning (IL-based), and cyclic incremental learning (CIL-based). We evaluate these approaches in terms of prediction accuracy and capacity to characterize human decision-making strategies. The FL-based model achieves performance most comparable to that of the CL-based model. This indicates that FL has value in scaling data science methods to data collected in computational modeling contexts when data sharing is not convenient, practical or permissible. Frontiers Media S.A. 2022-08-25 /pmc/articles/PMC9453750/ /pubmed/36092038 http://dx.doi.org/10.3389/fpsyg.2022.943198 Text en Copyright © 2022 Zhang, Vashisht, Totev, Trinh and Ward. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Psychology
Zhang, Lili
Vashisht, Himanshu
Totev, Andrey
Trinh, Nam
Ward, Tomas
A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
title A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
title_full A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
title_fullStr A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
title_full_unstemmed A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
title_short A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
title_sort comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making
topic Psychology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9453750/
https://www.ncbi.nlm.nih.gov/pubmed/36092038
http://dx.doi.org/10.3389/fpsyg.2022.943198
work_keys_str_mv AT zhanglili acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT vashishthimanshu acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT totevandrey acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT trinhnam acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT wardtomas acomparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT zhanglili comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT vashishthimanshu comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT totevandrey comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT trinhnam comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking
AT wardtomas comparisonofdistributedmachinelearningmethodsforthesupportofmanylabscollaborationsincomputationalmodelingofdecisionmaking