Cargando…

Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space

Neural decoding can be conceptualized as the problem of mapping brain responses back to sensory stimuli via a feature space. We introduce (i) a novel experimental paradigm that uses well-controlled yet highly naturalistic stimuli with a priori known feature representations and (ii) an implementation...

Descripción completa

Detalles Bibliográficos
Autores principales: Dado, Thirza, Güçlütürk, Yağmur, Ambrogioni, Luca, Ras, Gabriëlle, Bosch, Sander, van Gerven, Marcel, Güçlü, Umut
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Nature Publishing Group UK 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8741893/
https://www.ncbi.nlm.nih.gov/pubmed/34997012
http://dx.doi.org/10.1038/s41598-021-03938-w
_version_ 1784629589057732608
author Dado, Thirza
Güçlütürk, Yağmur
Ambrogioni, Luca
Ras, Gabriëlle
Bosch, Sander
van Gerven, Marcel
Güçlü, Umut
author_facet Dado, Thirza
Güçlütürk, Yağmur
Ambrogioni, Luca
Ras, Gabriëlle
Bosch, Sander
van Gerven, Marcel
Güçlü, Umut
author_sort Dado, Thirza
collection PubMed
description Neural decoding can be conceptualized as the problem of mapping brain responses back to sensory stimuli via a feature space. We introduce (i) a novel experimental paradigm that uses well-controlled yet highly naturalistic stimuli with a priori known feature representations and (ii) an implementation thereof for HYPerrealistic reconstruction of PERception (HYPER) of faces from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring fMRI data as participants perceive face images synthesized by the generator network of a GAN. We show that the latent vectors used for generation effectively capture the same defining stimulus properties as the fMRI measurements. As such, these latents (conditioned on the GAN) are used as the in-between feature representations underlying the perceived images that can be predicted in neural decoding for (re-)generation of the originally perceived stimuli, leading to the most accurate reconstructions of perception to date.
format Online
Article
Text
id pubmed-8741893
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Nature Publishing Group UK
record_format MEDLINE/PubMed
spelling pubmed-87418932022-01-10 Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space Dado, Thirza Güçlütürk, Yağmur Ambrogioni, Luca Ras, Gabriëlle Bosch, Sander van Gerven, Marcel Güçlü, Umut Sci Rep Article Neural decoding can be conceptualized as the problem of mapping brain responses back to sensory stimuli via a feature space. We introduce (i) a novel experimental paradigm that uses well-controlled yet highly naturalistic stimuli with a priori known feature representations and (ii) an implementation thereof for HYPerrealistic reconstruction of PERception (HYPER) of faces from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring fMRI data as participants perceive face images synthesized by the generator network of a GAN. We show that the latent vectors used for generation effectively capture the same defining stimulus properties as the fMRI measurements. As such, these latents (conditioned on the GAN) are used as the in-between feature representations underlying the perceived images that can be predicted in neural decoding for (re-)generation of the originally perceived stimuli, leading to the most accurate reconstructions of perception to date. Nature Publishing Group UK 2022-01-07 /pmc/articles/PMC8741893/ /pubmed/34997012 http://dx.doi.org/10.1038/s41598-021-03938-w Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Article
Dado, Thirza
Güçlütürk, Yağmur
Ambrogioni, Luca
Ras, Gabriëlle
Bosch, Sander
van Gerven, Marcel
Güçlü, Umut
Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space
title Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space
title_full Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space
title_fullStr Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space
title_full_unstemmed Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space
title_short Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space
title_sort hyperrealistic neural decoding for reconstructing faces from fmri activations via the gan latent space
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8741893/
https://www.ncbi.nlm.nih.gov/pubmed/34997012
http://dx.doi.org/10.1038/s41598-021-03938-w
work_keys_str_mv AT dadothirza hyperrealisticneuraldecodingforreconstructingfacesfromfmriactivationsviatheganlatentspace
AT gucluturkyagmur hyperrealisticneuraldecodingforreconstructingfacesfromfmriactivationsviatheganlatentspace
AT ambrogioniluca hyperrealisticneuraldecodingforreconstructingfacesfromfmriactivationsviatheganlatentspace
AT rasgabrielle hyperrealisticneuraldecodingforreconstructingfacesfromfmriactivationsviatheganlatentspace
AT boschsander hyperrealisticneuraldecodingforreconstructingfacesfromfmriactivationsviatheganlatentspace
AT vangervenmarcel hyperrealisticneuraldecodingforreconstructingfacesfromfmriactivationsviatheganlatentspace
AT gucluumut hyperrealisticneuraldecodingforreconstructingfacesfromfmriactivationsviatheganlatentspace