Cargando…

Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models

Studies of the primate visual system have begun to test a wide range of complex computational object-vision models. Realistic models have many parameters, which in practice cannot be fitted using the limited amounts of brain-activity data typically available. Task performance optimization (e.g. usin...

Descripción completa

Detalles Bibliográficos
Autores principales: Khaligh-Razavi, Seyed-Mahdi, Henriksson, Linda, Kay, Kendrick, Kriegeskorte, Nikolaus
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Academic Press 2017
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5341758/
https://www.ncbi.nlm.nih.gov/pubmed/28298702
http://dx.doi.org/10.1016/j.jmp.2016.10.007
_version_ 1782513033235398656
author Khaligh-Razavi, Seyed-Mahdi
Henriksson, Linda
Kay, Kendrick
Kriegeskorte, Nikolaus
author_facet Khaligh-Razavi, Seyed-Mahdi
Henriksson, Linda
Kay, Kendrick
Kriegeskorte, Nikolaus
author_sort Khaligh-Razavi, Seyed-Mahdi
collection PubMed
description Studies of the primate visual system have begun to test a wide range of complex computational object-vision models. Realistic models have many parameters, which in practice cannot be fitted using the limited amounts of brain-activity data typically available. Task performance optimization (e.g. using backpropagation to train neural networks) provides major constraints for fitting parameters and discovering nonlinear representational features appropriate for the task (e.g. object classification). Model representations can be compared to brain representations in terms of the representational dissimilarities they predict for an image set. This method, called representational similarity analysis (RSA), enables us to test the representational feature space as is (fixed RSA) or to fit a linear transformation that mixes the nonlinear model features so as to best explain a cortical area’s representational space (mixed RSA). Like voxel/population-receptive-field modelling, mixed RSA uses a training set (different stimuli) to fit one weight per model feature and response channel (voxels here), so as to best predict the response profile across images for each response channel. We analysed response patterns elicited by natural images, which were measured with functional magnetic resonance imaging (fMRI). We found that early visual areas were best accounted for by shallow models, such as a Gabor wavelet pyramid (GWP). The GWP model performed similarly with and without mixing, suggesting that the original features already approximated the representational space, obviating the need for mixing. However, a higher ventral-stream visual representation (lateral occipital region) was best explained by the higher layers of a deep convolutional network and mixing of its feature set was essential for this model to explain the representation. We suspect that mixing was essential because the convolutional network had been trained to discriminate a set of 1000 categories, whose frequencies in the training set did not match their frequencies in natural experience or their behavioural importance. The latter factors might determine the representational prominence of semantic dimensions in higher-level ventral-stream areas. Our results demonstrate the benefits of testing both the specific representational hypothesis expressed by a model’s original feature space and the hypothesis space generated by linear transformations of that feature space.
format Online
Article
Text
id pubmed-5341758
institution National Center for Biotechnology Information
language English
publishDate 2017
publisher Academic Press
record_format MEDLINE/PubMed
spelling pubmed-53417582017-03-13 Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models Khaligh-Razavi, Seyed-Mahdi Henriksson, Linda Kay, Kendrick Kriegeskorte, Nikolaus J Math Psychol Article Studies of the primate visual system have begun to test a wide range of complex computational object-vision models. Realistic models have many parameters, which in practice cannot be fitted using the limited amounts of brain-activity data typically available. Task performance optimization (e.g. using backpropagation to train neural networks) provides major constraints for fitting parameters and discovering nonlinear representational features appropriate for the task (e.g. object classification). Model representations can be compared to brain representations in terms of the representational dissimilarities they predict for an image set. This method, called representational similarity analysis (RSA), enables us to test the representational feature space as is (fixed RSA) or to fit a linear transformation that mixes the nonlinear model features so as to best explain a cortical area’s representational space (mixed RSA). Like voxel/population-receptive-field modelling, mixed RSA uses a training set (different stimuli) to fit one weight per model feature and response channel (voxels here), so as to best predict the response profile across images for each response channel. We analysed response patterns elicited by natural images, which were measured with functional magnetic resonance imaging (fMRI). We found that early visual areas were best accounted for by shallow models, such as a Gabor wavelet pyramid (GWP). The GWP model performed similarly with and without mixing, suggesting that the original features already approximated the representational space, obviating the need for mixing. However, a higher ventral-stream visual representation (lateral occipital region) was best explained by the higher layers of a deep convolutional network and mixing of its feature set was essential for this model to explain the representation. We suspect that mixing was essential because the convolutional network had been trained to discriminate a set of 1000 categories, whose frequencies in the training set did not match their frequencies in natural experience or their behavioural importance. The latter factors might determine the representational prominence of semantic dimensions in higher-level ventral-stream areas. Our results demonstrate the benefits of testing both the specific representational hypothesis expressed by a model’s original feature space and the hypothesis space generated by linear transformations of that feature space. Academic Press 2017-02 /pmc/articles/PMC5341758/ /pubmed/28298702 http://dx.doi.org/10.1016/j.jmp.2016.10.007 Text en © 2016 The Author(s) http://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
spellingShingle Article
Khaligh-Razavi, Seyed-Mahdi
Henriksson, Linda
Kay, Kendrick
Kriegeskorte, Nikolaus
Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
title Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
title_full Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
title_fullStr Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
title_full_unstemmed Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
title_short Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
title_sort fixed versus mixed rsa: explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5341758/
https://www.ncbi.nlm.nih.gov/pubmed/28298702
http://dx.doi.org/10.1016/j.jmp.2016.10.007
work_keys_str_mv AT khalighrazaviseyedmahdi fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels
AT henrikssonlinda fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels
AT kaykendrick fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels
AT kriegeskortenikolaus fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels