Cargando…
Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models
Studies of the primate visual system have begun to test a wide range of complex computational object-vision models. Realistic models have many parameters, which in practice cannot be fitted using the limited amounts of brain-activity data typically available. Task performance optimization (e.g. usin...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Academic Press
2017
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5341758/ https://www.ncbi.nlm.nih.gov/pubmed/28298702 http://dx.doi.org/10.1016/j.jmp.2016.10.007 |
_version_ | 1782513033235398656 |
---|---|
author | Khaligh-Razavi, Seyed-Mahdi Henriksson, Linda Kay, Kendrick Kriegeskorte, Nikolaus |
author_facet | Khaligh-Razavi, Seyed-Mahdi Henriksson, Linda Kay, Kendrick Kriegeskorte, Nikolaus |
author_sort | Khaligh-Razavi, Seyed-Mahdi |
collection | PubMed |
description | Studies of the primate visual system have begun to test a wide range of complex computational object-vision models. Realistic models have many parameters, which in practice cannot be fitted using the limited amounts of brain-activity data typically available. Task performance optimization (e.g. using backpropagation to train neural networks) provides major constraints for fitting parameters and discovering nonlinear representational features appropriate for the task (e.g. object classification). Model representations can be compared to brain representations in terms of the representational dissimilarities they predict for an image set. This method, called representational similarity analysis (RSA), enables us to test the representational feature space as is (fixed RSA) or to fit a linear transformation that mixes the nonlinear model features so as to best explain a cortical area’s representational space (mixed RSA). Like voxel/population-receptive-field modelling, mixed RSA uses a training set (different stimuli) to fit one weight per model feature and response channel (voxels here), so as to best predict the response profile across images for each response channel. We analysed response patterns elicited by natural images, which were measured with functional magnetic resonance imaging (fMRI). We found that early visual areas were best accounted for by shallow models, such as a Gabor wavelet pyramid (GWP). The GWP model performed similarly with and without mixing, suggesting that the original features already approximated the representational space, obviating the need for mixing. However, a higher ventral-stream visual representation (lateral occipital region) was best explained by the higher layers of a deep convolutional network and mixing of its feature set was essential for this model to explain the representation. We suspect that mixing was essential because the convolutional network had been trained to discriminate a set of 1000 categories, whose frequencies in the training set did not match their frequencies in natural experience or their behavioural importance. The latter factors might determine the representational prominence of semantic dimensions in higher-level ventral-stream areas. Our results demonstrate the benefits of testing both the specific representational hypothesis expressed by a model’s original feature space and the hypothesis space generated by linear transformations of that feature space. |
format | Online Article Text |
id | pubmed-5341758 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2017 |
publisher | Academic Press |
record_format | MEDLINE/PubMed |
spelling | pubmed-53417582017-03-13 Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models Khaligh-Razavi, Seyed-Mahdi Henriksson, Linda Kay, Kendrick Kriegeskorte, Nikolaus J Math Psychol Article Studies of the primate visual system have begun to test a wide range of complex computational object-vision models. Realistic models have many parameters, which in practice cannot be fitted using the limited amounts of brain-activity data typically available. Task performance optimization (e.g. using backpropagation to train neural networks) provides major constraints for fitting parameters and discovering nonlinear representational features appropriate for the task (e.g. object classification). Model representations can be compared to brain representations in terms of the representational dissimilarities they predict for an image set. This method, called representational similarity analysis (RSA), enables us to test the representational feature space as is (fixed RSA) or to fit a linear transformation that mixes the nonlinear model features so as to best explain a cortical area’s representational space (mixed RSA). Like voxel/population-receptive-field modelling, mixed RSA uses a training set (different stimuli) to fit one weight per model feature and response channel (voxels here), so as to best predict the response profile across images for each response channel. We analysed response patterns elicited by natural images, which were measured with functional magnetic resonance imaging (fMRI). We found that early visual areas were best accounted for by shallow models, such as a Gabor wavelet pyramid (GWP). The GWP model performed similarly with and without mixing, suggesting that the original features already approximated the representational space, obviating the need for mixing. However, a higher ventral-stream visual representation (lateral occipital region) was best explained by the higher layers of a deep convolutional network and mixing of its feature set was essential for this model to explain the representation. We suspect that mixing was essential because the convolutional network had been trained to discriminate a set of 1000 categories, whose frequencies in the training set did not match their frequencies in natural experience or their behavioural importance. The latter factors might determine the representational prominence of semantic dimensions in higher-level ventral-stream areas. Our results demonstrate the benefits of testing both the specific representational hypothesis expressed by a model’s original feature space and the hypothesis space generated by linear transformations of that feature space. Academic Press 2017-02 /pmc/articles/PMC5341758/ /pubmed/28298702 http://dx.doi.org/10.1016/j.jmp.2016.10.007 Text en © 2016 The Author(s) http://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). |
spellingShingle | Article Khaligh-Razavi, Seyed-Mahdi Henriksson, Linda Kay, Kendrick Kriegeskorte, Nikolaus Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models |
title | Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models |
title_full | Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models |
title_fullStr | Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models |
title_full_unstemmed | Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models |
title_short | Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models |
title_sort | fixed versus mixed rsa: explaining visual representations by fixed and mixed feature sets from shallow and deep computational models |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5341758/ https://www.ncbi.nlm.nih.gov/pubmed/28298702 http://dx.doi.org/10.1016/j.jmp.2016.10.007 |
work_keys_str_mv | AT khalighrazaviseyedmahdi fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels AT henrikssonlinda fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels AT kaykendrick fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels AT kriegeskortenikolaus fixedversusmixedrsaexplainingvisualrepresentationsbyfixedandmixedfeaturesetsfromshallowanddeepcomputationalmodels |