Cargando…

Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries

Cultural similarities and differences in facial expressions have been a controversial issue in the field of facial communications. A key step in addressing the debate regarding the cultural dependency of emotional expression (and perception) is to characterize the visual features of specific facial...

Descripción completa

Detalles Bibliográficos
Autores principales: Inagaki, Mikio, Ito, Tatsuro, Shinozaki, Takashi, Fujita, Ichiro
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9670153/
https://www.ncbi.nlm.nih.gov/pubmed/36405116
http://dx.doi.org/10.3389/fpsyg.2022.988302
_version_ 1784832279819845632
author Inagaki, Mikio
Ito, Tatsuro
Shinozaki, Takashi
Fujita, Ichiro
author_facet Inagaki, Mikio
Ito, Tatsuro
Shinozaki, Takashi
Fujita, Ichiro
author_sort Inagaki, Mikio
collection PubMed
description Cultural similarities and differences in facial expressions have been a controversial issue in the field of facial communications. A key step in addressing the debate regarding the cultural dependency of emotional expression (and perception) is to characterize the visual features of specific facial expressions in individual cultures. Here we developed an image analysis framework for this purpose using convolutional neural networks (CNNs) that through training learned visual features critical for classification. We analyzed photographs of facial expressions derived from two databases, each developed in a different country (Sweden and Japan), in which corresponding emotion labels were available. While the CNNs reached high rates of correct results that were far above chance after training with each database, they showed many misclassifications when they analyzed faces from the database that was not used for training. These results suggest that facial features useful for classifying facial expressions differed between the databases. The selectivity of computational units in the CNNs to action units (AUs) of the face varied across the facial expressions. Importantly, the AU selectivity often differed drastically between the CNNs trained with the different databases. Similarity and dissimilarity of these tuning profiles partly explained the pattern of misclassifications, suggesting that the AUs are important for characterizing the facial features and differ between the two countries. The AU tuning profiles, especially those reduced by principal component analysis, are compact summaries useful for comparisons across different databases, and thus might advance our understanding of universality vs. specificity of facial expressions across cultures.
format Online
Article
Text
id pubmed-9670153
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-96701532022-11-18 Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries Inagaki, Mikio Ito, Tatsuro Shinozaki, Takashi Fujita, Ichiro Front Psychol Psychology Cultural similarities and differences in facial expressions have been a controversial issue in the field of facial communications. A key step in addressing the debate regarding the cultural dependency of emotional expression (and perception) is to characterize the visual features of specific facial expressions in individual cultures. Here we developed an image analysis framework for this purpose using convolutional neural networks (CNNs) that through training learned visual features critical for classification. We analyzed photographs of facial expressions derived from two databases, each developed in a different country (Sweden and Japan), in which corresponding emotion labels were available. While the CNNs reached high rates of correct results that were far above chance after training with each database, they showed many misclassifications when they analyzed faces from the database that was not used for training. These results suggest that facial features useful for classifying facial expressions differed between the databases. The selectivity of computational units in the CNNs to action units (AUs) of the face varied across the facial expressions. Importantly, the AU selectivity often differed drastically between the CNNs trained with the different databases. Similarity and dissimilarity of these tuning profiles partly explained the pattern of misclassifications, suggesting that the AUs are important for characterizing the facial features and differ between the two countries. The AU tuning profiles, especially those reduced by principal component analysis, are compact summaries useful for comparisons across different databases, and thus might advance our understanding of universality vs. specificity of facial expressions across cultures. Frontiers Media S.A. 2022-11-03 /pmc/articles/PMC9670153/ /pubmed/36405116 http://dx.doi.org/10.3389/fpsyg.2022.988302 Text en Copyright © 2022 Inagaki, Ito, Shinozaki and Fujita. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Psychology
Inagaki, Mikio
Ito, Tatsuro
Shinozaki, Takashi
Fujita, Ichiro
Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries
title Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries
title_full Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries
title_fullStr Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries
title_full_unstemmed Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries
title_short Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries
title_sort convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries
topic Psychology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9670153/
https://www.ncbi.nlm.nih.gov/pubmed/36405116
http://dx.doi.org/10.3389/fpsyg.2022.988302
work_keys_str_mv AT inagakimikio convolutionalneuralnetworksrevealdifferencesinactionunitsoffacialexpressionsbetweenfaceimagedatabasesdevelopedindifferentcountries
AT itotatsuro convolutionalneuralnetworksrevealdifferencesinactionunitsoffacialexpressionsbetweenfaceimagedatabasesdevelopedindifferentcountries
AT shinozakitakashi convolutionalneuralnetworksrevealdifferencesinactionunitsoffacialexpressionsbetweenfaceimagedatabasesdevelopedindifferentcountries
AT fujitaichiro convolutionalneuralnetworksrevealdifferencesinactionunitsoffacialexpressionsbetweenfaceimagedatabasesdevelopedindifferentcountries