Cargando…
Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition
The emotional changes in facial micro-expressions are combinations of action units. The researchers have revealed that action units can be used as additional auxiliary data to improve facial micro-expression recognition. Most of the researchers attempt to fuse image features and action unit informat...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10528512/ https://www.ncbi.nlm.nih.gov/pubmed/37761545 http://dx.doi.org/10.3390/e25091246 |
_version_ | 1785111271455064064 |
---|---|
author | Yang, Hongling Xie, Lun Pan, Hang Li, Chiqin Wang, Zhiliang Zhong, Jialiang |
author_facet | Yang, Hongling Xie, Lun Pan, Hang Li, Chiqin Wang, Zhiliang Zhong, Jialiang |
author_sort | Yang, Hongling |
collection | PubMed |
description | The emotional changes in facial micro-expressions are combinations of action units. The researchers have revealed that action units can be used as additional auxiliary data to improve facial micro-expression recognition. Most of the researchers attempt to fuse image features and action unit information. However, these works ignore the impact of action units on the facial image feature extraction process. Therefore, this paper proposes a local detail feature enhancement model based on a multimodal dynamic attention fusion network (MADFN) method for micro-expression recognition. This method uses a masked autoencoder based on learnable class tokens to remove local areas with low emotional expression ability in micro-expression images. Then, we utilize the action unit dynamic fusion module to fuse action unit representation to improve the potential representation ability of image features. The state-of-the-art performance of our proposed model is evaluated and verified on SMIC, CASME II, SAMM, and their combined 3DB-Combined datasets. The experimental results demonstrated that the proposed model achieved competitive performance with accuracy rates of 81.71%, 82.11%, and 77.21% on SMIC, CASME II, and SAMM datasets, respectively, that show the MADFN model can help to improve the discrimination of facial image emotional features. |
format | Online Article Text |
id | pubmed-10528512 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-105285122023-09-28 Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition Yang, Hongling Xie, Lun Pan, Hang Li, Chiqin Wang, Zhiliang Zhong, Jialiang Entropy (Basel) Article The emotional changes in facial micro-expressions are combinations of action units. The researchers have revealed that action units can be used as additional auxiliary data to improve facial micro-expression recognition. Most of the researchers attempt to fuse image features and action unit information. However, these works ignore the impact of action units on the facial image feature extraction process. Therefore, this paper proposes a local detail feature enhancement model based on a multimodal dynamic attention fusion network (MADFN) method for micro-expression recognition. This method uses a masked autoencoder based on learnable class tokens to remove local areas with low emotional expression ability in micro-expression images. Then, we utilize the action unit dynamic fusion module to fuse action unit representation to improve the potential representation ability of image features. The state-of-the-art performance of our proposed model is evaluated and verified on SMIC, CASME II, SAMM, and their combined 3DB-Combined datasets. The experimental results demonstrated that the proposed model achieved competitive performance with accuracy rates of 81.71%, 82.11%, and 77.21% on SMIC, CASME II, and SAMM datasets, respectively, that show the MADFN model can help to improve the discrimination of facial image emotional features. MDPI 2023-08-22 /pmc/articles/PMC10528512/ /pubmed/37761545 http://dx.doi.org/10.3390/e25091246 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Yang, Hongling Xie, Lun Pan, Hang Li, Chiqin Wang, Zhiliang Zhong, Jialiang Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition |
title | Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition |
title_full | Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition |
title_fullStr | Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition |
title_full_unstemmed | Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition |
title_short | Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition |
title_sort | multimodal attention dynamic fusion network for facial micro-expression recognition |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10528512/ https://www.ncbi.nlm.nih.gov/pubmed/37761545 http://dx.doi.org/10.3390/e25091246 |
work_keys_str_mv | AT yanghongling multimodalattentiondynamicfusionnetworkforfacialmicroexpressionrecognition AT xielun multimodalattentiondynamicfusionnetworkforfacialmicroexpressionrecognition AT panhang multimodalattentiondynamicfusionnetworkforfacialmicroexpressionrecognition AT lichiqin multimodalattentiondynamicfusionnetworkforfacialmicroexpressionrecognition AT wangzhiliang multimodalattentiondynamicfusionnetworkforfacialmicroexpressionrecognition AT zhongjialiang multimodalattentiondynamicfusionnetworkforfacialmicroexpressionrecognition |