Cargando…

Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications

With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, f...

Descripción completa

Detalles Bibliográficos
Autores principales: Kim, Chunghwan, Cha, Ho-Seung, Kim, Junghwan, Kwak, HwyKuen, Lee, WooJin, Im, Chang-Hwan
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10099104/
https://www.ncbi.nlm.nih.gov/pubmed/37050641
http://dx.doi.org/10.3390/s23073580
_version_ 1785024977096933376
author Kim, Chunghwan
Cha, Ho-Seung
Kim, Junghwan
Kwak, HwyKuen
Lee, WooJin
Im, Chang-Hwan
author_facet Kim, Chunghwan
Cha, Ho-Seung
Kim, Junghwan
Kwak, HwyKuen
Lee, WooJin
Im, Chang-Hwan
author_sort Kim, Chunghwan
collection PubMed
description With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user’s facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user’s face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users’ facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user’s facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user’s eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user’s facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.
format Online
Article
Text
id pubmed-10099104
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-100991042023-04-14 Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications Kim, Chunghwan Cha, Ho-Seung Kim, Junghwan Kwak, HwyKuen Lee, WooJin Im, Chang-Hwan Sensors (Basel) Article With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user’s facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user’s face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users’ facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user’s facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user’s eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user’s facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before. MDPI 2023-03-29 /pmc/articles/PMC10099104/ /pubmed/37050641 http://dx.doi.org/10.3390/s23073580 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Kim, Chunghwan
Cha, Ho-Seung
Kim, Junghwan
Kwak, HwyKuen
Lee, WooJin
Im, Chang-Hwan
Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications
title Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications
title_full Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications
title_fullStr Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications
title_full_unstemmed Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications
title_short Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications
title_sort facial motion capture system based on facial electromyogram and electrooculogram for immersive social virtual reality applications
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10099104/
https://www.ncbi.nlm.nih.gov/pubmed/37050641
http://dx.doi.org/10.3390/s23073580
work_keys_str_mv AT kimchunghwan facialmotioncapturesystembasedonfacialelectromyogramandelectrooculogramforimmersivesocialvirtualrealityapplications
AT chahoseung facialmotioncapturesystembasedonfacialelectromyogramandelectrooculogramforimmersivesocialvirtualrealityapplications
AT kimjunghwan facialmotioncapturesystembasedonfacialelectromyogramandelectrooculogramforimmersivesocialvirtualrealityapplications
AT kwakhwykuen facialmotioncapturesystembasedonfacialelectromyogramandelectrooculogramforimmersivesocialvirtualrealityapplications
AT leewoojin facialmotioncapturesystembasedonfacialelectromyogramandelectrooculogramforimmersivesocialvirtualrealityapplications
AT imchanghwan facialmotioncapturesystembasedonfacialelectromyogramandelectrooculogramforimmersivesocialvirtualrealityapplications