Cargando…

Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities

Silent speech decoding is a novel application of the Brain–Computer Interface (BCI) based on articulatory neuromuscular activities, reducing difficulties in data acquirement and processing. In this paper, spatial features and decoders that can be used to recognize the neuromuscular signals are inves...

Descripción completa

Detalles Bibliográficos
Autores principales: Wang, You, Zhang, Ming, Wu, RuMeng, Gao, Han, Yang, Meng, Luo, Zhiyuan, Li, Guang
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7407985/
https://www.ncbi.nlm.nih.gov/pubmed/32664599
http://dx.doi.org/10.3390/brainsci10070442
_version_ 1783567732434796544
author Wang, You
Zhang, Ming
Wu, RuMeng
Gao, Han
Yang, Meng
Luo, Zhiyuan
Li, Guang
author_facet Wang, You
Zhang, Ming
Wu, RuMeng
Gao, Han
Yang, Meng
Luo, Zhiyuan
Li, Guang
author_sort Wang, You
collection PubMed
description Silent speech decoding is a novel application of the Brain–Computer Interface (BCI) based on articulatory neuromuscular activities, reducing difficulties in data acquirement and processing. In this paper, spatial features and decoders that can be used to recognize the neuromuscular signals are investigated. Surface electromyography (sEMG) data are recorded from human subjects in mimed speech situations. Specifically, we propose to utilize transfer learning and deep learning methods by transforming the sEMG data into spectrograms that contain abundant information in time and frequency domains and are regarded as channel-interactive. For transfer learning, a pre-trained model of Xception on the large image dataset is used for feature generation. Three deep learning methods, Multi-Layer Perception, Convolutional Neural Network and bidirectional Long Short-Term Memory, are then trained using the extracted features and evaluated for recognizing the articulatory muscles’ movements in our word set. The proposed decoders successfully recognized the silent speech and bidirectional Long Short-Term Memory achieved the best accuracy of 90%, outperforming the other two algorithms. Experimental results demonstrate the validity of spectrogram features and deep learning algorithms.
format Online
Article
Text
id pubmed-7407985
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-74079852020-08-12 Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities Wang, You Zhang, Ming Wu, RuMeng Gao, Han Yang, Meng Luo, Zhiyuan Li, Guang Brain Sci Article Silent speech decoding is a novel application of the Brain–Computer Interface (BCI) based on articulatory neuromuscular activities, reducing difficulties in data acquirement and processing. In this paper, spatial features and decoders that can be used to recognize the neuromuscular signals are investigated. Surface electromyography (sEMG) data are recorded from human subjects in mimed speech situations. Specifically, we propose to utilize transfer learning and deep learning methods by transforming the sEMG data into spectrograms that contain abundant information in time and frequency domains and are regarded as channel-interactive. For transfer learning, a pre-trained model of Xception on the large image dataset is used for feature generation. Three deep learning methods, Multi-Layer Perception, Convolutional Neural Network and bidirectional Long Short-Term Memory, are then trained using the extracted features and evaluated for recognizing the articulatory muscles’ movements in our word set. The proposed decoders successfully recognized the silent speech and bidirectional Long Short-Term Memory achieved the best accuracy of 90%, outperforming the other two algorithms. Experimental results demonstrate the validity of spectrogram features and deep learning algorithms. MDPI 2020-07-11 /pmc/articles/PMC7407985/ /pubmed/32664599 http://dx.doi.org/10.3390/brainsci10070442 Text en © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Wang, You
Zhang, Ming
Wu, RuMeng
Gao, Han
Yang, Meng
Luo, Zhiyuan
Li, Guang
Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities
title Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities
title_full Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities
title_fullStr Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities
title_full_unstemmed Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities
title_short Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities
title_sort silent speech decoding using spectrogram features based on neuromuscular activities
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7407985/
https://www.ncbi.nlm.nih.gov/pubmed/32664599
http://dx.doi.org/10.3390/brainsci10070442
work_keys_str_mv AT wangyou silentspeechdecodingusingspectrogramfeaturesbasedonneuromuscularactivities
AT zhangming silentspeechdecodingusingspectrogramfeaturesbasedonneuromuscularactivities
AT wurumeng silentspeechdecodingusingspectrogramfeaturesbasedonneuromuscularactivities
AT gaohan silentspeechdecodingusingspectrogramfeaturesbasedonneuromuscularactivities
AT yangmeng silentspeechdecodingusingspectrogramfeaturesbasedonneuromuscularactivities
AT luozhiyuan silentspeechdecodingusingspectrogramfeaturesbasedonneuromuscularactivities
AT liguang silentspeechdecodingusingspectrogramfeaturesbasedonneuromuscularactivities