Cargando…

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose...

Descripción completa

Detalles Bibliográficos
Autores principales: Hollenstein, Nora, Renggli, Cedric, Glaus, Benjamin, Barrett, Maria, Troendle, Marius, Langer, Nicolas, Zhang, Ce
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8314009/
https://www.ncbi.nlm.nih.gov/pubmed/34326723
http://dx.doi.org/10.3389/fnhum.2021.659410
_version_ 1783729461853683712
author Hollenstein, Nora
Renggli, Cedric
Glaus, Benjamin
Barrett, Maria
Troendle, Marius
Langer, Nicolas
Zhang, Ce
author_facet Hollenstein, Nora
Renggli, Cedric
Glaus, Benjamin
Barrett, Maria
Troendle, Marius
Langer, Nicolas
Zhang, Ce
author_sort Hollenstein, Nora
collection PubMed
description Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available.
format Online
Article
Text
id pubmed-8314009
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-83140092021-07-28 Decoding EEG Brain Activity for Multi-Modal Natural Language Processing Hollenstein, Nora Renggli, Cedric Glaus, Benjamin Barrett, Maria Troendle, Marius Langer, Nicolas Zhang, Ce Front Hum Neurosci Human Neuroscience Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available. Frontiers Media S.A. 2021-07-13 /pmc/articles/PMC8314009/ /pubmed/34326723 http://dx.doi.org/10.3389/fnhum.2021.659410 Text en Copyright © 2021 Hollenstein, Renggli, Glaus, Barrett, Troendle, Langer and Zhang. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Human Neuroscience
Hollenstein, Nora
Renggli, Cedric
Glaus, Benjamin
Barrett, Maria
Troendle, Marius
Langer, Nicolas
Zhang, Ce
Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
title Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
title_full Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
title_fullStr Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
title_full_unstemmed Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
title_short Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
title_sort decoding eeg brain activity for multi-modal natural language processing
topic Human Neuroscience
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8314009/
https://www.ncbi.nlm.nih.gov/pubmed/34326723
http://dx.doi.org/10.3389/fnhum.2021.659410
work_keys_str_mv AT hollensteinnora decodingeegbrainactivityformultimodalnaturallanguageprocessing
AT rengglicedric decodingeegbrainactivityformultimodalnaturallanguageprocessing
AT glausbenjamin decodingeegbrainactivityformultimodalnaturallanguageprocessing
AT barrettmaria decodingeegbrainactivityformultimodalnaturallanguageprocessing
AT troendlemarius decodingeegbrainactivityformultimodalnaturallanguageprocessing
AT langernicolas decodingeegbrainactivityformultimodalnaturallanguageprocessing
AT zhangce decodingeegbrainactivityformultimodalnaturallanguageprocessing