Cargando…
Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8314009/ https://www.ncbi.nlm.nih.gov/pubmed/34326723 http://dx.doi.org/10.3389/fnhum.2021.659410 |
_version_ | 1783729461853683712 |
---|---|
author | Hollenstein, Nora Renggli, Cedric Glaus, Benjamin Barrett, Maria Troendle, Marius Langer, Nicolas Zhang, Ce |
author_facet | Hollenstein, Nora Renggli, Cedric Glaus, Benjamin Barrett, Maria Troendle, Marius Langer, Nicolas Zhang, Ce |
author_sort | Hollenstein, Nora |
collection | PubMed |
description | Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available. |
format | Online Article Text |
id | pubmed-8314009 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-83140092021-07-28 Decoding EEG Brain Activity for Multi-Modal Natural Language Processing Hollenstein, Nora Renggli, Cedric Glaus, Benjamin Barrett, Maria Troendle, Marius Langer, Nicolas Zhang, Ce Front Hum Neurosci Human Neuroscience Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available. Frontiers Media S.A. 2021-07-13 /pmc/articles/PMC8314009/ /pubmed/34326723 http://dx.doi.org/10.3389/fnhum.2021.659410 Text en Copyright © 2021 Hollenstein, Renggli, Glaus, Barrett, Troendle, Langer and Zhang. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Human Neuroscience Hollenstein, Nora Renggli, Cedric Glaus, Benjamin Barrett, Maria Troendle, Marius Langer, Nicolas Zhang, Ce Decoding EEG Brain Activity for Multi-Modal Natural Language Processing |
title | Decoding EEG Brain Activity for Multi-Modal Natural Language Processing |
title_full | Decoding EEG Brain Activity for Multi-Modal Natural Language Processing |
title_fullStr | Decoding EEG Brain Activity for Multi-Modal Natural Language Processing |
title_full_unstemmed | Decoding EEG Brain Activity for Multi-Modal Natural Language Processing |
title_short | Decoding EEG Brain Activity for Multi-Modal Natural Language Processing |
title_sort | decoding eeg brain activity for multi-modal natural language processing |
topic | Human Neuroscience |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8314009/ https://www.ncbi.nlm.nih.gov/pubmed/34326723 http://dx.doi.org/10.3389/fnhum.2021.659410 |
work_keys_str_mv | AT hollensteinnora decodingeegbrainactivityformultimodalnaturallanguageprocessing AT rengglicedric decodingeegbrainactivityformultimodalnaturallanguageprocessing AT glausbenjamin decodingeegbrainactivityformultimodalnaturallanguageprocessing AT barrettmaria decodingeegbrainactivityformultimodalnaturallanguageprocessing AT troendlemarius decodingeegbrainactivityformultimodalnaturallanguageprocessing AT langernicolas decodingeegbrainactivityformultimodalnaturallanguageprocessing AT zhangce decodingeegbrainactivityformultimodalnaturallanguageprocessing |