Cargando…
Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches
Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn pl...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7025497/ https://www.ncbi.nlm.nih.gov/pubmed/32116634 http://dx.doi.org/10.3389/fnbot.2020.00002 |
_version_ | 1783498522232881152 |
---|---|
author | Alaçam, Özge Li, Xingshan Menzel, Wolfgang Staron, Tobias |
author_facet | Alaçam, Özge Li, Xingshan Menzel, Wolfgang Staron, Tobias |
author_sort | Alaçam, Özge |
collection | PubMed |
description | Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn plays a role in driving the focus of attention in the visual environment. However, language and vision are two different representational modalities, which accommodate different aspects and granularities of conceptualizations. To integrate them into a single, coherent system solution is still a challenge, which could profit from inspiration by human crossmodal processing. Based on fundamental psycholinguistic insights into the nature of situated language comprehension, we derive a set of performance characteristics facilitating the robustness of language understanding, such as crossmodal reference resolution, attention guidance, or predictive processing. Artificial systems for language comprehension should meet these characteristics in order to be able to perform in a natural and smooth manner. We discuss how empirical findings on the crossmodal support of language comprehension in humans can be applied in computational solutions for situated language comprehension and how they can help to mitigate the shortcomings of current approaches. |
format | Online Article Text |
id | pubmed-7025497 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-70254972020-02-28 Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches Alaçam, Özge Li, Xingshan Menzel, Wolfgang Staron, Tobias Front Neurorobot Neuroscience Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn plays a role in driving the focus of attention in the visual environment. However, language and vision are two different representational modalities, which accommodate different aspects and granularities of conceptualizations. To integrate them into a single, coherent system solution is still a challenge, which could profit from inspiration by human crossmodal processing. Based on fundamental psycholinguistic insights into the nature of situated language comprehension, we derive a set of performance characteristics facilitating the robustness of language understanding, such as crossmodal reference resolution, attention guidance, or predictive processing. Artificial systems for language comprehension should meet these characteristics in order to be able to perform in a natural and smooth manner. We discuss how empirical findings on the crossmodal support of language comprehension in humans can be applied in computational solutions for situated language comprehension and how they can help to mitigate the shortcomings of current approaches. Frontiers Media S.A. 2020-01-31 /pmc/articles/PMC7025497/ /pubmed/32116634 http://dx.doi.org/10.3389/fnbot.2020.00002 Text en Copyright © 2020 Alaçam, Li, Menzel and Staron. http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Neuroscience Alaçam, Özge Li, Xingshan Menzel, Wolfgang Staron, Tobias Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
title | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
title_full | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
title_fullStr | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
title_full_unstemmed | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
title_short | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
title_sort | crossmodal language comprehension—psycholinguistic insights and computational approaches |
topic | Neuroscience |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7025497/ https://www.ncbi.nlm.nih.gov/pubmed/32116634 http://dx.doi.org/10.3389/fnbot.2020.00002 |
work_keys_str_mv | AT alacamozge crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches AT lixingshan crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches AT menzelwolfgang crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches AT starontobias crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches |