Cargando…

Jointly Linking Visual and Textual Entity Mentions with Background Knowledge

“A picture is worth a thousand words”, the adage reads. However, pictures cannot replace words in terms of their ability to efficiently convey clear (mostly) unambiguous and concise knowledge. Images and text, indeed, reveal different and complementary information that, if combined, result in more i...

Descripción completa

Detalles Bibliográficos
Autores principales: Dost, Shahi, Serafini, Luciano, Rospocher, Marco, Ballan, Lamberto, Sperduti, Alessandro
Formato: Online Artículo Texto
Lenguaje:English
Publicado: 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7298199/
http://dx.doi.org/10.1007/978-3-030-51310-8_24
_version_ 1783547168321175552
author Dost, Shahi
Serafini, Luciano
Rospocher, Marco
Ballan, Lamberto
Sperduti, Alessandro
author_facet Dost, Shahi
Serafini, Luciano
Rospocher, Marco
Ballan, Lamberto
Sperduti, Alessandro
author_sort Dost, Shahi
collection PubMed
description “A picture is worth a thousand words”, the adage reads. However, pictures cannot replace words in terms of their ability to efficiently convey clear (mostly) unambiguous and concise knowledge. Images and text, indeed, reveal different and complementary information that, if combined, result in more information than the sum of that contained in the single media. The combination of visual and textual information can be obtained through linking the entities mentioned in the text with those shown in the pictures. To further integrate this with agent background knowledge, an additional step is necessary. That is, either finding the entities in the agent knowledge base that correspond to those mentioned in the text or shown in the picture or, extending the knowledge base with the newly discovered entities. We call this complex task Visual-Textual-Knowledge Entity Linking (VTKEL). In this paper, after providing a precise definition of the VTKEL task, we present a dataset composed of about 30K commented pictures, annotated with visual and textual entities, and linked to the YAGO ontology. Successively, we develop a purely unsupervised algorithm for the solution of the VTKEL tasks. The evaluation on the VTKEL dataset shows promising results.
format Online
Article
Text
id pubmed-7298199
institution National Center for Biotechnology Information
language English
publishDate 2020
record_format MEDLINE/PubMed
spelling pubmed-72981992020-06-17 Jointly Linking Visual and Textual Entity Mentions with Background Knowledge Dost, Shahi Serafini, Luciano Rospocher, Marco Ballan, Lamberto Sperduti, Alessandro Natural Language Processing and Information Systems Article “A picture is worth a thousand words”, the adage reads. However, pictures cannot replace words in terms of their ability to efficiently convey clear (mostly) unambiguous and concise knowledge. Images and text, indeed, reveal different and complementary information that, if combined, result in more information than the sum of that contained in the single media. The combination of visual and textual information can be obtained through linking the entities mentioned in the text with those shown in the pictures. To further integrate this with agent background knowledge, an additional step is necessary. That is, either finding the entities in the agent knowledge base that correspond to those mentioned in the text or shown in the picture or, extending the knowledge base with the newly discovered entities. We call this complex task Visual-Textual-Knowledge Entity Linking (VTKEL). In this paper, after providing a precise definition of the VTKEL task, we present a dataset composed of about 30K commented pictures, annotated with visual and textual entities, and linked to the YAGO ontology. Successively, we develop a purely unsupervised algorithm for the solution of the VTKEL tasks. The evaluation on the VTKEL dataset shows promising results. 2020-05-26 /pmc/articles/PMC7298199/ http://dx.doi.org/10.1007/978-3-030-51310-8_24 Text en © Springer Nature Switzerland AG 2020 This article is made available via the PMC Open Access Subset for unrestricted research re-use and secondary analysis in any form or by any means with acknowledgement of the original source. These permissions are granted for the duration of the World Health Organization (WHO) declaration of COVID-19 as a global pandemic.
spellingShingle Article
Dost, Shahi
Serafini, Luciano
Rospocher, Marco
Ballan, Lamberto
Sperduti, Alessandro
Jointly Linking Visual and Textual Entity Mentions with Background Knowledge
title Jointly Linking Visual and Textual Entity Mentions with Background Knowledge
title_full Jointly Linking Visual and Textual Entity Mentions with Background Knowledge
title_fullStr Jointly Linking Visual and Textual Entity Mentions with Background Knowledge
title_full_unstemmed Jointly Linking Visual and Textual Entity Mentions with Background Knowledge
title_short Jointly Linking Visual and Textual Entity Mentions with Background Knowledge
title_sort jointly linking visual and textual entity mentions with background knowledge
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7298199/
http://dx.doi.org/10.1007/978-3-030-51310-8_24
work_keys_str_mv AT dostshahi jointlylinkingvisualandtextualentitymentionswithbackgroundknowledge
AT serafiniluciano jointlylinkingvisualandtextualentitymentionswithbackgroundknowledge
AT rospochermarco jointlylinkingvisualandtextualentitymentionswithbackgroundknowledge
AT ballanlamberto jointlylinkingvisualandtextualentitymentionswithbackgroundknowledge
AT sperdutialessandro jointlylinkingvisualandtextualentitymentionswithbackgroundknowledge