Survey on evaluation methods for dialogue systems

In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very c...

Descripción completa

Detalles Bibliográficos
Autores principales: Deriu, Jan, Rodrigo, Alvaro, Otegi, Arantxa, Echegoyen, Guillermo, Rosset, Sophie, Agirre, Eneko, Cieliebak, Mark
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer Netherlands 2020
Materias:
Article
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7817575/
https://www.ncbi.nlm.nih.gov/pubmed/33505103
http://dx.doi.org/10.1007/s10462-020-09866-x
_version_ 1783638668833980416
author Deriu, Jan
Rodrigo, Alvaro
Otegi, Arantxa
Echegoyen, Guillermo
Rosset, Sophie
Agirre, Eneko
Cieliebak, Mark
author_facet Deriu, Jan
Rodrigo, Alvaro
Otegi, Arantxa
Echegoyen, Guillermo
Rosset, Sophie
Agirre, Eneko
Cieliebak, Mark
author_sort Deriu, Jan
collection PubMed
description In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost- and time-intensive. Thus, much work has been put into finding methods which allow a reduction in involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented, conversational, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then present the evaluation methods regarding that class.
format Online
Article
Text
id pubmed-7817575
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher Springer Netherlands
record_format MEDLINE/PubMed
spelling pubmed-78175752021-01-25 Survey on evaluation methods for dialogue systems Deriu, Jan Rodrigo, Alvaro Otegi, Arantxa Echegoyen, Guillermo Rosset, Sophie Agirre, Eneko Cieliebak, Mark Artif Intell Rev Article In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost- and time-intensive. Thus, much work has been put into finding methods which allow a reduction in involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented, conversational, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then present the evaluation methods regarding that class. Springer Netherlands 2020-06-25 2021 /pmc/articles/PMC7817575/ /pubmed/33505103 http://dx.doi.org/10.1007/s10462-020-09866-x Text en © The Author(s) 2020 Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
spellingShingle Article
Deriu, Jan
Rodrigo, Alvaro
Otegi, Arantxa
Echegoyen, Guillermo
Rosset, Sophie
Agirre, Eneko
Cieliebak, Mark
Survey on evaluation methods for dialogue systems
title Survey on evaluation methods for dialogue systems
title_full Survey on evaluation methods for dialogue systems
title_fullStr Survey on evaluation methods for dialogue systems
title_full_unstemmed Survey on evaluation methods for dialogue systems
title_short Survey on evaluation methods for dialogue systems
title_sort survey on evaluation methods for dialogue systems
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7817575/
https://www.ncbi.nlm.nih.gov/pubmed/33505103
http://dx.doi.org/10.1007/s10462-020-09866-x
work_keys_str_mv AT deriujan surveyonevaluationmethodsfordialoguesystems
AT rodrigoalvaro surveyonevaluationmethodsfordialoguesystems
AT otegiarantxa surveyonevaluationmethodsfordialoguesystems
AT echegoyenguillermo surveyonevaluationmethodsfordialoguesystems
AT rossetsophie surveyonevaluationmethodsfordialoguesystems
AT agirreeneko surveyonevaluationmethodsfordialoguesystems
AT cieliebakmark surveyonevaluationmethodsfordialoguesystems

Ejemplares similares