Cargando…

Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks

OBJECTIVES: Text classification is a recurrent goal in machine learning projects and a typical task in crowdsourcing platforms. Hybrid approaches, leveraging crowdsourcing and machine learning, work better than either in isolation and help to reduce crowdsourcing costs. One way to mix crowd and mach...

Descripción completa

Detalles Bibliográficos
Autores principales: Ramírez, Jorge, Baez, Marcos, Casati, Fabio, Benatallah, Boualem
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6925844/
https://www.ncbi.nlm.nih.gov/pubmed/31864400
http://dx.doi.org/10.1186/s13104-019-4858-z
_version_ 1783481988579065856
author Ramírez, Jorge
Baez, Marcos
Casati, Fabio
Benatallah, Boualem
author_facet Ramírez, Jorge
Baez, Marcos
Casati, Fabio
Benatallah, Boualem
author_sort Ramírez, Jorge
collection PubMed
description OBJECTIVES: Text classification is a recurrent goal in machine learning projects and a typical task in crowdsourcing platforms. Hybrid approaches, leveraging crowdsourcing and machine learning, work better than either in isolation and help to reduce crowdsourcing costs. One way to mix crowd and machine efforts is to have algorithms highlight passages from texts and feed these to the crowd for classification. In this paper, we present a dataset to study text highlighting generation and its impact on document classification. DATA DESCRIPTION: The dataset was created through two series of experiments where we first asked workers to (i) classify documents according to a relevance question and to highlight parts of the text that supported their decision, and on a second phase, (ii) to assess document relevance but supported by text highlighting of varying quality (six human-generated and six machine-generated highlighting conditions). The dataset features documents from two application domains: systematic literature reviews and product reviews, three document sizes, and three relevance questions of different levels of difficulty. We expect this dataset of 27,711 individual judgments from 1851 workers to benefit not only this specific problem domain, but the larger class of classification problems where crowdsourced datasets with individual judgments are scarce.
format Online
Article
Text
id pubmed-6925844
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-69258442019-12-30 Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks Ramírez, Jorge Baez, Marcos Casati, Fabio Benatallah, Boualem BMC Res Notes Data Note OBJECTIVES: Text classification is a recurrent goal in machine learning projects and a typical task in crowdsourcing platforms. Hybrid approaches, leveraging crowdsourcing and machine learning, work better than either in isolation and help to reduce crowdsourcing costs. One way to mix crowd and machine efforts is to have algorithms highlight passages from texts and feed these to the crowd for classification. In this paper, we present a dataset to study text highlighting generation and its impact on document classification. DATA DESCRIPTION: The dataset was created through two series of experiments where we first asked workers to (i) classify documents according to a relevance question and to highlight parts of the text that supported their decision, and on a second phase, (ii) to assess document relevance but supported by text highlighting of varying quality (six human-generated and six machine-generated highlighting conditions). The dataset features documents from two application domains: systematic literature reviews and product reviews, three document sizes, and three relevance questions of different levels of difficulty. We expect this dataset of 27,711 individual judgments from 1851 workers to benefit not only this specific problem domain, but the larger class of classification problems where crowdsourced datasets with individual judgments are scarce. BioMed Central 2019-12-21 /pmc/articles/PMC6925844/ /pubmed/31864400 http://dx.doi.org/10.1186/s13104-019-4858-z Text en © The Author(s) 2019 Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
spellingShingle Data Note
Ramírez, Jorge
Baez, Marcos
Casati, Fabio
Benatallah, Boualem
Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
title Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
title_full Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
title_fullStr Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
title_full_unstemmed Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
title_short Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
title_sort crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
topic Data Note
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6925844/
https://www.ncbi.nlm.nih.gov/pubmed/31864400
http://dx.doi.org/10.1186/s13104-019-4858-z
work_keys_str_mv AT ramirezjorge crowdsourceddatasettostudythegenerationandimpactoftexthighlightinginclassificationtasks
AT baezmarcos crowdsourceddatasettostudythegenerationandimpactoftexthighlightinginclassificationtasks
AT casatifabio crowdsourceddatasettostudythegenerationandimpactoftexthighlightinginclassificationtasks
AT benatallahboualem crowdsourceddatasettostudythegenerationandimpactoftexthighlightinginclassificationtasks