Cargando…

Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in per...

Descripción completa

Detalles Bibliográficos
Autores principales: Samimi, Parnia, Ravana, Sri Devi
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Hindawi Publishing Corporation 2014
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4055211/
https://www.ncbi.nlm.nih.gov/pubmed/24977172
http://dx.doi.org/10.1155/2014/135641
_version_ 1782320619625381888
author Samimi, Parnia
Ravana, Sri Devi
author_facet Samimi, Parnia
Ravana, Sri Devi
author_sort Samimi, Parnia
collection PubMed
description Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment.
format Online
Article
Text
id pubmed-4055211
institution National Center for Biotechnology Information
language English
publishDate 2014
publisher Hindawi Publishing Corporation
record_format MEDLINE/PubMed
spelling pubmed-40552112014-06-29 Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review Samimi, Parnia Ravana, Sri Devi ScientificWorldJournal Review Article Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment. Hindawi Publishing Corporation 2014 2014-05-19 /pmc/articles/PMC4055211/ /pubmed/24977172 http://dx.doi.org/10.1155/2014/135641 Text en Copyright © 2014 P. Samimi and S. D. Ravana. https://creativecommons.org/licenses/by/3.0/ This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
spellingShingle Review Article
Samimi, Parnia
Ravana, Sri Devi
Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
title Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
title_full Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
title_fullStr Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
title_full_unstemmed Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
title_short Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
title_sort creation of reliable relevance judgments in information retrieval systems evaluation experimentation through crowdsourcing: a review
topic Review Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4055211/
https://www.ncbi.nlm.nih.gov/pubmed/24977172
http://dx.doi.org/10.1155/2014/135641
work_keys_str_mv AT samimiparnia creationofreliablerelevancejudgmentsininformationretrievalsystemsevaluationexperimentationthroughcrowdsourcingareview
AT ravanasridevi creationofreliablerelevancejudgmentsininformationretrievalsystemsevaluationexperimentationthroughcrowdsourcingareview