Cargando…

Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments

This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those com...

Descripción completa

Detalles Bibliográficos
Autores principales: Lin, Jimmy, Zhang, Qian
Formato: Online Artículo Texto
Lenguaje:English
Publicado: 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7148033/
http://dx.doi.org/10.1007/978-3-030-45442-5_6
_version_ 1783520516191027200
author Lin, Jimmy
Zhang, Qian
author_facet Lin, Jimmy
Zhang, Qian
author_sort Lin, Jimmy
collection PubMed
description This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those computational artifacts still functional? Perhaps not surprisingly, we are not able to replicate most of the retrieval runs encapsulated by those artifacts in a modern computational environment. We outline the various idiosyncratic reasons why, distilled into a series of “lessons learned” to help form an emerging set of best practices for the long-term sustainability of reproducibility efforts.
format Online
Article
Text
id pubmed-7148033
institution National Center for Biotechnology Information
language English
publishDate 2020
record_format MEDLINE/PubMed
spelling pubmed-71480332020-04-13 Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments Lin, Jimmy Zhang, Qian Advances in Information Retrieval Article This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those computational artifacts still functional? Perhaps not surprisingly, we are not able to replicate most of the retrieval runs encapsulated by those artifacts in a modern computational environment. We outline the various idiosyncratic reasons why, distilled into a series of “lessons learned” to help form an emerging set of best practices for the long-term sustainability of reproducibility efforts. 2020-03-24 /pmc/articles/PMC7148033/ http://dx.doi.org/10.1007/978-3-030-45442-5_6 Text en © Springer Nature Switzerland AG 2020 This article is made available via the PMC Open Access Subset for unrestricted research re-use and secondary analysis in any form or by any means with acknowledgement of the original source. These permissions are granted for the duration of the World Health Organization (WHO) declaration of COVID-19 as a global pandemic.
spellingShingle Article
Lin, Jimmy
Zhang, Qian
Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
title Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
title_full Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
title_fullStr Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
title_full_unstemmed Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
title_short Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
title_sort reproducibility is a process, not an achievement: the replicability of ir reproducibility experiments
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7148033/
http://dx.doi.org/10.1007/978-3-030-45442-5_6
work_keys_str_mv AT linjimmy reproducibilityisaprocessnotanachievementthereplicabilityofirreproducibilityexperiments
AT zhangqian reproducibilityisaprocessnotanachievementthereplicabilityofirreproducibilityexperiments