Cargando…
Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those com...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7148033/ http://dx.doi.org/10.1007/978-3-030-45442-5_6 |
_version_ | 1783520516191027200 |
---|---|
author | Lin, Jimmy Zhang, Qian |
author_facet | Lin, Jimmy Zhang, Qian |
author_sort | Lin, Jimmy |
collection | PubMed |
description | This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those computational artifacts still functional? Perhaps not surprisingly, we are not able to replicate most of the retrieval runs encapsulated by those artifacts in a modern computational environment. We outline the various idiosyncratic reasons why, distilled into a series of “lessons learned” to help form an emerging set of best practices for the long-term sustainability of reproducibility efforts. |
format | Online Article Text |
id | pubmed-7148033 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
record_format | MEDLINE/PubMed |
spelling | pubmed-71480332020-04-13 Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments Lin, Jimmy Zhang, Qian Advances in Information Retrieval Article This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those computational artifacts still functional? Perhaps not surprisingly, we are not able to replicate most of the retrieval runs encapsulated by those artifacts in a modern computational environment. We outline the various idiosyncratic reasons why, distilled into a series of “lessons learned” to help form an emerging set of best practices for the long-term sustainability of reproducibility efforts. 2020-03-24 /pmc/articles/PMC7148033/ http://dx.doi.org/10.1007/978-3-030-45442-5_6 Text en © Springer Nature Switzerland AG 2020 This article is made available via the PMC Open Access Subset for unrestricted research re-use and secondary analysis in any form or by any means with acknowledgement of the original source. These permissions are granted for the duration of the World Health Organization (WHO) declaration of COVID-19 as a global pandemic. |
spellingShingle | Article Lin, Jimmy Zhang, Qian Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments |
title | Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments |
title_full | Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments |
title_fullStr | Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments |
title_full_unstemmed | Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments |
title_short | Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments |
title_sort | reproducibility is a process, not an achievement: the replicability of ir reproducibility experiments |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7148033/ http://dx.doi.org/10.1007/978-3-030-45442-5_6 |
work_keys_str_mv | AT linjimmy reproducibilityisaprocessnotanachievementthereplicabilityofirreproducibilityexperiments AT zhangqian reproducibilityisaprocessnotanachievementthereplicabilityofirreproducibilityexperiments |