Cargando…
Reproducibility is a Process, Not an Achievement: The Replicability of IR Reproducibility Experiments
This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those com...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7148033/ http://dx.doi.org/10.1007/978-3-030-45442-5_6 |
Sumario: | This paper espouses a view of reproducibility in the computational sciences as a process and not just a point-in-time “achievement”. As a concrete case study, we revisit the Open-Source IR Reproducibility Challenge from 2015 and attempt to replicate those experiments: four years later, are those computational artifacts still functional? Perhaps not surprisingly, we are not able to replicate most of the retrieval runs encapsulated by those artifacts in a modern computational environment. We outline the various idiosyncratic reasons why, distilled into a series of “lessons learned” to help form an emerging set of best practices for the long-term sustainability of reproducibility efforts. |
---|