Cargando…

Rucio beyond ATLAS

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and...

Descripción completa

Detalles Bibliográficos
Autores principales: Lassnig, Mario, Barisits, Martin-Stefan, Laycock, Paul, Vaandering, Eric Wayne, Illingworth, Robert
Lenguaje:eng
Publicado: 2019
Materias:
Acceso en línea:http://cds.cern.ch/record/2693661
_version_ 1780964043975032832
author Lassnig, Mario
Barisits, Martin-Stefan
Laycock, Paul
Vaandering, Eric Wayne
Illingworth, Robert
author_facet Lassnig, Mario
Barisits, Martin-Stefan
Laycock, Paul
Vaandering, Eric Wayne
Illingworth, Robert
author_sort Lassnig, Mario
collection CERN
description For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organizations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analyzed by researchers, or archived for safe long-term storage. To fulfill these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities which arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments CMS and Belle II, the neutrino experiment DUNE, as well as the LIGO and SKA astronomical observatories.
id cern-2693661
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
record_format invenio
spelling cern-26936612019-10-15T18:30:53Zhttp://cds.cern.ch/record/2693661engLassnig, MarioBarisits, Martin-StefanLaycock, PaulVaandering, Eric WayneIllingworth, RobertRucio beyond ATLASParticle Physics - ExperimentFor many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organizations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analyzed by researchers, or archived for safe long-term storage. To fulfill these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities which arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments CMS and Belle II, the neutrino experiment DUNE, as well as the LIGO and SKA astronomical observatories.ATL-SOFT-SLIDE-2019-769oai:cds.cern.ch:26936612019-10-15
spellingShingle Particle Physics - Experiment
Lassnig, Mario
Barisits, Martin-Stefan
Laycock, Paul
Vaandering, Eric Wayne
Illingworth, Robert
Rucio beyond ATLAS
title Rucio beyond ATLAS
title_full Rucio beyond ATLAS
title_fullStr Rucio beyond ATLAS
title_full_unstemmed Rucio beyond ATLAS
title_short Rucio beyond ATLAS
title_sort rucio beyond atlas
topic Particle Physics - Experiment
url http://cds.cern.ch/record/2693661
work_keys_str_mv AT lassnigmario ruciobeyondatlas
AT barisitsmartinstefan ruciobeyondatlas
AT laycockpaul ruciobeyondatlas
AT vaanderingericwayne ruciobeyondatlas
AT illingworthrobert ruciobeyondatlas