Cargando…

Resource control in ATLAS distributed data management: Rucio Accounting and Quotas

The ATLAS Distributed Data Management system manages more than 160PB of physics data across more than 130 sites globally. Rucio, the next generation Distributed Data Management system of the ATLAS experiment, replaced DQ2 in December 2014 and will manage the experiments data throughout Run 2 of the...

Descripción completa

Detalles Bibliográficos
Autores principales: Barisits, Martin-Stefan, Serfon, Cedric, Garonne, Vincent, Lassnig, Mario, Beermann, Thomas Alfons, Vigne, Ralph
Lenguaje:eng
Publicado: 2015
Materias:
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/664/6/062002
http://cds.cern.ch/record/2016949
_version_ 1780946712775360512
author Barisits, Martin-Stefan
Serfon, Cedric
Garonne, Vincent
Lassnig, Mario
Beermann, Thomas Alfons
Vigne, Ralph
author_facet Barisits, Martin-Stefan
Serfon, Cedric
Garonne, Vincent
Lassnig, Mario
Beermann, Thomas Alfons
Vigne, Ralph
author_sort Barisits, Martin-Stefan
collection CERN
description The ATLAS Distributed Data Management system manages more than 160PB of physics data across more than 130 sites globally. Rucio, the next generation Distributed Data Management system of the ATLAS experiment, replaced DQ2 in December 2014 and will manage the experiments data throughout Run 2 of the LHC and beyond. The previous data management system pursued a rather simplistic approach for resource management, but with the increased data volume and more dynamic handling of data workflows required by the experiment, a more elaborate approach to this issue is needed. Rucio was delivered with an initial quota system, but during the first months of operation it turned out to not fully satisfy the collaborations resource management needs. We consequently introduce a new concept of declaring quota policies (limits) for accounts in Rucio. This new quota concept is based on accounts and RSE (Rucio storage element) expressions, which allows the definition of hierarchical quotas in a dynamic way. This concept enables the operators of the data management system to implement very specific policies for users, physics groups and production systems while, at the same time, lowering the operational burden. This contribution describes the concept, architecture and workflow of the system and includes an evaluation measuring the performance of the system.
id cern-2016949
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2015
record_format invenio
spelling cern-20169492022-08-10T12:55:50Zdoi:10.1088/1742-6596/664/6/062002http://cds.cern.ch/record/2016949engBarisits, Martin-StefanSerfon, CedricGaronne, VincentLassnig, MarioBeermann, Thomas AlfonsVigne, RalphResource control in ATLAS distributed data management: Rucio Accounting and QuotasParticle Physics - ExperimentThe ATLAS Distributed Data Management system manages more than 160PB of physics data across more than 130 sites globally. Rucio, the next generation Distributed Data Management system of the ATLAS experiment, replaced DQ2 in December 2014 and will manage the experiments data throughout Run 2 of the LHC and beyond. The previous data management system pursued a rather simplistic approach for resource management, but with the increased data volume and more dynamic handling of data workflows required by the experiment, a more elaborate approach to this issue is needed. Rucio was delivered with an initial quota system, but during the first months of operation it turned out to not fully satisfy the collaborations resource management needs. We consequently introduce a new concept of declaring quota policies (limits) for accounts in Rucio. This new quota concept is based on accounts and RSE (Rucio storage element) expressions, which allows the definition of hierarchical quotas in a dynamic way. This concept enables the operators of the data management system to implement very specific policies for users, physics groups and production systems while, at the same time, lowering the operational burden. This contribution describes the concept, architecture and workflow of the system and includes an evaluation measuring the performance of the system.ATL-SOFT-PROC-2015-052oai:cds.cern.ch:20169492015-05-18
spellingShingle Particle Physics - Experiment
Barisits, Martin-Stefan
Serfon, Cedric
Garonne, Vincent
Lassnig, Mario
Beermann, Thomas Alfons
Vigne, Ralph
Resource control in ATLAS distributed data management: Rucio Accounting and Quotas
title Resource control in ATLAS distributed data management: Rucio Accounting and Quotas
title_full Resource control in ATLAS distributed data management: Rucio Accounting and Quotas
title_fullStr Resource control in ATLAS distributed data management: Rucio Accounting and Quotas
title_full_unstemmed Resource control in ATLAS distributed data management: Rucio Accounting and Quotas
title_short Resource control in ATLAS distributed data management: Rucio Accounting and Quotas
title_sort resource control in atlas distributed data management: rucio accounting and quotas
topic Particle Physics - Experiment
url https://dx.doi.org/10.1088/1742-6596/664/6/062002
http://cds.cern.ch/record/2016949
work_keys_str_mv AT barisitsmartinstefan resourcecontrolinatlasdistributeddatamanagementrucioaccountingandquotas
AT serfoncedric resourcecontrolinatlasdistributeddatamanagementrucioaccountingandquotas
AT garonnevincent resourcecontrolinatlasdistributeddatamanagementrucioaccountingandquotas
AT lassnigmario resourcecontrolinatlasdistributeddatamanagementrucioaccountingandquotas
AT beermannthomasalfons resourcecontrolinatlasdistributeddatamanagementrucioaccountingandquotas
AT vigneralph resourcecontrolinatlasdistributeddatamanagementrucioaccountingandquotas