Cargando…

Automatic rebalancing of data in ATLAS distributed data management

The ATLAS Distributed Data Management system stores more than 220PB of physics data across more than 130 sites globally. Rucio, the next generation data management system of the ATLAS collaboration, has now been successfully operated for two years. However, with the increasing workload and utilizati...

Descripción completa

Detalles Bibliográficos
Autores principales: Barisits, Martin-Stefan, Serfon, Cedric, Garonne, Vincent, Lassnig, Mario, Beermann, Thomas, Javurek, Tomas
Lenguaje:eng
Publicado: 2017
Materias:
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/898/6/062006
http://cds.cern.ch/record/2241285
_version_ 1780953186304000000
author Barisits, Martin-Stefan
Serfon, Cedric
Garonne, Vincent
Lassnig, Mario
Beermann, Thomas
Javurek, Tomas
author_facet Barisits, Martin-Stefan
Serfon, Cedric
Garonne, Vincent
Lassnig, Mario
Beermann, Thomas
Javurek, Tomas
author_sort Barisits, Martin-Stefan
collection CERN
description The ATLAS Distributed Data Management system stores more than 220PB of physics data across more than 130 sites globally. Rucio, the next generation data management system of the ATLAS collaboration, has now been successfully operated for two years. However, with the increasing workload and utilization, more automated and advanced methods of managing the data are needed. In this article we present an extension to the data management system, which is in charge of detecting and foreseeing storage elements reaching and surpassing their capacity limit. The system automatically and dynamically rebalances the data to other storage elements, while respecting and guaranteeing data distribution policies and ensuring the availability of the data. This concept not only lowers the operational burden, as these cumbersome procedures had previously to be done manually, but it also enables the system to use its distributed resources more efficiently, which not only affects the data management system itself, but in consequence also the workload management and production systems. This contribution describes the concept and architecture behind those components and shows the benefits made by the system.
id cern-2241285
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2017
record_format invenio
spelling cern-22412852019-10-15T15:17:44Zdoi:10.1088/1742-6596/898/6/062006http://cds.cern.ch/record/2241285engBarisits, Martin-StefanSerfon, CedricGaronne, VincentLassnig, MarioBeermann, ThomasJavurek, TomasAutomatic rebalancing of data in ATLAS distributed data managementParticle Physics - ExperimentThe ATLAS Distributed Data Management system stores more than 220PB of physics data across more than 130 sites globally. Rucio, the next generation data management system of the ATLAS collaboration, has now been successfully operated for two years. However, with the increasing workload and utilization, more automated and advanced methods of managing the data are needed. In this article we present an extension to the data management system, which is in charge of detecting and foreseeing storage elements reaching and surpassing their capacity limit. The system automatically and dynamically rebalances the data to other storage elements, while respecting and guaranteeing data distribution policies and ensuring the availability of the data. This concept not only lowers the operational burden, as these cumbersome procedures had previously to be done manually, but it also enables the system to use its distributed resources more efficiently, which not only affects the data management system itself, but in consequence also the workload management and production systems. This contribution describes the concept and architecture behind those components and shows the benefits made by the system.ATL-SOFT-PROC-2017-010oai:cds.cern.ch:22412852017-01-11
spellingShingle Particle Physics - Experiment
Barisits, Martin-Stefan
Serfon, Cedric
Garonne, Vincent
Lassnig, Mario
Beermann, Thomas
Javurek, Tomas
Automatic rebalancing of data in ATLAS distributed data management
title Automatic rebalancing of data in ATLAS distributed data management
title_full Automatic rebalancing of data in ATLAS distributed data management
title_fullStr Automatic rebalancing of data in ATLAS distributed data management
title_full_unstemmed Automatic rebalancing of data in ATLAS distributed data management
title_short Automatic rebalancing of data in ATLAS distributed data management
title_sort automatic rebalancing of data in atlas distributed data management
topic Particle Physics - Experiment
url https://dx.doi.org/10.1088/1742-6596/898/6/062006
http://cds.cern.ch/record/2241285
work_keys_str_mv AT barisitsmartinstefan automaticrebalancingofdatainatlasdistributeddatamanagement
AT serfoncedric automaticrebalancingofdatainatlasdistributeddatamanagement
AT garonnevincent automaticrebalancingofdatainatlasdistributeddatamanagement
AT lassnigmario automaticrebalancingofdatainatlasdistributeddatamanagement
AT beermannthomas automaticrebalancingofdatainatlasdistributeddatamanagement
AT javurektomas automaticrebalancingofdatainatlasdistributeddatamanagement