Cargando…
Automatic rebalancing of data in ATLAS distributed data management
The ATLAS Distributed Data Management system stores more than 220PB of physics data across more than 130 sites globally. Rucio, the next generation data management system of the ATLAS collaboration has now been successfully operated for over a year. However, with the forthcoming start of run-2 and i...
Autores principales: | , , , , |
---|---|
Lenguaje: | eng |
Publicado: |
2016
|
Materias: | |
Acceso en línea: | http://cds.cern.ch/record/2217112 |
_version_ | 1780952079672541184 |
---|---|
author | Barisits, Martin-Stefan Serfon, Cedric Garonne, Vincent Lassnig, Mario Beermann, Thomas |
author_facet | Barisits, Martin-Stefan Serfon, Cedric Garonne, Vincent Lassnig, Mario Beermann, Thomas |
author_sort | Barisits, Martin-Stefan |
collection | CERN |
description | The ATLAS Distributed Data Management system stores more than 220PB of physics data across more than 130 sites globally. Rucio, the next generation data management system of the ATLAS collaboration has now been successfully operated for over a year. However, with the forthcoming start of run-2 and its expected workload and utilization, more automated and advanced methods of managing the data are needed. In this article we present an extension to the data management system, which is in charge of detecting and foreseeing data imbalances as well as storage elements reaching and surpassing their capacity limit. The system automatically and dynamically rebalances the data to other storage elements, while respecting and guaranteeing data distribution policies and ensuring the availability of the data. This concept not only lowers the operational burden, as these cumbersome procedures had previously to be done manually, but it also enables the system to use its distributed resources more efficiently, which not only affects the data management system itself, but in consequence also the workload management and production systems. This contribution describes the concept and architecture behind those components and shows the benefits made by the system. |
id | cern-2217112 |
institution | Organización Europea para la Investigación Nuclear |
language | eng |
publishDate | 2016 |
record_format | invenio |
spelling | cern-22171122019-09-30T06:29:59Zhttp://cds.cern.ch/record/2217112engBarisits, Martin-StefanSerfon, CedricGaronne, VincentLassnig, MarioBeermann, ThomasAutomatic rebalancing of data in ATLAS distributed data managementParticle Physics - ExperimentThe ATLAS Distributed Data Management system stores more than 220PB of physics data across more than 130 sites globally. Rucio, the next generation data management system of the ATLAS collaboration has now been successfully operated for over a year. However, with the forthcoming start of run-2 and its expected workload and utilization, more automated and advanced methods of managing the data are needed. In this article we present an extension to the data management system, which is in charge of detecting and foreseeing data imbalances as well as storage elements reaching and surpassing their capacity limit. The system automatically and dynamically rebalances the data to other storage elements, while respecting and guaranteeing data distribution policies and ensuring the availability of the data. This concept not only lowers the operational burden, as these cumbersome procedures had previously to be done manually, but it also enables the system to use its distributed resources more efficiently, which not only affects the data management system itself, but in consequence also the workload management and production systems. This contribution describes the concept and architecture behind those components and shows the benefits made by the system.ATL-SOFT-SLIDE-2016-664oai:cds.cern.ch:22171122016-09-20 |
spellingShingle | Particle Physics - Experiment Barisits, Martin-Stefan Serfon, Cedric Garonne, Vincent Lassnig, Mario Beermann, Thomas Automatic rebalancing of data in ATLAS distributed data management |
title | Automatic rebalancing of data in ATLAS distributed data management |
title_full | Automatic rebalancing of data in ATLAS distributed data management |
title_fullStr | Automatic rebalancing of data in ATLAS distributed data management |
title_full_unstemmed | Automatic rebalancing of data in ATLAS distributed data management |
title_short | Automatic rebalancing of data in ATLAS distributed data management |
title_sort | automatic rebalancing of data in atlas distributed data management |
topic | Particle Physics - Experiment |
url | http://cds.cern.ch/record/2217112 |
work_keys_str_mv | AT barisitsmartinstefan automaticrebalancingofdatainatlasdistributeddatamanagement AT serfoncedric automaticrebalancingofdatainatlasdistributeddatamanagement AT garonnevincent automaticrebalancingofdatainatlasdistributeddatamanagement AT lassnigmario automaticrebalancingofdatainatlasdistributeddatamanagement AT beermannthomas automaticrebalancingofdatainatlasdistributeddatamanagement |