Cargando…
Extending Rucio with modern cloud storage support
Rucio is a software framework designed to facilitate scientific collaborations in efficiently organising, managing, and accessing extensive volumes of data through customizable policies. The framework enables data distribution across globally distributed locations and heterogeneous data centres, int...
Autores principales: | , , , , , , , |
---|---|
Lenguaje: | eng |
Publicado: |
2023
|
Materias: | |
Acceso en línea: | http://cds.cern.ch/record/2869028 |
_version_ | 1780978256083681280 |
---|---|
author | Barisits, Martin Barreiro Megino, Fernando Harald Elmsheuser, Johannes Lassnig, Mario Patrascoiu, Mihai Serfon, Cedric Vendrell Moya, Alba Wegner, Tobias |
author_facet | Barisits, Martin Barreiro Megino, Fernando Harald Elmsheuser, Johannes Lassnig, Mario Patrascoiu, Mihai Serfon, Cedric Vendrell Moya, Alba Wegner, Tobias |
author_sort | Barisits, Martin |
collection | CERN |
description | Rucio is a software framework designed to facilitate scientific collaborations in efficiently organising, managing, and accessing extensive volumes of data through customizable policies. The framework enables data distribution across globally distributed locations and heterogeneous data centres, integrating various storage and network technologies into a unified federated entity. Rucio offers advanced features like distributed data recovery and adaptive replication, and it exhibits high scalability, modularity, and extensibility. Originally developed to meet the requirements of the high-energy physics experiment ATLAS, Rucio has been continuously expanded to support LHC experiments and diverse scientific communities. Recent R\&D projects within these communities have evaluated the integration of both private and commercially-provided cloud storage systems, leading to the development of additional functionalities for seamless integration within Rucio. Furthermore, the underlying systems, FTS and GFAL/Davix, have been extended to cater to specific use cases. This contribution focuses on the technical aspects of this work, particularly the challenges encountered in building a generic interface for self-hosted cloud storage, such as MinIO or CEPH S3 Gateway, and established providers like Google Cloud Storage and Amazon Simple Storage Service. Additionally, the integration of decentralised clouds like SEAL is explored. Key aspects, including authentication and authorisation, direct and remote access, throughput and cost estimation, are highlighted, along with shared experiences in daily operations. |
id | cern-2869028 |
institution | Organización Europea para la Investigación Nuclear |
language | eng |
publishDate | 2023 |
record_format | invenio |
spelling | cern-28690282023-08-31T18:34:38Zhttp://cds.cern.ch/record/2869028engBarisits, MartinBarreiro Megino, Fernando HaraldElmsheuser, JohannesLassnig, MarioPatrascoiu, MihaiSerfon, CedricVendrell Moya, AlbaWegner, TobiasExtending Rucio with modern cloud storage supportParticle Physics - ExperimentRucio is a software framework designed to facilitate scientific collaborations in efficiently organising, managing, and accessing extensive volumes of data through customizable policies. The framework enables data distribution across globally distributed locations and heterogeneous data centres, integrating various storage and network technologies into a unified federated entity. Rucio offers advanced features like distributed data recovery and adaptive replication, and it exhibits high scalability, modularity, and extensibility. Originally developed to meet the requirements of the high-energy physics experiment ATLAS, Rucio has been continuously expanded to support LHC experiments and diverse scientific communities. Recent R\&D projects within these communities have evaluated the integration of both private and commercially-provided cloud storage systems, leading to the development of additional functionalities for seamless integration within Rucio. Furthermore, the underlying systems, FTS and GFAL/Davix, have been extended to cater to specific use cases. This contribution focuses on the technical aspects of this work, particularly the challenges encountered in building a generic interface for self-hosted cloud storage, such as MinIO or CEPH S3 Gateway, and established providers like Google Cloud Storage and Amazon Simple Storage Service. Additionally, the integration of decentralised clouds like SEAL is explored. Key aspects, including authentication and authorisation, direct and remote access, throughput and cost estimation, are highlighted, along with shared experiences in daily operations.ATL-SOFT-PROC-2023-016oai:cds.cern.ch:28690282023-08-31 |
spellingShingle | Particle Physics - Experiment Barisits, Martin Barreiro Megino, Fernando Harald Elmsheuser, Johannes Lassnig, Mario Patrascoiu, Mihai Serfon, Cedric Vendrell Moya, Alba Wegner, Tobias Extending Rucio with modern cloud storage support |
title | Extending Rucio with modern cloud storage support |
title_full | Extending Rucio with modern cloud storage support |
title_fullStr | Extending Rucio with modern cloud storage support |
title_full_unstemmed | Extending Rucio with modern cloud storage support |
title_short | Extending Rucio with modern cloud storage support |
title_sort | extending rucio with modern cloud storage support |
topic | Particle Physics - Experiment |
url | http://cds.cern.ch/record/2869028 |
work_keys_str_mv | AT barisitsmartin extendingruciowithmoderncloudstoragesupport AT barreiromeginofernandoharald extendingruciowithmoderncloudstoragesupport AT elmsheuserjohannes extendingruciowithmoderncloudstoragesupport AT lassnigmario extendingruciowithmoderncloudstoragesupport AT patrascoiumihai extendingruciowithmoderncloudstoragesupport AT serfoncedric extendingruciowithmoderncloudstoragesupport AT vendrellmoyaalba extendingruciowithmoderncloudstoragesupport AT wegnertobias extendingruciowithmoderncloudstoragesupport |