Cargando…

Extending Rucio with modern cloud storage support

Rucio is a software framework designed to facilitate scientific collaborations in efficiently organising, managing, and accessing extensive volumes of data through customizable policies. The framework enables data distribution across globally distributed locations and heterogeneous data centres, int...

Descripción completa

Detalles Bibliográficos
Autores principales: Barisits, Martin, Barreiro Megino, Fernando Harald, Elmsheuser, Johannes, Lassnig, Mario, Patrascoiu, Mihai, Serfon, Cedric, Vendrell Moya, Alba, Wegner, Tobias
Lenguaje:eng
Publicado: 2023
Materias:
Acceso en línea:http://cds.cern.ch/record/2869028
_version_ 1780978256083681280
author Barisits, Martin
Barreiro Megino, Fernando Harald
Elmsheuser, Johannes
Lassnig, Mario
Patrascoiu, Mihai
Serfon, Cedric
Vendrell Moya, Alba
Wegner, Tobias
author_facet Barisits, Martin
Barreiro Megino, Fernando Harald
Elmsheuser, Johannes
Lassnig, Mario
Patrascoiu, Mihai
Serfon, Cedric
Vendrell Moya, Alba
Wegner, Tobias
author_sort Barisits, Martin
collection CERN
description Rucio is a software framework designed to facilitate scientific collaborations in efficiently organising, managing, and accessing extensive volumes of data through customizable policies. The framework enables data distribution across globally distributed locations and heterogeneous data centres, integrating various storage and network technologies into a unified federated entity. Rucio offers advanced features like distributed data recovery and adaptive replication, and it exhibits high scalability, modularity, and extensibility. Originally developed to meet the requirements of the high-energy physics experiment ATLAS, Rucio has been continuously expanded to support LHC experiments and diverse scientific communities. Recent R\&D projects within these communities have evaluated the integration of both private and commercially-provided cloud storage systems, leading to the development of additional functionalities for seamless integration within Rucio. Furthermore, the underlying systems, FTS and GFAL/Davix, have been extended to cater to specific use cases. This contribution focuses on the technical aspects of this work, particularly the challenges encountered in building a generic interface for self-hosted cloud storage, such as MinIO or CEPH S3 Gateway, and established providers like Google Cloud Storage and Amazon Simple Storage Service. Additionally, the integration of decentralised clouds like SEAL is explored. Key aspects, including authentication and authorisation, direct and remote access, throughput and cost estimation, are highlighted, along with shared experiences in daily operations.
id cern-2869028
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2023
record_format invenio
spelling cern-28690282023-08-31T18:34:38Zhttp://cds.cern.ch/record/2869028engBarisits, MartinBarreiro Megino, Fernando HaraldElmsheuser, JohannesLassnig, MarioPatrascoiu, MihaiSerfon, CedricVendrell Moya, AlbaWegner, TobiasExtending Rucio with modern cloud storage supportParticle Physics - ExperimentRucio is a software framework designed to facilitate scientific collaborations in efficiently organising, managing, and accessing extensive volumes of data through customizable policies. The framework enables data distribution across globally distributed locations and heterogeneous data centres, integrating various storage and network technologies into a unified federated entity. Rucio offers advanced features like distributed data recovery and adaptive replication, and it exhibits high scalability, modularity, and extensibility. Originally developed to meet the requirements of the high-energy physics experiment ATLAS, Rucio has been continuously expanded to support LHC experiments and diverse scientific communities. Recent R\&D projects within these communities have evaluated the integration of both private and commercially-provided cloud storage systems, leading to the development of additional functionalities for seamless integration within Rucio. Furthermore, the underlying systems, FTS and GFAL/Davix, have been extended to cater to specific use cases. This contribution focuses on the technical aspects of this work, particularly the challenges encountered in building a generic interface for self-hosted cloud storage, such as MinIO or CEPH S3 Gateway, and established providers like Google Cloud Storage and Amazon Simple Storage Service. Additionally, the integration of decentralised clouds like SEAL is explored. Key aspects, including authentication and authorisation, direct and remote access, throughput and cost estimation, are highlighted, along with shared experiences in daily operations.ATL-SOFT-PROC-2023-016oai:cds.cern.ch:28690282023-08-31
spellingShingle Particle Physics - Experiment
Barisits, Martin
Barreiro Megino, Fernando Harald
Elmsheuser, Johannes
Lassnig, Mario
Patrascoiu, Mihai
Serfon, Cedric
Vendrell Moya, Alba
Wegner, Tobias
Extending Rucio with modern cloud storage support
title Extending Rucio with modern cloud storage support
title_full Extending Rucio with modern cloud storage support
title_fullStr Extending Rucio with modern cloud storage support
title_full_unstemmed Extending Rucio with modern cloud storage support
title_short Extending Rucio with modern cloud storage support
title_sort extending rucio with modern cloud storage support
topic Particle Physics - Experiment
url http://cds.cern.ch/record/2869028
work_keys_str_mv AT barisitsmartin extendingruciowithmoderncloudstoragesupport
AT barreiromeginofernandoharald extendingruciowithmoderncloudstoragesupport
AT elmsheuserjohannes extendingruciowithmoderncloudstoragesupport
AT lassnigmario extendingruciowithmoderncloudstoragesupport
AT patrascoiumihai extendingruciowithmoderncloudstoragesupport
AT serfoncedric extendingruciowithmoderncloudstoragesupport
AT vendrellmoyaalba extendingruciowithmoderncloudstoragesupport
AT wegnertobias extendingruciowithmoderncloudstoragesupport