Cargando…

Data allocation service ADAS for the data rebalancing of ATLAS

The distributed data management system Rucio manages all data of the ATLAS collaboration across the grid. Automation, such as data replication and data rebalancing are important to ensure proper operation and execution of the scientific workflow. In this proceedings, a new data allocation grid servi...

Descripción completa

Detalles Bibliográficos
Autores principales: Vamosi, Ralf, Lassnig, Mario, Schikuta, Erich
Lenguaje:eng
Publicado: 2018
Materias:
Acceso en línea:https://dx.doi.org/10.1051/epjconf/201921406012
http://cds.cern.ch/record/2649939
_version_ 1780960777008578560
author Vamosi, Ralf
Lassnig, Mario
Schikuta, Erich
author_facet Vamosi, Ralf
Lassnig, Mario
Schikuta, Erich
author_sort Vamosi, Ralf
collection CERN
description The distributed data management system Rucio manages all data of the ATLAS collaboration across the grid. Automation, such as data replication and data rebalancing are important to ensure proper operation and execution of the scientific workflow. In this proceedings, a new data allocation grid service based on machine learning is proposed. This learning agent takes subsets of the global datasets and proposes a better allocation based on the imposed cost metric, such as waiting time in the workflow. As a service, it can be modularized and can run independently of the existing rebalancing and replication mechanisms. Furthermore, it collects data from other services and learns better allocation while running in the background. Apart from the user selecting datasets, other data services may consult this meta-heuristic service for improved data placement. Network and storage utilization is also taken into account.
id cern-2649939
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2018
record_format invenio
spelling cern-26499392022-08-10T12:27:02Zdoi:10.1051/epjconf/201921406012http://cds.cern.ch/record/2649939engVamosi, RalfLassnig, MarioSchikuta, ErichData allocation service ADAS for the data rebalancing of ATLASParticle Physics - ExperimentThe distributed data management system Rucio manages all data of the ATLAS collaboration across the grid. Automation, such as data replication and data rebalancing are important to ensure proper operation and execution of the scientific workflow. In this proceedings, a new data allocation grid service based on machine learning is proposed. This learning agent takes subsets of the global datasets and proposes a better allocation based on the imposed cost metric, such as waiting time in the workflow. As a service, it can be modularized and can run independently of the existing rebalancing and replication mechanisms. Furthermore, it collects data from other services and learns better allocation while running in the background. Apart from the user selecting datasets, other data services may consult this meta-heuristic service for improved data placement. Network and storage utilization is also taken into account.ATL-SOFT-PROC-2018-057oai:cds.cern.ch:26499392018-12-05
spellingShingle Particle Physics - Experiment
Vamosi, Ralf
Lassnig, Mario
Schikuta, Erich
Data allocation service ADAS for the data rebalancing of ATLAS
title Data allocation service ADAS for the data rebalancing of ATLAS
title_full Data allocation service ADAS for the data rebalancing of ATLAS
title_fullStr Data allocation service ADAS for the data rebalancing of ATLAS
title_full_unstemmed Data allocation service ADAS for the data rebalancing of ATLAS
title_short Data allocation service ADAS for the data rebalancing of ATLAS
title_sort data allocation service adas for the data rebalancing of atlas
topic Particle Physics - Experiment
url https://dx.doi.org/10.1051/epjconf/201921406012
http://cds.cern.ch/record/2649939
work_keys_str_mv AT vamosiralf dataallocationserviceadasforthedatarebalancingofatlas
AT lassnigmario dataallocationserviceadasforthedatarebalancingofatlas
AT schikutaerich dataallocationserviceadasforthedatarebalancingofatlas