intelligent Data Delivery Service (iDDS)

Mostrar otras versiones (1)

The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and...

Descripción completa

Detalles Bibliográficos
Autores principales: Guan, Wen, Maeno, Tadashi, Padolski, Siarhei, Bockelman, Brian Paul, Wenaus, Torre, Barreiro Megino, Fernando Harald, Lin, Fa-Hui, Zhang, Rui
Lenguaje:eng
Publicado: 2022
Materias:
Particle Physics - Experiment
Acceso en línea:http://cds.cern.ch/record/2799118
_version_ 1780972521599795200
author Guan, Wen
Maeno, Tadashi
Padolski, Siarhei
Bockelman, Brian Paul
Wenaus, Torre
Barreiro Megino, Fernando Harald
Lin, Fa-Hui
Zhang, Rui
author_facet Guan, Wen
Maeno, Tadashi
Padolski, Siarhei
Bockelman, Brian Paul
Wenaus, Torre
Barreiro Megino, Fernando Harald
Lin, Fa-Hui
Zhang, Rui
author_sort Guan, Wen
collection CERN
description The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service that has been deployed to serve data carousel, hyperparameter optimization, multiple-steps DAG (Directed Acyclic Graphs) workflows and so on. Here we will present the motivation for iDDS, the architecture, use cases and current status for ATLAS and Rubin Observatory exercise, and plans for the future.
id cern-2799118
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2022
record_format invenio
spelling cern-27991182022-01-07T20:20:07Zhttp://cds.cern.ch/record/2799118engGuan, WenMaeno, TadashiPadolski, SiarheiBockelman, Brian PaulWenaus, TorreBarreiro Megino, Fernando HaraldLin, Fa-HuiZhang, Ruiintelligent Data Delivery Service (iDDS)Particle Physics - ExperimentThe intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service that has been deployed to serve data carousel, hyperparameter optimization, multiple-steps DAG (Directed Acyclic Graphs) workflows and so on. Here we will present the motivation for iDDS, the architecture, use cases and current status for ATLAS and Rubin Observatory exercise, and plans for the future.ATL-SOFT-SLIDE-2022-001oai:cds.cern.ch:27991182022-01-07
spellingShingle Particle Physics - Experiment
Guan, Wen
Maeno, Tadashi
Padolski, Siarhei
Bockelman, Brian Paul
Wenaus, Torre
Barreiro Megino, Fernando Harald
Lin, Fa-Hui
Zhang, Rui
intelligent Data Delivery Service (iDDS)
title intelligent Data Delivery Service (iDDS)
title_full intelligent Data Delivery Service (iDDS)
title_fullStr intelligent Data Delivery Service (iDDS)
title_full_unstemmed intelligent Data Delivery Service (iDDS)
title_short intelligent Data Delivery Service (iDDS)
title_sort intelligent data delivery service (idds)
topic Particle Physics - Experiment
url http://cds.cern.ch/record/2799118
work_keys_str_mv AT guanwen intelligentdatadeliveryserviceidds
AT maenotadashi intelligentdatadeliveryserviceidds
AT padolskisiarhei intelligentdatadeliveryserviceidds
AT bockelmanbrianpaul intelligentdatadeliveryserviceidds
AT wenaustorre intelligentdatadeliveryserviceidds
AT barreiromeginofernandoharald intelligentdatadeliveryserviceidds
AT linfahui intelligentdatadeliveryserviceidds
AT zhangrui intelligentdatadeliveryserviceidds

Ejemplares similares