Cargando…

Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware

The Message Queue (MQ) architecture is an asynchronous communication scheme that provides an attractive solution for certain scenarios in a distributed computing model. The introduction of MQ as an intermediate component in-between the interacting processes allows to decouple the end-points making t...

Descripción completa

Detalles Bibliográficos
Autores principales: Krzemien, Wojciech, Stagni, Federico, Haen, Christophe, Mathe, Zoltan, McNab, Andrew, Zdybal, Milosz
Lenguaje:eng
Publicado: 2019
Materias:
Acceso en línea:https://dx.doi.org/10.1051/epjconf/201921403018
http://cds.cern.ch/record/2666848
_version_ 1780962020502274048
author Krzemien, Wojciech
Stagni, Federico
Haen, Christophe
Mathe, Zoltan
McNab, Andrew
Zdybal, Milosz
author_facet Krzemien, Wojciech
Stagni, Federico
Haen, Christophe
Mathe, Zoltan
McNab, Andrew
Zdybal, Milosz
author_sort Krzemien, Wojciech
collection CERN
description The Message Queue (MQ) architecture is an asynchronous communication scheme that provides an attractive solution for certain scenarios in a distributed computing model. The introduction of MQ as an intermediate component in-between the interacting processes allows to decouple the end-points making the system more flexible and providing high scalability and redundancy. DIRAC is a general-purpose interware software for distributed computing systems, which offers a common interface to a number of heterogeneous providers and guarantees transparent and reliable usage of the resources. The DIRAC platform has been adapted by several scientific projects, including High Energy Physics communities like LHCb, the Linear Collider and Belle2.A Message Queue generic interface has been incorporated into the DIRAC framework to help solving the scalability challenges that must be addressed during LHC Run3, starting in 2021. It allows to use the MQ scheme for a message exchange among the DIRAC components or to communicate with third-party services. Within this contribution we describe the integration of MQ systems with DIRAC and several use cases are shown. Message Queues are foreseen to be used in the pilot logging system, and as a backbone of the DIRAC component logging system and monitoring.
id cern-2666848
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
record_format invenio
spelling cern-26668482022-03-05T04:02:57Zdoi:10.1051/epjconf/201921403018http://cds.cern.ch/record/2666848engKrzemien, WojciechStagni, FedericoHaen, ChristopheMathe, ZoltanMcNab, AndrewZdybal, MiloszAddressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interwarehep-exParticle Physics - Experimentcs.DCComputing and ComputersThe Message Queue (MQ) architecture is an asynchronous communication scheme that provides an attractive solution for certain scenarios in a distributed computing model. The introduction of MQ as an intermediate component in-between the interacting processes allows to decouple the end-points making the system more flexible and providing high scalability and redundancy. DIRAC is a general-purpose interware software for distributed computing systems, which offers a common interface to a number of heterogeneous providers and guarantees transparent and reliable usage of the resources. The DIRAC platform has been adapted by several scientific projects, including High Energy Physics communities like LHCb, the Linear Collider and Belle2.A Message Queue generic interface has been incorporated into the DIRAC framework to help solving the scalability challenges that must be addressed during LHC Run3, starting in 2021. It allows to use the MQ scheme for a message exchange among the DIRAC components or to communicate with third-party services. Within this contribution we describe the integration of MQ systems with DIRAC and several use cases are shown. Message Queues are foreseen to be used in the pilot logging system, and as a backbone of the DIRAC component logging system and monitoring.The Message Queue (MQ) architecture is an asynchronous communication scheme that provides an attractive solution for certain scenarios in a distributed computing model. The introduction of MQ as an intermediate component in-between the interacting processes allows to decouple the end-points making the system more flexible and providing high scalability and redundancy. DIRAC is a general-purpose interware software for distributed computing systems, which offers a common interface to a number of heterogeneous providers and guarantees transparent and reliable usage of the resources. The DIRAC platform has been adapted by several scientific projects, including High Energy Physics communities like LHCb, the Linear Collider and Belle2. A Message Queue generic interface has been incorporated into the DIRAC framework to help solving the scalability challenges that must be addressed during LHC Run3, starting in 2021. It allows to use the MQ scheme for a message exchange among the DIRAC components or to communicate with third-party services. Within this contribution we describe the integration of MQ systems with DIRAC and several use cases are shown. Message Queues are foreseen to be used in the pilot logging system, and as a backbone of the DIRAC component logging system and monitoring.arXiv:1902.09645oai:cds.cern.ch:26668482019
spellingShingle hep-ex
Particle Physics - Experiment
cs.DC
Computing and Computers
Krzemien, Wojciech
Stagni, Federico
Haen, Christophe
Mathe, Zoltan
McNab, Andrew
Zdybal, Milosz
Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware
title Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware
title_full Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware
title_fullStr Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware
title_full_unstemmed Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware
title_short Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware
title_sort addressing scalability with message queues: architecture and use cases for dirac interware
topic hep-ex
Particle Physics - Experiment
cs.DC
Computing and Computers
url https://dx.doi.org/10.1051/epjconf/201921403018
http://cds.cern.ch/record/2666848
work_keys_str_mv AT krzemienwojciech addressingscalabilitywithmessagequeuesarchitectureandusecasesfordiracinterware
AT stagnifederico addressingscalabilitywithmessagequeuesarchitectureandusecasesfordiracinterware
AT haenchristophe addressingscalabilitywithmessagequeuesarchitectureandusecasesfordiracinterware
AT mathezoltan addressingscalabilitywithmessagequeuesarchitectureandusecasesfordiracinterware
AT mcnabandrew addressingscalabilitywithmessagequeuesarchitectureandusecasesfordiracinterware
AT zdybalmilosz addressingscalabilitywithmessagequeuesarchitectureandusecasesfordiracinterware