Cargando…

A scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMS

The part of the CMS data acquisition (DAQ) system responsible for data readout and event building is a complex network of interdependent distributed applications. To ensure successful data taking, these programs have to be constantly monitored in order to facilitate the timeliness of necessary corre...

Descripción completa

Detalles Bibliográficos
Autor principal: Simelevicius, Dainius
Lenguaje:eng
Publicado: 2018
Materias:
Acceso en línea:http://cds.cern.ch/record/2798185
_version_ 1780972466410094592
author Simelevicius, Dainius
author_facet Simelevicius, Dainius
author_sort Simelevicius, Dainius
collection CERN
description The part of the CMS data acquisition (DAQ) system responsible for data readout and event building is a complex network of interdependent distributed applications. To ensure successful data taking, these programs have to be constantly monitored in order to facilitate the timeliness of necessary corrections in case of any deviation from specified behaviour. A large number of diverse monitoring data samples are periodically collected from multiple sources across the network. Monitoring data are kept in memory for online operations and optionally stored on disk for post-mortem analysis. We present a generic, reusable solution based on an open source NoSQL database, Elasticsearch, which is fully compatible and non-intrusive with respect to the existing system. The motivation is to benefit from an off-the-shelf software to facilitate the development, maintenance and support efforts. Elasticsearch provides failover and data redundancy capabilities as well as a programming language independent JSON-over-HTTP interface. The possibility of horizontal scaling matches the requirements of a DAQ monitoring system. The data load from all sources is balanced by redistribution over an Elasticsearch cluster that can be hosted on a computer cloud. In order to achieve the necessary robustness and to validate the scalability of the approach the above monitoring solution currently runs in parallel with an existing in-house developed DAQ monitoring system.
id cern-2798185
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2018
record_format invenio
spelling cern-27981852021-12-15T20:34:02Zhttp://cds.cern.ch/record/2798185engSimelevicius, DainiusA scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMSDetectors and Experimental TechniquesThe part of the CMS data acquisition (DAQ) system responsible for data readout and event building is a complex network of interdependent distributed applications. To ensure successful data taking, these programs have to be constantly monitored in order to facilitate the timeliness of necessary corrections in case of any deviation from specified behaviour. A large number of diverse monitoring data samples are periodically collected from multiple sources across the network. Monitoring data are kept in memory for online operations and optionally stored on disk for post-mortem analysis. We present a generic, reusable solution based on an open source NoSQL database, Elasticsearch, which is fully compatible and non-intrusive with respect to the existing system. The motivation is to benefit from an off-the-shelf software to facilitate the development, maintenance and support efforts. Elasticsearch provides failover and data redundancy capabilities as well as a programming language independent JSON-over-HTTP interface. The possibility of horizontal scaling matches the requirements of a DAQ monitoring system. The data load from all sources is balanced by redistribution over an Elasticsearch cluster that can be hosted on a computer cloud. In order to achieve the necessary robustness and to validate the scalability of the approach the above monitoring solution currently runs in parallel with an existing in-house developed DAQ monitoring system.CMS-CR-2018-272oai:cds.cern.ch:27981852018-10-12
spellingShingle Detectors and Experimental Techniques
Simelevicius, Dainius
A scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMS
title A scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMS
title_full A scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMS
title_fullStr A scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMS
title_full_unstemmed A scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMS
title_short A scalable online monitoring system based on Elasticsearch for distributed data acquisition in CMS
title_sort scalable online monitoring system based on elasticsearch for distributed data acquisition in cms
topic Detectors and Experimental Techniques
url http://cds.cern.ch/record/2798185
work_keys_str_mv AT simeleviciusdainius ascalableonlinemonitoringsystembasedonelasticsearchfordistributeddataacquisitionincms
AT simeleviciusdainius scalableonlinemonitoringsystembasedonelasticsearchfordistributeddataacquisitionincms