Cargando…
Evaluation of NoSQL databases for DIRAC monitoring and beyond
Nowadays, many database systems are available but they may not be optimized for storing time series data. Monitoring DIRAC jobs would be better done using a database optimised for storing time series data. So far it was done using a MySQL database, which is not well suited for such an application. T...
Autores principales: | , , , |
---|---|
Lenguaje: | eng |
Publicado: |
2015
|
Materias: | |
Acceso en línea: | https://dx.doi.org/10.1088/1742-6596/664/4/042036 http://cds.cern.ch/record/2134572 |
_version_ | 1780949907402653696 |
---|---|
author | Mathe, Z Ramo, A Casajus Stagni, F Tomassetti, L |
author_facet | Mathe, Z Ramo, A Casajus Stagni, F Tomassetti, L |
author_sort | Mathe, Z |
collection | CERN |
description | Nowadays, many database systems are available but they may not be optimized for storing time series data. Monitoring DIRAC jobs would be better done using a database optimised for storing time series data. So far it was done using a MySQL database, which is not well suited for such an application. Therefore alternatives have been investigated. Choosing an appropriate database for storing huge amounts of time series data is not trivial as one must take into account different aspects such as manageability, scalability and extensibility. We compared the performance of Elasticsearch, OpenTSDB (based on HBase) and InfluxDB NoSQL databases, using the same set of machines and the same data. We also evaluated the effort required for maintaining them. Using the LHCb Workload Management System (WMS), based on DIRAC as a use case we set up a new monitoring system, in parallel with the current MySQL system, and we stored the same data into the databases under test. We evaluated Grafana (for OpenTSDB) and Kibana (for ElasticSearch) metrics and graph editors for creating dashboards, in order to have a clear picture on the usability of each candidate. In this paper we present the results of this study and the performance of the selected technology. We also give an outlook of other potential applications of NoSQL databases within the DIRAC project. |
id | oai-inspirehep.net-1413867 |
institution | Organización Europea para la Investigación Nuclear |
language | eng |
publishDate | 2015 |
record_format | invenio |
spelling | oai-inspirehep.net-14138672022-08-10T13:00:54Zdoi:10.1088/1742-6596/664/4/042036http://cds.cern.ch/record/2134572engMathe, ZRamo, A CasajusStagni, FTomassetti, LEvaluation of NoSQL databases for DIRAC monitoring and beyondComputing and ComputersNowadays, many database systems are available but they may not be optimized for storing time series data. Monitoring DIRAC jobs would be better done using a database optimised for storing time series data. So far it was done using a MySQL database, which is not well suited for such an application. Therefore alternatives have been investigated. Choosing an appropriate database for storing huge amounts of time series data is not trivial as one must take into account different aspects such as manageability, scalability and extensibility. We compared the performance of Elasticsearch, OpenTSDB (based on HBase) and InfluxDB NoSQL databases, using the same set of machines and the same data. We also evaluated the effort required for maintaining them. Using the LHCb Workload Management System (WMS), based on DIRAC as a use case we set up a new monitoring system, in parallel with the current MySQL system, and we stored the same data into the databases under test. We evaluated Grafana (for OpenTSDB) and Kibana (for ElasticSearch) metrics and graph editors for creating dashboards, in order to have a clear picture on the usability of each candidate. In this paper we present the results of this study and the performance of the selected technology. We also give an outlook of other potential applications of NoSQL databases within the DIRAC project.oai:inspirehep.net:14138672015 |
spellingShingle | Computing and Computers Mathe, Z Ramo, A Casajus Stagni, F Tomassetti, L Evaluation of NoSQL databases for DIRAC monitoring and beyond |
title | Evaluation of NoSQL databases for DIRAC monitoring and beyond |
title_full | Evaluation of NoSQL databases for DIRAC monitoring and beyond |
title_fullStr | Evaluation of NoSQL databases for DIRAC monitoring and beyond |
title_full_unstemmed | Evaluation of NoSQL databases for DIRAC monitoring and beyond |
title_short | Evaluation of NoSQL databases for DIRAC monitoring and beyond |
title_sort | evaluation of nosql databases for dirac monitoring and beyond |
topic | Computing and Computers |
url | https://dx.doi.org/10.1088/1742-6596/664/4/042036 http://cds.cern.ch/record/2134572 |
work_keys_str_mv | AT mathez evaluationofnosqldatabasesfordiracmonitoringandbeyond AT ramoacasajus evaluationofnosqldatabasesfordiracmonitoringandbeyond AT stagnif evaluationofnosqldatabasesfordiracmonitoringandbeyond AT tomassettil evaluationofnosqldatabasesfordiracmonitoringandbeyond |