Cargando…

The design of a distributed key-value store for petascale hot storage in data acquisition systems

Data acquisition systems for high energy physics experiments read-out terabytes of data per second from a large number of electronic components. They are thus inherently distributed systems and require fast online data selection, otherwise requirements for permanent storage would be enormous. Still,...

Descripción completa

Detalles Bibliográficos
Autores principales: Cicalese, Danilo, Jereczek, Grzegorz, Le Goff, Fabrice, Lehmann Miotto, Giovanna, Love, Jeremy, Maciejewski, Maciej, Mommsen, Remigius K, Radtke, Jakub, Schmiegel, Jakub, Szychowska, Malgorzata
Lenguaje:eng
Publicado: 2019
Materias:
Acceso en línea:https://dx.doi.org/10.1051/epjconf/201921401014
http://cds.cern.ch/record/2702025
_version_ 1780964592774545408
author Cicalese, Danilo
Jereczek, Grzegorz
Le Goff, Fabrice
Lehmann Miotto, Giovanna
Love, Jeremy
Maciejewski, Maciej
Mommsen, Remigius K
Radtke, Jakub
Schmiegel, Jakub
Szychowska, Malgorzata
author_facet Cicalese, Danilo
Jereczek, Grzegorz
Le Goff, Fabrice
Lehmann Miotto, Giovanna
Love, Jeremy
Maciejewski, Maciej
Mommsen, Remigius K
Radtke, Jakub
Schmiegel, Jakub
Szychowska, Malgorzata
author_sort Cicalese, Danilo
collection CERN
description Data acquisition systems for high energy physics experiments read-out terabytes of data per second from a large number of electronic components. They are thus inherently distributed systems and require fast online data selection, otherwise requirements for permanent storage would be enormous. Still, incoming data need to be buffered while waiting for this selection to happen. Each minute of an experiment can produce hundreds of terabytes that cannot be lost before a selection decision is made. In this context, we present the design of DAQDB (Data Acquisition Database) — a distributed key-value store for high-bandwidth, generic data storage in event-driven systems. DAQDB offers not only high-capacity and low-latency buffer for fast data selection, but also opens a new approach in high-bandwidth data acquisition by decoupling the lifetime of the data analysis processes from the changing event rate due to the duty cycle of the data source. This is achievable by the option to extend its capacity even up to hundreds of petabytes to store hours of an experiment’s data. Our initial performance evaluation shows that DAQDB is a promising alternative to generic database solutions for the high luminosity upgrades of the LHC at CERN.
id oai-inspirehep.net-1760510
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
record_format invenio
spelling oai-inspirehep.net-17605102022-08-10T12:23:16Zdoi:10.1051/epjconf/201921401014http://cds.cern.ch/record/2702025engCicalese, DaniloJereczek, GrzegorzLe Goff, FabriceLehmann Miotto, GiovannaLove, JeremyMaciejewski, MaciejMommsen, Remigius KRadtke, JakubSchmiegel, JakubSzychowska, MalgorzataThe design of a distributed key-value store for petascale hot storage in data acquisition systemsComputing and ComputersData acquisition systems for high energy physics experiments read-out terabytes of data per second from a large number of electronic components. They are thus inherently distributed systems and require fast online data selection, otherwise requirements for permanent storage would be enormous. Still, incoming data need to be buffered while waiting for this selection to happen. Each minute of an experiment can produce hundreds of terabytes that cannot be lost before a selection decision is made. In this context, we present the design of DAQDB (Data Acquisition Database) — a distributed key-value store for high-bandwidth, generic data storage in event-driven systems. DAQDB offers not only high-capacity and low-latency buffer for fast data selection, but also opens a new approach in high-bandwidth data acquisition by decoupling the lifetime of the data analysis processes from the changing event rate due to the duty cycle of the data source. This is achievable by the option to extend its capacity even up to hundreds of petabytes to store hours of an experiment’s data. Our initial performance evaluation shows that DAQDB is a promising alternative to generic database solutions for the high luminosity upgrades of the LHC at CERN.oai:inspirehep.net:17605102019
spellingShingle Computing and Computers
Cicalese, Danilo
Jereczek, Grzegorz
Le Goff, Fabrice
Lehmann Miotto, Giovanna
Love, Jeremy
Maciejewski, Maciej
Mommsen, Remigius K
Radtke, Jakub
Schmiegel, Jakub
Szychowska, Malgorzata
The design of a distributed key-value store for petascale hot storage in data acquisition systems
title The design of a distributed key-value store for petascale hot storage in data acquisition systems
title_full The design of a distributed key-value store for petascale hot storage in data acquisition systems
title_fullStr The design of a distributed key-value store for petascale hot storage in data acquisition systems
title_full_unstemmed The design of a distributed key-value store for petascale hot storage in data acquisition systems
title_short The design of a distributed key-value store for petascale hot storage in data acquisition systems
title_sort design of a distributed key-value store for petascale hot storage in data acquisition systems
topic Computing and Computers
url https://dx.doi.org/10.1051/epjconf/201921401014
http://cds.cern.ch/record/2702025
work_keys_str_mv AT cicalesedanilo thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT jereczekgrzegorz thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT legofffabrice thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT lehmannmiottogiovanna thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT lovejeremy thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT maciejewskimaciej thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT mommsenremigiusk thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT radtkejakub thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT schmiegeljakub thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT szychowskamalgorzata thedesignofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT cicalesedanilo designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT jereczekgrzegorz designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT legofffabrice designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT lehmannmiottogiovanna designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT lovejeremy designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT maciejewskimaciej designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT mommsenremigiusk designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT radtkejakub designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT schmiegeljakub designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems
AT szychowskamalgorzata designofadistributedkeyvaluestoreforpetascalehotstorageindataacquisitionsystems