Cargando…

Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage

Data acquisition systems are a key component for successful data taking in any experiment. The DAQ is a complex distributed computing system and coordinates all operations, from the data selection stage of interesting events to storage elements. For the High Luminosity upgrade of the Large Hadron Co...

Descripción completa

Detalles Bibliográficos
Autores principales: Abed Abud, Adam, Cicalese, Danilo, Jereczek, Grzegorz, Goff, Fabrice Le, Lehmann Miotto, Giovanna, Love, Jeremy, Maciejewski, Maciej, Mommsen, Remigius K, Radtke, Jakub, Schmiegel, Jakub, Szychowska, Malgorzata
Lenguaje:eng
Publicado: 2020
Materias:
Acceso en línea:https://dx.doi.org/10.1051/epjconf/202024510004
http://cds.cern.ch/record/2752945
_version_ 1780969329099014144
author Abed Abud, Adam
Cicalese, Danilo
Jereczek, Grzegorz
Goff, Fabrice Le
Lehmann Miotto, Giovanna
Love, Jeremy
Maciejewski, Maciej
Mommsen, Remigius K
Radtke, Jakub
Schmiegel, Jakub
Szychowska, Malgorzata
author_facet Abed Abud, Adam
Cicalese, Danilo
Jereczek, Grzegorz
Goff, Fabrice Le
Lehmann Miotto, Giovanna
Love, Jeremy
Maciejewski, Maciej
Mommsen, Remigius K
Radtke, Jakub
Schmiegel, Jakub
Szychowska, Malgorzata
author_sort Abed Abud, Adam
collection CERN
description Data acquisition systems are a key component for successful data taking in any experiment. The DAQ is a complex distributed computing system and coordinates all operations, from the data selection stage of interesting events to storage elements. For the High Luminosity upgrade of the Large Hadron Collider, the experiments at CERN need to meet challenging requirements to record data with a much higher occupancy in the detectors. The DAQ system will receive and deliver data with a significantly increased trigger rate, one million events per second, and capacity, terabytes of data per second. An effective way to meet these requirements is to decouple real-time data acquisition from event selection. Data fragments can be temporarily stored in a large distributed key-value store. Fragments belonging to the same event can be then queried on demand, by the data selection processes. Implementing such a model relies on a proper combination of emerging technologies, such as persistent memory, NVMe SSDs, scalable networking, and data structures, as well as high performance, scalable software. In this paper, we present DAQDB (Data Acquisition Database) — an open source implementation of this design that was presented earlier, with an extensive evaluation of this approach, from the single node to the distributed performance. Furthermore, we complement our study with a description of the challenges faced and the lessons learned while integrating DAQDB with the existing software framework of the ATLAS experiment.
id oai-inspirehep.net-1832155
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2020
record_format invenio
spelling oai-inspirehep.net-18321552022-08-17T13:56:01Zdoi:10.1051/epjconf/202024510004http://cds.cern.ch/record/2752945engAbed Abud, AdamCicalese, DaniloJereczek, GrzegorzGoff, Fabrice LeLehmann Miotto, GiovannaLove, JeremyMaciejewski, MaciejMommsen, Remigius KRadtke, JakubSchmiegel, JakubSzychowska, MalgorzataLet’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storageComputing and ComputersData acquisition systems are a key component for successful data taking in any experiment. The DAQ is a complex distributed computing system and coordinates all operations, from the data selection stage of interesting events to storage elements. For the High Luminosity upgrade of the Large Hadron Collider, the experiments at CERN need to meet challenging requirements to record data with a much higher occupancy in the detectors. The DAQ system will receive and deliver data with a significantly increased trigger rate, one million events per second, and capacity, terabytes of data per second. An effective way to meet these requirements is to decouple real-time data acquisition from event selection. Data fragments can be temporarily stored in a large distributed key-value store. Fragments belonging to the same event can be then queried on demand, by the data selection processes. Implementing such a model relies on a proper combination of emerging technologies, such as persistent memory, NVMe SSDs, scalable networking, and data structures, as well as high performance, scalable software. In this paper, we present DAQDB (Data Acquisition Database) — an open source implementation of this design that was presented earlier, with an extensive evaluation of this approach, from the single node to the distributed performance. Furthermore, we complement our study with a description of the challenges faced and the lessons learned while integrating DAQDB with the existing software framework of the ATLAS experiment.oai:inspirehep.net:18321552020
spellingShingle Computing and Computers
Abed Abud, Adam
Cicalese, Danilo
Jereczek, Grzegorz
Goff, Fabrice Le
Lehmann Miotto, Giovanna
Love, Jeremy
Maciejewski, Maciej
Mommsen, Remigius K
Radtke, Jakub
Schmiegel, Jakub
Szychowska, Malgorzata
Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage
title Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage
title_full Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage
title_fullStr Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage
title_full_unstemmed Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage
title_short Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage
title_sort let’s get our hands dirty: a comprehensive evaluation of daqdb, key-value store for petascale hot storage
topic Computing and Computers
url https://dx.doi.org/10.1051/epjconf/202024510004
http://cds.cern.ch/record/2752945
work_keys_str_mv AT abedabudadam letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT cicalesedanilo letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT jereczekgrzegorz letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT gofffabricele letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT lehmannmiottogiovanna letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT lovejeremy letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT maciejewskimaciej letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT mommsenremigiusk letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT radtkejakub letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT schmiegeljakub letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage
AT szychowskamalgorzata letsgetourhandsdirtyacomprehensiveevaluationofdaqdbkeyvaluestoreforpetascalehotstorage