Cargando…
Exploring Object Stores for High-Energy Physics Data Storage
Over the last two decades, ROOT TTree has been used for storing over one exabyte of High-Energy Physics (HEP) events. The TTree columnar on-disk layout has been proved to be ideal for analyses of HEP data that typically require access to many events, but only a subset of the information stored for e...
Autores principales: | , |
---|---|
Lenguaje: | eng |
Publicado: |
2021
|
Materias: | |
Acceso en línea: | https://dx.doi.org/10.1051/epjconf/202125102066 http://cds.cern.ch/record/2775940 |
_version_ | 1780971605449506816 |
---|---|
author | López-Gómez, Javier Blomer, Jakob |
author_facet | López-Gómez, Javier Blomer, Jakob |
author_sort | López-Gómez, Javier |
collection | CERN |
description | Over the last two decades, ROOT TTree has been used for storing over one exabyte of High-Energy Physics (HEP) events. The TTree columnar on-disk layout has been proved to be ideal for analyses of HEP data that typically require access to many events, but only a subset of the information stored for each of them. Future colliders, and particularly HL-LHC, will bring an increase of at least one order of magnitude in the volume of generated data. Therefore, the use of modern storage hardware, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes more important. However, TTree was not designed to optimally exploit modern hardware and may become a bottleneck for data retrieval. The ROOT RNTuple I/O system aims at overcoming TTree’s limitations and at providing improved effciency for modern storage systems. In this paper, we extend RNTuple with a backend that uses Intel DAOS as the underlying storage, demonstrating that the RNTuple architecture can accommodate high-performance object stores. From the user perspective, data can be accessed with minimal changes to the code, that is by replacing a filesystem path by a DAOS URI. Our performance evaluation shows that the new backend can be used for realistic analyses, while outperforming the compatibility solution provided by the DAOS project. |
id | cern-2775940 |
institution | Organización Europea para la Investigación Nuclear |
language | eng |
publishDate | 2021 |
record_format | invenio |
spelling | cern-27759402023-01-31T09:51:18Zdoi:10.1051/epjconf/202125102066http://cds.cern.ch/record/2775940engLópez-Gómez, JavierBlomer, JakobExploring Object Stores for High-Energy Physics Data Storagehep-exParticle Physics - Experimentcs.DBComputing and ComputersOver the last two decades, ROOT TTree has been used for storing over one exabyte of High-Energy Physics (HEP) events. The TTree columnar on-disk layout has been proved to be ideal for analyses of HEP data that typically require access to many events, but only a subset of the information stored for each of them. Future colliders, and particularly HL-LHC, will bring an increase of at least one order of magnitude in the volume of generated data. Therefore, the use of modern storage hardware, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes more important. However, TTree was not designed to optimally exploit modern hardware and may become a bottleneck for data retrieval. The ROOT RNTuple I/O system aims at overcoming TTree’s limitations and at providing improved effciency for modern storage systems. In this paper, we extend RNTuple with a backend that uses Intel DAOS as the underlying storage, demonstrating that the RNTuple architecture can accommodate high-performance object stores. From the user perspective, data can be accessed with minimal changes to the code, that is by replacing a filesystem path by a DAOS URI. Our performance evaluation shows that the new backend can be used for realistic analyses, while outperforming the compatibility solution provided by the DAOS project.Over the last two decades, ROOT TTree has been used for storing over one exabyte of High-Energy Physics (HEP) events. The TTree columnar on-disk layout has been proved to be ideal for analyses of HEP data that typically require access to many events, but only a subset of the information stored for each of them. Future colliders, and particularly HL-LHC, will bring an increase of at least one order of magnitude in the volume of generated data. Therefore, the use of modern storage hardware, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes more important. However, TTree was not designed to optimally exploit modern hardware and may become a bottleneck for data retrieval. The ROOT RNTuple I/O system aims at overcoming TTree's limitations and at providing improved efficiency for modern storage systems. In this paper, we extend RNTuple with a backend that uses Intel DAOS as the underlying storage, demonstrating that the RNTuple architecture can accommodate high-performance object stores. From the user perspective, data can be accessed with minimal changes to the code, that is by replacing a filesystem path by a DAOS URI. Our performance evaluation shows that the new backend can be used for realistic analyses, while outperforming the compatibility solution provided by the DAOS project.arXiv:2107.07304oai:cds.cern.ch:27759402021 |
spellingShingle | hep-ex Particle Physics - Experiment cs.DB Computing and Computers López-Gómez, Javier Blomer, Jakob Exploring Object Stores for High-Energy Physics Data Storage |
title | Exploring Object Stores for High-Energy Physics Data Storage |
title_full | Exploring Object Stores for High-Energy Physics Data Storage |
title_fullStr | Exploring Object Stores for High-Energy Physics Data Storage |
title_full_unstemmed | Exploring Object Stores for High-Energy Physics Data Storage |
title_short | Exploring Object Stores for High-Energy Physics Data Storage |
title_sort | exploring object stores for high-energy physics data storage |
topic | hep-ex Particle Physics - Experiment cs.DB Computing and Computers |
url | https://dx.doi.org/10.1051/epjconf/202125102066 http://cds.cern.ch/record/2775940 |
work_keys_str_mv | AT lopezgomezjavier exploringobjectstoresforhighenergyphysicsdatastorage AT blomerjakob exploringobjectstoresforhighenergyphysicsdatastorage |