Cargando…

RNTuple performance: Status and Outlook

Upcoming HEP experiments, e.g. at the HL-LHC, are expected to increase the volume of generated data by at least one order of magnitude. In order to retain the ability to analyze the influx of data, full exploitation of modern storage hardware and systems, such as low-latency high-bandwidth NVMe devi...

Descripción completa

Detalles Bibliográficos
Autores principales: Lopez-Gomez, Javier, Blomer, Jakob
Lenguaje:eng
Publicado: 2023
Materias:
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/2438/1/012118
http://cds.cern.ch/record/2808833
_version_ 1780973117013753856
author Lopez-Gomez, Javier
Blomer, Jakob
author_facet Lopez-Gomez, Javier
Blomer, Jakob
author_sort Lopez-Gomez, Javier
collection CERN
description Upcoming HEP experiments, e.g. at the HL-LHC, are expected to increase the volume of generated data by at least one order of magnitude. In order to retain the ability to analyze the influx of data, full exploitation of modern storage hardware and systems, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes critical.To this end, the ROOT RNTuple I/O subsystem has been designed to address performance bottlenecks and shortcomings of ROOT’s current state of the art TTree I/O subsystem. RNTuple provides a backwards-incompatible redesign of the TTree binary format and access API that evolves the ROOT event data I/O for the challenges of the upcoming decades. It focuses on a compact data format, on performance engineering for modern storage hardware, for instance through making parallel and asynchronous I/O calls by default, and on robust interfaces that are easy to use correctly.In this contribution, we evaluate the RNTuple performance for typical HEP analysis tasks. We compare the throughput delivered by RNTuple to popular I/O libraries outside HEP, such as HDF5 and Apache Parquet. We demonstrate the advantages of RNTuple for HEP analysis workflows and provide an outlook on the road to its use in production.
id cern-2808833
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2023
record_format invenio
spelling cern-28088332023-03-05T03:26:44Zdoi:10.1088/1742-6596/2438/1/012118http://cds.cern.ch/record/2808833engLopez-Gomez, JavierBlomer, JakobRNTuple performance: Status and OutlookOther Fields of PhysicsAccelerators and Storage RingsComputing and ComputersUpcoming HEP experiments, e.g. at the HL-LHC, are expected to increase the volume of generated data by at least one order of magnitude. In order to retain the ability to analyze the influx of data, full exploitation of modern storage hardware and systems, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes critical.To this end, the ROOT RNTuple I/O subsystem has been designed to address performance bottlenecks and shortcomings of ROOT’s current state of the art TTree I/O subsystem. RNTuple provides a backwards-incompatible redesign of the TTree binary format and access API that evolves the ROOT event data I/O for the challenges of the upcoming decades. It focuses on a compact data format, on performance engineering for modern storage hardware, for instance through making parallel and asynchronous I/O calls by default, and on robust interfaces that are easy to use correctly.In this contribution, we evaluate the RNTuple performance for typical HEP analysis tasks. We compare the throughput delivered by RNTuple to popular I/O libraries outside HEP, such as HDF5 and Apache Parquet. We demonstrate the advantages of RNTuple for HEP analysis workflows and provide an outlook on the road to its use in production.Upcoming HEP experiments, e.g. at the HL-LHC, are expected to increase the volume of generated data by at least one order of magnitude. In order to retain the ability to analyze the influx of data, full exploitation of modern storage hardware and systems, such as low-latency high-bandwidth NVMe devices and distributed object stores, becomes critical. To this end, the ROOT RNTuple I/O subsystem has been designed to address performance bottlenecks and shortcomings of ROOT's current state of the art TTree I/O subsystem. RNTuple provides a backwards-incompatible redesign of the TTree binary format and access API that evolves the ROOT event data I/O for the challenges of the upcoming decades. It focuses on a compact data format, on performance engineering for modern storage hardware, for instance through making parallel and asynchronous I/O calls by default, and on robust interfaces that are easy to use correctly. In this contribution, we evaluate the RNTuple performance for typical HEP analysis tasks. We compare the throughput delivered by RNTuple to popular I/O libraries outside HEP, such as HDF5 and Apache Parquet. We demonstrate the advantages of RNTuple for HEP analysis workflows and provide an outlook on the road to its use in production.arXiv:2204.09043oai:cds.cern.ch:28088332023
spellingShingle Other Fields of Physics
Accelerators and Storage Rings
Computing and Computers
Lopez-Gomez, Javier
Blomer, Jakob
RNTuple performance: Status and Outlook
title RNTuple performance: Status and Outlook
title_full RNTuple performance: Status and Outlook
title_fullStr RNTuple performance: Status and Outlook
title_full_unstemmed RNTuple performance: Status and Outlook
title_short RNTuple performance: Status and Outlook
title_sort rntuple performance: status and outlook
topic Other Fields of Physics
Accelerators and Storage Rings
Computing and Computers
url https://dx.doi.org/10.1088/1742-6596/2438/1/012118
http://cds.cern.ch/record/2808833
work_keys_str_mv AT lopezgomezjavier rntupleperformancestatusandoutlook
AT blomerjakob rntupleperformancestatusandoutlook