Cargando…
A plugin-based approach to data analysis for the AMS experiment on the ISS
In many HEP experiments a typical data analysis workflow requires each user to read the experiment data in order to extract meaningful information and produce relevant plots for the considered analysis. Multiple users accessing the same data result in a redundant access to the data itself, which cou...
Autor principal: | |
---|---|
Lenguaje: | eng |
Publicado: |
2019
|
Materias: | |
Acceso en línea: | https://dx.doi.org/10.1051/epjconf/201921405038 http://cds.cern.ch/record/2826919 |
_version_ | 1780973870019248128 |
---|---|
author | Formato, Valerio |
author_facet | Formato, Valerio |
author_sort | Formato, Valerio |
collection | CERN |
description | In many HEP experiments a typical data analysis workflow requires each user to read the experiment data in order to extract meaningful information and produce relevant plots for the considered analysis. Multiple users accessing the same data result in a redundant access to the data itself, which could be factorized effectively improving the CPU efficiency of the analysis jobs and relieving stress from the storage infrastructure. To address this issue we present a modular and lightweight solution where the users code is embedded in different "analysis plugins" which are then collected and loaded at runtime for execution, where the data is read only once and shared between all the different plugins. This solution was developed for one of the data analysis groups within the AMS collaboration but is easily extendable to all kinds of analyses and workloads that need I/O access on AMS data or custom data formats and can even adapted with little effort to another HEP experiment data. This framework could then be easily embedded into a "analysis train" and we will discuss a possible implementation and different ways to optimise CPU efficiency and execution time. |
id | cern-2826919 |
institution | Organización Europea para la Investigación Nuclear |
language | eng |
publishDate | 2019 |
record_format | invenio |
spelling | cern-28269192022-09-13T20:18:47Zdoi:10.1051/epjconf/201921405038http://cds.cern.ch/record/2826919engFormato, ValerioA plugin-based approach to data analysis for the AMS experiment on the ISSComputing and ComputersIn many HEP experiments a typical data analysis workflow requires each user to read the experiment data in order to extract meaningful information and produce relevant plots for the considered analysis. Multiple users accessing the same data result in a redundant access to the data itself, which could be factorized effectively improving the CPU efficiency of the analysis jobs and relieving stress from the storage infrastructure. To address this issue we present a modular and lightweight solution where the users code is embedded in different "analysis plugins" which are then collected and loaded at runtime for execution, where the data is read only once and shared between all the different plugins. This solution was developed for one of the data analysis groups within the AMS collaboration but is easily extendable to all kinds of analyses and workloads that need I/O access on AMS data or custom data formats and can even adapted with little effort to another HEP experiment data. This framework could then be easily embedded into a "analysis train" and we will discuss a possible implementation and different ways to optimise CPU efficiency and execution time.oai:cds.cern.ch:28269192019 |
spellingShingle | Computing and Computers Formato, Valerio A plugin-based approach to data analysis for the AMS experiment on the ISS |
title | A plugin-based approach to data analysis for the AMS experiment on the ISS |
title_full | A plugin-based approach to data analysis for the AMS experiment on the ISS |
title_fullStr | A plugin-based approach to data analysis for the AMS experiment on the ISS |
title_full_unstemmed | A plugin-based approach to data analysis for the AMS experiment on the ISS |
title_short | A plugin-based approach to data analysis for the AMS experiment on the ISS |
title_sort | plugin-based approach to data analysis for the ams experiment on the iss |
topic | Computing and Computers |
url | https://dx.doi.org/10.1051/epjconf/201921405038 http://cds.cern.ch/record/2826919 |
work_keys_str_mv | AT formatovalerio apluginbasedapproachtodataanalysisfortheamsexperimentontheiss AT formatovalerio pluginbasedapproachtodataanalysisfortheamsexperimentontheiss |