Cargando…

A computational storage plugin implemented in EOS to support in-situ data processing on storage servers

<!--HTML-->Computational storage involves integrating compute resources with storage devices or systems to enable data processing within the storage device. This approach reduces data movement, enhances processing efficiency, and reduces costs. To facilitate in-situ data processing on storage...

Descripción completa

Detalles Bibliográficos
Autor principal: Cheng, Yaodong
Lenguaje:eng
Publicado: 2023
Materias:
Acceso en línea:http://cds.cern.ch/record/2857555
_version_ 1780977568925614080
author Cheng, Yaodong
author_facet Cheng, Yaodong
author_sort Cheng, Yaodong
collection CERN
description <!--HTML-->Computational storage involves integrating compute resources with storage devices or systems to enable data processing within the storage device. This approach reduces data movement, enhances processing efficiency, and reduces costs. To facilitate in-situ data processing on storage servers, we developed a computational storage plugin that can be added to EOS FST. This plugin enables users to deploy compute resources directly within the storage servers, allowing them to perform data processing operations on the data stored in the FST nodes without having to move the data to a separate computing system. This can reduce latency and improve overall performance, especially when processing large volumes of data. The plugin can be extended to support a variety of data processing tasks, including data filtering, compression, encryption, and machine learning. The computational storage function is defined in a configuration that can be implemented in scripting languages or evolved independently of the storage system in the form of containers. When an FST node receives a request to open a file, the plugin is executed first. It then calls the target program on the storage server by parsing the parameters of the command to open the file. At this time, the input file must be on the FTS storage server, and the plugin also writes the output file to the node. At the end of the task execution, the output file is automatically registered into the MGM server. Client access is fully compatible with XRootD's API and EOS commands. Users can add tasks and parameters to be performed in the open option. The plugin has been tested and applied in the data processing of the Large High Altitude Air Shower Observatory (LHAASO), and the results show that the efficiency of data decoding is more than 5 times higher than the original method.
id cern-2857555
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2023
record_format invenio
spelling cern-28575552023-05-04T18:19:46Zhttp://cds.cern.ch/record/2857555engCheng, YaodongA computational storage plugin implemented in EOS to support in-situ data processing on storage serversEOS 2023 WorkshopHEP Computing<!--HTML-->Computational storage involves integrating compute resources with storage devices or systems to enable data processing within the storage device. This approach reduces data movement, enhances processing efficiency, and reduces costs. To facilitate in-situ data processing on storage servers, we developed a computational storage plugin that can be added to EOS FST. This plugin enables users to deploy compute resources directly within the storage servers, allowing them to perform data processing operations on the data stored in the FST nodes without having to move the data to a separate computing system. This can reduce latency and improve overall performance, especially when processing large volumes of data. The plugin can be extended to support a variety of data processing tasks, including data filtering, compression, encryption, and machine learning. The computational storage function is defined in a configuration that can be implemented in scripting languages or evolved independently of the storage system in the form of containers. When an FST node receives a request to open a file, the plugin is executed first. It then calls the target program on the storage server by parsing the parameters of the command to open the file. At this time, the input file must be on the FTS storage server, and the plugin also writes the output file to the node. At the end of the task execution, the output file is automatically registered into the MGM server. Client access is fully compatible with XRootD's API and EOS commands. Users can add tasks and parameters to be performed in the open option. The plugin has been tested and applied in the data processing of the Large High Altitude Air Shower Observatory (LHAASO), and the results show that the efficiency of data decoding is more than 5 times higher than the original method.oai:cds.cern.ch:28575552023
spellingShingle HEP Computing
Cheng, Yaodong
A computational storage plugin implemented in EOS to support in-situ data processing on storage servers
title A computational storage plugin implemented in EOS to support in-situ data processing on storage servers
title_full A computational storage plugin implemented in EOS to support in-situ data processing on storage servers
title_fullStr A computational storage plugin implemented in EOS to support in-situ data processing on storage servers
title_full_unstemmed A computational storage plugin implemented in EOS to support in-situ data processing on storage servers
title_short A computational storage plugin implemented in EOS to support in-situ data processing on storage servers
title_sort computational storage plugin implemented in eos to support in-situ data processing on storage servers
topic HEP Computing
url http://cds.cern.ch/record/2857555
work_keys_str_mv AT chengyaodong acomputationalstoragepluginimplementedineostosupportinsitudataprocessingonstorageservers
AT chengyaodong eos2023workshop
AT chengyaodong computationalstoragepluginimplementedineostosupportinsitudataprocessingonstorageservers