Cargando…
Exploitation of network-segregated CPU resources in CMS
<!--HTML-->CMS is tackling the exploitation of CPU resources at HPC centers where compute nodes do not have network connectivity to the Internet. Pilot agents and payload jobs need to interact with external services from the compute nodes: access to the application software (cmvfs) and conditi...
Autor principal: | |
---|---|
Lenguaje: | eng |
Publicado: |
2021
|
Materias: | |
Acceso en línea: | http://cds.cern.ch/record/2767519 |
_version_ | 1780971308563038208 |
---|---|
author | Delgado Peris, Antonio |
author_facet | Delgado Peris, Antonio |
author_sort | Delgado Peris, Antonio |
collection | CERN |
description | <!--HTML-->CMS is tackling the exploitation of CPU resources at HPC centers where compute nodes do not have network connectivity to the Internet. Pilot agents and payload jobs need to interact with external services from the compute nodes: access to the application software (cmvfs) and conditions data (Frontier), management of input and output data files (data management services), and job management (HTCondor). Finding an alternative route to these services is challenging. Seamless integration in the CMS production system without causing any operational overhead is a key goal.
The case of the Barcelona Supercomputing Center (BSC), in Spain, is particularly challenging, due to its especially restrictive network setup. We describe in this paper the solutions developed within CMS to overcome these restrictions, and integrate this resource in production. Singularity containers with application software releases are built and pre-placed in the HPC facility shared file system, together with conditions data files. HTCondor has been extended to relay communications between running pilot jobs and HTCondor daemons through the HPC shared file system. This operation mode also allows piping input and output data files through the HPC file system.
Results, issues encountered during the integration process, and remaining concerns are discussed. |
id | cern-2767519 |
institution | Organización Europea para la Investigación Nuclear |
language | eng |
publishDate | 2021 |
record_format | invenio |
spelling | cern-27675192022-11-02T22:25:26Zhttp://cds.cern.ch/record/2767519engDelgado Peris, AntonioExploitation of network-segregated CPU resources in CMS25th International Conference on Computing in High Energy & Nuclear PhysicsConferences<!--HTML-->CMS is tackling the exploitation of CPU resources at HPC centers where compute nodes do not have network connectivity to the Internet. Pilot agents and payload jobs need to interact with external services from the compute nodes: access to the application software (cmvfs) and conditions data (Frontier), management of input and output data files (data management services), and job management (HTCondor). Finding an alternative route to these services is challenging. Seamless integration in the CMS production system without causing any operational overhead is a key goal. The case of the Barcelona Supercomputing Center (BSC), in Spain, is particularly challenging, due to its especially restrictive network setup. We describe in this paper the solutions developed within CMS to overcome these restrictions, and integrate this resource in production. Singularity containers with application software releases are built and pre-placed in the HPC facility shared file system, together with conditions data files. HTCondor has been extended to relay communications between running pilot jobs and HTCondor daemons through the HPC shared file system. This operation mode also allows piping input and output data files through the HPC file system. Results, issues encountered during the integration process, and remaining concerns are discussed.oai:cds.cern.ch:27675192021 |
spellingShingle | Conferences Delgado Peris, Antonio Exploitation of network-segregated CPU resources in CMS |
title | Exploitation of network-segregated CPU resources in CMS |
title_full | Exploitation of network-segregated CPU resources in CMS |
title_fullStr | Exploitation of network-segregated CPU resources in CMS |
title_full_unstemmed | Exploitation of network-segregated CPU resources in CMS |
title_short | Exploitation of network-segregated CPU resources in CMS |
title_sort | exploitation of network-segregated cpu resources in cms |
topic | Conferences |
url | http://cds.cern.ch/record/2767519 |
work_keys_str_mv | AT delgadoperisantonio exploitationofnetworksegregatedcpuresourcesincms AT delgadoperisantonio 25thinternationalconferenceoncomputinginhighenergynuclearphysics |