Cargando…

Exploitation of network-segregated CPU resources in CMS

<!--HTML-->CMS is tackling the exploitation of CPU resources at HPC centers where compute nodes do not have network connectivity to the Internet. Pilot agents and payload jobs need to interact with external services from the compute nodes: access to the application software (cmvfs) and conditi...

Descripción completa

Detalles Bibliográficos
Autor principal: Delgado Peris, Antonio
Lenguaje:eng
Publicado: 2021
Materias:
Acceso en línea:http://cds.cern.ch/record/2767519
_version_ 1780971308563038208
author Delgado Peris, Antonio
author_facet Delgado Peris, Antonio
author_sort Delgado Peris, Antonio
collection CERN
description <!--HTML-->CMS is tackling the exploitation of CPU resources at HPC centers where compute nodes do not have network connectivity to the Internet. Pilot agents and payload jobs need to interact with external services from the compute nodes: access to the application software (cmvfs) and conditions data (Frontier), management of input and output data files (data management services), and job management (HTCondor). Finding an alternative route to these services is challenging. Seamless integration in the CMS production system without causing any operational overhead is a key goal. The case of the Barcelona Supercomputing Center (BSC), in Spain, is particularly challenging, due to its especially restrictive network setup. We describe in this paper the solutions developed within CMS to overcome these restrictions, and integrate this resource in production. Singularity containers with application software releases are built and pre-placed in the HPC facility shared file system, together with conditions data files. HTCondor has been extended to relay communications between running pilot jobs and HTCondor daemons through the HPC shared file system. This operation mode also allows piping input and output data files through the HPC file system. Results, issues encountered during the integration process, and remaining concerns are discussed.
id cern-2767519
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2021
record_format invenio
spelling cern-27675192022-11-02T22:25:26Zhttp://cds.cern.ch/record/2767519engDelgado Peris, AntonioExploitation of network-segregated CPU resources in CMS25th International Conference on Computing in High Energy & Nuclear PhysicsConferences<!--HTML-->CMS is tackling the exploitation of CPU resources at HPC centers where compute nodes do not have network connectivity to the Internet. Pilot agents and payload jobs need to interact with external services from the compute nodes: access to the application software (cmvfs) and conditions data (Frontier), management of input and output data files (data management services), and job management (HTCondor). Finding an alternative route to these services is challenging. Seamless integration in the CMS production system without causing any operational overhead is a key goal. The case of the Barcelona Supercomputing Center (BSC), in Spain, is particularly challenging, due to its especially restrictive network setup. We describe in this paper the solutions developed within CMS to overcome these restrictions, and integrate this resource in production. Singularity containers with application software releases are built and pre-placed in the HPC facility shared file system, together with conditions data files. HTCondor has been extended to relay communications between running pilot jobs and HTCondor daemons through the HPC shared file system. This operation mode also allows piping input and output data files through the HPC file system. Results, issues encountered during the integration process, and remaining concerns are discussed.oai:cds.cern.ch:27675192021
spellingShingle Conferences
Delgado Peris, Antonio
Exploitation of network-segregated CPU resources in CMS
title Exploitation of network-segregated CPU resources in CMS
title_full Exploitation of network-segregated CPU resources in CMS
title_fullStr Exploitation of network-segregated CPU resources in CMS
title_full_unstemmed Exploitation of network-segregated CPU resources in CMS
title_short Exploitation of network-segregated CPU resources in CMS
title_sort exploitation of network-segregated cpu resources in cms
topic Conferences
url http://cds.cern.ch/record/2767519
work_keys_str_mv AT delgadoperisantonio exploitationofnetworksegregatedcpuresourcesincms
AT delgadoperisantonio 25thinternationalconferenceoncomputinginhighenergynuclearphysics