Cargando…

Integrating HPC into an agile and cloud-focused environment at CERN

CERN’s batch and grid services are mainly focused on High Throughput computing (HTC) for processing data from the Large Hadron Collider (LHC) and other experiments. However, part of the user community requires High Performance Computing (HPC) for massively parallel applications across many cores on...

Descripción completa

Detalles Bibliográficos
Autores principales: Llopis, Pablo, Lindqvist, Carolina, Høimyr, Nils, van der Ster, Dan, Ganz, Philippe
Lenguaje:eng
Publicado: 2019
Materias:
Acceso en línea:https://dx.doi.org/10.1051/epjconf/201921407025
http://cds.cern.ch/record/2698981
_version_ 1780964398914863104
author Llopis, Pablo
Lindqvist, Carolina
Høimyr, Nils
van der Ster, Dan
Ganz, Philippe
author_facet Llopis, Pablo
Lindqvist, Carolina
Høimyr, Nils
van der Ster, Dan
Ganz, Philippe
author_sort Llopis, Pablo
collection CERN
description CERN’s batch and grid services are mainly focused on High Throughput computing (HTC) for processing data from the Large Hadron Collider (LHC) and other experiments. However, part of the user community requires High Performance Computing (HPC) for massively parallel applications across many cores on MPI-enabled infrastructure. This contribution addresses the implementation of HPC infrastructure at CERN for Lattice QCD application development, as well as for different types of simulations for the accelerator and technology sector at CERN. Our approach has been to integrate the HPC facilities as far as possible with the HTC services in our data centre, and to take advantage of an agile infrastructure for updates, configuration and deployment. The HPC cluster has been orchestrated with the OpenStack Ironic component, and is hence managed with the same tools as the CERN internal OpenStack cloud. Experience and benchmarks of MPI applications across Infiniband with shared storage on CephFS is discussed, as well the setup of the SLURM scheduler for HPC jobs with a provision for backfill of HTC workloads.
id oai-inspirehep.net-1761589
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
record_format invenio
spelling oai-inspirehep.net-17615892022-08-10T12:26:40Zdoi:10.1051/epjconf/201921407025http://cds.cern.ch/record/2698981engLlopis, PabloLindqvist, CarolinaHøimyr, Nilsvan der Ster, DanGanz, PhilippeIntegrating HPC into an agile and cloud-focused environment at CERNComputing and ComputersCERN’s batch and grid services are mainly focused on High Throughput computing (HTC) for processing data from the Large Hadron Collider (LHC) and other experiments. However, part of the user community requires High Performance Computing (HPC) for massively parallel applications across many cores on MPI-enabled infrastructure. This contribution addresses the implementation of HPC infrastructure at CERN for Lattice QCD application development, as well as for different types of simulations for the accelerator and technology sector at CERN. Our approach has been to integrate the HPC facilities as far as possible with the HTC services in our data centre, and to take advantage of an agile infrastructure for updates, configuration and deployment. The HPC cluster has been orchestrated with the OpenStack Ironic component, and is hence managed with the same tools as the CERN internal OpenStack cloud. Experience and benchmarks of MPI applications across Infiniband with shared storage on CephFS is discussed, as well the setup of the SLURM scheduler for HPC jobs with a provision for backfill of HTC workloads.oai:inspirehep.net:17615892019
spellingShingle Computing and Computers
Llopis, Pablo
Lindqvist, Carolina
Høimyr, Nils
van der Ster, Dan
Ganz, Philippe
Integrating HPC into an agile and cloud-focused environment at CERN
title Integrating HPC into an agile and cloud-focused environment at CERN
title_full Integrating HPC into an agile and cloud-focused environment at CERN
title_fullStr Integrating HPC into an agile and cloud-focused environment at CERN
title_full_unstemmed Integrating HPC into an agile and cloud-focused environment at CERN
title_short Integrating HPC into an agile and cloud-focused environment at CERN
title_sort integrating hpc into an agile and cloud-focused environment at cern
topic Computing and Computers
url https://dx.doi.org/10.1051/epjconf/201921407025
http://cds.cern.ch/record/2698981
work_keys_str_mv AT llopispablo integratinghpcintoanagileandcloudfocusedenvironmentatcern
AT lindqvistcarolina integratinghpcintoanagileandcloudfocusedenvironmentatcern
AT høimyrnils integratinghpcintoanagileandcloudfocusedenvironmentatcern
AT vandersterdan integratinghpcintoanagileandcloudfocusedenvironmentatcern
AT ganzphilippe integratinghpcintoanagileandcloudfocusedenvironmentatcern