Cargando…

Memory handling in the ATLAS submission system from job definition to sites limits

In the past few years the increased luminosity of the LHC, changes in the linux kernel and a move to a 64bit architecture have affected the ATLAS jobs memory usage and the ATLAS workload management system had to be adapted to be more flexible and pass memory parameters to the batch systems, which in...

Descripción completa

Detalles Bibliográficos
Autores principales: Forti, Alessandra, Walker, Rodney, Maeno, Tadashi, Love, Peter, Rauschmayr, Nathalie, Filipcic, Andrej, Di Girolamo, Alessandro
Lenguaje:eng
Publicado: 2017
Materias:
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/898/5/052004
http://cds.cern.ch/record/2242868
_version_ 1780953282439544832
author Forti, Alessandra
Walker, Rodney
Maeno, Tadashi
Love, Peter
Rauschmayr, Nathalie
Filipcic, Andrej
Di Girolamo, Alessandro
author_facet Forti, Alessandra
Walker, Rodney
Maeno, Tadashi
Love, Peter
Rauschmayr, Nathalie
Filipcic, Andrej
Di Girolamo, Alessandro
author_sort Forti, Alessandra
collection CERN
description In the past few years the increased luminosity of the LHC, changes in the linux kernel and a move to a 64bit architecture have affected the ATLAS jobs memory usage and the ATLAS workload management system had to be adapted to be more flexible and pass memory parameters to the batch systems, which in the past wasn’t a necessity. This paper describes the steps required to add the capability to better handle memory requirements, included the review of how each component definition and parametrization of the memory is mapped to the other components, and what changes had to be applied to make the submission chain work. These changes go from the definition of tasks and the way tasks memory requirements are set using scout jobs, through the new memory tool developed to do that, to how these values are used by the submission component of the system and how the jobs are treated by the sites through the CEs, batch systems and ultimately the kernel.
id cern-2242868
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2017
record_format invenio
spelling cern-22428682019-10-15T15:17:06Zdoi:10.1088/1742-6596/898/5/052004http://cds.cern.ch/record/2242868engForti, AlessandraWalker, RodneyMaeno, TadashiLove, PeterRauschmayr, NathalieFilipcic, AndrejDi Girolamo, AlessandroMemory handling in the ATLAS submission system from job definition to sites limitsParticle Physics - ExperimentIn the past few years the increased luminosity of the LHC, changes in the linux kernel and a move to a 64bit architecture have affected the ATLAS jobs memory usage and the ATLAS workload management system had to be adapted to be more flexible and pass memory parameters to the batch systems, which in the past wasn’t a necessity. This paper describes the steps required to add the capability to better handle memory requirements, included the review of how each component definition and parametrization of the memory is mapped to the other components, and what changes had to be applied to make the submission chain work. These changes go from the definition of tasks and the way tasks memory requirements are set using scout jobs, through the new memory tool developed to do that, to how these values are used by the submission component of the system and how the jobs are treated by the sites through the CEs, batch systems and ultimately the kernel.ATL-SOFT-PROC-2017-018oai:cds.cern.ch:22428682017-01-24
spellingShingle Particle Physics - Experiment
Forti, Alessandra
Walker, Rodney
Maeno, Tadashi
Love, Peter
Rauschmayr, Nathalie
Filipcic, Andrej
Di Girolamo, Alessandro
Memory handling in the ATLAS submission system from job definition to sites limits
title Memory handling in the ATLAS submission system from job definition to sites limits
title_full Memory handling in the ATLAS submission system from job definition to sites limits
title_fullStr Memory handling in the ATLAS submission system from job definition to sites limits
title_full_unstemmed Memory handling in the ATLAS submission system from job definition to sites limits
title_short Memory handling in the ATLAS submission system from job definition to sites limits
title_sort memory handling in the atlas submission system from job definition to sites limits
topic Particle Physics - Experiment
url https://dx.doi.org/10.1088/1742-6596/898/5/052004
http://cds.cern.ch/record/2242868
work_keys_str_mv AT fortialessandra memoryhandlingintheatlassubmissionsystemfromjobdefinitiontositeslimits
AT walkerrodney memoryhandlingintheatlassubmissionsystemfromjobdefinitiontositeslimits
AT maenotadashi memoryhandlingintheatlassubmissionsystemfromjobdefinitiontositeslimits
AT lovepeter memoryhandlingintheatlassubmissionsystemfromjobdefinitiontositeslimits
AT rauschmayrnathalie memoryhandlingintheatlassubmissionsystemfromjobdefinitiontositeslimits
AT filipcicandrej memoryhandlingintheatlassubmissionsystemfromjobdefinitiontositeslimits
AT digirolamoalessandro memoryhandlingintheatlassubmissionsystemfromjobdefinitiontositeslimits