Cargando…

Continuous Analysis
 -- User Containers on the Grid

The ATLAS software infrastructure has undergone several changes towards the adoption of Continuous Integration methodology to develop and test software. The users community can benefit from a CI environment in several ways: they can develop their custom analysis, build and test it using revision con...

Descripción completa

Detalles Bibliográficos
Autores principales: Heinrich, Lukas, Forti, Alessandra, Nilsson, Paul, Maeno, Tadashi
Lenguaje:eng
Publicado: 2019
Materias:
Acceso en línea:http://cds.cern.ch/record/2666156
_version_ 1780961992168701952
author Heinrich, Lukas
Forti, Alessandra
Nilsson, Paul
Maeno, Tadashi
author_facet Heinrich, Lukas
Forti, Alessandra
Nilsson, Paul
Maeno, Tadashi
author_sort Heinrich, Lukas
collection CERN
description The ATLAS software infrastructure has undergone several changes towards the adoption of Continuous Integration methodology to develop and test software. The users community can benefit from a CI environment in several ways: they can develop their custom analysis, build and test it using revision control services such as GitLab. By providing targeted official base images ATLAS enables users to also build self-contained Linux container images as part of the CI pipelines, a crucial component for analysis preservation and re-use scenarios such as reinterpretation of searches for Beyond the Standard Model physics (RECAST). However, so far, the execution of preserved analyses was constrained to dedicated cloud infrastructure and not well-integrated into the wider WLCG computing model, where software distribution has so far relied on a combination of collaboration software distributed via the CVMFS filesystem and user software distributed ad-hoc by the workflow management system. We describe an integration of containerized workloads into the grid infrastructure enabling users to submit self-authored or externally provided container images. To that end, the pilot process executed on the worker node has been extended to utilize the userspace container runtime singularity to execute such workloads. Further, the PanDA job configuration as well as the user-facing command line interfaces have been adapted to allow a detailed specification of the runtime environment. Through this work a continuous grid analysis paradigm emerges, in which for each change in the revision control system, an automated pipeline of unit testing, image building and workload submission based on the freshly built image into the grid is triggered thus further streamlining physics analyses.
id cern-2666156
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
record_format invenio
spelling cern-26661562019-09-30T06:29:59Zhttp://cds.cern.ch/record/2666156engHeinrich, LukasForti, AlessandraNilsson, PaulMaeno, TadashiContinuous Analysis
 -- User Containers on the GridParticle Physics - ExperimentThe ATLAS software infrastructure has undergone several changes towards the adoption of Continuous Integration methodology to develop and test software. The users community can benefit from a CI environment in several ways: they can develop their custom analysis, build and test it using revision control services such as GitLab. By providing targeted official base images ATLAS enables users to also build self-contained Linux container images as part of the CI pipelines, a crucial component for analysis preservation and re-use scenarios such as reinterpretation of searches for Beyond the Standard Model physics (RECAST). However, so far, the execution of preserved analyses was constrained to dedicated cloud infrastructure and not well-integrated into the wider WLCG computing model, where software distribution has so far relied on a combination of collaboration software distributed via the CVMFS filesystem and user software distributed ad-hoc by the workflow management system. We describe an integration of containerized workloads into the grid infrastructure enabling users to submit self-authored or externally provided container images. To that end, the pilot process executed on the worker node has been extended to utilize the userspace container runtime singularity to execute such workloads. Further, the PanDA job configuration as well as the user-facing command line interfaces have been adapted to allow a detailed specification of the runtime environment. Through this work a continuous grid analysis paradigm emerges, in which for each change in the revision control system, an automated pipeline of unit testing, image building and workload submission based on the freshly built image into the grid is triggered thus further streamlining physics analyses.ATL-SOFT-SLIDE-2019-073oai:cds.cern.ch:26661562019-03-08
spellingShingle Particle Physics - Experiment
Heinrich, Lukas
Forti, Alessandra
Nilsson, Paul
Maeno, Tadashi
Continuous Analysis
 -- User Containers on the Grid
title Continuous Analysis
 -- User Containers on the Grid
title_full Continuous Analysis
 -- User Containers on the Grid
title_fullStr Continuous Analysis
 -- User Containers on the Grid
title_full_unstemmed Continuous Analysis
 -- User Containers on the Grid
title_short Continuous Analysis
 -- User Containers on the Grid
title_sort continuous analysis
 -- user containers on the grid
topic Particle Physics - Experiment
url http://cds.cern.ch/record/2666156
work_keys_str_mv AT heinrichlukas continuousanalysisusercontainersonthegrid
AT fortialessandra continuousanalysisusercontainersonthegrid
AT nilssonpaul continuousanalysisusercontainersonthegrid
AT maenotadashi continuousanalysisusercontainersonthegrid