Cargando…

Improving efficiency of analysis jobs in CMS

Data collected by the Compact Muon Solenoid experiment at the Large Hadron Collider are continuously analyzed by hundreds of physicists thanks to the CMS Remote Analysis Builder and the CMS global pool, exploiting the resources of the Worldwide LHC Computing Grid.Making an efficient use of such an e...

Descripción completa

Detalles Bibliográficos
Autor principal: Cristella, Leonardo
Lenguaje:eng
Publicado: SISSA 2019
Materias:
Acceso en línea:https://dx.doi.org/10.22323/1.351.0015
http://cds.cern.ch/record/2709163
_version_ 1780965074728386560
author Cristella, Leonardo
author_facet Cristella, Leonardo
author_sort Cristella, Leonardo
collection CERN
description Data collected by the Compact Muon Solenoid experiment at the Large Hadron Collider are continuously analyzed by hundreds of physicists thanks to the CMS Remote Analysis Builder and the CMS global pool, exploiting the resources of the Worldwide LHC Computing Grid.Making an efficient use of such an extensive and expensive system is crucial.Supporting a variety of workflows while preserving efficient resource usage poses special challenges, like: scheduling of jobs in a multicore/pilot model where several single core jobs with an undefined run time run inside pilot jobs with a fixed lifetime; avoiding that too many concurrent reads from same storage push jobs into I/O wait mode making CPU cycles go idle; monitoring user activity to detect low efficiency workflows and provide optimizations, etc.In this contribution we report on two novel complementary approaches adopted in CMS to improve the scheduling efficiency of user analysis jobs: automatic job splitting and automated run time estimates. They both aim at finding an optimal value for the scheduling run time. We also report on how we use the flexibility of the global CMS computing pool to select the amount, kind, and running locations of jobs exploiting remote access to the input data.
id oai-inspirehep.net-1766690
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
publisher SISSA
record_format invenio
spelling oai-inspirehep.net-17666902020-02-12T21:11:28Zdoi:10.22323/1.351.0015http://cds.cern.ch/record/2709163engCristella, LeonardoImproving efficiency of analysis jobs in CMSComputing and ComputersData collected by the Compact Muon Solenoid experiment at the Large Hadron Collider are continuously analyzed by hundreds of physicists thanks to the CMS Remote Analysis Builder and the CMS global pool, exploiting the resources of the Worldwide LHC Computing Grid.Making an efficient use of such an extensive and expensive system is crucial.Supporting a variety of workflows while preserving efficient resource usage poses special challenges, like: scheduling of jobs in a multicore/pilot model where several single core jobs with an undefined run time run inside pilot jobs with a fixed lifetime; avoiding that too many concurrent reads from same storage push jobs into I/O wait mode making CPU cycles go idle; monitoring user activity to detect low efficiency workflows and provide optimizations, etc.In this contribution we report on two novel complementary approaches adopted in CMS to improve the scheduling efficiency of user analysis jobs: automatic job splitting and automated run time estimates. They both aim at finding an optimal value for the scheduling run time. We also report on how we use the flexibility of the global CMS computing pool to select the amount, kind, and running locations of jobs exploiting remote access to the input data.SISSAoai:inspirehep.net:17666902019
spellingShingle Computing and Computers
Cristella, Leonardo
Improving efficiency of analysis jobs in CMS
title Improving efficiency of analysis jobs in CMS
title_full Improving efficiency of analysis jobs in CMS
title_fullStr Improving efficiency of analysis jobs in CMS
title_full_unstemmed Improving efficiency of analysis jobs in CMS
title_short Improving efficiency of analysis jobs in CMS
title_sort improving efficiency of analysis jobs in cms
topic Computing and Computers
url https://dx.doi.org/10.22323/1.351.0015
http://cds.cern.ch/record/2709163
work_keys_str_mv AT cristellaleonardo improvingefficiencyofanalysisjobsincms