Cargando…

Producing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool

The CMS experiment has an HTCondor Global Pool, composed of more than 200K CPU cores available for Monte Carlo production and the analysis of data. The submission of user jobs to this pool is handled by either CRAB3, the standard workflow management tool used by CMS users to submit analysis jobs req...

Descripción completa

Detalles Bibliográficos
Autor principal: Hurtado Anampa, Kenyi Paolo
Lenguaje:eng
Publicado: 2018
Materias:
Acceso en línea:http://cds.cern.ch/record/2648950
_version_ 1780960706955313152
author Hurtado Anampa, Kenyi Paolo
author_facet Hurtado Anampa, Kenyi Paolo
author_sort Hurtado Anampa, Kenyi Paolo
collection CERN
description The CMS experiment has an HTCondor Global Pool, composed of more than 200K CPU cores available for Monte Carlo production and the analysis of data. The submission of user jobs to this pool is handled by either CRAB3, the standard workflow management tool used by CMS users to submit analysis jobs requiring event processing of large amounts of data, or by CMS Connect, a service focused on final stage condor-like analysis jobs and applications that already have a workflow job manager in place. The latest scenario can bring cases in which workflows need further adjustments in order to efficiently work in a globally distributed pool of resources. For instance, the generation of matrix elements for high energy physics processes via Madgraph5\_aMC@NLO and the usage of tools not (yet) fully supported by the CMS software, such as TensorFlow with GPU support, are tasks with particular requirements. A special adaption, either at the pool factory level (advertising GPU resources) or at the execute level (e.g to handle special parameters that describe certain needs for the remote execute nodes during submission) is needed in order to adequately work in the CMS global pool. This contribution describes the challenges and efforts performed towards adapting such workflows so they can properly profit from the Global Pool via CMS Connect.
id cern-2648950
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2018
record_format invenio
spelling cern-26489502019-09-30T06:29:59Zhttp://cds.cern.ch/record/2648950engHurtado Anampa, Kenyi PaoloProducing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global PoolDetectors and Experimental TechniquesThe CMS experiment has an HTCondor Global Pool, composed of more than 200K CPU cores available for Monte Carlo production and the analysis of data. The submission of user jobs to this pool is handled by either CRAB3, the standard workflow management tool used by CMS users to submit analysis jobs requiring event processing of large amounts of data, or by CMS Connect, a service focused on final stage condor-like analysis jobs and applications that already have a workflow job manager in place. The latest scenario can bring cases in which workflows need further adjustments in order to efficiently work in a globally distributed pool of resources. For instance, the generation of matrix elements for high energy physics processes via Madgraph5\_aMC@NLO and the usage of tools not (yet) fully supported by the CMS software, such as TensorFlow with GPU support, are tasks with particular requirements. A special adaption, either at the pool factory level (advertising GPU resources) or at the execute level (e.g to handle special parameters that describe certain needs for the remote execute nodes during submission) is needed in order to adequately work in the CMS global pool. This contribution describes the challenges and efforts performed towards adapting such workflows so they can properly profit from the Global Pool via CMS Connect.CMS-CR-2018-288oai:cds.cern.ch:26489502018-10-18
spellingShingle Detectors and Experimental Techniques
Hurtado Anampa, Kenyi Paolo
Producing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool
title Producing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool
title_full Producing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool
title_fullStr Producing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool
title_full_unstemmed Producing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool
title_short Producing Madgraph5\_aMC@NLO gridpacks and using TensorFlow GPU resources in the CMS HTCondor Global Pool
title_sort producing madgraph5\_amc@nlo gridpacks and using tensorflow gpu resources in the cms htcondor global pool
topic Detectors and Experimental Techniques
url http://cds.cern.ch/record/2648950
work_keys_str_mv AT hurtadoanampakenyipaolo producingmadgraph5amcnlogridpacksandusingtensorflowgpuresourcesinthecmshtcondorglobalpool