Cargando…

Exploiting analytics techniques in CMS computing monitoring

The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been...

Descripción completa

Detalles Bibliográficos
Autores principales: Bonacorsi, D, Kuznetsov, V, Magini, N, Repečka, A, Vaandering, E
Lenguaje:eng
Publicado: 2017
Materias:
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/898/9/092030
http://cds.cern.ch/record/2296668
_version_ 1780956908820103168
author Bonacorsi, D
Kuznetsov, V
Magini, N
Repečka, A
Vaandering, E
author_facet Bonacorsi, D
Kuznetsov, V
Magini, N
Repečka, A
Vaandering, E
author_sort Bonacorsi, D
collection CERN
description The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.
id oai-inspirehep.net-1638624
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2017
record_format invenio
spelling oai-inspirehep.net-16386242021-02-09T10:06:32Zdoi:10.1088/1742-6596/898/9/092030http://cds.cern.ch/record/2296668engBonacorsi, DKuznetsov, VMagini, NRepečka, AVaandering, EExploiting analytics techniques in CMS computing monitoringComputing and ComputersThe CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.oai:inspirehep.net:16386242017
spellingShingle Computing and Computers
Bonacorsi, D
Kuznetsov, V
Magini, N
Repečka, A
Vaandering, E
Exploiting analytics techniques in CMS computing monitoring
title Exploiting analytics techniques in CMS computing monitoring
title_full Exploiting analytics techniques in CMS computing monitoring
title_fullStr Exploiting analytics techniques in CMS computing monitoring
title_full_unstemmed Exploiting analytics techniques in CMS computing monitoring
title_short Exploiting analytics techniques in CMS computing monitoring
title_sort exploiting analytics techniques in cms computing monitoring
topic Computing and Computers
url https://dx.doi.org/10.1088/1742-6596/898/9/092030
http://cds.cern.ch/record/2296668
work_keys_str_mv AT bonacorsid exploitinganalyticstechniquesincmscomputingmonitoring
AT kuznetsovv exploitinganalyticstechniquesincmscomputingmonitoring
AT maginin exploitinganalyticstechniquesincmscomputingmonitoring
AT repeckaa exploitinganalyticstechniquesincmscomputingmonitoring
AT vaanderinge exploitinganalyticstechniquesincmscomputingmonitoring