Cargando…

Collecting conditions usage metadata to optimize current and future ATLAS software and processing

Conditions data (for example: alignment, calibration, data quality) are used extensively in the processing of real and simulated data in ATLAS. The volume and variety of the conditions data needed by different types of processing are quite diverse, so optimizing its access requires a careful underst...

Descripción completa

Detalles Bibliográficos
Autores principales: Barberis, Dario, Formica, Andrea, Gallas, Elizabeth, Oda, Susumu, Rinaldi, Lorenzo, Rybkin, Grigori, Verducci, Monica
Lenguaje:eng
Publicado: 2016
Materias:
Acceso en línea:http://cds.cern.ch/record/2220759
_version_ 1780952207370223616
author Barberis, Dario
Formica, Andrea
Gallas, Elizabeth
Oda, Susumu
Rinaldi, Lorenzo
Rybkin, Grigori
Verducci, Monica
author_facet Barberis, Dario
Formica, Andrea
Gallas, Elizabeth
Oda, Susumu
Rinaldi, Lorenzo
Rybkin, Grigori
Verducci, Monica
author_sort Barberis, Dario
collection CERN
description Conditions data (for example: alignment, calibration, data quality) are used extensively in the processing of real and simulated data in ATLAS. The volume and variety of the conditions data needed by different types of processing are quite diverse, so optimizing its access requires a careful understanding of conditions usage patterns. These patterns can be quantified by mining representative log files from each type of processing and gathering detailed information about conditions usage for that type of processing into a central repository. In this presentation, we describe the systems developed to collect this conditions usage metadata per job type and describe a few specific (but very different) ways in which it has been used. For example, it can be used to cull specific conditions data into a much more compact package to be used by jobs doing similar types of processing: these customized collections can then be shipped with jobs to be executed on isolated worker nodes (such as HPC farms) that have no network access to conditions. Another usage is in the design of future ATLAS software: to provide Run 3 software developers essential information about the nature of current conditions accessed by software. This helps to optimize internal handling of conditions data to minimize its memory footprint while facilitating access to this data by the sub-processes that need it.
id cern-2220759
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2016
record_format invenio
spelling cern-22207592019-09-30T06:29:59Zhttp://cds.cern.ch/record/2220759engBarberis, DarioFormica, AndreaGallas, ElizabethOda, SusumuRinaldi, LorenzoRybkin, GrigoriVerducci, MonicaCollecting conditions usage metadata to optimize current and future ATLAS software and processingParticle Physics - ExperimentConditions data (for example: alignment, calibration, data quality) are used extensively in the processing of real and simulated data in ATLAS. The volume and variety of the conditions data needed by different types of processing are quite diverse, so optimizing its access requires a careful understanding of conditions usage patterns. These patterns can be quantified by mining representative log files from each type of processing and gathering detailed information about conditions usage for that type of processing into a central repository. In this presentation, we describe the systems developed to collect this conditions usage metadata per job type and describe a few specific (but very different) ways in which it has been used. For example, it can be used to cull specific conditions data into a much more compact package to be used by jobs doing similar types of processing: these customized collections can then be shipped with jobs to be executed on isolated worker nodes (such as HPC farms) that have no network access to conditions. Another usage is in the design of future ATLAS software: to provide Run 3 software developers essential information about the nature of current conditions accessed by software. This helps to optimize internal handling of conditions data to minimize its memory footprint while facilitating access to this data by the sub-processes that need it.ATL-SOFT-SLIDE-2016-733oai:cds.cern.ch:22207592016-09-30
spellingShingle Particle Physics - Experiment
Barberis, Dario
Formica, Andrea
Gallas, Elizabeth
Oda, Susumu
Rinaldi, Lorenzo
Rybkin, Grigori
Verducci, Monica
Collecting conditions usage metadata to optimize current and future ATLAS software and processing
title Collecting conditions usage metadata to optimize current and future ATLAS software and processing
title_full Collecting conditions usage metadata to optimize current and future ATLAS software and processing
title_fullStr Collecting conditions usage metadata to optimize current and future ATLAS software and processing
title_full_unstemmed Collecting conditions usage metadata to optimize current and future ATLAS software and processing
title_short Collecting conditions usage metadata to optimize current and future ATLAS software and processing
title_sort collecting conditions usage metadata to optimize current and future atlas software and processing
topic Particle Physics - Experiment
url http://cds.cern.ch/record/2220759
work_keys_str_mv AT barberisdario collectingconditionsusagemetadatatooptimizecurrentandfutureatlassoftwareandprocessing
AT formicaandrea collectingconditionsusagemetadatatooptimizecurrentandfutureatlassoftwareandprocessing
AT gallaselizabeth collectingconditionsusagemetadatatooptimizecurrentandfutureatlassoftwareandprocessing
AT odasusumu collectingconditionsusagemetadatatooptimizecurrentandfutureatlassoftwareandprocessing
AT rinaldilorenzo collectingconditionsusagemetadatatooptimizecurrentandfutureatlassoftwareandprocessing
AT rybkingrigori collectingconditionsusagemetadatatooptimizecurrentandfutureatlassoftwareandprocessing
AT verduccimonica collectingconditionsusagemetadatatooptimizecurrentandfutureatlassoftwareandprocessing