Support for HTCondor high-throughput computing workflows in the REANA reusable analysis platform

Mostrar otras versiones (1)

REANA is a reusable and reproducible data analysis platform allowing researchers to structure their analysis pipelines and run them on remote containerised compute clouds. REANA supports several different workflows systems (CWL, Serial, Yadage) and uses Kubernetes’ job execution backend. We have des...

Descripción completa

Detalles Bibliográficos
Autores principales: Maciulaitis, Rokas, Brener, Paul, Hampton, Scott, Hildreth, Mike, Hurtado Anampa, Kenyi Paolo, Johnson, Irena, Kankel, Cody, Okraska, Jan, Rodriguez Rodriguez, Diego, Simko, Tibor
Lenguaje:eng
Publicado: 2019
Materias:
Computing and Computers
Acceso en línea:http://cds.cern.ch/record/2696223
_version_ 1780964168026816512
author Maciulaitis, Rokas
Brener, Paul
Hampton, Scott
Hildreth, Mike
Hurtado Anampa, Kenyi Paolo
Johnson, Irena
Kankel, Cody
Okraska, Jan
Rodriguez Rodriguez, Diego
Simko, Tibor
author_facet Maciulaitis, Rokas
Brener, Paul
Hampton, Scott
Hildreth, Mike
Hurtado Anampa, Kenyi Paolo
Johnson, Irena
Kankel, Cody
Okraska, Jan
Rodriguez Rodriguez, Diego
Simko, Tibor
author_sort Maciulaitis, Rokas
collection CERN
description REANA is a reusable and reproducible data analysis platform allowing researchers to structure their analysis pipelines and run them on remote containerised compute clouds. REANA supports several different workflows systems (CWL, Serial, Yadage) and uses Kubernetes’ job execution backend. We have designed an abstract job execution component that extends the REANA platform job execution capabilities to support multiple compute backends. We have tested the abstract job execution component with HTCondor and verified the scalability of the designed solution. The results show that the REANA platform would be able to support hybrid scientific workflows where different parts of the analysis pipelines can be executed on multiple computing backends.
id cern-2696223
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
record_format invenio
spelling cern-26962232019-10-24T19:12:04Zhttp://cds.cern.ch/record/2696223engMaciulaitis, RokasBrener, PaulHampton, ScottHildreth, MikeHurtado Anampa, Kenyi PaoloJohnson, IrenaKankel, CodyOkraska, JanRodriguez Rodriguez, DiegoSimko, TiborSupport for HTCondor high-throughput computing workflows in the REANA reusable analysis platformComputing and ComputersREANA is a reusable and reproducible data analysis platform allowing researchers to structure their analysis pipelines and run them on remote containerised compute clouds. REANA supports several different workflows systems (CWL, Serial, Yadage) and uses Kubernetes’ job execution backend. We have designed an abstract job execution component that extends the REANA platform job execution capabilities to support multiple compute backends. We have tested the abstract job execution component with HTCondor and verified the scalability of the designed solution. The results show that the REANA platform would be able to support hybrid scientific workflows where different parts of the analysis pipelines can be executed on multiple computing backends.CERN-IT-2019-004oai:cds.cern.ch:26962232019-09-27
spellingShingle Computing and Computers
Maciulaitis, Rokas
Brener, Paul
Hampton, Scott
Hildreth, Mike
Hurtado Anampa, Kenyi Paolo
Johnson, Irena
Kankel, Cody
Okraska, Jan
Rodriguez Rodriguez, Diego
Simko, Tibor
Support for HTCondor high-throughput computing workflows in the REANA reusable analysis platform
title Support for HTCondor high-throughput computing workflows in the REANA reusable analysis platform
title_full Support for HTCondor high-throughput computing workflows in the REANA reusable analysis platform
title_fullStr Support for HTCondor high-throughput computing workflows in the REANA reusable analysis platform
title_full_unstemmed Support for HTCondor high-throughput computing workflows in the REANA reusable analysis platform
title_short Support for HTCondor high-throughput computing workflows in the REANA reusable analysis platform
title_sort support for htcondor high-throughput computing workflows in the reana reusable analysis platform
topic Computing and Computers
url http://cds.cern.ch/record/2696223
work_keys_str_mv AT maciulaitisrokas supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT brenerpaul supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT hamptonscott supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT hildrethmike supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT hurtadoanampakenyipaolo supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT johnsonirena supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT kankelcody supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT okraskajan supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT rodriguezrodriguezdiego supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform
AT simkotibor supportforhtcondorhighthroughputcomputingworkflowsinthereanareusableanalysisplatform

Ejemplares similares