Cargando…

EOS as storage back-end for JRC scientific data processing

<!--HTML-->The Joint Research Centre (JRC) of the European Commission has set up the JRC Earth Observation Data and Processing Platform (JEODPP) as an infrastructure to enable the JRC projects to process and analyze big geospatial data in support to EU policy needs. This platform is built upon...

Descripción completa

Detalles Bibliográficos
Autor principal: Burger, Armin
Lenguaje:eng
Publicado: 2019
Materias:
Acceso en línea:http://cds.cern.ch/record/2659480
_version_ 1780961475266871296
author Burger, Armin
author_facet Burger, Armin
author_sort Burger, Armin
collection CERN
description <!--HTML-->The Joint Research Centre (JRC) of the European Commission has set up the JRC Earth Observation Data and Processing Platform (JEODPP) as an infrastructure to enable the JRC projects to process and analyze big geospatial data in support to EU policy needs. This platform is built upon commodity hardware and has become fully operational mid 2016, with platform extensions in 2017 and 2018. EOS was deployed on the JEODPP as the core storage system and is used in production with strong support by the CERN EOS team. The JEODPP infrastructure is actively used by more than 25 JRC projects for data storage and various types of data processing and analysis. In order to serve the growing needs for data storage and processing capacity by the JRC projects, the platform has been extended in 2018. It currently consists of the EOS system as storage back-end with a total gross capacity of 13 PB, and processing and service nodes with a total of 1400 cores. A further extension to 17 PB gross storage and 1800 CPU cores are already in the pipeline. In 2018 the EOS service has been migrated to the new Citrine release. The presentation will give an overview about the implemented platform, the current status, issues identified during the migration to the Citrine release, and experiences made with EOS and FUSE client as main storage back-end. Testing of new functionalities, like the MGM namespace in QuarkDB, wider usage of the fusex client, or the workflow engine are envisaged for 2019.
id cern-2659480
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2019
record_format invenio
spelling cern-26594802022-11-02T22:12:03Zhttp://cds.cern.ch/record/2659480engBurger, ArminEOS as storage back-end for JRC scientific data processingEOS workshopHEP Computing<!--HTML-->The Joint Research Centre (JRC) of the European Commission has set up the JRC Earth Observation Data and Processing Platform (JEODPP) as an infrastructure to enable the JRC projects to process and analyze big geospatial data in support to EU policy needs. This platform is built upon commodity hardware and has become fully operational mid 2016, with platform extensions in 2017 and 2018. EOS was deployed on the JEODPP as the core storage system and is used in production with strong support by the CERN EOS team. The JEODPP infrastructure is actively used by more than 25 JRC projects for data storage and various types of data processing and analysis. In order to serve the growing needs for data storage and processing capacity by the JRC projects, the platform has been extended in 2018. It currently consists of the EOS system as storage back-end with a total gross capacity of 13 PB, and processing and service nodes with a total of 1400 cores. A further extension to 17 PB gross storage and 1800 CPU cores are already in the pipeline. In 2018 the EOS service has been migrated to the new Citrine release. The presentation will give an overview about the implemented platform, the current status, issues identified during the migration to the Citrine release, and experiences made with EOS and FUSE client as main storage back-end. Testing of new functionalities, like the MGM namespace in QuarkDB, wider usage of the fusex client, or the workflow engine are envisaged for 2019.oai:cds.cern.ch:26594802019
spellingShingle HEP Computing
Burger, Armin
EOS as storage back-end for JRC scientific data processing
title EOS as storage back-end for JRC scientific data processing
title_full EOS as storage back-end for JRC scientific data processing
title_fullStr EOS as storage back-end for JRC scientific data processing
title_full_unstemmed EOS as storage back-end for JRC scientific data processing
title_short EOS as storage back-end for JRC scientific data processing
title_sort eos as storage back-end for jrc scientific data processing
topic HEP Computing
url http://cds.cern.ch/record/2659480
work_keys_str_mv AT burgerarmin eosasstoragebackendforjrcscientificdataprocessing
AT burgerarmin eosworkshop