Cargando…

Evolution of the HEPS Jupyter-based remote data analysis System

<!--HTML-->High Energy Photon Source (HEPS) has the characteristic of large amount of data, high timeliness, and diverse requirements for scientific data analysis. Generally, researchers need to spend a lot of time in the configuration of the experimental environment. In response to the above...

Descripción completa

Detalles Bibliográficos
Autor principal: Liu, Zhibin
Lenguaje:eng
Publicado: 2021
Materias:
Acceso en línea:http://cds.cern.ch/record/2767292
_version_ 1780971288922161152
author Liu, Zhibin
author_facet Liu, Zhibin
author_sort Liu, Zhibin
collection CERN
description <!--HTML-->High Energy Photon Source (HEPS) has the characteristic of large amount of data, high timeliness, and diverse requirements for scientific data analysis. Generally, researchers need to spend a lot of time in the configuration of the experimental environment. In response to the above problems, we introduce a remote data analysis system for HEPS. The platform provides users a web-based interactive interface with Jupyter, which makes scientists are able to process data analysis anytime and anywhere. Particularly, we discuss the system architecture as well as the key points of this system. A solution of managing and scheduling heterogeneous computing resources (CPU and GPU) is proposed, which adopts Kubernetes to achieve centralized heterogeneous resources management and resource expansion on demand. An improved Kubernetes resource scheduler is discussed. The schedular dispatches resources to upper applications in combination with the cluster status, which can transparently and quickly deployment the data analysis environment for users in seconds and reach the maximum resource utilization. We also introduce an automated deployment solution to improve the work efficiency of developers and help deploy multidisciplinary applications faster and better in the production environment. A unified certification is illustrated to make sure the security of remote data access and data analysis. Finally, we will show the running status of the system.
id cern-2767292
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2021
record_format invenio
spelling cern-27672922022-11-02T22:25:36Zhttp://cds.cern.ch/record/2767292engLiu, ZhibinEvolution of the HEPS Jupyter-based remote data analysis System25th International Conference on Computing in High Energy & Nuclear PhysicsConferences<!--HTML-->High Energy Photon Source (HEPS) has the characteristic of large amount of data, high timeliness, and diverse requirements for scientific data analysis. Generally, researchers need to spend a lot of time in the configuration of the experimental environment. In response to the above problems, we introduce a remote data analysis system for HEPS. The platform provides users a web-based interactive interface with Jupyter, which makes scientists are able to process data analysis anytime and anywhere. Particularly, we discuss the system architecture as well as the key points of this system. A solution of managing and scheduling heterogeneous computing resources (CPU and GPU) is proposed, which adopts Kubernetes to achieve centralized heterogeneous resources management and resource expansion on demand. An improved Kubernetes resource scheduler is discussed. The schedular dispatches resources to upper applications in combination with the cluster status, which can transparently and quickly deployment the data analysis environment for users in seconds and reach the maximum resource utilization. We also introduce an automated deployment solution to improve the work efficiency of developers and help deploy multidisciplinary applications faster and better in the production environment. A unified certification is illustrated to make sure the security of remote data access and data analysis. Finally, we will show the running status of the system.oai:cds.cern.ch:27672922021
spellingShingle Conferences
Liu, Zhibin
Evolution of the HEPS Jupyter-based remote data analysis System
title Evolution of the HEPS Jupyter-based remote data analysis System
title_full Evolution of the HEPS Jupyter-based remote data analysis System
title_fullStr Evolution of the HEPS Jupyter-based remote data analysis System
title_full_unstemmed Evolution of the HEPS Jupyter-based remote data analysis System
title_short Evolution of the HEPS Jupyter-based remote data analysis System
title_sort evolution of the heps jupyter-based remote data analysis system
topic Conferences
url http://cds.cern.ch/record/2767292
work_keys_str_mv AT liuzhibin evolutionofthehepsjupyterbasedremotedataanalysissystem
AT liuzhibin 25thinternationalconferenceoncomputinginhighenergynuclearphysics