Cargando…

Data Federation Strategies for ATLAS using XRootD

In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user...

Descripción completa

Detalles Bibliográficos
Autores principales: Gardner, R, Campana, S, Duckeck, G, Elmsheuser, J, Hanushevski, A, Hönig, F, Iven, J, Legger, F, Vukotic, I, Yang, W
Lenguaje:eng
Publicado: 2013
Materias:
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/513/4/042049
http://cds.cern.ch/record/1622223
_version_ 1780933230670643200
author Gardner, R
Campana, S
Duckeck, G
Elmsheuser, J
Hanushevski, A
Hönig, F
Iven, J
Legger, F
Vukotic, I
Yang, W
author_facet Gardner, R
Campana, S
Duckeck, G
Elmsheuser, J
Hanushevski, A
Hönig, F
Iven, J
Legger, F
Vukotic, I
Yang, W
author_sort Gardner, R
collection CERN
description In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed.
id cern-1622223
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2013
record_format invenio
spelling cern-16222232019-09-30T06:29:59Zdoi:10.1088/1742-6596/513/4/042049http://cds.cern.ch/record/1622223engGardner, RCampana, SDuckeck, GElmsheuser, JHanushevski, AHönig, FIven, JLegger, FVukotic, IYang, WData Federation Strategies for ATLAS using XRootDDetectors and Experimental TechniquesIn the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed.ATL-SOFT-PROC-2013-039oai:cds.cern.ch:16222232013-10-30
spellingShingle Detectors and Experimental Techniques
Gardner, R
Campana, S
Duckeck, G
Elmsheuser, J
Hanushevski, A
Hönig, F
Iven, J
Legger, F
Vukotic, I
Yang, W
Data Federation Strategies for ATLAS using XRootD
title Data Federation Strategies for ATLAS using XRootD
title_full Data Federation Strategies for ATLAS using XRootD
title_fullStr Data Federation Strategies for ATLAS using XRootD
title_full_unstemmed Data Federation Strategies for ATLAS using XRootD
title_short Data Federation Strategies for ATLAS using XRootD
title_sort data federation strategies for atlas using xrootd
topic Detectors and Experimental Techniques
url https://dx.doi.org/10.1088/1742-6596/513/4/042049
http://cds.cern.ch/record/1622223
work_keys_str_mv AT gardnerr datafederationstrategiesforatlasusingxrootd
AT campanas datafederationstrategiesforatlasusingxrootd
AT duckeckg datafederationstrategiesforatlasusingxrootd
AT elmsheuserj datafederationstrategiesforatlasusingxrootd
AT hanushevskia datafederationstrategiesforatlasusingxrootd
AT honigf datafederationstrategiesforatlasusingxrootd
AT ivenj datafederationstrategiesforatlasusingxrootd
AT leggerf datafederationstrategiesforatlasusingxrootd
AT vukotici datafederationstrategiesforatlasusingxrootd
AT yangw datafederationstrategiesforatlasusingxrootd