Cargando…
Data Federation Strategies for ATLAS using XRootD
In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user...
Autores principales: | , , , , , , , , , |
---|---|
Lenguaje: | eng |
Publicado: |
2013
|
Materias: | |
Acceso en línea: | https://dx.doi.org/10.1088/1742-6596/513/4/042049 http://cds.cern.ch/record/1622223 |
_version_ | 1780933230670643200 |
---|---|
author | Gardner, R Campana, S Duckeck, G Elmsheuser, J Hanushevski, A Hönig, F Iven, J Legger, F Vukotic, I Yang, W |
author_facet | Gardner, R Campana, S Duckeck, G Elmsheuser, J Hanushevski, A Hönig, F Iven, J Legger, F Vukotic, I Yang, W |
author_sort | Gardner, R |
collection | CERN |
description | In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed. |
id | cern-1622223 |
institution | Organización Europea para la Investigación Nuclear |
language | eng |
publishDate | 2013 |
record_format | invenio |
spelling | cern-16222232019-09-30T06:29:59Zdoi:10.1088/1742-6596/513/4/042049http://cds.cern.ch/record/1622223engGardner, RCampana, SDuckeck, GElmsheuser, JHanushevski, AHönig, FIven, JLegger, FVukotic, IYang, WData Federation Strategies for ATLAS using XRootDDetectors and Experimental TechniquesIn the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed.ATL-SOFT-PROC-2013-039oai:cds.cern.ch:16222232013-10-30 |
spellingShingle | Detectors and Experimental Techniques Gardner, R Campana, S Duckeck, G Elmsheuser, J Hanushevski, A Hönig, F Iven, J Legger, F Vukotic, I Yang, W Data Federation Strategies for ATLAS using XRootD |
title | Data Federation Strategies for ATLAS using XRootD |
title_full | Data Federation Strategies for ATLAS using XRootD |
title_fullStr | Data Federation Strategies for ATLAS using XRootD |
title_full_unstemmed | Data Federation Strategies for ATLAS using XRootD |
title_short | Data Federation Strategies for ATLAS using XRootD |
title_sort | data federation strategies for atlas using xrootd |
topic | Detectors and Experimental Techniques |
url | https://dx.doi.org/10.1088/1742-6596/513/4/042049 http://cds.cern.ch/record/1622223 |
work_keys_str_mv | AT gardnerr datafederationstrategiesforatlasusingxrootd AT campanas datafederationstrategiesforatlasusingxrootd AT duckeckg datafederationstrategiesforatlasusingxrootd AT elmsheuserj datafederationstrategiesforatlasusingxrootd AT hanushevskia datafederationstrategiesforatlasusingxrootd AT honigf datafederationstrategiesforatlasusingxrootd AT ivenj datafederationstrategiesforatlasusingxrootd AT leggerf datafederationstrategiesforatlasusingxrootd AT vukotici datafederationstrategiesforatlasusingxrootd AT yangw datafederationstrategiesforatlasusingxrootd |