Cargando…

The Dynafed data federator as a grid site storage element

The Dynafed data federator is designed to present a dynamic and unified view of a distributed file repository. We describe our use of Dynafed to construct a production-ready WLCG storage element (SE) using existing grid storage endpoints as well as object storage. Dynafed is used as the primary SE f...

Descripción completa

Detalles Bibliográficos
Autores principales: Ebert, Marcus, Berghaus, Frank, Casteels, Kevin, Driemel, Colson, Keeble, Oliver, Leavett-Brown, Colin, Galindo, Fernando Fernandez, Furano, Fabrizio, Paterson, Michael, Seuster, Rolf, Sobie, Randall, Tafirout, Reda
Lenguaje:eng
Publicado: 2020
Materias:
Acceso en línea:https://dx.doi.org/10.1051/epjconf/202024504003
http://cds.cern.ch/record/2758816
_version_ 1780970194259148800
author Ebert, Marcus
Berghaus, Frank
Casteels, Kevin
Driemel, Colson
Keeble, Oliver
Leavett-Brown, Colin
Galindo, Fernando Fernandez
Furano, Fabrizio
Paterson, Michael
Seuster, Rolf
Sobie, Randall
Tafirout, Reda
author_facet Ebert, Marcus
Berghaus, Frank
Casteels, Kevin
Driemel, Colson
Keeble, Oliver
Leavett-Brown, Colin
Galindo, Fernando Fernandez
Furano, Fabrizio
Paterson, Michael
Seuster, Rolf
Sobie, Randall
Tafirout, Reda
author_sort Ebert, Marcus
collection CERN
description The Dynafed data federator is designed to present a dynamic and unified view of a distributed file repository. We describe our use of Dynafed to construct a production-ready WLCG storage element (SE) using existing grid storage endpoints as well as object storage. Dynafed is used as the primary SE for the Canadian distributed computing cloud systems for the Belle-II experiment, where we use it in production for reading input files. We have run up to 6,000 Belle-II jobs simultaneously on distributed cloud resources, requiring the transfer of approximately 60 TB of input data per day. Similarly, we have been using a Dynafed-based SE in pre-production testing for the ATLAS experiment. We will describe the configuration of Dynafed to make it suitable as a WLCG SE. In particular, we will highlight the improvements within Dynafed that make it possible to do checksum based file verification and 3rd-party file copy via WebDAV. We will also report on a new monitoring system and an automated system that collects storage information from all endpoints.
id oai-inspirehep.net-1831533
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2020
record_format invenio
spelling oai-inspirehep.net-18315332021-03-25T22:33:56Zdoi:10.1051/epjconf/202024504003http://cds.cern.ch/record/2758816engEbert, MarcusBerghaus, FrankCasteels, KevinDriemel, ColsonKeeble, OliverLeavett-Brown, ColinGalindo, Fernando FernandezFurano, FabrizioPaterson, MichaelSeuster, RolfSobie, RandallTafirout, RedaThe Dynafed data federator as a grid site storage elementComputing and ComputersThe Dynafed data federator is designed to present a dynamic and unified view of a distributed file repository. We describe our use of Dynafed to construct a production-ready WLCG storage element (SE) using existing grid storage endpoints as well as object storage. Dynafed is used as the primary SE for the Canadian distributed computing cloud systems for the Belle-II experiment, where we use it in production for reading input files. We have run up to 6,000 Belle-II jobs simultaneously on distributed cloud resources, requiring the transfer of approximately 60 TB of input data per day. Similarly, we have been using a Dynafed-based SE in pre-production testing for the ATLAS experiment. We will describe the configuration of Dynafed to make it suitable as a WLCG SE. In particular, we will highlight the improvements within Dynafed that make it possible to do checksum based file verification and 3rd-party file copy via WebDAV. We will also report on a new monitoring system and an automated system that collects storage information from all endpoints.oai:inspirehep.net:18315332020
spellingShingle Computing and Computers
Ebert, Marcus
Berghaus, Frank
Casteels, Kevin
Driemel, Colson
Keeble, Oliver
Leavett-Brown, Colin
Galindo, Fernando Fernandez
Furano, Fabrizio
Paterson, Michael
Seuster, Rolf
Sobie, Randall
Tafirout, Reda
The Dynafed data federator as a grid site storage element
title The Dynafed data federator as a grid site storage element
title_full The Dynafed data federator as a grid site storage element
title_fullStr The Dynafed data federator as a grid site storage element
title_full_unstemmed The Dynafed data federator as a grid site storage element
title_short The Dynafed data federator as a grid site storage element
title_sort dynafed data federator as a grid site storage element
topic Computing and Computers
url https://dx.doi.org/10.1051/epjconf/202024504003
http://cds.cern.ch/record/2758816
work_keys_str_mv AT ebertmarcus thedynafeddatafederatorasagridsitestorageelement
AT berghausfrank thedynafeddatafederatorasagridsitestorageelement
AT casteelskevin thedynafeddatafederatorasagridsitestorageelement
AT driemelcolson thedynafeddatafederatorasagridsitestorageelement
AT keebleoliver thedynafeddatafederatorasagridsitestorageelement
AT leavettbrowncolin thedynafeddatafederatorasagridsitestorageelement
AT galindofernandofernandez thedynafeddatafederatorasagridsitestorageelement
AT furanofabrizio thedynafeddatafederatorasagridsitestorageelement
AT patersonmichael thedynafeddatafederatorasagridsitestorageelement
AT seusterrolf thedynafeddatafederatorasagridsitestorageelement
AT sobierandall thedynafeddatafederatorasagridsitestorageelement
AT tafiroutreda thedynafeddatafederatorasagridsitestorageelement
AT ebertmarcus dynafeddatafederatorasagridsitestorageelement
AT berghausfrank dynafeddatafederatorasagridsitestorageelement
AT casteelskevin dynafeddatafederatorasagridsitestorageelement
AT driemelcolson dynafeddatafederatorasagridsitestorageelement
AT keebleoliver dynafeddatafederatorasagridsitestorageelement
AT leavettbrowncolin dynafeddatafederatorasagridsitestorageelement
AT galindofernandofernandez dynafeddatafederatorasagridsitestorageelement
AT furanofabrizio dynafeddatafederatorasagridsitestorageelement
AT patersonmichael dynafeddatafederatorasagridsitestorageelement
AT seusterrolf dynafeddatafederatorasagridsitestorageelement
AT sobierandall dynafeddatafederatorasagridsitestorageelement
AT tafiroutreda dynafeddatafederatorasagridsitestorageelement