Cargando…

Dynamic federations: storage aggregation using open tools and protocols

A number of storage elements now offer standard protocol interfaces like NFS 4.1/pNFS and WebDAV, for access to their data repositories, in line with the standardization effort of the European Middleware Initiative (EMI). Also the LCG FileCatalogue (LFC) can offer such features. Here we report on wo...

Descripción completa

Detalles Bibliográficos
Autores principales: Fabrizio Furano, F F, Patrick Fuhrmann, P F, Ricardo Brito da Rocha, R R, Adrien Devresse, A D, Oliver Keeble, O K, Alejandro Alvarez Ayllon, A A
Lenguaje:eng
Publicado: 2012
Materias:
Acceso en línea:http://cds.cern.ch/record/1460525
_version_ 1780925233223434240
author Fabrizio Furano, F F
Patrick Fuhrmann, P F
Ricardo Brito da Rocha, R R
Adrien Devresse, A D
Oliver Keeble, O K
Alejandro Alvarez Ayllon, A A
author_facet Fabrizio Furano, F F
Patrick Fuhrmann, P F
Ricardo Brito da Rocha, R R
Adrien Devresse, A D
Oliver Keeble, O K
Alejandro Alvarez Ayllon, A A
author_sort Fabrizio Furano, F F
collection CERN
description A number of storage elements now offer standard protocol interfaces like NFS 4.1/pNFS and WebDAV, for access to their data repositories, in line with the standardization effort of the European Middleware Initiative (EMI). Also the LCG FileCatalogue (LFC) can offer such features. Here we report on work that seeks to exploit the federation potential of these protocols and build a system that offers a unique view of the storage and metadata ensemble and the possibility of integration of other compatible resources such as those from cloud providers. The challenge, here undertaken by the providers of dCache and DPM, and pragmatically open to other Grid and Cloud storage solutions, is to build such a system while being able to accommodate name translations from existing catalogues (e.g. LFCs), experiment- based metadata catalogues, or stateless algorithmic name translations, also known as ”trivial file catalogues”. Such so-called storage federations of standard protocols-based storage elements give a unique view of their content, thus promoting simplicity in accessing the data they contain and offering new possibilities for resilience and data placement strategies. The goal is to consider HTTP and NFS4.1-based storage elements and metadata catalogues and make them able to cooperate through an architecture that properly feeds the redirection mechanisms that they are based upon, thus giving the functionalities of a ”loosely coupled” storage federation. One of the key requirements is to use standard clients (provided by OS’es or open source distributions, e.g. Web browsers) to access an already aggregated system; this approach is quite different from aggregating the repositories at the client side through some wrapper API, like for instance GFAL, or by developing new custom clients. Other technical challenges that will determine the success of this initiative include performance, latency and scalability, and the ability to create worldwide storage federations that are able to redirect clients to repositories that they can efficiently access, for instance trying to choose the endpoints that are closer or applying other criteria. We believe that the features of a loosely coupled federation of open-protocols-based storage elements will open many possibilities of evolving the current computing models without disrupting them, and, at the same time, will be able to operate with the existing infrastructures, follow their evolution path and add storage centers that can be acquired as a third-party service.
id cern-1460525
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2012
record_format invenio
spelling cern-14605252022-08-17T13:27:58Zhttp://cds.cern.ch/record/1460525engFabrizio Furano, F FPatrick Fuhrmann, P FRicardo Brito da Rocha, R RAdrien Devresse, A DOliver Keeble, O KAlejandro Alvarez Ayllon, A ADynamic federations: storage aggregation using open tools and protocolsComputing and ComputersA number of storage elements now offer standard protocol interfaces like NFS 4.1/pNFS and WebDAV, for access to their data repositories, in line with the standardization effort of the European Middleware Initiative (EMI). Also the LCG FileCatalogue (LFC) can offer such features. Here we report on work that seeks to exploit the federation potential of these protocols and build a system that offers a unique view of the storage and metadata ensemble and the possibility of integration of other compatible resources such as those from cloud providers. The challenge, here undertaken by the providers of dCache and DPM, and pragmatically open to other Grid and Cloud storage solutions, is to build such a system while being able to accommodate name translations from existing catalogues (e.g. LFCs), experiment- based metadata catalogues, or stateless algorithmic name translations, also known as ”trivial file catalogues”. Such so-called storage federations of standard protocols-based storage elements give a unique view of their content, thus promoting simplicity in accessing the data they contain and offering new possibilities for resilience and data placement strategies. The goal is to consider HTTP and NFS4.1-based storage elements and metadata catalogues and make them able to cooperate through an architecture that properly feeds the redirection mechanisms that they are based upon, thus giving the functionalities of a ”loosely coupled” storage federation. One of the key requirements is to use standard clients (provided by OS’es or open source distributions, e.g. Web browsers) to access an already aggregated system; this approach is quite different from aggregating the repositories at the client side through some wrapper API, like for instance GFAL, or by developing new custom clients. Other technical challenges that will determine the success of this initiative include performance, latency and scalability, and the ability to create worldwide storage federations that are able to redirect clients to repositories that they can efficiently access, for instance trying to choose the endpoints that are closer or applying other criteria. We believe that the features of a loosely coupled federation of open-protocols-based storage elements will open many possibilities of evolving the current computing models without disrupting them, and, at the same time, will be able to operate with the existing infrastructures, follow their evolution path and add storage centers that can be acquired as a third-party service.CERN-IT-Note-2012-025Submitted to Computing in High Energy and Nuclear Physics (CHEP) 2012oai:cds.cern.ch:14605252012-07-09
spellingShingle Computing and Computers
Fabrizio Furano, F F
Patrick Fuhrmann, P F
Ricardo Brito da Rocha, R R
Adrien Devresse, A D
Oliver Keeble, O K
Alejandro Alvarez Ayllon, A A
Dynamic federations: storage aggregation using open tools and protocols
title Dynamic federations: storage aggregation using open tools and protocols
title_full Dynamic federations: storage aggregation using open tools and protocols
title_fullStr Dynamic federations: storage aggregation using open tools and protocols
title_full_unstemmed Dynamic federations: storage aggregation using open tools and protocols
title_short Dynamic federations: storage aggregation using open tools and protocols
title_sort dynamic federations: storage aggregation using open tools and protocols
topic Computing and Computers
url http://cds.cern.ch/record/1460525
work_keys_str_mv AT fabriziofuranoff dynamicfederationsstorageaggregationusingopentoolsandprotocols
AT patrickfuhrmannpf dynamicfederationsstorageaggregationusingopentoolsandprotocols
AT ricardobritodarocharr dynamicfederationsstorageaggregationusingopentoolsandprotocols
AT adriendevressead dynamicfederationsstorageaggregationusingopentoolsandprotocols
AT oliverkeebleok dynamicfederationsstorageaggregationusingopentoolsandprotocols
AT alejandroalvarezayllonaa dynamicfederationsstorageaggregationusingopentoolsandprotocols