Cargando…

DIRAC file replica and metadata catalog

File replica and metadata catalogs are essential parts of any distributed data management system, which are largely determining its functionality and performance. A new File Catalog (DFC) was developed in the framework of the DIRAC Project that combines both replica and metadata catalog functionalit...

Descripción completa

Detalles Bibliográficos
Autores principales: Tsaregorodtsev, A, Poss, S
Lenguaje:eng
Publicado: 2012
Materias:
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/396/3/032108
http://cds.cern.ch/record/1565915
_version_ 1780930947789619200
author Tsaregorodtsev, A
Poss, S
author_facet Tsaregorodtsev, A
Poss, S
author_sort Tsaregorodtsev, A
collection CERN
description File replica and metadata catalogs are essential parts of any distributed data management system, which are largely determining its functionality and performance. A new File Catalog (DFC) was developed in the framework of the DIRAC Project that combines both replica and metadata catalog functionality. The DFC design is based on the practical experience with the data management system of the LHCb Collaboration. It is optimized for the most common patterns of the catalog usage in order to achieve maximum performance from the user perspective. The DFC supports bulk operations for replica queries and allows quick analysis of the storage usage globally and for each Storage Element separately. It supports flexible ACL rules with plug-ins for various policies that can be adopted by a particular community. The DFC catalog allows to store various types of metadata associated with files and directories and to perform efficient queries for the data based on complex metadata combinations. Definition of file ancestor-descendent relation chains is also possible. The DFC catalog is implemented in the general DIRAC distributed computing framework following the standard grid security architecture. In this paper we describe the design of the DFC and its implementation details. The performance measurements are compared with other grid file catalog implementations. The experience of the DFC Catalog usage in the CLIC detector project are discussed.
id cern-1565915
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2012
record_format invenio
spelling cern-15659152022-08-17T13:25:21Zdoi:10.1088/1742-6596/396/3/032108http://cds.cern.ch/record/1565915engTsaregorodtsev, APoss, SDIRAC file replica and metadata catalogComputing and ComputersFile replica and metadata catalogs are essential parts of any distributed data management system, which are largely determining its functionality and performance. A new File Catalog (DFC) was developed in the framework of the DIRAC Project that combines both replica and metadata catalog functionality. The DFC design is based on the practical experience with the data management system of the LHCb Collaboration. It is optimized for the most common patterns of the catalog usage in order to achieve maximum performance from the user perspective. The DFC supports bulk operations for replica queries and allows quick analysis of the storage usage globally and for each Storage Element separately. It supports flexible ACL rules with plug-ins for various policies that can be adopted by a particular community. The DFC catalog allows to store various types of metadata associated with files and directories and to perform efficient queries for the data based on complex metadata combinations. Definition of file ancestor-descendent relation chains is also possible. The DFC catalog is implemented in the general DIRAC distributed computing framework following the standard grid security architecture. In this paper we describe the design of the DFC and its implementation details. The performance measurements are compared with other grid file catalog implementations. The experience of the DFC Catalog usage in the CLIC detector project are discussed.oai:cds.cern.ch:15659152012
spellingShingle Computing and Computers
Tsaregorodtsev, A
Poss, S
DIRAC file replica and metadata catalog
title DIRAC file replica and metadata catalog
title_full DIRAC file replica and metadata catalog
title_fullStr DIRAC file replica and metadata catalog
title_full_unstemmed DIRAC file replica and metadata catalog
title_short DIRAC file replica and metadata catalog
title_sort dirac file replica and metadata catalog
topic Computing and Computers
url https://dx.doi.org/10.1088/1742-6596/396/3/032108
http://cds.cern.ch/record/1565915
work_keys_str_mv AT tsaregorodtseva diracfilereplicaandmetadatacatalog
AT posss diracfilereplicaandmetadatacatalog