The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data

Mostrar otras versiones (3)

Modern scientific experiments collect vast amounts of data that must be catalogued to meet multiple use cases and search criteria. In particular, high-energy physics experiments currently in operation produce several billion events per year. A database with the references to the files including each...

Descripción completa

Detalles Bibliográficos
Autores principales: Barberis, D, Cranshaw, J, Dimitrov, G, Favareto, A, Fernandez Casani, A, Gonzalez de la Hoz, S, Hrivnac, J, Malon, D, Nowak, M, Salt Cairols, J, Sanchez, J, Sorokoletov, R, Zhang, Q
Lenguaje:eng
Publicado: 2013
Materias:
Detectors and Experimental Techniques
Acceso en línea:https://dx.doi.org/10.1088/1742-6596/513/4/042002
http://cds.cern.ch/record/1606543
_version_ 1780931688891678720
author Barberis, D
Cranshaw, J
Dimitrov, G
Favareto, A
Fernandez Casani, A
Gonzalez de la Hoz, S
Hrivnac, J
Malon, D
Nowak, M
Salt Cairols, J
Sanchez, J
Sorokoletov, R
Zhang, Q
author_facet Barberis, D
Cranshaw, J
Dimitrov, G
Favareto, A
Fernandez Casani, A
Gonzalez de la Hoz, S
Hrivnac, J
Malon, D
Nowak, M
Salt Cairols, J
Sanchez, J
Sorokoletov, R
Zhang, Q
author_sort Barberis, D
collection CERN
description Modern scientific experiments collect vast amounts of data that must be catalogued to meet multiple use cases and search criteria. In particular, high-energy physics experiments currently in operation produce several billion events per year. A database with the references to the files including each event in every stage of processing is necessary in order to retrieve the selected events from data storage systems. The ATLAS EventIndex project is studying the best way to store the necessary information using modern data storage technologies (Hadoop, HBase etc.) that allow saving in memory key-value pairs and select the best tools to support this application from the point of view of performance, robustness and ease of use. This paper describes the initial design and performance tests and the project evolution towards deployment and operation during 2014.
id cern-1606543
institution Organización Europea para la Investigación Nuclear
language eng
publishDate 2013
record_format invenio
spelling cern-16065432019-09-30T06:29:59Zdoi:10.1088/1742-6596/513/4/042002http://cds.cern.ch/record/1606543engBarberis, DCranshaw, JDimitrov, GFavareto, AFernandez Casani, AGonzalez de la Hoz, SHrivnac, JMalon, DNowak, MSalt Cairols, JSanchez, JSorokoletov, RZhang, QThe ATLAS EventIndex: an event catalogue for experiments collecting large amounts of dataDetectors and Experimental TechniquesModern scientific experiments collect vast amounts of data that must be catalogued to meet multiple use cases and search criteria. In particular, high-energy physics experiments currently in operation produce several billion events per year. A database with the references to the files including each event in every stage of processing is necessary in order to retrieve the selected events from data storage systems. The ATLAS EventIndex project is studying the best way to store the necessary information using modern data storage technologies (Hadoop, HBase etc.) that allow saving in memory key-value pairs and select the best tools to support this application from the point of view of performance, robustness and ease of use. This paper describes the initial design and performance tests and the project evolution towards deployment and operation during 2014.ATL-SOFT-PROC-2013-012oai:cds.cern.ch:16065432013-10-07
spellingShingle Detectors and Experimental Techniques
Barberis, D
Cranshaw, J
Dimitrov, G
Favareto, A
Fernandez Casani, A
Gonzalez de la Hoz, S
Hrivnac, J
Malon, D
Nowak, M
Salt Cairols, J
Sanchez, J
Sorokoletov, R
Zhang, Q
The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
title The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
title_full The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
title_fullStr The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
title_full_unstemmed The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
title_short The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
title_sort atlas eventindex: an event catalogue for experiments collecting large amounts of data
topic Detectors and Experimental Techniques
url https://dx.doi.org/10.1088/1742-6596/513/4/042002
http://cds.cern.ch/record/1606543
work_keys_str_mv AT barberisd theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT cranshawj theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT dimitrovg theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT favaretoa theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT fernandezcasania theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT gonzalezdelahozs theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT hrivnacj theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT malond theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT nowakm theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT saltcairolsj theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT sanchezj theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT sorokoletovr theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT zhangq theatlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT barberisd atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT cranshawj atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT dimitrovg atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT favaretoa atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT fernandezcasania atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT gonzalezdelahozs atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT hrivnacj atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT malond atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT nowakm atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT saltcairolsj atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT sanchezj atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT sorokoletovr atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata
AT zhangq atlaseventindexaneventcatalogueforexperimentscollectinglargeamountsofdata

Ejemplares similares