Cargando…

The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data

Modern scientific experiments collect vast amounts of data that must be catalogued to meet multiple use cases and search criteria. In particular, high-energy physics experiments currently in operation produce several billion events per year. A database with the references to the files including each...

Descripción completa

Detalles Bibliográficos
Autores principales: Barberis, D, Cranshaw, J, Dimitrov, G, Favareto, A, Fernández Casaní, A, González de la Hoz, S, Hřivnáč, J, Malon, D, Nowak, M, Salt Cairols, J, Sánchez, J, Sorokoletov, R, Zhang, Q
Lenguaje:eng
Publicado: 2013
Materias:
Acceso en línea:http://cds.cern.ch/record/1606537
Descripción
Sumario:Modern scientific experiments collect vast amounts of data that must be catalogued to meet multiple use cases and search criteria. In particular, high-energy physics experiments currently in operation produce several billion events per year. A database with the references to the files including each event in every stage of processing is necessary in order to retrieve the selected events from data storage systems. The ATLAS EventIndex project is studying the best way to store the necessary information using modern data storage technologies (Hadoop, HBase etc.) that allow saving in memory key-value pairs and select the best tools to support this application from the point of view of performance, robustness and ease of use. This paper describes the initial design and performance tests and the project evolution towards deployment and operation during 2014.