Experience with highly-parallel software for the storage system of the ATLAS Experiment at CERN

The ATLAS experiment is observing proton-proton collisions delivered by the LHC accelerator at a centre of mass energy of 7 TeV. The ATLAS Trigger and Data Acquisition (TDAQ) system selects interesting events on-line in a three-level trigger system in order to store them at a budgeted rate of several hundred Hz, for an average event size of ~1.2 MB. This paper focuses on the TDAQ data-logging system and in particular on the implementation and performance of a novel SW design, reporting on the effort of exploiting the full power of recently installed multi-core hardware. In this respect, the main challenge presented by the data-logging workload is the conflict between the largely parallel nature of the event processing, especially the recently introduced on-line event-compression, and the constraint of sequential file writing and checksum evaluation. This is furtherly complicated by the necessity of operating in a fully data-driven mode, to cope with continuously evolving trigger and detector configurations. The novel SW design is based on a thread-pool, implemented in C++ using modern parallel programming tools and techniques, as provided by libraries like TBB(1) and Boost(2). Lockless patterns, atomic instructions and concurrent containers have been employed to provide an efficient implementation able to cope with the above requirements. In this paper we report on the design of the new ATLAS on-line storage software. In particular we will discuss our development experience using recent concurrency-oriented libraries. Finally we will show the new system performance with respect to the old, single-threaded software design. (1) http://threadingbuildingblocks.org/ (2) http://www.boost.org/