Optimising the HLT Farm at the LHCb Experiment

The LHCb experiment at CERN uses a server farm to filter the events consisting of up to 4400 cores, which receives events at a rate of 1MHz and filters them with a trigger application that has an output of around 2 kHz. Configuring and starting O(5000) instances of a very large trigger application, each consisting of hundreds of shared libraries is a formidable task and for the efficient operation of the experiment it is essential that this can be done very quickly and reliably. The architecture of the system and the mechanisms for distributing the software used by the trigger application affect the overall performance and the flexibility to make changes and run the applications with different trigger configurations. We analysed the performance of different alternatives for distributing the software in the farm and we evaluated them in the context of the real world tasks that are executed and also in terms of the overhead in the system, such as the network traffic and file system operations. We present the strategies that allowed us to optimize the configuration and the tools that can be used to perform this kind of benchmarks in similar systems.