Monitoring the data flow of LHCb’s data acquisition system

The data acquisition system of the Large Hadron Collider beauty (LHCb) experiment need to read out huge amount of data. Monitoring is done for each subsystem but there exist no system to monitor the overall data flow. The aim of this work has been to design a system where the data rates can be vied continuously and making it possible to do an exact consistency check after the run to ensure no data are lost. This involves collecting and processing all necessary data from each subsystem and integrate it into the experiment control system for displaying it to the operators. The challenges are to communicate and collect data from all stages of the data acquisitions system which uses different techniques and data formats. The size of the system also makes it a challenge to gather all statistics in real time. The system must also be able to support partitioning. The result was to build a data flow monitoring system, that acquire statistics from all stages of the data acquisition, process it and display it in the experiment control system of LHCb. The system can be used to locate and troubleshoot problems in the data acquisition, but also makes it possible to make a consistency check that the data amounts are the same at all stages to ensure no data are lost. The statistics is collected using a message passing tool called Distributed Information Management System (DIM), which proved to be effective and flexible in this kind of distri buted environments. The statistics are displayed using a main panel to give an overview of the current data flow in the system, and several subpanels to give detailed statistics at each stage. The project is successful and the tool is used daily to monitor the data acquisition in LHCb.