Networks for the ATLAS LHC Detector: Requirements, Design and Validation

ATLAS (A Toroidal LHC ApparatuS) is one of the four experiments approved for operation on the LHC (Large Hadron Collider) accelerator at CERN. Inside the ATLAS detector, the proton-proton bunch crossings will occur at approximately 40 MHz, generating about 1.6 Mbyte data for each event. The handling of the resulting data rate (64 Tbyte/s) is a serious challenge. As most of the collisions are "ordinary" (i.e. contain irrelevant physics data) a three layer Trigger and Data Acquisition system (TDAQ) is used to select the interesting events in real-time before recording them on mass storage. The total data rate which needs to be permanently stored is thus reduced to approximately 320 Mbyte/s. The second level trigger is a distributed system, implemented using approximately 1000 end-nodes interconnected by a high speed Ethernet network (the DataFlow network). It receives event data at a rate of up to 100 kHz, analyzes it, and reduces the rate to 3.3 kHz; a cross-sectional network bandwidth of approximately 10 Gbyte/s is required for this task. As opposed to general purpose networks (e.g. campus or enterprise networks), tight requirements with respect to message delivery (loss rate and latency) must be met by the ATLAS DataFlow network, in order to assure the real-time operation of the trigger and data acquisition system. Both packet loss and latencies in the network should be minimized in order to efficiently exploit the processing power and buffering resources deployed in the TDAQ system. The contribution of this thesis relies in the design, testing and validation of this complex high-speed Ethernet network. The design of the network has been tightly coupled with that of the TDAQ applications relying on it for passing control and data messages. We have established the traffic patterns and analytical queuing models that allow a good prediction of the network behaviour (in terms of packet loss and latency) as a function of the average utilization of the links and buffering capability of the switching devices. We have identified the Ethernet features that characterize the performance of switches from the TDAQ perspective. Requirements have been derived, and a comprehensive methodology for ascertaining the compliance of devices to our requirements has been established. Furthermore, we have carried out measurements that validated the DataFlow system performance and scalability on a fully functional 10% scale system.