FELIX: A high-throughput network approach for interfacing to front end electronics for ATLAS upgrades

The ATLAS experiment at CERN is planning full deployment of a new unified link technology for connecting detector front end electronics on the timescale of the LHC Run 4 (2025). It is estimated that roughly 8000 GBT (GigaBit Transceiver) links, with transfer rates probably up to 9.6 Gbps, will replace existing links used for readout, detector control and distribution of timing and trigger information. In particular the links used for readout are often detector-specific. Already in Run 3 this technology will be deployed in conjunction with new muon detectors, additional muon first-level triggering electronics and new on-detector and off-detector liquid argon calorimeter electronics to be used for first level triggering. A total of roughly 2000 GBT links or GBT-like links (for connecting to off-detector trigger electronics) will be needed. A new class of devices will need to be developed to interface many GBT links to the rest of the trigger, data-acquisition and detector control systems. In this paper we present FELIX, the Front End LInk eXchange. The purpose of FELIX is to route data from and to multiple GBT links via a high-performance general purpose network capable of a total throughput that could be of O(20 Tbps). FELIX is designed to be used as a multi-purpose routing device; data can be forwarded to or from multiple destinations connected to the network, e.g. data-acquisition, monitoring, detector control, calibration, or configuration systems. The system will be capable of handling special bi-directional low-latency channels, such as needed for transferring Timing, Trigger and Control, TTC, signals via GBT links or for GBT-to-GBT Direct-Output-Low-Latency connections to first level trigger electronics. The software layer will integrate support for data sampling, quality-of-service prioritization, load balancing based on a round-robin scheme and automatic failover. FELIX is the central element of a new readout architecture that replaces the legacy point-to-point links with a switched network. The new architecture will be more dynamic, flexible and adaptable. By replacing point-to-point links with a switched network, the number of single points-of-failure can be reduced and a more robust and fault-tolerant system can be built, also implementing traffic equalization schemes. In preparation for the deployment in Run 3 and to support the ongoing detector and trigger electronics development, a FELIX technology demonstrator is planned to be available early 2015, using commercial-off-the-shelf server PC technology in combination with a commercial FPGA-based PCIe Gen3 I/O card interfacing to up to 24 GBT links and with TTC connectivity provided by an FMC-based mezzanine card. Dedicated firmware for the Xilinx Virtex-7 FPGA installed on the I/O card alongside an interrupt-driven Linux kernel driver and user-space software will provide the required functionality. On the network side, the FELIX demonstrator connects to an Ethernet-based network. Extensions to other high-performance network technologies, such as Infiniband are possible and will be discussed in the paper. In this paper we introduce a new approach to interfacing to on-detector and trigger electronics on the basis of GBT link or GBT-like link technology for ATLAS in Run 3 and Run 4, highlighting the innovative elements and advantages. We will then derive the functional and performance requirements on FELIX and present the design and implementation of the FELIX demonstrator. Furthermore, we will show throughput performance results as well as networking and data processing benchmarks. We intend to show that our design is a viable solution for a multi-purpose routing device in the anticipated ATLAS architecture of LHC Run 4.