Utilizing HPC Network Technologies in High Energy Physics Experiments

Because of their performance characteristics high-performance fabrics like Infiniband or OmniPath are interesting technologies for many local area network applications, including data acquisition systems for high-energy physics experiments like the ATLAS experiment at CERN. This paper analyzes existing APIs for high-performance fabrics and evaluates their suitability for data acquisition systems in terms of performance and domain applicability. The study finds that existing software APIs for high-performance interconnects are focused on applications in high-performance computing with specific workloads and are not compatible with the requirements of data acquisition systems. To evaluate the use of high-performance interconnects in data acquisition systems a custom library, NetIO, is presented and compared against existing technologies. NetIO has a message queue-like interface which matches the ATLAS use case better than traditional HPC APIs like MPI. The architecture of NetIO is based on a interchangeable back-end system with which different interconnects can be supported. A libfabric-based back-end supports a wide range of fabric technologies including Infiniband. On the front-end side NetIO supports several high-level communication patterns that are found in typical data acquisition applications like client/server and publish/subscribe. Unlike other frameworks NetIO distinguishes between high-throughput and low-latency communication, which is essential for applications with heterogeneous traffic patterns like data acquisition systems. Benchmarks of NetIO in comparison with the message queue implementation ZeroMQ are presented. NetIO reaches up to 2x higher throughput than ZeroMQ on Ethernet and higher throughput on FDR Infiniband. Latencies are competitive in comparison with ZeroMQ.