The ALICE O2 common driver for the C-RORC and CRU read-out cards

ALICE (A Large Ion Collider Experiment) is the heavy-ion detector designed to study the strongly interacting state of matter realized in relativistic heavy-ion collisions at the CERN Large Hadron Collider (LHC). A major upgrade of the experiment is planned during the 2019-2020 long shutdown. In order to cope with a data rate 100 times higher than during LHC Run 1 and with the continuous read-out of the Time Projection Chamber (TPC), it is necessary to upgrade the Online and Offline Computing to a new common system called O2. The O2 read-out chain will use commodity x86 Linux servers equipped with custom PCIe FPGA-based read-out cards. This paper discusses the driver architecture for the cards that will be used in O2: the PCIe v2 x8, Xilinx Virtex 6 based C-RORC (Common Readout Receiver Card) and the PCIe v3 x16, Intel Arria 10 based CRU (Common Readout Unit). Access to the PCIe cards is provided via three layers of software. Firstly, the low-level PCIe (PCI Express) layer responsible for the userspace interface for low-level operations such as memory mapping the PCIe BAR (Base Address Registers) and creating scatter-gather lists, which is provided by the PDA (Portable Driver Architecture) library developed by the Frankfurt Institute for Advanced Studies (FIAS). Above that sits our userspace driver which implements synchronization, controls the read-out card – e.g. resetting and configuring the card, providing it with bus addresses to transfer data to and checking for data arrival – and presents a uniform, high-level C++ interface that abstracts over the differences between the C-RORC and CRU. This interface – of which direct usage is principally intended for high-performance read-out processes – allows users to configure and use the various aspects of the read-out cards, such as configuration, DMA transfers and commands to the front-end. The top layer consists of a Python wrapper and command-line utilities that are provided to facilitate scripting and executing tasks from a shell, such as card resetting; performing benchmarks; reading or writing registers; and running test suites. Additionally, the paper presents the results of benchmarks in various test environments. Finally, we present our plans for future development, testing and integration.