Experience with Shifter Assistant

The Trigger and DAQ (TDAQ) system of the ATLAS experiment is a complex distributed computing system, composed of O(60000) of applications running on more than 2500 computers. The system is operated by a crew of operators on shift. An important aspect of operations is to minimize the downtime of the system caused by runtime failures, such as human errors, unawareness, miscommunication, etc. The paper describes recent developments in one of “intelligent” TDAQ frameworks, the Shifter Assistant (SA) and summarizes the experience of it’s use in operations of ATLAS in the course of LHC Run 2. SA is a framework whose main aim is to automatize routine system checks, error detection and diagnosis, events correlation etc. in order to help the operators to react on runtime problems promptly and effectively. The tool is based on CEP (Complex Event Processing) technology. It constantly processes this stream of operational events (O(100kHz)) over a set of “directives” (or rules) in the knowledge base, producing human-oriented alerts and making shifters aware about the operational issues. More then 200 directives were developed by TDAQ and ATLAS detector experts for different domains. In this paper we also describe different types of directives which were developed in course of Run 2, and present few examples of most interesting ones, demonstrating the power of CEP for this type of applications.