Automating usability of ATLAS Distributed Computing resources

The automation of ATLAS Distributed Computing (ADC) operations is essential to reduce\nmanpower costs and allow performance-enhancing actions, which improve the reliability of\nthe system. In this perspective a crucial case is the automatic exclusion/recovery of ATLAS\ncomputing sites storage resources, which are continuously exploited at the edge of their\ncapabilities. It is challenging to adopt unambiguous decision criteria for storage resources\nwho feature non-homogeneous types, sizes and roles. The recently developed Storage Area\nAutomatic Blacklisting (SAAB) tool has provided a suitable solution, by employing an\ninference algorithm which processes SAM (Site Availability Test) site-by-site SRM tests\noutcome. SAAB accomplishes both the tasks of providing global monitoring as well as\nautomatic operations on single sites. The implementation of the SAAB tool has been the first\nstep in a comprehensive review of the storage areas monitoring and central management at\nall levels. Such review has involved the reordering and optimization of SAM tests deployment \nand the inclusion of SAAB results in the ATLAS Site Status Board with both dedicated metrics\nand views. The final structure allows monitoring the storage resources statuses with fine\ntime-granularity and automatic actions to be taken in foreseen cases, like automatic\nexclusion/recovery and notifications to sites. Hence, the human actions are restricted to\ntickets tracking and exchanging, where and when needed. In this work we show SAAB\nworking principles and features. We present also the decrease of human interactions\nachieved within the ATLAS Computing Operation team. The automation results in a prompt\nreaction to failures, which grants the optimization of resource exploitation.