ART - ATLAS Release Tester using the Grid

The ART system is designed to run test jobs on the Grid after an ATLAS nightly release has been built. The choice was taken to exploit the Grid as a backend as it offers a huge resource pool, suitable for a deep set of integration tests, and running the tests could be delegated to the highly scalable ATLAS production system (PanDA). The challenge of enabling the Grid as a test environment is met through the use of the CVMFS file system for the software and input data files. Test jobs are submitted to the Grid by the gitlab-ci system which itself is triggered at end of a release build. Jobs can be adorned with special headers that direct the system how to run the specific test, allowing many options to be customised. The gitlab-ci waits for exit status and output files and copied back from the Grid to an EOS area accessible by the users. All gittlab-ci jobs run in ART's virtual machines using docker images for their ATLAS setup. ART jobs can be tracked by using the PanDA system. ART can also be used to run short test jobs locally. It uses the same art command-line interface, where the back-end is replaced to access a local machine for job submission rather than the Grid. This allows developers to ensure their tests work correctly before adding them to the system. In both the Grid and local machine options, running and result copying is completely parallelized. ART is written in python, complete with its own local and Grid tests to give approximately 90% code coverage of the ART tool itself. ART is in production since one year now and full replaces and augments the former system.