Building and Using Containers on HPCs for the ATLAS Experiment

The HPC environment presents several challenges to the ATLAS experiment in running their automated computational workflow smoothly and efficiently, in particular on issues such as software distribution and IO load. A vital component of the LHC Computing Grid, CVMFS is not always available in HPC environments. ATLAS computing has experimented with all-inclusive containers, and later developed an environment to produce such containers for both Shifter and Singularity. The all-inclusive containers include most of the recent ATLAS software releases, database releases, and other tools extracted from CVMFS. This helped ATLAS to distribute software automatically to HPCs with an environment identical to those in CVMFS. It also significantly reduced the metadata I/O load to HPCs’ shared file systems. The production operation at NERSC has proved that by using this type of containers, we can both transparently fit into the previously developed ATLAS operation methods, and at the same time scale up to run many more jobs.