Development of a container-based virtual appliance for developing and running HEP data analyses at CERN

The High Energy Physics (HEP) research community studies fundamental physics by analyzing petabytes of data produced in particle colliders and taken by particle detectors every year. To process this amount of data, the analysis code developed by scientists is executed on a global network of computing resources. Scientific software maintainers (so-called “software librarians”) bundle a compatible set of needed scientific software into a versioned release. These large and frequently changing stacks are commonly distributed to scientists and worker nodes using the CernVM-File System (CernVM-FS). This distributed file system enables on-demand loading of binaries installed on centrally managed software repositories. The minimal Linux image CernVM provides a portable and reproducible runtime environment for developing and running scientific software. Its key ingredient is the tight coupling with the CernVM-FS client to provide access to the base platform (operating system and tools) as well as the scientific software. To his day, CernVM images are designed to use full virtualization. The goal of this project is to develop a new version of CernVM, CernVM 5, that should deliver all the benefits of the CernVM appliance and should be equally practical as a Linux container and as a full virtual machine. To this end, the CernVM 5 container image consists of a “Just Enough Operating System (JeOS)”, with its contents defined by the HEP_OSlibs meta-package which is commonly used as a base platform in HEP. CernVM 5 further aims to smoothly integrate the CernVM-FS client in various container runtimes. Lastly, CernVM 5 uses special build tools and post-build processing to ensure that scientific software stacks using their custom compilers and build chains can coexist with standard system applications. As a result, CernVM 5 aims to provide a single, minimal container image that can be used as a virtual appliance for mounting the CernVM-FS client and for running and developing HEP application software.