The ATLAS Data Carousel Project Status and Plans

The High Luminosity upgrade to the LHC, which aims for a ten-fold increase in the luminosity of proton-proton collisions at an energy of 14 TeV, is expected to start operation in 2028/29, and will deliver an unprecedented volume of scientific data at the multi-exabyte scale. This amount of data has to be stored and the corresponding storage system must ensure fast and reliable data delivery for processing by scientific groups distributed all over the world. The present LHC computing and data management model will not be able to provide the required infrastructure growth even taking into account the expected hardware technology evolution. To address this challenge, the Data Carousel R&D project was launched by the ATLAS experiment in the fall of 2018. By Data Carousel we mean on-demand reading of selected data from tape without pre-staging. Data Carousel uses a sliding window buffer whose size can be tuned to suit available resources and production requirements. The Data Carousel in ATLAS is the orchestration between the workflow management systems, the distributed data management and the tape services. We successfully and quickly passed the R&D project phases involving FTS, dCache, CTA, Rucio, PanDA/JEDI, ATLAS Computing Operations and the WLCG centers. Our current goal is to simultaneously run major ATLAS production workflows in Data Carousel mode with respect to dynamic computing shares and sliding window size. We are also working on tape throughput estimation, in anticipation for HL-LHC. Joint-tape throughput tests with other LHC experiments have also been conducted. Data Carousel technology may be applicable to other scientific communities, such as SKA, DUNE, Vera Rubin Observatory, BELLE II, and NICA to manage large-scale data volumes between different QoS storage elements. State-of-the-art data and workflow management technologies are under active development and their status will be presented, as well as the ATLAS data carousel plans.