Data-driven auto-configuration of the ATLAS reconstruction software

The central data reconstruction of the ATLAS experiment of LHC is a very challenging task, involving large-scale computing and a wide variety of data formats, applications and software versions. In 2009-2010, the ATLAS detector has recorded hundreds of millions of collision, single beam and cosmic events, during an unstable commisioning period that gradually evolved to a stable operation mode aiming at physics. In parallel, the ATLAS Collaboration processed comparable amounts of simulated data, and also produced a collection of sophisticated derived datasets. To handle all this complexity, we have developed a powerful data-driven auto-configuration mechanism and a unified configuration interface that provides a lot of flexibility: "Reco_trf". The auto-configuration mechanism consists of inspecting the metadata of each job's input file to automatically derive the configuration parameters relevant for the input format and the requested tasks. This also simplifies considerably the configuration of jobs from ordinary users, who can use the same script to run without modification on real or simulated data, on files belonging to different major production, using raw or derived input data of any format. Possible intermediate algorithms are automatically scheduled according to the content of the input file. Reco_trf is a so-called "job transformation" interface used for all centralized production tasks at CERN's Tier0 and on the Grid, and is also largely used by normal users. Reco_trf adds a lot of flexibility in the Production systems by allowing the execution of arbitrary python commands without building new software releases, while still bookkeeping this information in the production databases.