Reliability Engineering for ATLAS Petascale Data Processing on the Grid

The ATLAS detector is in its third year of continuous LHC running taking data for physics analysis. A starting point for ATLAS physics analysis is reconstruction of the raw data. First-pass processing takes place shortly after data taking, followed later by reprocessing of the raw data with updated software and calibrations to improve the quality of the reconstructed data for physics analysis. Data reprocessing involves a significant commitment of computing resources and is conducted on the Grid. The reconstruction of one petabyte of ATLAS data with 1B collision events from the LHC takes about three million core-hours. Petascale data processing on the Grid involves millions of data processing jobs. At such scales, the reprocessing must handle a continuous stream of failures. Automatic job resubmission recovers transient failures at the cost of CPU time used by the failed jobs. Orchestrating ATLAS data processing applications to ensure efficient usage of tens of thousands of CPU-cores, reliability engineering minimizes the reprocessing duration and failure recovery costs. In 2010 reprocessing, the cost to recover transient failures was 6% of the CPU time used for reconstruction. In 2011 reprocessing, the cost used to recover transient failures was reduced to 4% of the CPU time used for the reconstruction. We present reliability engineering analysis of the ATLAS petascale data processing on the Grid.