CMSSW Performance Monitoring on processors based on the Intel Core and Nehalem Microarchitectures

CPU clock frequency is not likely to be increased significantly in the coming years, and data analysis speed can be improved by using more processors or buying new machines, only if one is willing to change the programming paradigm to a parallel one. Therefore, performance monitoring procedures and tools are needed to help programmers to optimize existing software running on current and future hardware, without having to redesign it completely. Low level information from hardware performance counters is vital to spot specific performance problems slowing program execution. HEP software is often huge and complex, and existing tools are unable to give results with the required granularity. I will report on the approach I have chosen to solve this problem on CMSSW. It involves decomposing the application into parts and monitoring each one of them separately. Both counting and sampling methods are used to allow an analysis with the required custom granularity: from global level, up to the function level. A set of tools (based on perfmon2 - a software interface to hardware counters) has been developed and deployed. This document is a report concerning the work done at CERN on performance monitoring of CMSSW, using a 4-way analysis based on hardware performance counters. I will present the Intel Core and Nehalem microarchitectures and the methodology used to analyse the software, the Cycle Accounting Analysis. Next I will describe the 4 analysis approaches used (Overall, Symbol, Module Level and Modular Symbol Level analysis) and their complementary usefulness in performance evaluation. A few paragraphs are dedicated to a basic explanation of the inner workings of the tools implemented