Xrootd Monitoring for the CMS experiment

During spring and summer 2011 CMS deployed Xrootd front-end servers on all US T1 and T2 sites. This allows for remote access to all experiment data and is used for user-analysis, visualization, running of jobs at T2s and T3s when data is not available at local sites, and as a fail-over mechanism for data-access in CMSSW jobs. Monitoring of Xrootd infrastructure is implemented on three levels. On the first level, service and data availability checks are performed by Nagios probes. The second level uses Xrootd report stream; a relatively simple stream processor is used to aggregate data from all sites and to feed the needed data into MonALISA service and further into MonALISA repository providing web interface and long-term storage. The third level uses detailed monitoring stream of Xrootd servers configured to include detailed information about users, opened files and individual data transfers. A custom application was developed in C++ to process this information and to, first, provide a real-time view of the system usage and, second, to store data into ROOT trees for detailed analysis. Detailed monitoring allows us to determine hot data-samples, to detect abuses of the system, including sub-optimal usage of the Xrootd protocol and ROOT tree-caching mechanism. Data from all three levels is also exported to CMS monitoring aggregators, Dashboard and Data Popularity Framework.