Collecting and Storing Data Flow Monitoring in Elasticsearch

A very large amount of data is produced from the online data flow monitoring for the CMS data acquisition system. However, there are only a small portion of data is stored permanently in the relational database. This is because of the high cost needed while relying on the dedicated infrastructure as well as the issues in its performance itself. A new approach needs to be found in order to confront such a big volume of data known as “Big Data”. The Big Data [1] is the term given to the very large and complex data sets that cannot be handled by the traditional data processing application [2] in terms of capturing, storing, managing, and analyzing. The sheer size of the data [3] in CMS data acquisition system is one of the major challenges, and is the one of the most easily recognized. New technology need to be used as the alternative of the traditional databases initial evaluation to handle this problem as more data need to be stored permanently and can be easily retrieved. This report consists of the introduction of Elasticsearch as well as its application in collecting and storing data flow monitoring.