Building of CRAB Spark Data Pipeline

CMS Remote Analysis Builder (CRAB) has an important role to help CMS users submit analysis jobs to the LHC Computing Grid. CRAB team want to analyze the users usage to prioritize the features, to know which one should be deprecated or should be upgraded. This project focuses on building a the data pipeline to provide a daily update of this kind of information to the CRAB team. The usage data is processed through PySpark, saved into Opensearch and eventually visualized with Grafana. Moreover, manual scripts are provided for the CRAB team to reproduce it later when questions change.