Enhancements in Functionality of the Interactive Visual Explorer for ATLAS Computing Metadata

The development of the Interactive Visual Explorer (InVEx), a visual analytics tool for ATLAS computing metadata, includes research of various approaches for data handling both on server and on client sides. InVEx is implemented as a web-based application which aims at the enhancing of analytical and visualization capabilities of the existing monitoring tools, and facilitate the process of data analysis with the interactivity and human supervision. The development of InVEx started with the implementation of a 3-dimensional interactive tool for cluster analysis (for the k-means and DBSCAN algorithms), and its further evolvement is closely linked to the needs of ATLAS computing experts, providing metadata analysis to ensure the stability and efficiency of the distributed computing environment functionality. In the process of the integration of InVEx with ATLAS computing metadata sources we faced two main challenges: 1) big data volumes needed to be analyzed in the real time mode (as an example, one ATLAS computing task may contain tens of thousands jobs, each having over two hundred of different parameters), and 2) machine learning clustering algorithms alone are not sufficient for visual cluster analysis - the ability of user-defined clusterization/grouping (by nominal or ordinal parameters) should be added, to make the process of data analysis more manageable. The current work is focused on the architecture enhancements of the InVEx application. First, we will describe the user-manageable data preparation method for cluster analysis. Then, we will present the Level-of-Detail method for the interactive visual data analysis. Beginning with the low detalization, when all data are grouped (by clusterization algorithms or by parameters) and aggregated, we provide users with means to look deeply into this data, incrementally increasing the level of detalization. And finally, the development of data storage format for InVEx is adapted for the Level-of-Detail method to keep all stages of data derivation sequence.