MASADA: A Modeling and Simulation Automated Data Analysis framework for continuous data-intensive validation of simulation models

Complex networked computer systems are usually subjected to upgrades and enhancements on a continuous basis. Modeling and simulation of such systems helps with guiding their engineering processes, in particular when testing candi- date design alternatives directly on the real system is not an option. Models are built and simulation exercises are run guided by specific research and/or design questions. A vast amount of operational conditions for the real system need to be assumed in order to focus on the relevant questions at hand. A typical boundary condition for computer systems is the exogenously imposed workload. Meanwhile, in typical projects huge amounts of monitoring information are logged and stored with the purpose of studying the system’s performance in search for improvements. Also research questions change as systems’ operational conditions vary throughout its lifetime. This context poses many challenges to determine the validity of simulation models. As the behavioral empirical base of the system grows, the question can be posed of whether a model that was deemed valid at one point in time can be regarded as invalid in the context of novel, unprecedented operation conditions. In this work we present a framework and a prototype tool that can help with answering this question in a systematic and automated way. The idea is to select a systems’ operation time interval and get a simulation automatically parameterized, run, and validated, producing a set of results to determine the degree of approximation against available metrics of the real system. We tested our framework and tool in a real case study for TDAQ, the trigger and data acquisition system of the ATLAS particle physics experiment at CERN’s Large Hadron Collider. The results show that with a modest amount of manual set up efforts MASADA can provide a sound mechanism for validating our simulation model for the TDAQ network on a continuous basis, as new evidence becomes available when new collisions take place at CERN’s collider.