CERNBox logs ingestion and analysis

CERNBox is a platform that provides cloud data storage to CERN users. The files in CERNBox are stored in the EOS storage system. This service is built upon hundreds of physical machines. CERNBox is being migrated to containers technology in order to enhance operations and deployment of new software versions. A benefit of it, is that one can run multiple containers of the same daemon to achieve redundancy and high-availability of the service. Additionally, containers are very light, they start in matter of seconds, and use a portion of the disk and memory speed. Logging is, in the simplest sense, about recording discrete events. It is an essential part for the monitoring process as they provide detailed information about events and errors which can help to understand what happens inside the cluster and how is the flow of events, so we can spot the different responses of the server and check when error occur. All the CERNBox service components have been migrated to containers in order to experiment with container technologies and evaluate their use in production scenarios. At the time, logging is confined to the single containers scratch space, while it is needed to centralize logs instead. Most of container services provide their own logging mechanism, but usually this is not enough for a complete logging solution. For example, in case of a container crash, you won't be able to access your application's logs. As such, it's required that logs are stored in a central location, and have a life-cycle separately from nodes and containers. Also, it's necessary to use a centrally provided monitoring system. Logging is already enabled for three CERNBox components: cboxredirectd, nginx-eos and nginx-web. As a design principle, log ingestion must be decoupled from the application, thus it will be required only a single multi-purpose log ingestion service for all the CERNBox components. This translates into having a single Docker image to handle all the CERNBox components logs and their ingestion to the central monitoring infrastructure. After saving the logs, there's another question rising: What are the users doing? For answering this, it's necessary to start digging into the logs data.