QA and Testing in CERNBOX: the cornerstone of service development and operation

<!--HTML-->QA and Testing in CERNBOX (https://cernbox.cern.ch) presents a serious challenge and is critically important: - the service deals with user data directly on their local computers (synchronization clients), at present ~1000 clients connecting daily - the very nature of synchronization is to propagate changes across computers which also means propagating problems if they occur - there is a wide range of supported platforms (MacOSX, Windows, major Linux distributions, mobile platforms) - OS semantics are often incompatible or conflicting: for example HFS is case preserving, NTFS supports the legacy of 8.3 DOS file format, etc. - the operational environment varies enormously, for example: from fast, reliable network inside computing center to unreliable, high-latency, ad-hoc connections from airports - etc. From service development perspective, CERNBOX integrates several complex components which development cycles are disjoint and geographically distributed: - PB-range storage backend (EOS) developed and operated in-house - synchronization clients and web-access layer (ownCloud) developed externally To handle some of the development, integration and operational challenges we have developed a testing framework called Smashbox. In this presentation I will summarize how we use Smashbox for QA and Testing of critical data handling paths of CERNBOX with some real-life examples. References: - Source code: https://github.com/cernbox/smashbox - Mysteries of Dropbox: https://indico.cern.ch/event/336753/session/1/contribution/28 - Smashbox in action: https://owncloud.org/blog/smashbox-in-action