Assessing the Usefulness of Assurance Cases: an Experience with the CERN Large Hadron Collider

Assurance cases are structured arguments designed to show that a system functions properly in its operational environment. They are mandated by safety standards and are largely used in the industrial domain; however, they are typically proprietary and not publicly available. Therefore, the benefits of assurance case development are usually not rigorously documented, measured, or assessed. In this paper, we present an assurance case for the CERN Large Hadron Collider (LHC) Machine Protection System (MPS). We relied on open-source documentation for its creation and used eliminative argumentation, a well-known methodology for assurance case development. The development involved four authors with considerable experience in assurance case development, three of whom work for Critical System Labs, a small enterprise specializing in assurance case development. The process required approximately three months and led to an assurance case with 506 nodes. The results have been validated with CERN experts. Our experience shows that (a) the effort (cost and time) required to develop our assurance case is negligible compared to the time needed to develop the system, (b) eliminative argumentation helped identify 10 defeaters not detailed in the documentation we used for creation of the assurance case (and in general can identify correct defeaters with high precision and recall). In the paper, describe our experience and also discuss how the LHC assurance case helped accurately identify Key Performance Indicators for the Machine Protection System.