Large-scale upgrade campaigns of SCADA systems at CERN - organisation, tools and lessons learned

The paper describes planning and execution of large-scale maintenance campaigns of SCADA systems for CERN's accelerator and technical infrastructure. These activities, required to keep up with the pace of development of the controlled systems and rapid evolution of software, are constrained by many factors, such as availability for operation and planned interventions on equipment. Experience gathered throughout the past ten years of maintenance campaigns for the SCADA Applications Service at CERN, covering over 230 systems distributed across almost 120 servers, is presented. Further improvements for the procedures and tools are proposed to adapt to the increasing number of applications in the service and reduce maintenance effort and required downtime.