CMS HCAL VTRx-induced communication loss and mitigation

The Compact Muon Solenoid Experiment Phase 1 Hadron Calorimeter upgrade implemented the first large-scale application of VTRxs modules, radiation and magnetic field tolerant optical transceivers. During Run 2 of the Large Hadron Collider, the CMS Hadron Calorimeter endcap experienced a failure of control communication that revealed a variable manufacturing weakness in the VTRx and affected nearly half of the communication links in the Hadron Calorimeter endcaps. The CMS Hadron Calorimeter team provided the first observation of this phenomenon, linked the loss to the VTRx, revealed its temperature dependence, and pioneered the mitigation tactic adopted by other LHC experiments. We will present these aspects of the larger VTRx investigation.