Yesterday afternoon cells 3 to 6 went down at the same time. They all hang off one Stratix 5700 in the MCC room. HMIs went grey, each cell has a CompactLogix 1769 L33ER and they all lost their remote I/O, 16#0204 on everything. Every port light on the switch was flashing solid, all of them at once.
A contractor was in the room running a new drop for a label printer. He had patched a cable between two ports on that same Stratix to test the run. When we pulled that cable everything came back inside a minute and it has been fine since.
Nobody documented this plant so I do not know the firmware on anything. What I tried: pulled the patch, power cycled the switch after, checked every cell came back.
My guess is the switch has a fault and the extra cable pushed it over, or the printer PC brought a virus in. Should we replace the Stratix?