Came in this morning and the mixer line is down. ControlLogix 1756-L71, 1756-EN2T in slot 1, switch is a Cisco IE 3400. The EN2T display scrolls the IP fine but the link LED is off and the switch port LED is amber.
I moved the EN2T's cable to a spare port on the same switch and it came straight back. Moved it back, dead again. IT's console says err-disabled, "psecure-violation". They did a port security rollout Friday night and the other 30 ports on that switch are fine. A new cable made no difference.
IT says the PLC NIC is faulty and sending "bad MACs". It runs fine on the spare port so I'm not sure about that. I swapped that EN2T for a spare two weeks ago, if that matters. Would a module swap two weeks back leave the old MAC stuck on that port, and would IT be able to see that from their end?
Not a faulty NIC, a NIC doesn't send a bad MAC, it sends its MAC. Ask IT for show port-security interface on that port. It lists the allowed MAC and the last violation MAC. If they differ, there's your answer. Also ask whether it's port security with sticky MAC or 802.1X, they're different things and IT sometimes says one and means the other.
Got it from IT. Port security, sticky. Secure MAC on the port is one address. Last source address on the violation is a different one, and its the one printed on my 1756-EN2T. The secure one, I'm pretty sure that's the old EN2T from two weeks ago. Aging is 0, so it never forgets. Makes sense now.
Two things, in order. 1. That secure MAC is the dead module, so the port's locked to something sitting on a shelf. Sticky learns whatever is live, and Friday your new one was live, so ask IT whether the rollout pushed a MAC list from an older inventory. 2. Every spare swap on every PLC port will do the same until the process changes, which is the real problem. Ask IT for errdisable recovery too, so a violation clears itself after a few minutes instead of a phone call.
IT cleared the violation and did a shut and no shut on the port, it came up, mixer ran the night shift. They also admitted the MAC list came from a scan they did in July, before my swap. Then at 06:00 the cleaning crew power cycled the panel like every morning and it's err-disabled again. Same old MAC listed as secure. So the clear didn't stick, or the sticky did. Either way, no luck.
The shut and no shut clears the violation, not the sticky entry. The old MAC is still in the running config, so the next link up fails the same way. IT needs to remove the stale sticky mac-address line for that port and put the 1756-EN2T's MAC in as a static entry rather than letting it re-learn. Then write the spare procedure down: swap module, send IT the new MAC, they update the port, five minutes. Or exempt the PLC ports from sticky and keep a static allow list per port from the start. There's a writeup on the OT side of this here: https://plctr.com/implementing-cybersecurity-measures-for-plc-systems/
IT removed the stale sticky line, put my EN2T's MAC in as a static entry on that port, and turned on errdisable recovery for psecure-violation at 300 s. Cleaning crew cycled the panel Thursday and Friday, port came up both times.
So IT pushed port security Friday night with a July MAC list, the old EN2T, and my swapped module was the violation. The first clear left the old entry in place. The static MAC and a written swap procedure fixed it. Every spare now goes on a form to IT with its MAC before it goes in.
IT still wants 802.1X on those ports and I've no idea whether an EN2T can even do that. Thanks northbay, eddieb.
Moved to Networks & Comms. Marking solved.
plctr.com team