Historian server lost all four of its OPC DA sources at 6.10 Monday morning. Two KEPServerEX boxes, an RSLinx Classic OPC server on the old packaging PLC and a vendor DA server on the compressor. About 8,000 tags, the lot of them bad quality from the same minute.
All four boxes ping and the DA servers show running, local clients on each one still see data. IT pushed Windows updates Sunday night to the historian box and to the two Kepware boxes and nobody thought to tell me so that is the only thing that has changed.
I restarted the collector service twice, nothing, rebooted a Kepware box, nothing, and reconnecting a link by hand gets as far as connecting and then drops.
Four vendors dying at once says it isn't the servers. Is there something in a Windows update that shuts DA out from across the network and leaves the local clients alone?
Are those four links all remote DCOM, historian box to server box? And what's in the historian's Windows event log at 6.10, System and Application? When a link drops on my remote sites it's always in the log, even when the collector says nothing.
All remote DCOM, yes. The System log on the historian box, and on the one Kepware box I checked, both have a run of DistributedCOM 10036 entries from 6.10: "The server-side authentication level policy does not allow the user ... from address ... to activate DCOM server. Please raise the activation authentication level at least to RPC_C_AUTHN_LEVEL_PKT_INTEGRITY in client application."
That's the DCOM hardening, KB5004442. Once the server side enforces it, a DA client coming in below Packet Integrity gets refused. Something in Sunday's batch tipped one side over, and it hits every DA link the same way, which is why all four went at 06:10.
Two things.
1. On the historian box, dcomcnfg, Component Services, My Computer, Properties, Default Properties, set Default Authentication Level to Packet Integrity. Restart the collector.
2. If a server still refuses, do the same on that server box, and set it on the specific OPC server's DCOM entry under DCOM Config too.
Long term, these links belong on OPC UA. There's a writeup on DA versus UA and why DCOM keeps doing this: https://plctr.com/introduction-to-opc-open-platform-communications-for-plc-integration/
Packet Integrity set on the historian box, collector restarted. Both Kepware links came straight back, good quality. RSLinx Classic and the compressor vendor server are still bad, same 10036 in the log as the 6.10 run, so I'm stuck on those two.
RSLinx Classic wants the setting on its own box as well, we had to do both ends for ours. The vendor server on the compressor might be too old to do Packet Integrity at all, in which case it's a UA wrapper or a gateway in front of it, no registry trick fixes that one.
RSLinx came back after Packet Integrity on the packaging PC and a reboot. The compressor server is from 2009 and refused every level, so it's gone behind the Kepware OPC DA client driver on the nearer box and the historian reads it over UA from there. All 8,000 tags good since Tuesday.
So it was the DCOM hardening, DA clients refused below Packet Integrity after Sunday's updates. Authentication level on both ends fixed three, the fourth went behind a UA gateway.
What I still don't get is why it worked until Sunday. IT say the registry override went away years ago, so I'm not sure what was holding it up. Thanks northbay, wandak.
Marking solved.
plctr.com team