Five stages drawn left to right showing a finished pallet moving from the rung that counts it, through the result area in the controller, the OPC UA server, the MES adapter and into the MES, with four numbered red break points marking where the result or its acknowledgement can be lost

MES Handshakes: Work Order Down, Result Up, and the Acknowledgement Nobody Designs

by

MES Handshakes: Work Order Down, Result Up, and the Acknowledgement Nobody Designs

The MES was unreachable from 01:40 to 05:00 and case packer CP-3 ran the whole way through it. Eighteen pallets came off in those 200 minutes, 864 cases, and when the MES came back it had a record of one of them.

So, the answer first. An MES handshake is three things, not two: a work order coming down, a result going up, and a decision about what the machine does when the result is sent and nothing comes back. The third one has exactly three answers — hold, buffer and retry, or carry on and forget — each defensible, each with a different failure, and the one you get by not choosing is the one that quietly deletes records.

CP-3 had the third. Nobody picked it.

This is the case packer on the carton line behind transfer TR-14, the two-belt machine the January articles worked through. A 1756-L83E carries all four machines on that line; CP-3’s logic sits in a 20 ms periodic task, it takes cartons off the transfer’s discharge and builds a pallet of 48 cases every 11 minutes or so, and the MES is a room away over a link that is up something like 99.7 per cent of a month. That last number is the one people quote when they decide the acknowledgement does not need designing. Three tenths of a per cent of a month is a little over two hours.

Five stages drawn left to right showing a finished pallet moving from the rung that counts it, through the result area in the controller, the OPC UA server, the MES adapter and into the MES, with four numbered red break points marking where the result or its acknowledgement can be lost

Four joins carry the result up. Only the last one sends anything back, and that return leg is the half of the handshake almost nobody writes a specification for.

What the two halves actually carry

Two user-defined types, thirteen members between them, fifteen OPC UA nodes once the structures themselves are counted.

The order side is small: Ord.ID, Ord.SKU, Ord.Qty_Cases, Ord.Per_Pallet and then Ord.New, which the MES writes last. The result side is Res.Seq, Res.Order_ID, Res.Good, Res.Reject, Res.Finished and Res.Ready, which the controller writes last. The two acknowledgements are Ord.Ack and Res.Ack_Seq, and both of them are DINTs rather than BOOLs. That is not a style preference and it is the single decision this article exists to argue for.

Write the flag last. Both directions, every time.

The reason is that an OPC UA write of a structure is not atomic from the reader’s point of view unless somebody arranged for it to be. If the MES writes Ord.New before it writes Ord.Qty_Cases, and the controller’s 20 ms task happens to land between the two, CP-3 starts an order for whatever quantity was left in the tag from last time. Writing the trigger last costs nothing and removes the whole class of problem. On the controller side the same discipline applies to the copy that builds the result: publication 1756-RM018A-EN-P puts COP and CPS side by side and gives a table for choosing between them, and the row that matters here is data another task can overwrite, where the answer is CPS, because tasks that try to interrupt a CPS are delayed until it has finished. There is a caveat printed right under that table which people miss, and it is the one relevant to a machine with an MES on it: the instruction protects you from other tasks in the controller, not from a remote client writing the source while the copy runs. For that, the manual tells you to write interlock code yourself. Ord.New and Res.Ready are that interlock.

Advertisement

A table of the thirteen members of the order and result structures, each row giving the data type, whether the MES or the case packer writes it, the OPC UA access setting and what the member is for, with the three acknowledgement rows Ord.Ack, Res.Seq and Res.Ack_Seq picked out in colour against the ten ordinary data members

Three marked rows carry the whole acknowledgement. Two of them are numbers rather than bits, and that is what makes an acknowledgement arriving four seconds late completely harmless instead of a double count.

Why the acknowledgement has to be a number

A bit can only say “something happened”. A number says which thing.

Res.Seq increments on every posted result, never repeats and never resets, not at shift change and not at midnight. Res.Ack_Seq is the sequence number the MES has actually committed to its database. The controller clears Res.Ready when Res.Ack_Seq equals Res.Seq and at no other time — not on a timer, not on a rising edge of some Ack bit, not when the comms status tag says the link is healthy. Everything good follows from that one comparison. An acknowledgement that arrives after the timer expired matches an old sequence number and is discarded rather than crediting the wrong pallet. An acknowledgement that arrives twice is idempotent, because the second one compares equal to a value that has already been handled. A retry from the MES adapter, which is the most common thing to happen on a flaky link, cannot produce a duplicate row, because the number in it is the number that is already stored.

If this sounds over-engineered, the people who wrote the OPC UA specification came to the same conclusion. Part 4, the services part, defines the Publish service as doing two jobs at once: asking the server for notifications, and acknowledging receipt of the ones already delivered, by sequence number, for one or more subscriptions at a time. What the server does with an unacknowledged message is keep it in a retransmission queue that the client can call Republish against. And the subscription state table in the same part has a state called LATE, entered when the publishing timer expires with notifications ready and no publish request queued to carry them, alongside a lifetime counter that eventually terminates the subscription altogether if the client never comes back. A protocol designed by committee over fifteen years arrived at a sequence number, a queue, a named state for “nothing came back” and a rule for giving up. Your case packer needs the same four things and can have them in about thirty rungs.

Three ways out of LATE, and what each one costs

CP-3 reached LATE at 01:46 and again every eleven minutes until 05:00. Here is what each of the three policies would have done with those 200 minutes.

Carry on and forget is one result area, overwritten by each new pallet. The machine never stops, the operators never see anything, and the MES gets whatever happened to be sitting in the tag when it reconnected. That is one record out of eighteen. Seventeen pallets, 816 cases, exist physically in a warehouse and do not exist in the system that is supposed to know where stock is, and nothing anywhere logged a problem. This is what CP-3 had, and it is what you get by default, because a result area that is simply written by the logic and read by whoever is listening behaves exactly this way and nobody had to choose it.

Hold until acknowledged is the opposite. The pallet in progress finishes, posts, gets nothing back, and CP-3 refuses the next work order. It had six minutes of the current order left when the link went, so the machine was stopped for 194 minutes of the outage: seventeen and a half pallets, 847 cases that were never made. Nothing was lost because nothing was produced. This is the correct answer on a pharmaceutical line where an unrecorded unit is worse than an unmade one, and it is an expensive and slightly absurd answer on a carton line packing tinned tomatoes. Both of those are real plants. The policy is not a technical decision and it should not be made by whoever writes the rung.

Advertisement

Buffer and retry is an array. Res_Buf holds twenty entries of 28 bytes, Buf_In advances on each unacknowledged result, Buf_Out drains when acknowledgements resume, and the machine never stops. Twenty entries at eleven minutes a pallet covers 220 minutes, so this outage used eighteen of them and finished with two spare and twenty minutes of margin. That margin is the entire design and it is a number you choose rather than a box you tick: an hour of tolerance is six entries, four hours is 22, a full twelve-hour shift is 66 and still under two kilobytes of tag memory. There is no version of this that is expensive. What there is, and what almost nobody writes, is the rung for what happens when Buf_In catches Buf_Out — because a full buffer puts you straight back to choosing between hold and forget, only now at three in the morning with no warning and with somebody’s array wrapping silently.

Pick the depth from the longest outage you are willing to survive, then add a quarter.

A two-column comparison of the same 200 minute outage under each policy, showing what the machine did and what it cost, with buffer depths for four and twelve hour outages and a final row recording that no operator saw anything under any of the three

The bottom row is the one that should be uncomfortable. Under all three policies the machine looked completely normal to the person standing next to it for the whole of those 200 minutes.

A state diagram of the result handshake, with IDLE, POSTED and ACKED running across the top row, LATE sitting below them where the acknowledgement timer expires, and three branches out of LATE into HOLD, BUFFER and FORGET, each labelled with its policy letter, showing return paths from BUFFER and HOLD and none at all from FORGET

Three ways out of LATE. Draw this before you write a rung, because the version in your head has only the top row in it and the top row is the part that already works.

Where the ten seconds in T_Ack comes from

The timer that decides you are in LATE has a preset, and the preset has arithmetic behind it or it is a guess.

Measure the round trip first. On CP-3 the result reaches the MES adapter on the next publishing cycle of a subscription set to 1000 ms, the adapter maps it and commits it, and the MES writes Res.Ack_Seq back down. Typical is 340 ms because the publishing cycle usually catches it early; the worst observed over a fortnight of logging was 2.1 seconds, which was a database checkpoint. T_Ack is set to 10 seconds, which is 4.8 times the worst measurement, and that ratio is the number to defend rather than the ten. Too short and the buffer fills with results that were about to be acknowledged. Too long and a real outage sits undetected for minutes while the machine builds pallets whose records are going nowhere. The one preset that is definitely wrong is one shorter than the publishing interval, and it is a mistake worth naming because it is easy to make: if the subscription publishes every second and T_Ack is 500 ms, the machine is in LATE on every single pallet and the buffer is doing all the work while the link is perfectly healthy.

Five timing traces over thirty seconds covering two pallet cycles. In the first, a pallet finishes at one second, the result posts, an acknowledgement arrives at 2.7 seconds and the ready flag clears. In the second, the pallet finishes at thirteen seconds, the ready flag stays up, the ten second acknowledgement timer expires at 23.1 seconds and the buffer count goes above zero

The second cycle is the one to design. From 13.1 to 23.1 seconds the machine is running normally and one finished pallet exists only in a DINT with nowhere yet to go.

The order coming down has the same problem, backwards

Almost every project designs the order handshake properly and then leaves one hole in it.

Ord.Ack echoes Ord.ID once CP-3 has copied the order into its own working tags, which lets the MES know the order was taken and which order was taken. That much is normal. What gets skipped is the refusal: an order for 73 cases per pallet on a machine whose magazine holds 48 is not something to acknowledge and then quietly clamp, and it is not something to reject silently either. CP-3 writes Ord.Ack to the negative of the order ID and Ord.Reject_Code to a number between 1 and 6, and the MES shows the operator a line saying which rule the order broke. Six reasons, one DINT, and the alternative is a pallet built wrong four hours later by somebody who never saw a message.

Set Ord.Ack to zero at start-up, not to the last order it handled.

What everybody checks first

Records go missing, and the first move is always the MES queue and the network.

Both are healthy, which proves nothing at all, because by the time anybody looks the link has been back for six hours and the queue has drained. The second move is to watch Res.Ready online, and that also finds nothing, because on a working machine Res.Ready is at 1 for about 400 milliseconds out of every eleven minutes and every time you glance at it you see a 0, which is what a correct machine looks like and also exactly what a machine that never posts anything looks like. The thing that finds it in one minute is Res.Seq. It is a number that only ever goes up, the MES knows the last one it stored, and the difference between the two is the count of records that went somewhere else. If that difference is 17 you do not have a network problem, you have a policy you never chose. Our page on reading production counts and downtime reasons out of the PLC covers what to do with those numbers once they arrive intact, and the wider PLC to MES integration article has the tag-level picture this one sits inside.

Gaps in a monotonic sequence are the cheapest alarm in this whole system.

The machine-to-machine version of the same argument is worth reading alongside it, because the shapes are identical: two conveyor zones disagreeing about who owns a box is a handshake with no acknowledgement designed into it, and it fails by stalling rather than by losing records. So is a setpoint arriving from outside, which the handshake for writing a setpoint back into the PLC works through from the other end. And if the buffer is the policy you pick, buffering plant data through an outage and backfilling it afterwards is the same idea one layer up, with a queue on disk instead of an array in a controller.

Advertisement

What to do on Monday

Go and find Res.Ready on your own machine, or whatever it is called there, and work out what clears it. If the answer is a timer, a comms-healthy bit, or nothing at all, you have the forget policy whether you meant to or not. Then ask the operations manager, not the controls engineer, which of the three answers the plant wants when the MES is gone for two hours, and write the number they give you on the drawing next to the buffer depth. If the buffer depth is what comes back, size it from the longest outage in last year’s IT incident log rather than from a round number, and put a rung on Buf_In catching Buf_Out before you put anything else on the machine.