Notes from the seams

Almost every perception or grasp-planning model in production reports a confidence value alongside its output. Almost every safety architecture built around that model treats low confidence as the trigger for a fallback behaviour. The step nobody checks is whether the number means what the architecture assumes it means.

Why the number is usually wrong in a specific way

Modern classifiers are frequently overconfident exactly where it matters most — on inputs that sit outside the training distribution. A network rarely outputs "I don't know." It outputs a class and a high score, because nothing in standard training ever taught it to recognise novelty. This is precisely the failure mode ISO/IEC TS 25223 is being written to address: not whether the model is accurate, but whether its stated uncertainty is calibrated — whether a 90% confidence score is actually right 90% of the time, including on the inputs the model has never met.

Four questions before a confidence threshold becomes a safety function

Was calibration measured, not assumed? Expected calibration error or an equivalent metric, on a held-out set that includes near-distribution and out-of-distribution examples — not just the validation split from the same collection campaign as training.

Does the threshold hold under distribution shift? A threshold tuned on indoor lighting rarely survives a change in camera, a new SKU, or a different floor surface. If the threshold moves, the safety function built on it moves with it — silently.

Is the failure asymmetry represented? A missed grasp and a dropped, misidentified, or crushed object are not equivalent errors. The confidence threshold that minimises overall error is rarely the threshold that minimises harm.

What happens below threshold? "Fall back to a safe state" is not a specification until the safe state, the transition time, and the behaviour if the robot is mid-task are all defined and themselves verified.

The Machinery Regulation replaces the old Machinery Directive, and the headline change is not procedural — it is scope. Where the Directive's software provisions were written for an era of fixed control logic, the Regulation names categories that did not meaningfully exist when it was drafted the first time around: autonomous mobile machinery, AI-based safety systems, and industrial robots and cobots, alongside machinery with remote-operation features and connected, software-defined equipment.

Why "we already do functional safety" is not the same as ready

A robot manufacturer with a solid IEC 61508 programme can still find itself outside conformity, because the Regulation is not only asking whether the hardware fails safely. It is asking questions the classic functional-safety lifecycle was never built to answer on its own: what happens when a safety-relevant function is implemented in AI, what evidence supports that function's trustworthiness, and how changes to that function after the machine is placed on the market are governed.

This is exactly the seam our AI-safety and certification practices exist to close — the Regulation does not care whether your evidence lives in three different teams' folders, only whether it adds up to a coherent conformity argument.

2027 is not as far away as it looks

Robotics hardware and safety architecture decisions have long lead times. A conformity gap discovered during a 2026 design freeze is a very different problem than one discovered in a 2025 concept review. The practical advice is the same one we give on every regulatory deadline: read the annex now, map your current architecture and evidence against the named categories, and treat any gap as an architecture decision, not a documentation task to close later.

We are regularly shown a shared drive full of test reports, review minutes and component datasheets and told "the safety case is all in there." It usually isn't — not because the engineering is weak, but because a folder of evidence is not the same thing as an argument. UL 4600's structure — claim, argument, evidence — exists precisely because that difference is where certification reviews stall.

The three parts, and where each one breaks

A claim is a specific, falsifiable statement: "the robot will not exceed 15 N of contact force during unexpected human contact in collaborative mode." Claims break when they are vague enough that no evidence could ever contradict them.

The argument is the reasoning that connects evidence to the claim: why these tests, this analysis, and this operational history are sufficient to support it. Arguments break silently — a document simply asserts the claim is "demonstrated by testing" without saying which testing, under what conditions, or why that testing generalises to the claimed scope.

The evidence is the test report, the analysis, the field data. Evidence breaks when it quietly supports a narrower claim than the one it's cited against — a force measurement taken at one joint configuration, cited in support of a claim about the whole workspace.

A test you can run on your own case today

Pick any claim in your safety documentation. Ask: if I deleted the evidence behind it, would a reader who has never met your team notice that the claim is now unsupported? If the honest answer is "probably not, because the evidence and the claim aren't explicitly linked," that is the gap a UL 4600-structured case is built to close.

It's a reasonable instinct to wait for a standard to be finished before building a programme around it — clause numbers change, scope shifts, an early guess can look foolish in hindsight. For TS 22440 and TS 25223, both still in development, we advise against waiting anyway.

The questions don't move even when the clause numbers do

ISO/IEC TR 5469, published and stable since 2024, already describes the engineering questions the later specifications will formalise: what safety requirements look like for a system whose behaviour is learned rather than written, how AI verification integrates into a functional-safety lifecycle, and how a system's confidence in its own output should be treated as a safety property. Those questions do not change while the specification text is being drafted — only the precise, citable requirement wording does.

Teams that build their AI safety case around TR 5469's structure now are not gambling on a moving target. They are doing the engineering work the eventual specification will ask for anyway, and positioning themselves to map existing evidence onto new clause numbers when they publish — instead of starting the underlying analysis from a standing start under a submission deadline.

The precedent from SOTIF

We watched the same pattern play out with ISO 21448 in automotive: teams that structured their validation around the underlying scenario-space reasoning years before the standard reached its current published form adapted fastest once consensus formed. Teams that waited spent their first eighteen months translating existing but unstructured work into the standard's vocabulary, instead of producing new evidence.

The model that was evaluated is rarely the exact function that commands the motors. Between the two sit quantisation, a vendor inference runtime, a motion planner that translates a grasp pose into joint trajectories, and a motor controller with its own timing and fault behaviour. Each of these changes the function. Most of them are treated as a build step rather than a safety-relevant transformation.

Three questions that usually have no owner

What did quantisation cost you, and where? Compressing a model for an embedded accelerator typically costs a small amount of aggregate accuracy. That aggregate hides the distribution: the loss concentrates in low-confidence, rare, or novel-object cases — the same cases most likely to matter for a safety claim. If the deployed model was not re-evaluated on the stratified protocol, the safety evidence describes a different function than the one running on the robot.

Is the inference and motion toolchain qualified? A compiler or motion planner that transforms a safety-relevant function is a software tool in the IEC 61508 sense, with a tool confidence level and potentially a qualification requirement. Vendor toolchains for ML accelerators and modern motion stacks are frequently newer and less transparent than the tooling an organisation qualified carefully for its classic control loops.

What happens when the actuator executes a corrupted command? A plausible-looking but wrong model output, combined with a random hardware fault in the motion pipeline, does not announce itself. The model's own sensitivity to corrupted computation is an input to the dependent failure analysis — and it is usually missing from both the ML evaluation and the classic FMEDA.

How to close it

• Treat the deployed, on-target artefact as the item under evaluation — not a workstation build with a different numerical path.
• Classify the inference and motion toolchain formally, and qualify it or argue the confidence level down with your own verification evidence.
• Include the actuation pipeline in the dependent failure analysis, with the model treated as a component whose behaviour under corrupted computation has been characterised.
• Make timing and determinism explicit requirements — a motion pipeline that occasionally exceeds its control-loop budget under load is a functional safety finding, whatever its accuracy.