Notes from the seams

Mean average precision is a model selection metric. It was designed to rank candidate architectures against a fixed benchmark, and it does that well. It was never designed to support a claim about harm, and it fails at that job in four specific ways.

1. It aggregates away the cases that matter

Safety risk is concentrated in rare conditions: unusual poses, occlusion, low illumination, small and distant objects, under-represented road user types. Those are precisely the cases with the fewest samples, so they contribute least to an aggregate score. A model can improve its headline number while degrading on every safety-critical slice. Stratify by the ODD dimensions in your operational design domain and report each one separately — including the ones with uncomfortable numbers.

2. It has no notion of consequence asymmetry

Missing a pedestrian and hallucinating a phantom obstacle are both errors and are not remotely equivalent. One risks a collision; the other risks a rear-end from an unnecessary brake event — which is also a hazard, just a different one. Your metrics need to separate false negatives from false positives per class, and both need to be traceable to the hazardous events they contribute to.

3. It says nothing about the distribution it was measured on

A number without its test distribution is uninterpretable. If the evaluation set was drawn from the same collection campaign as the training data, it measures interpolation, not generalisation. The assessor's question is not "what is your accuracy" but "on what distribution, how was it constructed, and how does it relate to the ODD you intend to operate in".

4. It is a point estimate with no uncertainty

For a rare class with a few hundred test instances, the confidence interval on your detection rate is far wider than the improvement you are celebrating between releases. Report intervals. Justify sample sizes. If the interval is too wide to support the claim, the honest conclusion is that you need more data, not a rounder number.

None of this means model metrics are useless. It means they are an input to safety evidence, not the evidence itself. The evidence is the argument that connects them to a hazard.

The Autonomous Driving Act amended the German road traffic act (StVG §1d–§1l) to permit vehicles with an autonomous driving function to operate in a defined operational area. The implementing ordinance — the AFGBV — turns that permission into a set of concrete obligations. Teams tend to read it as a homologation formality. It is closer to a set of architectural requirements with a legal deadline attached.

The defined operational area is a technical artefact

The festgelegter Betriebsbereich is not marketing language for "the city we launch in". It is a bounded region within which you have to demonstrate the function is safe, and outside which you must not operate. That makes your ODD specification a regulatory document. If your ODD lives as prose in a concept slide deck, you do not yet have one — you need a structured, testable definition with explicit exit conditions and a demonstrable behaviour at every boundary.

The risk-minimal state has to be engineered, not asserted

The vehicle must be able to place itself in a risikominimaler Zustand independently. This is a full function with its own hazard analysis, not a fallback paragraph. What is the state? How is the decision to enter it made, and by what evidence? What if the situation degrades during the manoeuvre? What if the fault is in the component that is supposed to detect the need for it? Each of those questions is a safety goal, and each of them cuts across all three standards.

Technical supervision is a system, not a person

The Technische Aufsicht can deactivate manoeuvres and release alternatives, but not drive. That produces a set of requirements most programmes discover late: a communication link with defined availability and latency, a situational awareness interface with real human factors work behind it, defined behaviour on link loss, and — critically — a human oversight function whose own failure modes are part of your hazard analysis. This is also where the EU AI Act's human oversight obligations start interacting with your safety concept.

Assessment is independent and evidence-hungry

The route to approval runs through a technical service and the KBA, and it expects an argument, not a document library. Data recording obligations, incident reporting and the ongoing duties on the operator mean the safety case does not end at start of production — it becomes a living artefact fed by field data.

The controllability parameter in ISO 26262 asks how likely it is that the driver, or another road user, can act to avoid the harm. It is one of the three inputs to ASIL, and it is the one that has historically absorbed the most optimism. A C1 rather than a C3 is frequently the difference between QM and ASIL C — between a normal development and a programme with a safety manager, an assessor and a very different cost base.

The inherited assumption

For decades that optimism was defensible. There was a driver, they were legally required to be attentive, they had hands near the wheel, and the failure modes in question produced symptoms a human could react to within a second or two. The classification rationale often did not even state the assumption, because it was universal.

Then the same platform got a conditional automation feature. The driver is now permitted to look away. Their reaction time is no longer one second; it is the takeover transition time plus the time to build situational awareness, and the literature on that is sobering. Meanwhile the classification rationale from the previous programme was inherited wholesale, because the hardware did not change and nobody flagged the item definition as modified.

What a valid re-assessment looks like

• State the assumed driver state explicitly for every controllability rating — attentive, permitted to be disengaged, absent, or remotely supervising.
• Use a takeover time budget with a source, not a reflex estimate. Include the perception-of-situation phase, not just the hands-on-wheel phase.
• Check whether the controllability argument depends on a warning strategy that itself has an ASIL — if the warning fails silently, the controllability claim fails with it.
• For Level 4, remove the driver from the argument entirely and ask what the system does instead. That is usually where the risk-minimal state requirement appears.
• Re-run the HARA on item definition change, not on hardware change. This is the process gap that lets stale ratings survive.

The statistical argument against mileage-based validation is well known: demonstrating a failure rate comparable to human drivers by observation alone requires a number of kilometres that no programme can afford, and every software change resets the clock. The argument is correct, it has been made repeatedly for a decade, and programmes still report progress in kilometres. It is worth asking why.

The reason is that mileage is easy to measure and coverage is not. Replacing it requires you to define what the space is that you are covering — and that definition is uncomfortable, because it forces you to admit how much of it you have not characterised.

What a coverage argument actually contains

A parameterised scenario space derived from the ODD. Not a list of test cases — a description of the dimensions along which scenarios vary, with the ranges each one can take inside your operational domain.

A sampling strategy with a rationale. Combinatorial coverage on the discrete dimensions, importance sampling towards criticality on the continuous ones, and an explicit statement of which interactions you are and are not exploring.

A criticality-aware allocation. Uniform sampling wastes almost all of its budget on benign scenarios. The scenarios worth simulating a thousand times are the ones near the performance boundary, which means you need a way to find that boundary — usually search-based or adversarial scenario generation.

A residual claim. The coverage argument ends by stating what fraction of the space is characterised, what is known to be unsafe and has been addressed, and what the remaining uncharacterised space is estimated to contain. That last number is Area 3, and it is the number the assessor came for.

The model that was evaluated is rarely the model that ships. Between the two sit quantisation, operator fusion, graph rewriting, pruning, a vendor compiler and a runtime scheduler. Each of these changes the numerical behaviour of the function. Most of them are treated as a build step.

Three questions that usually have no owner

What did quantisation cost you, and where? Post-training quantisation to INT8 typically costs a small amount of aggregate accuracy. That aggregate hides the distribution: the loss concentrates in low-confidence, low-contrast, small-object cases — the same safety-critical slices discussed elsewhere on this blog. If the deployed model was not re-evaluated on the stratified protocol, the safety evidence describes a different function than the one in the vehicle.

Is the toolchain qualified? A compiler that transforms a safety-related function is a software tool in the ISO 26262 sense, with a tool confidence level and potentially a qualification requirement. Vendor toolchains for ML accelerators are frequently newer, less mature and less transparent than the compilers the same organisation qualified carefully for its classic ECUs.

What happens when the accelerator computes the wrong answer? Random hardware faults in accelerator memory or MAC arrays do not announce themselves. They produce a plausible tensor with wrong values, and the network's output degrades gracefully — which is exactly the worst behaviour, because there is no exception to catch. This is a dependent failure analysis question, and the model's own sensitivity to corrupted computation is an input to it.

How to close it

• Treat the deployed artefact as the item under evaluation, and re-run the safety-relevant evaluation on target — not on a workstation with a different numerical path.
• Classify the ML toolchain formally and qualify it, or argue the confidence level down with evidence from your own verification.
• Include the inference platform in the dependent failure analysis, with the model treated as a component whose failure behaviour under corrupted computation has been characterised.
• Make timing and determinism explicit requirements. A perception pipeline that occasionally exceeds its budget under load is a functional safety finding, whatever its accuracy.