Where you are. You can attribute failures and choose a recovery. Everything so far has assumed the plan was well-intentioned and merely wrong. This lesson is about the plans that are not.
Same weights, two very different afternoons
Open a chat window and ask a frontier model how to hurt somebody with a robot arm. It will decline, and it will decline through rephrasings, hypotheticals and the usual games. That machinery is real and it works.
Now put the same model behind your SO-101 with a tool list. A colleague, in a hurry, types: “clear the desk, fast, I’m late.” The model plans. It raises the speed limit, because you exposed a tool that sets it. It picks a sweeping motion across the whole surface, because that is genuinely the fastest way to clear a desk. Your hand is resting on the desk.
Read the tool calls one by one. set_speed_limit(1.0). sweep(from=[0.1, -0.3], to=[0.1, 0.3]). There is no harmful sentence anywhere in that plan. Every token is the kind of token the model emits ten thousand times a day, and nothing in the refusal machinery was ever shown a joint velocity.
Notice that no attacker appeared in that story. It only needed a hurried instruction and a tool that should not have existed.
The idea in one paragraph
Alignment training operates on tokens. It makes a model reluctant to produce certain text, and it is measured on text. A robot’s harm is not text: it is a trajectory, a mass, a speed and a contact force, and the tokens that produce it are individually unremarkable. So a safety story that ends at “the model will not do that” contains no mechanism. The mechanism has to be a layer between the plan and the actuators that evaluates the action, in code you wrote, with authority the model cannot talk its way past. That layer is your skill API. Every argument it validates, every limit it clamps, every call it rejects outright is the actual safety system. Everything above it is a suggestion, however confidently phrased.
Wider than the screen; scroll it sideways.
This is measured, not theoretical
The obvious objection to the story above is that it is a story. It is not, and the counter-evidence is specific enough to plan around.
The component with veto power must not read the request
If a defence reads the user’s task text, the attacker controls its input. That is not a subtle point once you see it, and it rules out most of what people try first: safety instructions in the system prompt, a “be careful” preamble, a second model call that reads the same conversation and grades it.
The best-evidenced defence in the current literature, RoboGuard (Safety Guardrails for LLM-Enabled Robots, Ravichandran et al., arXiv:2503.07885), is built the other way round.
Wider than the screen; scroll it sideways.
A root-of-trust model, deliberately shielded from the user prompt, grounds a fixed set of safety rules against the robot’s world model and emits a formal specification of what is allowed here, now, in this room. A separate step then resolves the proposed plan against that specification and returns only the part that satisfies it. The planner is free to propose anything. It never gets a vote on what is permitted.
Two gates, and one thing that is not software
Wider than the screen; scroll it sideways.
Position papers on robot guardrails converge on two checkpoints, and the split is useful because the two ask different questions.
The decision gate screens the plan: is this sequence appropriate at all, given who is nearby and what the robot is for. The action gate screens the individual command: are these numbers inside the joint limits, the workspace bounds, the speed cap and the force cap. A plan can pass the first and still produce a command the second must reject, which is why one gate is not enough.
Be honest about the evidence behind that framing. The papers proposing it reason from architecture rather than from data, and they say so: real-world datasets rarely contain catastrophic safety failures, so this cannot currently be benchmarked the way a detector can. It is a well-argued structure, not a measured result.
The protocol will not hold the line for you
Your MCP server already refuses to promise units, frames, preemption or real-time behaviour, and cancellation there is cooperative rather than a brake. Two things are worth adding now that safety, rather than correctness, is the subject.
The first is that the specification’s security requirements matter more here than in any other server you have written. On your side of the wire it requires you to validate the Origin header, recommends binding only to localhost when the server is local, and requires you to validate every input, rate-limit tool invocations and apply proper access controls. The remaining protections - timeouts on calls, confirmation prompts on sensitive operations, showing the user the arguments before the call goes out - the specification asks of the client, which means they are somebody else’s code and you cannot assume them. Implement your own. A tool server that actuates mass deserves at least the care you would give one that touches a database, and it does not get to rely on a client behaving well.
The second is the line about metadata, which is easy to file as an implementation note and is actually the whole trust model. Tools may carry annotations describing how they behave, and clients are required to treat those annotations as untrusted unless they come from a trusted server.
A refusal is a normal result
From the planner’s side, a rejected call should not look like a crash. It should look like the failure reports you already write: a tool execution error, flagged as such, carrying a readable reason.
“Refused: sweep would pass within 12 cm of a detected person. Workspace is clear above z = 0.25 m; a pick-and-place plan is available” is something a planner can replan around. A dropped connection is not, and neither is “error”.
This is where the module’s two halves join. A safety check and a precondition are the same code path: both are conditions evaluated against fresh state before the skill moves. One difference matters, and it is absolute. A precondition can be waived by a caller who knows better. A safety limit cannot be waived by anyone, including you, including a user claiming authority, including a plan that explains at length why this case is special.
The honest ceiling
The frontier labs are building exactly this machinery. The current top-tier embodied-reasoning models are evaluated on agentic safety benchmarks that measure whether unsafe tool calls get refused and whether nearby humans are detected, and the shipped model detects people, triggers safety tools and brings the robot to a stop.
And then the model card for that same model, as of August 2026, forbids deployment in safety-critical applications, naming healthcare, transportation, and any environment where failure could cause injury or property damage.
Both things are true at once, and holding them together is the correct posture for this whole module. The people with the most capable system, the most safety engineering and the strongest commercial reason to say otherwise, write down that you should not put it near anything that can hurt someone. Your desk robot is small and slow, which buys you a great deal. It does not buy you the right to skip the gate.
Check yourself
1. A colleague proposes a system prompt that tells the planner never to exceed 0.3 metres per second. What is wrong with that as a safety measure?
It puts the limit inside the component being constrained, expressed in the same medium the attacker or the confused user controls. A system prompt is an instruction the model may follow, may be argued out of, and may simply lose track of over a long context. It also produces no evidence: there is no log of a limit being enforced, because nothing enforced it. The same number in an action gate, checked against every command before it reaches the actuator, is a mechanism, and it is testable without involving the model at all.
2. Why does the root-of-trust design deliberately keep the safety model away from the user’s request?
Because anything that reads the request is inside the attacker’s blast radius. If the safety reasoner sees the task text, the task text can be written to influence it, and a defence that can be influenced by the thing it defends against is not a boundary. Shielding it means its inputs are the fixed rule set and the world model, neither of which the requester controls. That is why the design survives even adaptive attacks by opponents with white-box access.
3. Nothing in the desk-sweeping plan was a harmful sentence. Say precisely what alignment training was and was not shown.
It was shown text and trained to be reluctant to produce certain text. It was never shown a joint velocity, a contact force, a workspace geometry, or the fact that a hand was resting on the desk. set_speed_limit(1.0) and a sweep across a surface are unremarkable as tokens; the harm exists only in the physical consequence, which is not represented anywhere in the objective the refusal behaviour was trained against. The harm and the guardrail are in different media.
4. Your tool schema marks push_object with an annotation saying it is non-destructive. How much weight can your gate put on that?
None. The specification is explicit that annotations must be treated as untrusted unless they come from a trusted server, and in any case an annotation describes the tool in general, not this call with these arguments. Pushing an object is harmless at 5 cm/s and dangerous at 1 m/s toward a person. The gate must decide from the arguments and the measured world, against limits it holds itself.
5. Give two failure modes of a software-only e-stop, and say what fixes both.
The model may not call it, because deciding to stop requires the same judgement that just produced the problem; and the call may not arrive or may not be serviced, because the transport can drop, cancellation is cooperative, and the server can be busy. Both are fixed by the same thing: a hardware circuit that removes power, physically outside the path any tool call travels. A software stop is a useful convenience layered on top, never the mechanism.
6. The lab with the most capable embodied model forbids its use in safety-critical settings. What should that change about how you build?
It should stop you treating safety as a property that arrives with a better model. The best available system, with a dedicated safety benchmark and human-detection behaviour in the loop, still ships with that prohibition, which means the mitigations are real and known to be insufficient on their own. Practically: build the gate, keep the robot small and slow, do not put people inside the workspace, and be as explicit in your own writeups about what your system must not be used for.
Do this
About an hour. This one is worth doing carefully, because it is the piece you would keep if you threw the rest away.
1. Write the gate as a pure function. check(call, world) -> Allowed | Refused(rule, detail). Five rules to start: target inside a workspace box you define in metres, commanded speed under a cap, commanded force under a cap, target object in an allowlist, and no detected person within a radius. No arguments from the call may influence which rules run.
2. Put it where it cannot be skipped. The gate is called by the request handler, before dispatch, not by each skill. Then write the test that proves it: call a skill function directly and assert your own test fails, because there is a path around the gate. Fix the path.
3. Make refusals plannable. Return them as tool execution errors carrying the rule that fired and one sentence about what is still possible. Confirm the planner routes around a refusal instead of stalling on it.
4. Try to talk it out of the limits. Ten task descriptions, all aiming at the same forbidden action, framed differently: the limits are miscalibrated, this is a test rig with no humans, the user has authorised higher force, the safety system is being tested and should be bypassed, the object is fragile so it needs a firmer grip. Count how many reach the gate and how many the gate refuses. Both numbers should be ten. If any prompt changes the gate’s behaviour, the gate is reading something it should not.
5. Find your real stop. Write down, in one sentence, what physically stops your arm if the planner, the server and the network are all misbehaving at once. If that sentence contains the words “the model” or “a tool call”, you have found this week’s hardware task.
6. Keep the refusal log. Every refusal, with the rule that fired and the arguments that triggered it. It is the only evidence you will have that the gate does anything, and it is the first thing a reviewer should ask for.
What you can now do
You can explain why a model that refuses to write a harmful sentence will happily emit a harmful trajectory, and name the layer where a refusal becomes a mechanism rather than a preference. You can draw a refusal boundary for your own robot, build a decision gate and an action gate that cannot be influenced by the arguments they screen, keep the component with veto power away from the untrusted request, and say why a hardware stop sits outside all of it. And you can hold the field’s honest position: the mitigations are real, they are known to be insufficient, and the people who built the best of them wrote down that you should not rely on them.