Recovery is a choice, not a reflex

Where you are. You can attribute a failed run to the skill, the plan, or the world model. This lesson turns that judgement into machinery: what watches, how often, and what it is allowed to do about what it sees.

The three-line fix that made the desk worse

You have a flaky grasp. About one attempt in four closes on air. The obvious fix is three lines long and you have written it a thousand times:

for attempt in range(3):
    result = call_skill("pick", object_id="pen")
    if result["postcondition_held"]:
        break

It works. Grasp reliability at the task level goes from 75% to about 98%, the demo stops embarrassing you, and you ship it.

Two weeks later the arm knocks the tray while placing a mug. The tray ends up two centimetres left of where the planner thinks it is. The next call is place(pen, tray). The gripper opens over empty desk, the pen lands beside the tray, and the postcondition check says the gripper is empty and the pen is not in the tray. Failure. Retry.

The arm picks the pen up off the desk and drops it in exactly the same wrong place. Twice more. Three pens on the desk would have been better than one pen dropped from height three times, and by the third attempt the pen has rolled somewhere the camera cannot see.

The retry loop did what you asked. It assumed the failure was random. It was not.

The idea in one paragraph

Recovery is two machines, not one. Something has to notice, and something has to choose a response. The noticing is the half people skip, because a tool result is sitting right there and it has a status in it, and the component reporting that status is the same one that might have failed. A real monitor observes the world, and it runs at three different scopes: once per skill, once per plan, and repeatedly over the whole run. The choosing is a ladder of responses ordered by cost and blast radius, from retry at the top to stopping at the bottom, and each rung encodes an assumption about which layer broke. Match the rung to the failure class and recovery works. Reach for the same rung every time and you have not built recovery, you have built a louder version of the original failure.

Three monitors, three cadences

Each monitor catches exactly one of the three failure classes, and they get progressively more expensive in the same order.

The load-bearing property is not the cadence. It is that none of the three reads the skill’s own report.

The ladder

Six rungs, and each one is a hypothesis about what went wrong.

• Retry. Assumes the failure was random and the world is unchanged. Correct only for class one, and only when the world has not moved.
• Re-perceive, then retry. Assumes the skill is fine and the belief that fed it went stale. This is the rung that would have saved the pen.
• Substitute a skill. Assumes the approach is wrong for this object. Suction instead of a pinch grasp, push instead of lift.
• Observe, then replan. Assumes the plan was wrong. The observation is not optional, for the reason in the last lesson.
• Ask the human. Assumes no sensor available to you can settle the question. Ambiguous instruction, unrecognised object, two equally good interpretations.
• Stop and hold. Assumes the next action is unsafe or unrecoverable. Not a failure of the system, a correct output of it.

Two rules keep the ladder from becoming a maze. You may only move down the ladder within one failure, never back up, or you will oscillate between retrying and replanning forever. And every rung below the first must be preceded by an observation, because every one of them is a decision about a world you have just been told you misunderstand.

A cap on each rung, not only on the loop

Your planner loop already has three caps on the whole run: consecutive failures, total calls, and wall clock. Those stop a run that has gone wrong. They do not stop a run that spends its entire budget on the top rung, which is what a bare retry loop does by construction.

So each rung gets its own cap as well, decided before the run: at most two retries per skill call, then move down; at most two replans per task, then ask. A retry in software costs latency and a log line. A retry on a robot costs a physical action against a world with no undo, and each one changes the state the next attempt will be planned against, so the per-rung cap is doing safety work as well as budget work.

The stage the loop was missing

Compare that with the planner loop you already built. It has perceive, plan, call and monitor, and replanning as an edge off the monitor. What it does not have is a box between noticing and responding, so the edge always leads to the same place.

Attribute is that box, and adding it is the whole change this lesson makes. Once the loop knows which layer broke, the edge out of monitoring has six destinations instead of one, and picking between them is a lookup rather than a guess.

The dashed line is there as a reminder rather than a new idea. You already know that replanning without re-perceiving is worse than the failure it was recovering from. The failure taxonomy explains why the rule has no exceptions: the class it fails on is the class where the belief itself is the broken component, and that is precisely the class you cannot identify without looking.

After the client dies

One recovery case is worth separating out because the protocol shapes it. Your planner process can crash while an arm is mid-motion, and a durable task id is what lets the restarted process ask what became of that call rather than waking up blind in front of a robot in an unknown pose.

What the task id does not carry is permission. A fresh process holding a completed task result knows one call’s outcome and nothing about the desk, and its belief is now older than every action taken since it was written. Treat a reconnect as a class-three failure by default: the world model is presumed wrong, the first action after reconnecting is an observation, and no irreversible skill runs until that observation is in hand.

For the ask-the-human rung, a server can return an input-required result and have the client reissue the original call with the answer attached. Useful, but the mechanism is not the safeguard. The safeguard is that the server refuses to proceed without the answer, rather than the planner being asked nicely to check first.

A monitor that fires early is also a failure

Every monitor has two error rates, and teams measure one of them.

Missing a real failure is the one everybody counts. Stopping a skill that was going to succeed is the one that quietly caps your performance, and it is easy to build: a goal check evaluated a beat too early, a progress classifier that has not seen this stage before, a drift monitor that flags every lighting change.

So report both numbers for every monitor you add: how often it caught a real failure, and how often it stopped a run that was on track. If you cannot state the second number, you do not know whether the monitor is helping, and the published record says your intuition about which way it goes is not reliable.

Check yourself

Do this

About an hour, building directly on the classifier from lesson 9.

1. Wire the ladder to the classifier. Map each attribution to a starting rung: skill_failed starts at retry, world_model_wrong starts at re-perceive, plan_wrong starts at observe-then-replan. Log which rung fired on every failure.

2. Enforce monotone descent and budgets. Two retries per call, two replans per task, twelve actions total, then stop. Assert in code that the rung index never decreases within one failure, so the rule is mechanical rather than remembered.

3. Run the ablation. Thirty episodes with retry-only recovery, thirty with the full ladder, same random seeds and the same three injected faults. Report task success, mean actions per episode, and how often the run ended by hitting a budget rather than by finishing.

4. Measure the monitor, not just the robot. Have the goal monitor evaluate one step early, on purpose, ten per cent of the time. Report how many otherwise-successful episodes it aborted. That is your false-abort rate, and it is the number that will not appear in anyone else’s demo.

5. Kill the client mid-motion. Add a two-second place and terminate the calling process one second in. Restart it. Have the new process resume as a class-three failure: observe before anything else, and refuse every skill you marked irreversible until that observation lands. Then compare its rebuilt belief against truth and print the disagreement. That disagreement is what a naive resume would have planned against.

What you can now do

You can build a recovery system that chooses rather than reacts: three monitors at three cadences, none of which reads the component under test, feeding an attribution that selects one rung of a six-rung ladder. You can defend every rung by naming the assumption it makes, cap all of them with budgets that terminate, and place your one expensive check where irreversibility makes it worth the price. And you can state both error rates for every monitor you add, including the one that quietly costs you runs you would otherwise have won.