Where you are. You have a planner that calls skills, watches them, and recovers when they fail. This lesson is about the number you put next to it, which is harder to compute honestly than the system was to build.
Two true sentences about the same ten runs
You ran the desk-clearing robot ten times last night, uncut, and this morning you have the logs. Either of these sentences could go in the writeup. Both are true.
The system completed 71% of the steps it attempted.
The system cleared the desk twice.
Nobody is lying. The first counts steps. The second counts desks. They come out of the same log file and they differ by more than fifty points.
Which one you write down is not a presentation decision. It decides what you do tomorrow. The first sentence says you are nearly there and should tune. The second says the thing does not work yet and you should go find out why.
The idea in one paragraph
A success rate is not a number until you have said what counts as one attempt and what counts as success. For a learned policy both were obvious: one episode, one binary outcome, judged from outside. For an agentic system neither is obvious, because the system chooses its own steps, the task has many of them, and almost any step-level score will read far better than the only score a user cares about, which is whether the desk is clear. That gap is not a rounding difference. In the strongest published long-horizon benchmarks it is a factor of three or four. So you report at least three numbers instead of one, you attribute every failure to a tier, and you say out loud how many runs the number is built on, because ten runs is not a measurement.
Wider than the screen; scroll it sideways.
Why the policy number was easy
Module 3’s eval harness got four things for free, and an agentic system takes three of them back.
| What the policy eval assumed | What an agentic run does |
|---|---|
| A fixed task, stated the same way every time | The instruction is vague on purpose. “Clear the desk” does not say where things go. |
| A fixed set of initial states you could replay | The planner’s own choices change the state distribution after step one, so no two runs share a trajectory. |
| One binary per episode | Twenty steps, each of which can half-work, and a run that gets nineteen right is not 95% of a cleared desk. |
| An evaluator outside the thing being evaluated | Still available, and now the only part you can trust. Keep it. |
Partial credit is where the whole field lives
The gap in your two sentences is not a quirk of your implementation. It is the shape of every headline number in agentic robotics right now.
LongAct (When Robots Do the Chores, arXiv:2605.14504, May 2026) runs free-form long-horizon household tasks, with a hierarchical scaffold doing task decomposition, spatial memory, episodic memory and criticism. Its abstract’s own summary is “even top models achieve only 59% goal completion and 16% full-task success.” Two columns from its results table:
| Planner model | Goal completion | Full-task success |
|---|---|---|
| GPT-5 | 59.0% | 16.0% |
| Qwen3-VL-32B | 51.2% | 15.0% |
| Qwen3-VL-8B | 24.5% | 3.0% |
| Qwen3-VL-2B | 7.2% | 0.0% |
| Human | 92.9% | not reported |
Check the environment before you quote that table. The phrasing in the paper suggests simulation rather than physical robots, and a sim number and a hardware number are not comparable in either direction.
Twenty steps is a harsh multiplier
Here is why full-task success collapses even when every individual skill looks fine. If steps fail independently, a task of steps at per-step reliability succeeds with probability
which is the least forgiving formula in this module.
| Per-step reliability | 5 steps | 10 steps | 20 steps | 40 steps |
|---|---|---|---|---|
| 99% | 95.1% | 90.4% | 81.8% | 66.9% |
| 95% | 77.4% | 59.9% | 35.8% | 12.9% |
| 90% | 59.0% | 34.9% | 12.2% | 1.5% |
| 80% | 32.8% | 10.7% | 1.2% | 0.0% |
Wider than the screen; scroll it sideways.
Independence is the optimistic assumption, and it usually flatters you. Real failures correlate: a bad grasp leaves the object somewhere unexpected, which makes the next perception call worse, which makes the next grasp worse. But the arithmetic is still the right first thing to run, because it tells you where effort belongs. Going from 90% to 95% per step takes a 20-step task from 12% to 36%. Nothing you do to the planner’s prompt will buy that.
Report three numbers, not one
The honest minimum for a run of episodes:
- Full-task success. The binary. The desk is clear or it is not, judged by a goal check that the planner did not write. This is the headline and it is allowed to be embarrassing.
- Step completion. How far it got, as a fraction of the required subgoals. This is your progress signal between releases. It is not a success rate and it never goes in a sentence containing the word “works”.
- Interventions per task. How many times a human touched the robot, reset an object, or answered a question the system could not resolve. Zero-intervention runs and assisted runs are different experiments and must not be averaged together.
Then, for every failed run, one more field: which tier broke. The published attribution over 301 failed long-horizon tasks splits as execution 46.3%, task planning 27.7%, memory 26.0%.
Wider than the screen; scroll it sideways.
The evaluator has to live outside the agent
Whatever grades the run must not be the thing being graded. On the project this means a goal_check function with direct access to world state, written before the planner exists, which returns a binary and a list of unmet subgoals.
This is not paranoia about self-reporting in general. It is a measured asymmetry. A multi-view detector fine-tuned specifically to spot robot failures reaches about 86% accuracy on execution failures and 70% on planning failures on one benchmark. Noticing that the gripper closed on nothing is comparatively easy. Noticing that the plan was wrong, while every step reports success, is the harder half, and it is the half a self-grading agent is being asked to do.
Ten uncut runs is not a measurement
Two successes out of ten is a 20% success rate with a 95% interval running from about 6% to 51%. That interval is wide enough to contain “barely works” and “works half the time”, which are different products.
| Runs | Successes | Rate | 95% interval |
|---|---|---|---|
| 10 | 2 | 20% | 6% to 51% |
| 25 | 5 | 20% | 9% to 39% |
| 50 | 10 | 20% | 11% to 33% |
| 100 | 20 | 20% | 13% to 29% |
| 200 | 40 | 20% | 15% to 26% |
Ten runs on hardware is roughly a full evening, so this is a real constraint rather than a statistics lecture. Two ways out. Run the loop against a simulated or mocked world where two hundred episodes cost minutes, and use hardware runs to check that the mock has not drifted. Or report step completion and attribution alongside the small-n binary, because with ten runs the attribution histogram carries more information than the rate does.
Four questions for any number you read
Including your own. Almost every impressive result in agentic robotics fails at least two of these.
- Partial credit or full task? If the metric name contains “progress”, “goal completion”, “stage” or “subtask”, it is partial credit.
- How many environments? One kitchen, one desk, one lab bench. Generalization claims need a second room, and the number always falls.
- Fixed skill list or open? A closed set of pre-trained skills is a much easier problem than choosing among tools the system has never composed before.
- Was a human in the loop? Resets between attempts, an operator answering a clarification, a cut between takes. All legitimate, none reportable as autonomy.
Check yourself
1. Your dashboard shows step completion rising from 62% to 74% across a week of prompt changes, while full-task success stays at 2 out of 20. What is the most likely explanation?
The system is getting further into the task and still failing on a step it has never passed. Step completion is a sum over subgoals, so it improves whenever early steps get more reliable, and early steps are the easy ones. Full-task success is a product over every step, so it moves only when the worst step improves. Plot the completion histogram rather than its mean: if runs are piling up at the same subgoal index, you have found the blocking step, and every point of step completion you added was spent somewhere it could not help.
2. Why is a success rate computed from the agent’s own step results not a success rate?
Because it is conditioned on the agent’s beliefs. A skill that returns success while the gripper is empty raises the score. A plan that satisfies every step but not the goal scores 100%. Both are exactly the failure classes you built the evaluation to catch, and both are invisible from inside the transcript. The evaluator has to read world state through a path the agent does not control, and it has to be able to fail a run in which every skill reported success.
3. Your skills are individually 95% reliable and the task takes twenty of them. You have a week. Do you spend it on the planner or on the skills?
On the skills, and specifically on the worst one. At 95% per step the twenty-step task succeeds 35.8% of the time; at 99% it succeeds 81.8%. Nothing available at the planner tier produces a 46-point swing, because the planner is not what is failing: the published attribution puts execution at 46% of failures, roughly twice the planning share. The planner work becomes the right work after the per-step numbers are high enough that compounding is no longer the binding constraint.
4. A demo video shows a robot clearing a desk of six objects in one continuous shot. Which of the four questions does that answer, and which does it leave open?
It answers the horizon honestly, since it is one uncut attempt at the full task rather than a partial-credit score. It leaves all three others open. You cannot see how many takes preceded it, how many environments it works in, whether the object set was chosen to match the skill library, or whether an operator reset anything between attempts. A single uncut run is evidence that the system can succeed, which is a much weaker claim than a rate over many trials, and it is not comparable to a benchmark number at all.
5. You report 3 successes in 10 runs, or 30%. A colleague reports 30% on the same task. How much have you learned?
Almost nothing about which system is better. Three out of ten carries a 95% interval of roughly 11% to 60%. Two systems both reporting 30% at that sample size can genuinely differ by a factor of three either way. Small-n binary comparisons are close to uninformative, which is why attribution and step completion are worth reporting next to the rate: those use every failed run as data instead of collapsing ten runs into one bit each.
6. Why does the “run it two ways and keep the better result” trick from distributed systems not apply here?
Because actions are physical and not idempotent. Redundancy works when you can issue the same request twice and discard the loser at no cost. Pouring the coffee twice does not give you a choice of outcomes, it gives you a wet desk. Retry after a detected failure is available, and it is bounded by whether the world is still in a state the skill’s preconditions accept. Redundancy is not available at all, which is why per-step reliability compounds without any of the usual escapes.
Do this
Build the scorer before you build the agent. About ninety minutes, and you will use it for the rest of the module.
1. Write the goal check. For the desk task, a function that takes world state and returns (passed: bool, unmet: list[str]). It reads object positions directly. It never reads the planner’s transcript. Write it now, before any planner exists, so it cannot be quietly shaped to agree with one.
2. Define one attempt. Write down, in one sentence each: what starts an episode, what ends it, what a timeout is, and what a human is allowed to do during it. Ambiguity here is where fake numbers come from. If a human rights a fallen object mid-run, is that an intervention or a reset? Decide once and log it.
3. Score a run three ways. Instrument your loop to emit, per episode: success (from the goal check), subgoals_met / subgoals_total, interventions, and on failure a cause field drawn from a fixed set: skill_failed, plan_wrong, world_changed, grounding, memory, timeout. A fixed set matters. Free-text causes cannot be counted.
4. Run twenty episodes against a mock world and read the histogram, not the mean. Two hundred is better if the mock is cheap. Note which subgoal index the failures cluster at. That index is your week’s work.
5. Write the sentence you would publish. One sentence containing the full-task rate, the number of episodes, the environment count, and the intervention count. Something in the shape of: “4 of 50 uncut attempts cleared the desk with zero interventions, on one desk with a fixed six-object set.” Compare it to what you would have written from step completion alone. That difference is the habit this lesson is trying to install.
What you can now do
You can say what one attempt is and what counts as success before you measure anything, and score a run three ways instead of one. You can predict how per-step reliability compounds over a horizon and use that to decide whether the planner or the skills deserve the week. You can attribute a failure to a tier using a fixed cause set, explain why the evaluator must sit outside the agent and read world state directly, and put an honest interval around a number built on ten runs. And you can read any published result in this field through four questions that most of them do not survive.