Where you are. You have every part: a skill API, preconditions you impose yourself, an MCP server, a planner loop, a failure taxonomy, monitoring, a refusal boundary, an honest evaluation, and a latency budget. This is where they become one program, and where you find out which of your recovery mechanisms is actually doing the work. No robot is required for any of it, on either path from Module 4 lesson 0; if you have one, there is a second acceptance criterion at the end.
Watch someone clear your desk
Ask a person to do it while you sit there.
They will ask you a question first. “Where does this go?” The instruction did not say, and they will not guess about the thing they cannot classify. Then they will pick up the mug and hesitate, because it is still half full, and tipping it into the sink is a different action from carrying it. Then they will file everything, and at the end they will do the thing that matters most: they will look at the desk again, rather than at their memory of it.
Three behaviours, and only the third is invisible. Nobody notices someone glancing at a desk. But if you take that glance away and leave everything else, the same person will confidently declare a desk clear while a mug sits on it, because a colleague put it down two minutes ago and nothing told them.
That glance is worth about eighty points of success rate. You are going to measure it.
The idea in one paragraph
The milestone assembles the whole module into one loop: perceive, plan, call a skill, read what came back, decide whether the goal is met, recover when it is not. The world is a mock desk behind four skills with the same result shape your MCP server returns, so the agent you write here runs unchanged against the real one. The harness has three dials, one per failure class: unreliable grasping, unreliable perception, and a world that changes without telling you. The deliverable is not the demo. It is a table of nine numbers showing which recovery mechanism buys back which failure class, and one row of that table shows a system whose step-completion metric stays high while half its desks stay dirty.
Wider than the screen; scroll it sideways.
What you are building
Four skills, one desk, one loop, one evaluator that the loop cannot see.
look() returns the objects it believes are on the desk. pick(item_id) closes the gripper on one. place(bin) drops the held object into a named bin. home() parks the arm. Every result comes back in the shape you already know: content for the model, structuredContent for your code, and isError set when the call succeeded but the action did not. Calling a skill that does not exist raises instead, because that is a protocol error rather than a tool error, and the two want different handling.
Each desk has five objects. Four of them have a bin. The fifth is a gadget, and there is no bin for a gadget on purpose.
Six rules
| Rule | Why it is a rule |
|---|---|
The agent reaches the world only through server.call(...). | Every shortcut into ground truth silently repairs a failure class you were trying to measure. One peek at the desk object and your grounding number becomes fiction. |
goal_check reads the desk, never the transcript. | A grader built from the agent’s own results cannot fail a run in which every skill reported success, which is one of the three things you are here to catch. |
| Every failed run gets exactly one cause from a fixed set. | Free-text causes cannot be counted. The histogram is the most informative output of the whole project. |
A look result is a belief. Re-perceive after every failure and before declaring success. | This is the glance. It is one line of code and it is the largest single effect in the results table below. |
| Charge planner latency for every decision. | The idle fraction is a deliverable, and you cannot compute it if you did not count the calls. |
| Every retry is bounded, and exhausting the bound is a reported failure rather than an exception. | An unbounded retry loop against a wedged skill looks like a hang, and a hang has no cause field. |
Build it in five checkpoints
Each one runs before you start the next.
| Checkpoint | You are done when |
|---|---|
| 1. Drive the skills by hand | In a REPL you can look, pick something, place it, and produce a failed pick whose isError text you have read. |
2. goal_check and the evaluator | You can hand-play a desk to completion and to failure, and the grader agrees with you both times. |
| 3. The plan-once loop | look once, execute the whole plan, never look again. It runs end to end and succeeds sometimes. |
4. Read isError and retry | Bounded retry on a failed pick. At the default dials this should be very close to perfect, and you can predict the number before you run it. |
| 5. The three-dial sweep | Three loops crossed with three dial settings, run at 500 episodes each. Nine success rates, nine step-completion rates, one cause histogram. |
Checkpoint 4 is where the arithmetic from the evaluation lesson pays out, so predict before you measure. With four fileable objects and a grasp probability of 0.85, a loop that never retries succeeds when all four grasps land: . Give it four attempts per object and each object succeeds with probability , so the task succeeds of the time. My reference loops measured 49.6% and 99.8% over 500 episodes each. If your checkpoint-4 number is not within a couple of points of 99.8%, you have a bug, and you know its size before you go looking.
Three dials, three failure classes
Wider than the screen; scroll it sideways.
--grasp-psets the probability that a pick succeeds. When it fails, the gripper is empty, the object may have shifted, and the result says so. This is “the skill failed”.--drift-psets the probability that, after any call, somebody puts another object on the desk. Nothing reports it. This is “the world changed”.--miss-psets the probability thatlookomits an object or returns a confident wrong label. This is a grounding error, and it is the one that produces “the plan was wrong” without anything having gone wrong.
The experiment that is the point
Three loops. The first plans once and executes. The second reads isError and retries the same pick up to four times, but never perceives again. The third perceives at the top of every cycle. Five hundred episodes each, at three dial settings.
| Loop | grasp 0.85 | grasp 0.85, drift 0.15 | grasp 0.85, miss 0.10 |
|---|---|---|---|
| Plan once, ignore results | 49.6% | 10.8% | 22.6% |
Retry on isError | 99.8% | 14.8% | 36.0% |
| Re-perceive every cycle | 100.0% | 95.8% | 50.2% |
Those are my reference numbers, and the interesting content is in the differences rather than the values.
Reading isError is worth fifty points, and only against one failure class. Column one goes from 49.6% to 99.8% for the price of an if statement. Column two barely moves: 10.8% to 14.8%. Retry answers “do it again”, and “do it again” is no answer at all when the problem is that there is now an object you never saw.
Re-perception is what buys the world back. Column two goes to 95.8% the moment the loop looks again at the top of every cycle. Nothing else in the project produces an eighty-point swing. That is the glance from the hook, priced.
Nothing in the loop fixes grounding. Column three tops out at 50.2% with the best loop. If the detector never reports an object, no amount of replanning, retrying or re-looking recovers it, because every look is drawn from the same broken distribution. The only fixes live below the agent: a better detector, a second view, a different sensor, or a task design where a missed object is cheap.
The two acceptance criteria
This milestone is the one place in the course where the simulation reader is not substituting for anything. The mock desk is the assignment on both paths, and it is the whole assignment on one of them. The hardware criterion is an extension, and it is held to the same standard as everything in Module 4 rather than treated as a victory lap.
Criterion one, required on both paths. The mock desk, in full:
python clear_the_desk.py --demoruns one episode and prints every call, everyisError, and the grader’s verdict with its list of unmet subgoals.python clear_the_desk.py -n 500 --seed 1prints full-task success with its interval, step completion, interventions per task, the idle fraction, and the cause histogram.- Checkpoint 4 lands within a couple of points of the number you predicted from .
- The nine-cell table exists, at 500 episodes per cell, with your own numbers.
- The gadget is still on the desk in every successful run.
- A writeup of about a page: the table, which mechanism bought which column, the idle fraction at your planner latency, and one paragraph on what you would have to change to move the third column.
That last paragraph is the assignment. Everything above it is the setup.
Criterion two, if you have the arm from Module 4. Swap SkillServer for the MCP server you
built, point the agent at your own policies, and clear an actual desk. Same rigour as the mock,
which means these five things rather than a video:
- Ten uncut runs in one sitting, camera on throughout, desks dealt from a written list of starting layouts fixed before the first run rather than arranged by feel.
- The same goal check, reading the desk rather than the transcript. On hardware that means you, judging against a criterion written down first, or a camera check the agent cannot reach. A grader built from the agent’s own results fails here for exactly the reason it fails in the mock.
- Causes from the same fixed list, one per failed run, plus a new bucket named unanticipated whose contents you write out individually. That bucket is the deliverable.
- Interventions counted, every time your hand entered the workspace, with what it was for.
- The comparison stated: your mock table’s default-dial cell against your ten real runs, with both intervals, and a sentence on which of the three failure classes your real desk actually produced.
Expect the numbers to be much worse and the failures to be less tidy. A mock desk has exactly three failure modes because you wrote three, and a real one has a long tail you did not think of.
Check yourself
1. Your plan-once loop measures 38% at the default dials instead of the predicted 52%. Where do you look first?
At the gadget. The prediction counts only the four fileable objects, so a loop that also tries to file the gadget is attempting five picks and failing a subgoal that has nothing to do with grasping. The second candidate is the retry-adjacent case: a pick that fails and leaves the object displaced, followed by a place call issued anyway on an empty gripper, which returns isError and may leave your bookkeeping believing an object was filed. Both show up immediately in the cause histogram if you attribute honestly, which is why the histogram comes before the tuning.
2. Retry takes the first column from 49.6% to 99.8% and the second from 10.8% to only 14.8%. Explain the asymmetry in one sentence, then say what the second column needs.
Retry re-executes an action whose failure was reported, and the second column’s failure is never reported at all, so there is nothing to trigger the retry. The world changed underneath a plan that was correct when it was made, every skill returned success, and the loop finished satisfied. What that column needs is re-perception: a fresh look at the top of each cycle so the plan is rebuilt against the desk as it is now, plus a final check before declaring success, since the change can land after the last skill call.
3. Why can no amount of work on the planner move the third column?
Because the corruption happens before the planner sees anything. A missed object is absent from every observation the planner will ever receive, and a mislabelled one arrives with a confident wrong label. The planner then produces a perfectly sound plan for the world it was shown. Replanning draws another sample from the same broken detector, so it is not information, it is a second guess with the same bias. The fixes all sit below the agent: a better detector, a second viewpoint, a depth or weight sensor that fails differently from the camera, or making a missed object cheap rather than fatal.
4. Your goal check reads the agent’s list of completed steps instead of the desk. Which of the three columns still measures correctly?
None of them, and the first one most deceptively. With a transcript-based grader, any run whose skills all returned success is graded as a pass, so the drift column goes to roughly 100% and the grounding column reports success on desks with objects still on them. Even the grasp column is wrong, because a place issued on an empty gripper returns isError in this harness but a bookkeeping bug on your side could still mark the subgoal done. The grader has to read state through a path the agent does not control; that is the only property that makes it a grader.
5. At a planner latency of 1.2 s the reference loop spent about 32% of its wall clock thinking. Name two changes that reduce it and say what each costs.
Merge skills, so pick and place become one file(item_id) call: one decision instead of two, roughly 7.5 s of motion per decision instead of under 4, and the idle fraction falls accordingly. The cost is resolution: the planner can no longer intervene between the grasp and the placement, so a slip mid-sequence is handled by the skill or not at all. Alternatively, drop the look call from cycles where nothing has failed and nothing has been placed since the last one. The cost there is the drift column, which is precisely the thing re-perception was buying, so measure before and after rather than assuming.
6. Someone proposes reporting the mean of the nine cells as “the system’s success rate”. What is wrong with that, beyond the obvious?
Two things. The dial settings are chosen by you, so the mean is a statement about your choice of adversity rather than about the system, and it moves whenever you add a harder cell. More seriously, averaging hides the only structure in the table: three failure classes that respond to three different mechanisms. A single number cannot tell a reader that retry buys column one and does nothing for column two, which is the entire result. Report the table. If one number is genuinely required, report the worst cell and name its dial setting.
Do this
Build it. The scaffold is at project/clear_the_desk.py, and there is no solution file, deliberately.
The scaffold gives you a complete world, a complete four-skill server with the three dials wired in, the tool schemas, a Wilson-interval helper and the report printer. It stubs exactly three things with NotImplementedError: goal_check, Agent.run and run_episode. Fill them in against the five checkpoints, in order, running the thing at every stage.
Standard library only, so 500 episodes cost seconds. Four commands drive the whole project:
python clear_the_desk.py --demo
python clear_the_desk.py -n 500 --seed 1
python clear_the_desk.py -n 500 --seed 1 --drift-p 0.15
python clear_the_desk.py -n 500 --seed 1 --miss-p 0.10Run each of those against each of your three loops. That is the nine-cell table. Use 500 episodes rather than 50: at 50, a true 50% rate reports anywhere from 36% to 64%, which is wide enough to hide the effect you are looking for.
Then write the page. The table, which mechanism bought which column, your idle fraction, and the honest paragraph about the third column. Keep it. It is the writeup that makes the difference between having built an agentic robot and being able to say what you learned by building one.
What you can now do
You have built a complete planner-over-skills system: a loop that perceives, plans, calls typed skills, reads structured failures, recovers, and stops when a grader it does not control says the goal is met. You can predict its success rate from per-step reliability before you run it, and explain a gap between prediction and measurement. You can separate three failure classes experimentally rather than by argument, and you know from your own numbers that bounded retry answers one of them, re-perception answers another, and the third is not an agent problem at all. And you can put a defensible number next to the whole thing, with an interval, a step-completion figure beside it, and a cause histogram underneath.