Where you are. You can assemble, calibrate, teleoperate, record, train, deploy, diagnose and publish, or you can do the simulation-path equivalent of each. This is where all of it becomes one result with your name on it. There are two acceptance criteria below and neither is the easy one; pick the path you chose in lesson 0 and hold yourself to that column.
Trial seven
It is a Sunday afternoon. The camera has been running for eleven minutes. On the table there is a bowl, a cube, and a strip of masking tape with four small crosses drawn on it.
Trial seven. You put the cube on the third cross, step back, and start the rollout. The arm goes over, dips, closes on the cube, lifts it, swings across, and lets go about two centimetres short of the bowl. The cube bounces off the rim and rolls under the monitor.
You do not stop the camera. You do not delete the file. You reach for the cube, put it back on the third cross, and write failure / dropped-early on the sheet next to trial seven, because that is what happened, and because there are thirteen trials left and the number at the end has to mean something.
That is the whole milestone. The robot is the easy part.
The idea in one paragraph
Take one task, get a policy to do it reliably enough to clear a bar you set in advance, and prove it with an uncut scored run under a protocol you wrote before the first trial. Nothing here is new. It is the data-quality loop turned two or three times, with symptom-driven debugging in between and an honest evaluation at the end. What is new is that nobody tells you when it is done, the failures are yours, and the number you publish is the first robotics claim you will have made that somebody else could check. The bar differs by path because the constraints differ: the hardware reader is limited by wall-clock and pays for it in a wide interval, and the simulation reader is limited by nothing and therefore owes a narrow one.
Wider than the screen; scroll it sideways.
Pick a task you can already do by hand
Task selection decides most of your outcome before you record anything, and the criterion is not “interesting”, it is “repeatable by you”.
Wider than the screen; scroll it sideways.
Good candidates look like a cube into a bowl from four marked start zones, or a two-colour sort into two bins. Both have a real distribution of initial conditions, a clean success criterion, and failure modes worth studying.
Freeze the rig before you record anything
Everything the cameras can see is part of what the policy learns. So fix it, then leave it alone for the duration of the project.
- Clamp the cameras. Not “place them carefully”: clamp them, and mark the clamp positions.
- Mark the table. Tape, a printed grid, crosses at the start zones. You need the same physical setup at evaluation time as at recording time, and you cannot get it back by memory.
- Control the light. Artificial, diffuse, no window contribution, no moving shadows. Record every episode for a dataset in one session under that light.
- Save a reference frame from each camera at recording time, so you can restore the view later.
- Do the camera acceptance test, which is the best single check in this module: look only at the camera frames and ask whether you could do the task from those views alone. If the answer is no, no policy will do better.
Then write the numbers down: camera models, positions, the LeRobot version, the arm ids you calibrated with. These become the model card later, and half of them are impossible to recover after the fact.
Round one is for learning, not for succeeding
Record about 50 episodes to a coverage plan, train, and roll out. Expect this round to fail, possibly completely. That is the normal result on a first real dataset, and going in expecting it is what stops you from concluding that the method is broken or that you need a bigger model.
What you want out of round one is not a success rate. It is ten to twenty watched failures, sorted into buckets, and one sentence naming the cause of the largest bucket. If that bucket turns out to be “ignored the object”, you go back to the cameras rather than the dataset.
What counts as a trial
Before your first evaluation, decide what a trial is. This takes ten minutes and determines whether the number means anything.
Wider than the screen; scroll it sideways.
The void branch is the one that needs discipline. A trial where the bowl slid off the table, or the arm was still moving from the previous run, is legitimately void. A trial where the robot simply failed is not, however much you feel it was about to work. Voids are counted and published, which is what keeps them honest: a run with six voids and fourteen scored trials is a different result from one with twenty scored trials, and a reader is entitled to know which they are looking at.
The twenty trials
One sitting. Camera running from before trial one until after the last. Same reset procedure every time. Conditions dealt in advance by the harness, in a fixed random order, so you cannot unconsciously favour the placements the policy is good at.
Tag every failure the moment it happens, not from memory afterwards. Three near-identical failures are indistinguishable an hour later, and the bucket counts are the most actionable output of the whole exercise.
What 70% is and is not
Seventy per cent over twenty trials means fourteen successes, and the honest interval around it runs from about 48% to 85%. The lower end of that band is below a coin flip.
That is not a reason to be embarrassed and it is not a reason to inflate the number. It is a reason to say it properly and, if you have the time, to run more trials: thirty trials at 70% gives roughly 52-83%, and forty gives roughly 55-82%. The target for this milestone is at least 70% over at least 20 trials, and the “at least” on the trial count is worth more than the one on the rate.
The two acceptance criteria
Both columns are the same experiment run under different constraints. Read down the one you chose.
| Hardware path | Simulation path | |
|---|---|---|
| Entry test | teleoperate the task 10 out of 10, smoothly, same grasp each time | demonstrate the task 10 out of 10 through the keyboard rig, same approach each time |
| Rig freeze | clamped cameras, marked table, one lighting condition, reference frames saved | scene file frozen and hashed, simulator version pinned, camera poses and light in the file |
| Dataset | about 50 episodes to a coverage plan, one session | two datasets of 50 episodes each on the same task and schedule: one teleoperated by your hand, one from the scripted controller. Both hashed. |
| Training | one policy | one policy per dataset, identical recipe and seed |
| Scored trials | 20 consecutive, one sitting, camera on throughout | 200 per policy, initial conditions dealt from a seed never trained on |
| The bar | at least 70%, interval reported | the better policy at 70% or above with the lower bound of its Wilson interval above 60% |
| The extra experiment | none | the hand-versus-scripted comparison, both rates with intervals |
| Robustness | not required; you get drift for free | the same 200-trial protocol re-run under four perturbations never trained on, one at a time |
| Real-data component | it is all real data | train the same architecture on a pinned public SO-101 dataset and report chunk-level action error on a held-out split |
| Publication | dataset, checkpoint, protocol, uncut video, failure reel | the same five, plus scene hash, simulator version and every seed |
Two rows deserve their reason stated.
Why 200 trials rather than 20. The hardware reader’s trial count is set by wall-clock: 20 trials is an hour of resetting a bench, and near 70% the interval runs from about 48% to 85%. In simulation a trial costs milliseconds and a reset is a function call, so the same interval would be laziness wearing the hardware reader’s excuse. At 200 trials a 70% rate carries an interval of roughly 63% to 76%, which is a claim with edges.
Why two datasets. What makes a good demonstration is the lesson the simulation path most obviously loses, because a scripted controller has no bad days. Recording both a hand-driven and a scripted dataset and evaluating them under one protocol turns that loss into a measurement: you find out, on your own task, how much of the final success rate was demonstration consistency rather than model capacity. The hardware reader cannot run this experiment at all, because they have no way to produce the scripted half.
Check yourself
1. Why is the entry test “can you teleoperate it ten times out of ten” rather than “is this task interesting”?
Because behaviour cloning copies your demonstrations, including the parts you did not mean. A task you can only do seven times in ten by hand produces a dataset where three episodes in ten contain a recovery, a hesitation, or a different strategy, and the policy learns all of them as if they were the task. Your own reliability is a ceiling on the policy’s reliability, not a starting point it improves on. The interesting-ness of the task can come later, once you have a working loop; a task you cannot demonstrate consistently gives you nothing to debug with, because you can never tell whether a failure came from the model or from the data.
2. The bowl slides off the table mid-trial. On the next trial the robot grasps and drops the cube. Which is void, and why does collapsing the two destroy the number in both directions?
The first is void: the policy was never tested, because the task became impossible for reasons outside it. The second is a failure: the policy was tested and did not succeed. Collapse them one way, by voiding anything that went badly, and the rate measures your patience rather than the robot, drifting upward every time you feel a trial was unfair. Collapse them the other way, by scoring broken-rig trials as failures, and you are measuring the table rather than the policy, which makes the number impossible to compare across sessions. Publishing both counts separately is what keeps the distinction honest, since a reader can see how many voids you allowed yourself.
3. Your twenty trials give 70%, but the per-zone breakdown is 90% in two zones and 20% in the other two. What do you publish, and what do you record next?
Publish both: the headline rate with its interval, and the breakdown, because the aggregate describes a policy that does not exist. What exists is one behaviour that works and one that does not, averaged by however often you happened to test each zone. Next, record ten to twenty episodes into the two weak zones and check three things about them first: whether they were underrepresented in the dataset, whether the object is fully visible from both cameras there, and whether they sit near the edge of the arm’s reach. The third is not a data problem and more episodes will not touch it.
4. Why does the harness deal the initial conditions in advance, in a fixed random order, instead of letting you place the object where seems natural each trial?
Because you know which placements the policy handles well, and you cannot un-know it. Placing by feel produces a distribution quietly weighted toward the easy cases, which inflates the rate without any moment where you chose to cheat. Dealing in advance also makes the run reproducible: the same seed produces the same sequence, so a second evaluation after a change tests the same conditions rather than a new sample, and the comparison between rounds means something. The fixed order additionally lets you resume an interrupted session without re-rolling the trials you already did.
5. You are the only person involved and you are trying to be honest. Why does the success criterion still have to be written before trial one?
Because the drift is not a decision you make, it is a decision you fail to notice making. The cube lands half on the rim and settles inside; on trial three you would have called that a failure, and on trial twelve, with the rate looking thin, you will call it a success and experience it as reading the criterion rather than editing it. Honesty is not the safeguard, because the mechanism operates below the level where honesty applies. A sentence fixed in advance, and re-read before every judgement, converts the question from “was that good enough” into “does it match the sentence”, which is a test rather than an opinion.
6. Round one comes back at 0%. Name two things you check before recording a single additional episode.
First, look at the camera frames and ask whether you could do the task from those views alone; a policy that ignores the object is almost always a framing or lighting problem, and fifty more episodes through the same bad view produce a bigger dataset with the same defect. Second, replay a recorded episode on the arm: if the replay does not reproduce the motion you demonstrated, your calibration has changed since recording and every episode in the dataset now means something different from what it meant, so no amount of new data fixes it until that is resolved. Both checks take minutes and both can invalidate an afternoon of recording you were about to do.
Do this
The specification is in project/README.md, with the checklist you tick off. The harness is project/run_eval.py and the report generator is project/eval_report.py. Neither talks to the robot: you run the rollout in another terminal and type the verdict, because the thing scoring the experiment should not be the thing running it. On the simulation path the same two scripts work unchanged, with the rollout driven by your own script rather than by your hand, and the trial count raised.
1. Choose the task and pass the entry test. Demonstrate it ten times by hand. Count. If you get ten out of ten smoothly, with the same approach each time, proceed. If not, simplify the task until you do, then proceed. On the simulation path the demonstrations come through the keyboard rig, and passing this test is harder than it sounds.
2. Freeze the rig and write it down. Clamp, mark, light, reference frames, and the configuration notes that become your model card. On the simulation path: hash the scene file, pin the simulator version, and record the camera poses and light directions that live inside it. Same job, different artefacts.
3. Fill in the four decisions in run_eval.py. The task name, the success criterion, the failure buckets, and the initial-condition distribution. Then check the deal looks sane:
python run_eval.py --dry-run --trials 20Commit protocol.md at this point, before any trials exist. The commit timestamp is your evidence that the criterion came first.
4. Turn the loop until you clear 70%. Record to a coverage plan, train, roll out, watch twenty, bucket the failures, aim the next twenty episodes at the biggest bucket. Change one thing per round. Keep a log of round number, what changed, the rate, and the bucket counts.
5. Run the scored trials in one sitting, recording throughout. Twenty on hardware with the camera on; two hundred per policy in simulation with the screen recording running.
python run_eval.py --trials 20 --out results.jsonl --notes "act-40k, lamp on"
python eval_report.py results.jsonl --group cellRead the per-condition breakdown before the headline. An even rate across conditions is a policy; 90% in two zones and 20% in the others is two behaviours averaged together, and it will not survive contact with anybody else’s table.
5s. Simulation path only: the two extra experiments. First, run step 5 again for the policy trained on the other dataset, so you have the hand-versus-scripted comparison with an interval on each. Second, re-run the whole 200-trial protocol under four perturbations you never trained on, changing exactly one per run: object mass by a factor of 1.5, friction by 0.7, one camera moved two centimetres, and the light direction rotated. Five rows, one table. The degradation across them is your substitute for the drift a bench gives away for free.
5r. Simulation path only: touch real data once. Train the same architecture on a pinned public SO-101 dataset such as lerobot/svla_so101_pickplace, and report chunk-level action error on a held-out episode split. Then write the paragraph explaining why that number is not a success rate, and what you would need to turn it into one.
6. Publish the artifacts and the writeup. Dataset, checkpoint with a pinned version, protocol, uncut recording, failure reel, plus the scene hash, simulator version and seeds if you ran in simulation. Lead the post with the interval and the venue. Include the rounds that failed, especially round one.
What you can now do
You have taken a task from nothing to a measured success rate: chosen a task scoped to what you can demonstrate reliably, frozen a rig so that recording and evaluation conditions match, turned the data-quality loop until the number moved, and defended the number with a protocol written in advance, an uncut scored run and a published failure reel. If you took the simulation path you also have two things the hardware reader does not: a measurement of what demonstration consistency alone was worth, and a robustness table across perturbations you chose.
That last part is what separates this from a demo. You now have a result somebody else can disagree with, which is the only kind worth having, and a dataset - yours, or a pinned public revision - that Module 5 will fine-tune a vision-language-action model on.