Where you are. You have recorded a dataset, trained a policy, and put it on the arm. This lesson is about what to do with the number that came back.
Twenty tries, two cubes
The arm sets off toward the cube. It looks good. It is smooth, it is committed, it does not hesitate, and for about a second and a half you feel like a roboticist.
Then it dips a couple of centimetres to the left of the cube, closes on nothing, lifts, and dips again. And again. It pecks at the table like a bird until the time runs out.
You reset and run it again. Same thing. You run it twenty times and it picks the cube up twice.
The next hour decides whether you get good at this. One version of that hour doubles the training steps. Another downloads a bigger model. Both are the wrong hour, because the arm is doing exactly what it was shown, at high fidelity, with total confidence. You are not looking at a model that failed to learn. You are looking at your dataset, played back.
The idea in one paragraph
A policy trained by imitation is a compressed copy of your demonstrations, so its success rate is mostly a property of your data and only a little a property of your model. That makes improvement a measurement problem rather than a training problem. You run the policy enough times to see how it fails, sort those failures into named buckets, and spend the next batch of episodes on the biggest bucket. Record, train, evaluate, diagnose, record again. The step everybody skips is diagnose, and skipping it converts the loop into a treadmill where the dataset grows every weekend and the success rate does not move.
Wider than the screen; scroll it sideways.
Expect the first policy to fail completely
You built an evaluation harness in Module 3 and ran policies in simulation, where a mediocre first attempt still lands somewhere above zero. Real hardware does not grant you that. A first real policy commonly does nothing useful at all, and the reason is not mysterious: it takes only one broken assumption anywhere in the chain from camera placement to demonstration consistency to move the whole thing below the threshold where anything succeeds.
Wider than the screen; scroll it sideways.
The failure bucket is the entire signal
Sort the failures while they are fresh. Not “it did not work” but which of a small number of shapes it took. LeRobot’s own guidance maps three of those shapes onto three completely different causes, and it is worth learning by heart because they point at different parts of the building.
| What you see | What it usually is | What to record next |
|---|---|---|
| Fails at one specific stage every time, and the earlier stages look fine | Coverage of that stage. The demos rushed through it or varied there. | 10 to 20 more episodes targeting that stage, set up so the episode starts near it |
| Flaps, oscillates, or dithers instead of committing | Inconsistent demonstrations, or not enough training | Re-record your worst episodes; check the training loss actually flattened |
| Ignores the object entirely, or wanders off | Camera framing or lighting, not a model problem | Nothing. Fix the rig first; more episodes through a bad view are wasted |
That third row is the one that costs weekends. A policy that never looks at the object is not undertrained. It cannot see the object, or it can see it under one lighting condition and not the one in front of it now. Recording another 50 episodes through the same bad view produces a larger dataset with the same defect, and you will have spent an afternoon to learn nothing.
Coverage, one axis at a time
LeRobot’s documented rule for the first dataset is deliberately narrow: the first 50 episodes should be a constrained version of the task. One object, fixed position, fixed cameras, one operator. Only after that first policy trains and gets evaluated do you add diversity, and then along one axis at a time: more positions, then more lighting, then more objects, then more operators.
The reason to change one axis at a time is diagnostic, not superstitious. If you widen positions and lighting together and the rate drops, you have no way to attribute it. Changing one axis per round is the difference between an experiment and a hobby.
The most common coverage defect is not a missing axis, it is an uneven one. Left to yourself you will put the object roughly where it is convenient, which is the middle of the table, and after 50 episodes the corners will have three episodes between them. Your average success rate will look fine and will collapse the moment anybody else places the cube.
Wider than the screen; scroll it sideways.
The fix is mechanical. Divide the workspace into cells, decide how many episodes each one gets, and deal them in a shuffled order so that you cannot drift back toward the middle without noticing. code/coverage_grid.py does both halves of that: plan emits the shuffled schedule, and audit reads back where the object actually was and tells you the imbalance ratio. Both grids in the figure above are its real output.
The episodes you should not keep
A bad episode is worse than a missing one. A missing episode teaches nothing; a bad one actively teaches the wrong thing, and imitation learning has no mechanism for noticing that you did not mean it.
LeRobot’s advice here is unusually direct: do 5 to 10 demonstrations without recording to build a deliberate, repeatable strategy first; hesitant or inconsistent demos teach the model hesitation; use the same grasp, the same approach vector and the same timing every time; and optimise for speed only after the strategy is settled. While recording, the ← or r key discards the episode you just did and re-records it. Use it the instant you feel the demonstration go wrong, rather than telling yourself you will filter it later.
Later filtering is a real option and worth knowing about, but it is expensive and imprecise. Deleting an episode you already recorded requires you to remember which one it was, and by the time you are looking at a dataset viewer, three near-identical clips of you fumbling a grasp all look the same.
Why the curve has steps in it
Plot success rate against episode count over several rounds and you will not get a smooth rise. You will get a flat stretch, a jump, another flat stretch. That shape is a consequence of coverage being discrete.
A policy that has never seen the left third of the table fails there essentially all of the time, no matter how many more episodes you record on the right. That third contributes a fixed block of zeroes to your average, and the block does not shrink gradually as the dataset grows. It disappears in one step, on the round where you finally record there. The same is true of every distinct condition you have missed: each is a step waiting to be taken, and adding volume in conditions you already cover moves nothing.
This is what makes the diagnose step pay for itself so heavily. Volume moves you along a flat stretch. Coverage moves you off it.
When to stop turning the loop
Two signals end a round of data collection, and they are different.
The first is that the marginal batch stops moving the number. You recorded 20 more targeted episodes, retrained, evaluated, and the rate is within its own noise of where it was. That is not necessarily failure; twenty trials is a wide measurement, and you will learn how wide in publishing hardware work.
The second is more useful: the largest failure bucket becomes something data cannot fix. The gripper fingers cannot close on that shape. The wrist camera is blocked by the fingers at exactly the moment the grasp happens. The object rolls. When the biggest bucket is a property of the rig, you stop collecting and change the rig, because no volume of demonstrations teaches a mechanism to do something it cannot do.
Without hardware
The loop is identical and runs far faster, so the discipline it needs is different.
- Instead of twenty watched trials, run two hundred and bucket the failures automatically where the predicate can tell them apart, by hand where it cannot.
- Measure this: the bucket histogram before and after each batch of new episodes, and re-run the whole evaluation after every change, which the hardware reader cannot afford.
- What you lose, and it is the big one on this path: the world is now editable. When the biggest bucket points at the scene rather than the data, you can widen the gripper, lower the friction or move the object closer and the number goes up. That is the simulation path’s characteristic self-deception and it has no hardware analogue. Rule: the scene file is frozen at the same moment the dataset is, and any change to it restarts the experiment and gets written down.
Check yourself
1. Your policy reaches confidently toward the cube and closes on empty air, every time. Someone suggests training for another 40,000 steps. Why is that probably the wrong move, and what would you do instead?
Because the failure is consistent and confident, which is the signature of a policy that learned something precisely, not one that has not learned yet. An undertrained policy dithers and produces different failures on different runs. This one is executing a wrong belief crisply. That points at the setup rather than the optimiser: the camera view, the calibration, or a systematic offset between recording and evaluation conditions. Instead of training longer, watch ten trials, note whether the miss is in the same direction every time, and measure it. A repeatable offset is a geometry bug, and debugging by symptom takes it apart.
2. You have 50 episodes and a 20% success rate. You have time to record 20 more. What has to happen before you record a single one?
You have to watch failures and bucket them. Twenty episodes aimed at the largest bucket is worth several times twenty episodes added to the middle of the distribution, and you cannot aim without a bucket count. Concretely: run at least 10 to 20 trials, tag every failure with one label, count the labels, and check whether the top label is a data problem at all. If it is “never approached the object”, the answer is not more episodes; it is fixing the camera view before recording anything.
3. Why does LeRobot recommend that the first 50 episodes deliberately restrict the task, when the goal is a policy that generalises?
Because diversity you add before you have a working policy is diversity you cannot interpret. A constrained first dataset gives you a baseline that either works or does not, with few enough variables that a failure is attributable. Once something works in the narrow case, each axis you widen produces a measurable change you can attribute to that axis. Starting wide gives you a policy that fails everywhere and no way to tell which of six differences caused it. It is the same reason you do not change six things at once when debugging anything.
4. Your policy is at 70% overall. Broken down by start zone it is 95% in four zones and 10% in two. Why is 70% the wrong number to act on, and what do you record next?
Because 70% describes a policy that does not exist. What exists is a policy that works well in most of the space and not at all in part of it, and the aggregate is just a weighted average of those two behaviours, weighted by how often you happened to test each. Acting on it leads you to broad improvements when the whole deficit is concentrated. Record into the two failing zones, ten to twenty episodes each, and check whether they are underrepresented in the dataset, out of camera view, or at the edge of the arm’s reach. The last of those three is not a data problem.
5. Why is an episode you recorded badly and kept worse than an episode you never recorded?
Because behaviour cloning fits what it is shown without any signal about what you intended. A missing episode leaves a gap; a bad one places a target in the wrong spot, and the model will spend capacity reproducing your hesitation, your correction wobble, or the pause where you looked at the screen. This is why LeRobot’s advice is to practise 5 to 10 demonstrations with recording off, and to discard immediately with the ← or r key rather than planning to filter later. Consistency in the demonstrations is the actual product; the episode count is just how much of it you have.
6. When should you stop collecting data and change the hardware instead?
When the largest failure bucket describes something the mechanism cannot do. Fingers that cannot close around that shape, a wrist camera that is occluded by the gripper at the moment of grasp, an object that rolls out of a stable pose, a start zone at the edge of reach. Demonstrations teach a policy which motions to produce; they cannot teach a mechanism a capability it lacks. The tell is that even your own teleoperated demonstrations fail or feel awkward in that bucket, which is why the milestone in this module asks you to confirm you can teleoperate the task ten times out of ten before recording anything.
Do this
One full turn of the loop, on your own arm and your own dataset. Budget an afternoon.
1. Audit what you already have. Write down where the object started in each recorded episode. If you did not log it, watch the first frame of each episode in the dataset viewer and estimate the cell. Then:
python code/coverage_grid.py audit --csv recorded.csv --cols 3 --rows 2Note the imbalance ratio and any empty cells. Most first datasets come back somewhere between 5x and 30x imbalanced. That number, not the episode count, is what your policy’s edge-of-table behaviour is made of.
2. Run 20 trials and bucket every failure. Before the first trial, write down four to six failure labels and the sentence you will use to judge success. Then run, and tag as you go. Do not tag from memory afterwards.
3. Count the buckets and pick one. Take the largest. Write one sentence naming the cause you believe is behind it and how you would tell if you were wrong. If the largest bucket is “ignored the object”, stop and fix the cameras before recording anything.
4. Record 20 targeted episodes. Aimed at that bucket, set up so each episode starts close to the failing situation rather than at the beginning of the task. If the bucket was a coverage hole, use the planner to deal the placements:
python code/coverage_grid.py plan --episodes 20 --cols 3 --rows 2 --seed 15. Retrain, re-evaluate, and write down both numbers. Old rate, new rate, and the bucket counts for each. Keep this in a running log. Three rounds in, that log is the most valuable file in your project, because it is the only record of which interventions actually moved anything.
What you can now do
You can read a real evaluation as a diagnostic rather than a grade: separate a confident, repeatable failure from an untrained one, sort failures into buckets, and identify which bucket the next twenty episodes should target. You can audit a dataset’s coverage numerically instead of by feel, deal placements to a plan so the coverage stays even, and explain why the success-versus-data curve moves in steps rather than smoothly. And you can recognise the point where more data stops being the answer and the rig has to change.