Where you are. You can drive the follower with your hand and you know how fast you are allowed to move. This lesson is about what your hand should do, because everything it does becomes a training label.
The arm that pecks at the table
The policy is trained. You clear the desk, put the brick down, and start it.
The arm sets off, comes down toward the brick, stops a couple of centimetres short, lifts, comes down again, stops, lifts. Down, up, down, up. It never closes the gripper. After thirty seconds you switch it off.
Nothing is broken. The training ran to completion, the loss came down, and the arm is reaching for the right place. So go and watch your own recordings back at four times speed. Somewhere around the twelfth one you came in, did not like the angle, backed off an inch, and came in again. You did it in maybe eight of your fifty recordings. You never once thought of it as part of the demonstration; it was you correcting yourself.
The network cannot see the difference. From where it sits, backing off is simply a thing that happens when a gripper is two centimetres above a brick.
The idea in one paragraph
A demonstration is not a video of you doing a task. It is a stream of labelled examples. Thirty times a second the cameras and the joint sensors write down the situation, and whatever the arm did next is written down as the correct answer, and a policy is fitted to that. It has no access to your reasons. So if the same situation sometimes gets one response from you and sometimes another, then “sometimes this, sometimes that” is the honest summary of your behaviour, and the model will reproduce it - as a blend of the two, as a coin flip, or as a stall. Which makes the quality of a demonstration something other than whether it succeeded. It is whether it matches your other demonstrations.
Nine hundred rows an episode
Wider than the screen; scroll it sideways.
Do the arithmetic once and it stops being abstract. A thirty-second episode recorded at 30 frames a second is 900 rows. Fifty of them is 45,000 rows, which is the worked example LeRobot’s own guidance uses when it converts a recording session into a number of training steps.
Every one of those rows carries the same weight. There is no column for “this bit was sloppy” and no column for “I was thinking here”. A two-second pause while you glance at the screen is sixty rows whose recorded correct answer is stay still, in a situation that looks like an approach.
Two good habits average into one bad one
Wider than the screen; scroll it sideways.
Suppose the object sits behind a mug. On thirty of your episodes you went round the left of the mug; on the other twenty you went round the right. Both routes work. Both are things a competent operator would do.
The model sees one situation with two different correct answers. Depending on how the policy represents its output, that comes back as an average - straight through the mug - or as an unlucky choice halfway through, where it commits left for four frames and right for the next four and ends up doing neither.
The practical consequence is uncomfortable but simple. Pick the left route or the right route, and then always take it. You are allowed to be arbitrary. You are not allowed to be inconsistent.
Success is not the label
This is why the discard key exists during recording, and why LeRobot’s guidance is to use it the moment a demonstration goes wrong rather than planning to filter later. Recording a real dataset covers the mechanics; the habit is what matters, and the habit is: the instant you think “I’ll clean that one up later”, drop it now.
What “the same” actually means
Five things to hold fixed. They are all boring, which is the point.
| Hold fixed | What that means in your hand | What varying it teaches |
|---|---|---|
| Grasp point | The same face of the object, the same distance from its edge, every time | That any part of the object is a target, so the gripper aims at the average of them, which is often thin air |
| Approach vector | The same direction of arrival: straight down, or in from the front, but never both | That approach direction is free, so it gets averaged into a diagonal that fits neither |
| Phase timing | Roughly the same seconds spent approaching, gripping, lifting, carrying | That duration is arbitrary, which makes the policy’s sense of “how long until the next thing” unreliable |
| Speed | Inside the budget you measured last lesson, so the follower never trails | The lag, overshoot and settle of an arm being driven too fast, since the follower is what the camera recorded |
| Your body | Same chair, same side of the desk, same hand on the leader | Nothing directly, but changing it changes everything above without you noticing |
What is allowed to vary is where the object starts, and that variation should be planned rather than incidental. Planning it is a whole discipline of its own and it belongs to designing your first task.
Rehearse with the recording off
Wider than the screen; scroll it sideways.
LeRobot’s guidance is direct about this and worth taking literally: do five to ten demonstrations without recording first, and build a deliberate, repeatable strategy. Hesitant or inconsistent demonstrations teach the model hesitation. Use the same grasp, the same approach vector and the same timing. Optimise for speed only once the strategy is settled.
The way to make that concrete is to write the strategy down before you rehearse it, as named phases with a target duration each:
approach 2.0s come straight down, 3cm in front of the near face
grip 1.0s close fully, then pause half a second on the object
lift 1.0s straight up 10cm, no lateral drift
carry 3.0s one smooth arc to above the bowl
release 0.8s open fully, hold position
retreat 1.0s straight back to homeThe card does one more thing for you: it turns “was that a good demonstration?” into a question with an answer. Either the phases happened in order and roughly on time, or they did not.
A number instead of a feeling
Watching your own episodes back is necessary, and it is not enough, because the fifteenth one looks fine when you are tired. So measure them. code/demo_smoothness.py computes four numbers per episode, none of which is an absolute standard:
- detour - distance actually travelled through joint space divided by the straight-line distance. Backing off and coming in again inflates it.
- dwell - the fraction of the episode spent nearly still, measured against that episode’s own busy speed rather than a fixed cutoff.
- reversals per second - how often joints changed direction, past a deadband that ignores encoder dither.
- interior stops - how many separate times the whole arm came to rest and set off again, ignoring the rest at each end. This one is the most legible: zero is a single committed motion.
They are deliberately relative. The script ranks your episodes against your own median and flags the outliers, because there is no published threshold for “smooth enough” and inventing one would be worse than useless.
Without hardware
Consistency is measurable in a file, which makes this lesson one of the better ones on the simulation path.
- Instead of your own session, score somebody else’s.
lerobot/svla_so101_pickplaceis 50 demonstrations by one hand: episode lengths run 183 to 306 frames, median 230, checked on 10 August 2026. - Measure this: the length distribution, the per-joint trajectory spread across episodes at matched progress fractions, and the ratio of longest to shortest episode, which is about 1.7. Then name the episodes you would have discarded, and write the rule you discarded by before you look at which ones it catches.
- Try
lerobot/svla_so100_stackingnext. It has 56 episodes, three of them under 30 frames and one exactly one frame long. Nobody pruned it for you, which is the point.
Check yourself
1. Your fifty episodes all succeed. Why might the trained policy still hesitate at exactly the same point every time?
Because success is measured at the end of an episode and training happens per frame. If most of your episodes contain a pause at the same stage - lining up the grasp, say - then the rows recorded during those pauses all say “in a situation that looks like this, do not move”. At 30 rows a second even a short hesitation repeated across many episodes is a large, coherent block of training data that all points the same way. The policy learns the pause as faithfully as it learns the reach.
2. Going round the left of an obstacle and going round the right are both correct. Why is doing both worse than doing either?
Because the model is fitting a function from what it observes to what to do, and both routes start from the same observation. Two different correct answers for one input is a multimodal target. A policy that averages produces a path between the two, which goes through the obstacle. A policy that can represent both modes has to spend data learning each of them, and can still switch modes mid-motion and produce a trajectory that belongs to neither. Picking one arbitrary route and sticking to it costs you nothing and removes the whole problem.
3. Why is a messy episode that succeeded worse for your dataset than an episode you never recorded?
A missing episode leaves a gap: less data, and the model is no more wrong for it. A messy episode adds hundreds of rows that assert a wrong action was correct in a situation you will encounter again. Behaviour cloning has no signal about your intent and no weighting to downplay the bad rows; they train the model exactly as hard as the good ones. Which is why the discard key is meant to be used in the moment rather than in a filtering pass you will not do.
4. You are told your demonstrations must be “consistent”. Name four specific things that has to mean, in your hand.
The same grasp point on the object. The same approach direction. Roughly the same seconds spent in each phase. And a speed inside the budget where the follower does not trail your hand. A fifth, less obvious one is your own body position, because changing where you sit silently changes all four of the others.
5. The smoothness script says one of your five episodes has twice the median detour and two interior stops when every other episode had none. What have you learned, and what have you not?
You have learned that this episode is unlike the others and is worth watching back; probably you hesitated or re-gripped. You have not learned that it failed, that the other four were good, or that any of them will produce a working policy. The metrics are relative to your own median, so if all five episodes share the same bad habit, all five score consistently and nothing is flagged. Consistency is necessary, not sufficient.
6. Why does this lesson come before the one that records fifty episodes, rather than after?
Because demonstration habits are cheap to fix before recording and expensive afterwards. Fifty episodes is a couple of hours of your afternoon, and a habit you did not notice is present in all of them, which means the fix is re-recording rather than editing. Rehearsing costs ten minutes and catches it while the cost is still ten minutes.
Do this
About an hour, mostly away from the keyboard. Only steps 2 and 5 involve running anything, and nothing here was tested by the author against a physical arm.
1. Write the strategy card. On an actual index card, next to the arm. Phases down the left, target seconds down the right, one line of what your hand does for each. Use the template above as a starting point and change every line of it, because your task is not that one.
2. See what the metrics do, with no hardware. From module-04-hardware/code/:
python demo_smoothness.pyThree synthetic rehearsed demonstrations and one hesitant one, all reaching the same end pose in the same six seconds. Look at which numbers separate them and which do not. Then read hesitant() in the source and note how small the change actually is: one pause, one back-off, one second approach.
3. Rehearse ten times with the recording off. Stopwatch running, strategy card visible, saying the phase names out loud as you hit them. Write down each duration.
4. Set your own tolerance and hold to it. Something like: three consecutive runs whose total duration is within 15% of each other, with the phases in order and no re-grips. The exact number is yours; the discipline is that you set it before you start, not after you see your times. If you cannot hit it, your strategy is too hard for your hand and the fix is to simplify the strategy rather than to try harder.
5. Wire up the loader. Fill in load_episode() in demo_smoothness.py against whatever LeRobot version you installed. Print one row and one column name before you trust the shape, since a silently transposed array makes every metric meaningless without raising anything.
6. Carry one rule into the recording lesson. Record five episodes, not fifty. Score them, watch the flagged one back, fix whatever it shows you, and only then record the rest. The two hours you would otherwise spend collecting a dataset with a habit in it are the most expensive two hours in this module.
What you can now do
You can explain why an episode that succeeded may still be bad training data, and why two valid strategies mixed together are worse than either one alone. You can write a demonstration down as timed phases before performing it, rehearse until you can reproduce it, and use the discard key without arguing with yourself. And you can score your own episodes on four relative measures and tell the difference between an episode that is unusual and an episode that is wrong, which are not the same thing and get confused constantly.