Where you are. You have a written task specification, a taped bench and a schedule that says how many episodes go in each cell. This lesson is the session itself.
Do it ten times with your own hand
Leave the robot off. Put the brick in the first cell, pick it up with your fingers, drop it in the bowl, and put it back. Do that ten times without stopping.
Watch what happens across the ten. The first one is careful. By the fourth you have found a groove: same approach angle, same height, same pause before the release. By the eighth your hand has started cutting the corner. Somewhere around the sixth you paused for half a second because your phone buzzed.
Every one of those ten was a success. Every one of them was also different, and if a camera had been running, all ten differences would now be in a file with equal weight, described to the model as the correct thing to do.
That is the job this afternoon. Not “do the task sixty times.” Do the same task sixty times.
The idea in one paragraph
Recording is a loop: reset the scene, demonstrate, then decide immediately whether that episode goes in the dataset or in the bin. What you are producing is a labelled dataset in which your hand is the label, so every hesitation, every second grasp attempt and every mid-episode change of mind is a training signal telling the model that hesitating there is correct. The tooling is simple; the discipline is the whole thing. Hold the light, the cameras, the operator and the strategy fixed for the entire session, vary only which cell the object starts in, and throw episodes away the moment they go wrong rather than promising yourself you will clean the dataset later. You will not clean it later.
Wider than the screen; scroll it sideways.
During teleoperation the command in each frame is wherever you had the leader arm at that instant. So the model’s target is literally your hand’s position, sampled thirty times a second. That is worth holding onto, because it explains every piece of advice that follows: improving the dataset means improving what your hand did, not what the software did.
Dry runs first, with nothing recording
Before the first episode goes to disk, do five to ten demonstrations with the recorder off. The documented instruction is exactly this: “Do 5-10 demos without recording. Build a deliberate, repeatable strategy.”
You are looking for a routine you can reproduce: the same approach vector, the same grasp height, the same pause length, the same path to the bowl. When you can do three in a row that feel identical, you have a strategy. Start recording then.
Wider than the screen; scroll it sideways.
The command
Authenticate once, then record. The Hugging Face CLI is hf, not the older huggingface-cli.
hf auth login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential
HF_USER=$(NO_COLOR=1 hf auth whoami | awk -F': *' 'NR==1 {print $2}')
lerobot-record \
--robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=my_follower \
--robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30} }" \
--teleop.type=so101_leader --teleop.port=/dev/ttyACM1 --teleop.id=my_leader \
--dataset.repo_id=${HF_USER}/so101_brick_v1 \
--dataset.num_episodes=60 \
--dataset.episode_time_s=25 \
--dataset.reset_time_s=10 \
--dataset.single_task="put the red brick in the bowl" \
--dataset.streaming_encoding=true \
--dataset.no_stamp=true \
--display_data=true| Flag | What it controls | Default if you omit it |
|---|---|---|
--dataset.num_episodes | how many episodes before the session ends | 50 |
--dataset.episode_time_s | the recording window per episode | 60 |
--dataset.reset_time_s | the pause between episodes | 60 |
--dataset.fps | control-loop rate the episode is sampled at | 30 |
--dataset.single_task | the sentence stored with every frame | empty, and you want to supply it |
--dataset.push_to_hub | upload at the end | true; set False to stay local |
--dataset.streaming_encoding | encode video during capture instead of writing PNGs first | false, and the tool logs a hint telling you to turn it on |
--dataset.no_stamp | keep your repository id exactly as typed | false, which means your id gets a timestamp appended - see below |
--display_data | live streaming of the observations to Rerun | false |
The two --robot.id and --teleop.id strings must be the same ones you calibrated with. They are the key that finds your calibration file, and a typo does not error; it silently sends you back through calibration.
Locally the dataset lands in ~/.cache/huggingface/lerobot/{repo-id}, using whatever id the run actually settled on. The camera names you choose here, front and wrist above, are part of the dataset, and the same names have to appear when you deploy the policy.
The loop, and the key that matters
Wider than the screen; scroll it sideways.
Three keys run the whole session.
| Key | What it does |
|---|---|
right arrow, or n | keep this episode and move to the next |
left arrow, or r | throw this episode away and re-record it |
escape, or q | end the session, encode the video, upload |
These now work over SSH and on a headless machine as well as on a desktop session, because LeRobot falls back to reading the controlling terminal when no global keyboard backend is available. Tutorials written before mid-2026 will tell you the arrow keys need a graphical display; that is no longer true. Driving the arm by keyboard is a different feature and still does need one.
The letters are the better habit of the two. LeRobot’s own source note says so: an arrow key travels as a multi-byte escape sequence that a laggy SSH or VNC link can split, delay or swallow, while n, r and q are single bytes that survive.
Record in batches, and verify early
Do not record sixty episodes and then look at them. Record ten, stop, and check two things.
Look at the data. Open the dataset visualiser at huggingface.co/spaces/lerobot/visualize_dataset and paste your repository id - the real one, including any timestamp the recorder added. LeRobot’s own guidance says to do this always, before training. You are checking that both camera streams are present, that the object is visible in both, that frames are not dropped and that the episodes are the length you expected.
Replay one. This is the round-trip test:
lerobot-replay --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=my_follower \
--dataset.repo_id=${HF_USER}/so101_brick_v1 --dataset.episode=0--dataset.repo_id here is the name the recorder actually used. If you did not pass --dataset.no_stamp=true, it carries the timestamp suffix, and a replay against the name you typed fails before the arm moves.
The follower re-executes the recorded commands with nothing learned and nothing inferred. If a replayed episode does not do roughly what you did, the problem is in recording or calibration, and no amount of training will repair it. It is also the first time you will see how much of your demonstration was you correcting mid-motion.
What goes wrong mid-session
Two failures show up during recording rather than before it, and both look like software bugs.
The first is a camera that renegotiates its format and throws a width or height mismatch, or simply disconnects every few episodes. Two identical webcams on one USB bus are the usual cause, because they present the same descriptors and their device indices shuffle. Referring to cameras by path rather than index, as the examples do, removes half of that problem; the details live in cameras and viewpoints.
The second is a bus read failure, Failed to sync read 'Present_Position', usually when several joints move at once. LeRobot’s own source comment explains it: Feetech buses can occasionally return a corrupted status packet, which is why both the follower and leader configs carry num_read_retries: int = 2. Occasional retries are normal. A steady stream of them is a physical problem, and debugging by symptom is where that goes.
Without hardware
This is the lesson the simulation path replaces most directly, and it is worth doing both halves rather than one.
- Instead of recording real episodes, pin a public one.
lerobot/svla_so101_pickplaceis 50 episodes, 11,939 frames at 30 fps, six-dimensional state and action, two 480 by 640 cameras, one task string. Note the revision hash; that is your session log. - Instead of teleoperating the follower, teleoperate the MuJoCo scene with the keyboard rig from Module 2 and record 50 episodes yourself, using this lesson’s discard discipline unchanged.
- Measure this: your keep rate on the teleoperated set, and, on the downloaded set, the episodes you would have discarded with the rule written first. The milestone compares the two datasets, so record the hand-driven one even though the scripted controller is easier.
- What you lose: the hand. A scripted controller is more consistent than any human and keyboard teleoperation is inconsistent in ways no human is, so neither reproduces the eighth-repetition groove this lesson opens with.
Check yourself
1. You fumble a grasp, recover in two seconds, and complete the task successfully. Keep it or discard it?
Discard it. Success is not the criterion; consistency is. That episode contains two seconds of recovery flailing labelled as correct behaviour at that observation, and the model has no way to know you did not mean it. Since your hand is the label, an inconsistent hand is an inconsistent label. The forty seconds of re-recording is much cheaper than a policy that learned to wiggle.
2. Why replay an episode on the arm when you can watch the same episode in the dataset visualiser?
The visualiser tells you what was written to disk. The replay tells you whether those recorded commands, sent back to this robot in this calibration, reproduce the motion. It closes the loop through the hardware, so it catches calibration drift, a servo in an error state, and a follower that was not tracking the leader as closely as you believed. Two different questions, and the second one is the one that silently invalidates a whole afternoon.
3. You record thirty episodes on Saturday afternoon and thirty on Sunday evening under different lights. What did you actually build?
Two datasets stuck together, with lighting confounded with whatever else changed between the sessions, such as camera nudges or a different grasp habit. That is not fatal - lighting variation is a diversity axis you would eventually want - but you added it accidentally and at the same time as everything else, so if the policy behaves oddly you cannot attribute it. Diversity you chose is data. Diversity that happened to you is noise.
4. The two --robot.id strings are just labels. Why does getting one wrong cost you a session?
The id is the primary key for the calibration file. If it does not match an existing calibration, LeRobot does not fail; it finds nothing and takes you through calibration again. You end up with a second calibration under a second name, and episodes recorded before and after now sit in slightly different joint coordinates inside one dataset. Use one id per arm, forever, and back up the calibration JSON.
5. Your first ten episodes averaged 22 seconds and your last ten average 15. Nothing failed. Is this a problem?
Probably yes. You sped up, which means the late episodes have a different velocity profile, different pause lengths and probably a tighter approach than the early ones. The dataset now contains two styles, and the model averages them. It is the same multimodality problem as approaching from two sides, arriving through fatigue rather than choice. Either re-record the early batch at the new pace or accept the slower pace and hold it.
Do this
The full session, roughly two to three hours including breaks.
1. Warm up unrecorded. Five to ten demonstrations with no recorder. Stop when three in a row feel identical.
2. Record the first ten. Follow the order printed by recording_plan.py so cells interleave. Judge each episode before you reset. Log it as you go:
python code/session_log.py keep --cell A1
python code/session_log.py discard --cell A2 --reason grasp-missed3. Stop and verify. Open the dataset visualiser and check both camera streams. Replay episode 0 on the arm. Only continue if both look right; this is the cheapest moment to discover a problem.
4. Record the rest, in batches of about fifteen with a two-minute break between. Run python code/session_log.py report at each break. Watch the keep rate over the last ten: when it drops well below your session average, that is tired hands, and the episodes you keep from here are the ones that will hurt.
5. Write down two numbers when you finish: how long the session actually took, and your final keep rate. Practitioner reports put a fifty-episode session at around two hours, three to four times the arithmetic; you now have your own figure, which is the one that matters for planning the next session.
6. Push it. Leave --dataset.push_to_hub at its default, or upload afterwards. The dataset is the artefact this module exists to produce, and Module 5 fine-tunes a foundation model on it.
What you can now do
You can run a full recording session end to end: warm up unrecorded, hold the light and the cameras and the strategy fixed, vary only the start cell, and judge every episode in the moment with the discard key. You can verify a batch in the visualiser and round-trip it through lerobot-replay before committing an afternoon, and you can read a mid-session camera or bus error as a physical problem rather than a code problem. You have a real dataset, recorded by your own hand, in the same format the rest of the course consumes.