Where you are. You have an assembled, calibrated arm, a leader you can drive smoothly, and cameras you have already placed and checked. This lesson decides what the arm is going to learn to do.
Count the things that could be different
Put a red brick on the desk in front of the follower arm and a bowl to its right. Take one frame from the front camera and look at it.
Now, without moving anything, start counting what could be different the next time you set this up. The brick two centimetres to the left. The brick turned ninety degrees. The bowl nudged when you reached past it. The blinds open instead of shut. Your laptop lid throwing a shadow it is not throwing now. A friend driving the leader arm, holding it slightly differently, going slightly faster.
You will get to a dozen without trying. Each one is a scene the robot might face. Your one frame covers exactly one of them.
Everything else on that list is something the robot has never seen, and a robot that has never seen something does not degrade gracefully. It does something confident and wrong.
The idea in one paragraph
A task is not a sentence. It is the set of scenes in which that sentence has to work, and the dataset you record is a sample from that set. Every variable you leave free, where the object starts, how the light falls, who is driving, multiplies the size of the set your sixty episodes have to cover. So your first task should be constrained until it feels like cheating: one object, one small marked start region, one lighting condition, one operator, one grasp strategy. Then you relax exactly one constraint at a time and record more. This inverts the instinct you brought from software, where you design for the general case and specialise later. Here the general case is what you earn, one axis at a time, and starting general is the most common way a first policy ends up at zero.
Wider than the screen; scroll it sideways.
What is hard for a policy is not what is hard for you
Your intuition about difficulty was trained on your own body, which has tactile sensing, a lifetime of object priors and instant error recovery. The arm has none of those. It has two camera views, six joint angles, and whatever regularities were present in your demonstrations.
| Cheap, and right for a first task | Expensive, and worth postponing |
|---|---|
| One rigid, matte object, wider than a centimetre and narrower than the gripper’s opening | Transparent, shiny, thin or deformable objects |
| A target with centimetres of slack, like a bowl | A target with millimetres of slack, like a peg in a hole |
| One sensible way to do it | Several equally good strategies, chosen at random by you |
| Ten to thirty seconds of motion | A minute of chained sub-tasks |
| Failure costs a reset | Failure costs a broken thing, or a hand |
| A reset you can do in under ten seconds, one-handed | A reset that needs two hands and a ruler |
The multi-strategy row is the one that surprises people. If half your demonstrations approach the brick from the left and half from the right, both sets are correct and the policy is being taught two contradictory answers to the same picture. A model that regresses to the mean will drive straight down the middle and grasp nothing. This is exactly the multimodality problem you met in Module 3, and Diffusion Policy exists to handle it; you should not be leaning on that in week one. Pick one approach and use it every time.
The constrained core
The LeRobot guidance on this is blunt and worth keeping verbatim: “First 50 episodes = constrained version of the task: one object, fixed position, fixed camera setup, one operator.” Then, and only then, “add diversity along one axis at a time: more positions, then more lighting, then more objects, then more operators.”
Wider than the screen; scroll it sideways.
The word “positions” in that sequence is doing a lot of work, so pin it down physically. Tape out a small region on the desk and divide it into cells. Six cells in a two-by-three grid is a good starting shape: small enough that ten episodes each is a manageable session, large enough that the policy has to actually look at the brick rather than memorising one trajectory.
Wider than the screen; scroll it sideways.
Three checks before you spend an afternoon
Each takes a few minutes. Skipping them is how a two-hour session turns into a dataset that cannot be learned from.
Reach every cell by hand. Drive the leader arm and touch the centre of all six cells, then the bowl, with the gripper in the orientation you will actually grasp in. If a corner cell makes the arm strain or fold awkwardly, move the region. Module 1’s workspace lesson told you the reachable set has a shape and an edge; this is that lesson arriving with tape on it.
Do the task from the camera view alone. The documented test is the right one: “A good rule of thumb is you should be able to do the task yourself by only looking at the camera images.” Put the live camera windows on your screen, turn away from the bench, and teleoperate the whole task watching only the screens. If you cannot, the cameras are wrong, and the fix belongs in cameras and viewpoints, not in more data.
Check your grasp tolerance. Grasp the object deliberately a centimetre off-centre, then a centimetre high. If either fails outright, your task has no margin, and the policy’s small pose errors will land outside it every time. Widening the margin is a hardware fix: a bigger object, a bowl instead of a bin, or the compliant gripper the SO-ARM101 repository ships as a printable option, whose flexible finger forgives exactly this kind of error.
Write the specification down
Not for ceremony. Three weeks from now you will want to know what changed between the dataset that worked and the one that did not, and your memory will be confidently wrong.
| Field | Example | Why it earns its line |
|---|---|---|
| Task sentence | “put the red brick in the bowl” | It goes into --dataset.single_task verbatim and travels with every frame |
| Object | one 3 cm matte red brick | Swapping it later is an axis you relax deliberately |
| Start region | six cells, taped, photographed | Turns “various positions” into a number per cell |
| Target | bowl, outline taped to the desk | If it moves between recording and evaluation, your numbers are meaningless |
| Success criterion | brick inside the bowl and released, at rest | Must be binary and visible on video, or you cannot count it |
| Episode length | 25 s | Sets --dataset.episode_time_s, and the docs’ band is 20 to 45 s |
| Reset procedure | lift brick out, place in next cell | Its cost sets the pace of the whole session |
| Lighting | ceiling lamp on, blinds down | Written down so you can reproduce it after dark |
| Operator | you, one grasp strategy | A second pair of hands is an axis, not a convenience |
Name the axis you will relax first
Before you record anything, write down which constraint comes off first, and why. Usually it is start positions, because that is the variation your task most obviously needs and the cheapest one to generate. But if your real goal is a robot that works in the evening, lighting may deserve to go first, and knowing that now changes how you schedule the sessions.
The reason to commit in advance is that after your first evaluation you will be tempted to change everything at once. A written next-axis is what stops a debugging session from becoming a redesign.
Without hardware
- Instead of the bench, specify the task in the Module 2 scene: object, start regions, distractors, and a success predicate written as code before anything is trained.
- Measure this: the initial-condition distribution as a seeded list, and the count of distinct scenes your one sentence has to cover. Writing the predicate as a function is strictly harder than writing it as prose, which is the upgrade this path gives you.
- What you lose: the properties you did not specify. A physical brick has a mass, a friction coefficient and a shine whether or not you thought about them. A scene file has only what you typed, so a simulated task is never harder than your imagination.
Check yourself
1. Your task sentence is “put the brick in the bowl” and your first policy works only when the brick starts in the middle of the desk. Nothing is broken. What happened?
The dataset covered one region of the scene set and the policy interpolates confidently inside it. A brick outside that region produces an observation the model never saw, and there is nothing in behaviour cloning that makes an unseen input produce a cautious output. It produces a confident wrong one. The fix is coverage, not a bigger model: more cells, ten episodes each.
2. Half your demonstrations approach the brick from the left and half from the right. Both grasps work when you do them. Why is this worse than picking one?
The same camera image is now labelled with two contradictory actions. A policy trained to regress a single action per observation will land somewhere between the two, which is a path neither demonstration ever took and which grasps nothing. Multimodal demonstrations need a policy class that can represent multiple modes, which is precisely what Diffusion Policy is for. On a first dataset the cheap fix is to remove the multimodality at the source: one approach, every time.
3. You can do your task easily, watching the bench with your own eyes. Why is that not evidence that the task is feasible?
Because you are using senses the robot does not have: depth from binocular vision, an object model built over decades, and touch. The policy only sees the camera frames and its own joint angles. The documented test replaces your evidence with the robot’s: do the whole task watching only the camera windows. If that is hard for you, the cameras are the problem, and no quantity of episodes will fix it.
4. Why does the reset procedure belong in the task specification at all? It is not part of what the robot learns.
Because it sets the cost of every episode, and cost sets how much data you will actually collect. A ten-second one-handed reset lets you record sixty episodes in a sitting. A reset that needs two hands, a ruler and a re-tape turns the same sixty into a different afternoon, and it is the thing you will quietly start skipping when you get tired - which puts drift into the very positions you were trying to control.
5. You want a robot that sorts two objects into two bins. What do you record first, and why is that not the task you want?
Record one object into one bin, from a small taped region, sixty times. It is not the task you want, and it is the task whose failure teaches you the most per hour, because everything downstream - camera placement, demonstration consistency, the training recipe, the evaluation protocol - is shared with the task you want, and all of it is easier to debug when the task itself is not a suspect. Sorting is the second object plus a decision, which is two axes at once.
Do this
About forty minutes at the bench, plus five at the keyboard.
1. Write the specification. Fill in the nine-row table above for your own task, in a file next to your dataset. One paragraph, no more.
2. Run the three checks. Reach every cell with the leader arm. Do the whole task watching only the camera windows. Grasp the object a centimetre off-centre both ways. Write down what you had to change, because that list is your first real finding.
3. Tape the bench. Start region, cells, bowl outline, camera feet. Then photograph the whole bench and each camera’s view, and put those photos next to the specification. Before every evaluation from now on you will compare the live view to that photo.
4. Generate the recording schedule.
python code/recording_plan.py \
--episodes 60 --cells A1 A2 A3 B1 B2 B3 \
--episode-seconds 25 --task "put the red brick in the bowl"It prints episodes per cell, an interleaved order so your fatigue does not concentrate in one corner of the region, the frame count, and the lerobot-record command with your numbers already in it. It also prints a wall-clock budget that is roughly three to four times the arithmetic, which is what practitioner reports suggest a real session costs; treat that as an expectation to test rather than a measurement, and record your own actual time.
5. Name the axis. One sentence at the bottom of the specification: which constraint comes off first, and why.
What you can now do
You can turn a one-sentence goal into a written task specification with a binary success criterion, a taped start region and a reset you can actually sustain. You can say which properties of a task are cheap for a policy and which are expensive, and why multimodal demonstrations poison a first dataset. You can run the three feasibility checks before spending an afternoon recording, and you have a schedule that spreads your episodes evenly over the region rather than over your patience.