Where you are. You can drive the leader arm and watch the follower copy it. This lesson fixes the two viewpoints your policy is going to live inside for the rest of the module.
Turn the monitor around
Set the pair up the way you normally do. Now move your chair so the desk is behind you and the only thing in front of your face is the laptop screen with the live camera feed on it. Ask someone to put the block somewhere on the table without telling you where.
Drive the leader arm and pick it up.
The first thing you will do is lean sideways to peek. Do not. The second thing is the interesting one. As the gripper closes, the fingers slide across the block and take it out of view, and from that angle you cannot tell whether you are five millimetres short or five millimetres long. You close anyway. You lift. Nothing comes with you.
The arm was fine. The light was fine. Your hands were fine. The task was simply not doable from those pictures.
Whatever you just went through, a trained policy goes through on every episode of its life, and it never gets to lean sideways.
The idea in one paragraph
A behaviour-cloning policy receives exactly two things at each step: the camera images and the arm’s own joint positions. That is the whole universe it has. So the camera rig is not equipment that sits beside the robot, it is the observation space - the formal statement of what is knowable - and choosing it is a modelling decision, not a setup chore. Two consequences follow, and they are the lesson. Anything the cameras cannot resolve is not merely hard to learn, it is absent from the data, so no architecture and no amount of training recovers it. And anything about the rig that differs between recording and evaluation shifts the input distribution, which is the covariate shift you met in Module 3, except this time you caused it yourself by knocking the tripod with your elbow.
Wider than the screen; scroll it sideways.
What is not in there
Read the figure’s right-hand column again, because the omissions do more work than the inclusions. There is no object pose. No depth, unless you bought a depth camera and configured it. No contact force. No timestamp of the light level. And, importantly for the next lesson, nothing anywhere in the observation says where the camera is.
Module 0 said a sensor hands you an array with a unit, a shape and a rate, never a fact. This is that statement with money on it. The policy has to induce every fact it acts on from those two arrays, and it can only induce facts that the arrays actually distinguish.
Two cameras, two jobs
The documented default is two cameras: one fixed, looking at the scene from the front or from above, and one on the wrist. The LeRobot guidance is blunt about the trade, saying that multi-view often outperforms single-view while a single fixed camera also works if you want to keep things simple.
Wider than the screen; scroll it sideways.
They answer different questions. The fixed camera answers where is the block on the table, and it answers it in a way that never changes meaning, because the camera never moves. The wrist camera answers where is the block relative to my fingers, right now, and it answers at high resolution because it is close to the action. At a third of a metre, a 640-pixel-wide frame with a typical 60 degree field of view resolves roughly 0.6 mm of table per pixel; the same camera 8 cm from the object resolves several times finer.
The five-minute test
The acceptance test in the LeRobot docs is the one you already ran in the hook, and it is worth stating as a rule: you should be able to do the task yourself by only looking at the camera images. The cheat sheet puts it as a question - can you do the task from the camera view alone? If not, your cameras are wrong, and you fix them before recording.
Run it three times, and run it at the hardest object position rather than the comfortable one in the middle of the table. Five minutes of blind teleoperation costs almost nothing. Recording a dataset costs an afternoon, and training on it costs a night.
Lighting beats resolution
The docs say it in one line: lighting matters more than resolution, so use diffuse consistent light and avoid moving shadows. Here is why that outranks megapixels.
A camera does not report light. It reports the output of an auto-exposure and white-balance loop that is continuously renormalising the scene. When a cloud passes, or you switch on a desk lamp, or the sun tracks across the window, the loop reacts and every pixel in the frame changes at once. The policy has no channel through which to learn that the light changed rather than the world. It sees a different world.
Diffuse also means no hard shadow edges moving with the arm. A hard shadow cast by the gripper is a high-contrast feature that tracks the gripper, which is a beautiful thing for a network to latch onto and a useless one, because it disappears the moment you move the lamp.
Resolution, frame rate and the bus
The documented starting point across the LeRobot teleoperation, recording and cheat-sheet examples is 640 by 480 at 30 frames per second. Resist the urge to raise it. Policies resize their inputs anyway, so the extra pixels usually buy nothing at the output; what they reliably buy is traffic on a bus that also carries your servo commands.
Wider than the screen; scroll it sideways.
There is one error you will meet early and misread. When the driver silently negotiates a resolution other than the one you asked for, LeRobot refuses with a message of the form frame width=... or height=... do not match configured width=... or height=.... That is not a LeRobot bug and not a broken camera. It is the webcam telling you it will not do what you asked, usually because another stream already took the bandwidth.
Names that survive a reboot
Two identical webcams present identical USB descriptors, so the operating system has nothing to tell them apart by, and the /dev/videoN numbers can shuffle between reboots and replugs. Notice that the official rollout example addresses cameras by path rather than by integer index. Follow that. On Linux, a udev rule mapping a physical port to a stable name is twenty minutes well spent; buying two different camera models is five minutes well spent and works everywhere.
The rig is part of the dataset
Everything above collapses into one habit: after the rig passes the five-minute test, freeze it, and treat any later change as a change to your data.
| What changed | What the policy sees | The symptom you get |
|---|---|---|
| Fixed camera nudged 1 cm sideways | every pixel in the frame shifted by roughly 16 px at 35 cm | reaches confidently, and consistently off to one side |
| Room light changed | all pixel values remapped at once | worked at noon, fails at dusk, nothing in the logs |
| Camera key renamed or devices swapped | wrist image in the scene slot | incoherent motion, no error raised |
| Object starts outside the frame | no information at all about it | the policy ignores the object and runs its average trajectory |
| Wrist camera reseated by a few degrees | the grasp view it learned is gone | approaches well, misses at the last centimetre |
That first row is not a metaphor. The next lesson does the arithmetic that produces the 16 pixels, and explains why a policy has no way to distinguish it from the block having moved.
Without hardware
- Instead of placing cameras, inspect a rig somebody else froze.
lerobot/svla_so101_pickplaceships two 480 by 640 streams namedupandside; pull the MP4s and run this lesson’s acceptance test on them. Could you do the task from those two views alone? - Instead of the physical rig, use the MuJoCo cameras from Module 2. Render a scene, move one camera two centimetres, and re-render.
- Measure this: the acceptance-test verdict for both, written down, plus the fraction of frames in which the object is occluded in at least one view.
- What you lose: everything that makes a rig drift. A clamp does not creep in simulation and the sun does not cross a window, so the rig changes that silently destroy a working policy have no analogue here.
Check yourself
1. Your policy reaches for the block confidently and stops about 3 cm short every single time. The arm calibrates fine and teleoperation is precise. Which parts of the rig do you suspect, and in what order?
Consistency is the clue: a systematic offset points at something that changed uniformly rather than at noise. Check whether either camera has moved since recording, starting with the fixed one, since a small translation shifts the whole frame and the policy reads that as the block sitting somewhere else. Then check the object’s height, since the depth the demonstrations were recorded at is baked into the mapping. Then check whether the two camera keys are still bound to the same physical devices. A model-side fix is the last thing to reach for, because nothing about a systematic spatial offset suggests undertraining.
2. Why is raising the cameras to 1920 by 1080 usually the wrong answer to “my policy is imprecise”?
Because the policy resizes its inputs, so the extra pixels rarely reach the network, while the extra bandwidth definitely reaches the bus that also carries servo traffic. Imprecision almost always comes from what the viewpoint fails to show - occlusion at the moment of grasp, an object partly out of frame, shadows moving with the arm - or from demonstrations that were themselves imprecise. Higher resolution addresses none of those. The documented starting point is 640 by 480 at 30 fps, and it is the right place to stay until you have a reason tied to something you actually observed.
3. What exactly is wrong with recording a dataset over three evenings beside a window?
You have made time of day a hidden variable and then failed to sample it evenly. The auto-exposure loop renormalises the whole frame when the ambient light changes, so identical scenes at different hours produce systematically different arrays, and the policy has no way to represent “the light changed” separately from “the world changed”. You end up with a dataset whose lighting correlates with whatever else drifted across those three evenings, and a policy that appears to work at one hour and not another. Controlled artificial light and one sitting per dataset removes the variable rather than modelling it.
4. Two identical webcams, and the rig worked yesterday. Today the policy does nothing sensible. What is the likely cause, and why is there no error message?
The device indices probably swapped on reboot, so the image LeRobot labels as the scene view is now the wrist view and the other way round. There is no error because both cameras open successfully at the requested resolution and both deliver valid frames of the correct shape. Every contract the software can check is satisfied; the only thing that is wrong is the meaning of the arrays, which no type system here encodes. Address cameras by stable path rather than by index, or use two different models so they can be told apart.
5. The fixed camera and the wrist camera both see the block. What does each give you that the other cannot?
The fixed camera gives a view whose meaning is constant for the whole episode, so a position in that image maps to a position on the table regardless of what the arm is doing; that is what lets the policy know where to go from any starting configuration. The wrist camera gives resolution and a gripper-relative view: because it travels with the hand, the same picture means the same thing about the grasp no matter where on the table the grasp happens, which is what makes the final centimetres precise. Losing the fixed camera costs you global positioning, and losing the wrist camera costs you the endgame.
Do this
A structured rig audit, about forty minutes, before you record anything. Commands are LeRobot 0.6.x. Nothing here has been run against hardware by the author of this course, so trust your own output over this page.
1. Enumerate the cameras.
lerobot-find-cameras opencvThe opencv argument is optional; omit it and the tool probes every backend it knows. It writes sample frames to outputs/captured_images, relative to whatever directory you ran it from, and --output-dir moves that. Note the index or path for each device, and label the physical cameras with tape so you can tell later which is which.
2. Verify what you actually get, one camera at a time.
python code/camera_check.py --cameras 0 --seconds 5
python code/camera_check.py --cameras 1 --seconds 5
python code/camera_check.py --cameras 0 1 --seconds 10 # now both at onceRead the three lines that matter: negotiated resolution, measured frame rate, and brightness drift while nothing moves. A camera that passes alone and fails in company is a bus problem, not a camera problem.
3. Mount, then mark. Clamp both cameras so a firm nudge does not move them. Put tape on the table marking the region where the object is allowed to start, and check that region is fully inside the fixed camera’s frame with room to spare.
4. Run the blind test three times, at the hardest object position, using teleoperation with the display on:
lerobot-teleoperate \
--robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=my_follower \
--robot.cameras="{ top: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \
--teleop.type=so101_leader --teleop.port=/dev/ttyACM1 --teleop.id=my_leader \
--display_data=trueThree successes out of three, screen only, no peeking. Fewer than that and you move a camera rather than proceeding.
5. Save the reference frames and photograph the rig. camera_check.py writes one reference frame per camera into code/reference_frames/. Keep them. Take a phone photo of the whole desk from behind, including where the clamps sit, and put it beside them. Before every evaluation later in this module you will compare against both.
6. Write down the camera block exactly as you will use it, keys included, and paste it unchanged into every later command. Retyping it is how the keys drift.
What you can now do
You can lay out a two-camera rig and say what each viewpoint is for, apply the five-minute blind test that rejects a bad one before it costs you an afternoon, and explain why lighting outranks resolution and why 640 by 480 at 30 fps is the sensible default rather than a limitation. You can recognise the USB and device-naming failures by their symptoms, and you can list the rig changes that break a working policy without producing a single error message.