Where you are. You can run the collect-train-evaluate-diagnose loop and turn failures into a collection plan. This lesson is the map from what you are looking at to where to look.
Three centimetres to the left, forever
Trial one. The arm sets off, dips, and closes three centimetres to the left of the cube.
Trial two: three centimetres to the left. Trial three, four, five, six: three centimetres to the left. You could sit there all afternoon and it would never be right, and it would never be differently wrong. There is a small worn patch on the table where the fingers keep touching down.
Now run the other film in your head. Same arm, same policy, same cube, but this time each miss lands somewhere new. Left. Short. Right. Directly on top and it closes too early. Left again, further this time.
Both of those are “the policy misses the cube”. They have nothing else in common. The first one is a machine doing precisely the wrong thing; the second is a machine that does not know where the thing is. Fixing either has no effect at all on the other, and the fastest way to lose a weekend on this arm is to start work before you know which film you are in.
The idea in one paragraph
Your robot is a stack of five layers: wiring and power, calibration, cameras, data, policy. Any of them can produce “it does not work”, and the symptom is evidence about which layer rather than about how bad things are. Two properties of the symptom carry almost all the information. First, is the failure repeatable and identical, or repeatable and varied? Identical means geometry, which is layers one to three; varied means the policy genuinely does not know, which is layer four. Second, does it correlate with anything outside the robot, such as the time of day or which USB port you used? Correlation with the environment means your evaluation conditions have drifted away from your recording conditions. Debugging real hardware is mostly the discipline of extracting those two facts before touching anything.
Wider than the screen; scroll it sideways.
Measure the repeatability instead of eyeballing it
“It always misses left” is a feeling. Turn it into two numbers and it becomes a decision.
Put a ruler or a printed grid on the table. Run ten trials. For each one, write down where the object actually was and where the gripper actually closed, in millimetres, in the same frame every time. Then split that error into a bias, the average offset, and a scatter, the spread around that average.
Wider than the screen; scroll it sideways.
code/reach_bias.py does the arithmetic and prints the verdict, including the case people get wrong most often: a bias that is smaller than its own standard error, which means you have not measured a lean at all, only a small sample of a noisy process. Run --demo systematic and --demo scattered to see both shapes before you collect real numbers.
The rule it applies is simple. Bias much larger than scatter is geometry. Scatter much larger than bias is data. When they are comparable, fix the geometry first, because it is cheaper and because removing it changes your estimate of the scatter.
Rule out the bottom of the stack first, every time
Layers one to three are cheap to check and produce failures that look sophisticated. Check them before you form any theory about the policy.
LeRobot’s own triage starts with the motor LEDs, and it is worth doing before you open a terminal.
| What the LEDs do | What it means |
|---|---|
| Steady red, the whole chain from gripper to base | Wiring is fine. Move up a layer. |
| One or more dark, or the chain stops part-way | A wiring fault at that point. Reseat the 3-pin cables, check the controller board’s supply, make sure each motor is clicked fully in. |
| Blinking | The motor is in an error state, usually overload or the wrong supply voltage. |
Two probes that cut the stack in half
Do not debug the whole system. Cut it.
Teleoperate. Move the leader arm and watch the follower. If the follower tracks it correctly through the full range, then power, wiring, motor IDs and calibration are all sound, and you have eliminated the bottom two layers in about thirty seconds. If the follower lags, jumps, hits a limit early or sits at a visibly different pose from the leader, stop. The problem is below the policy and no amount of retraining touches it.
Replay. lerobot-replay plays a recorded episode back through the follower. This is the sharpest probe in the toolbox because it removes the policy entirely while keeping everything else. If a recorded episode replays and the arm does roughly what you did, your recording pipeline and your calibration agree with each other. If replay puts the gripper somewhere the demonstration never went, your calibration has changed since you recorded, and every episode in that dataset now means something different from what it meant.
That is the bisection. Teleoperation tests the arm. Replay tests the arm plus the recording. A rollout tests all of it, which is why a rollout is the last probe you run and the least informative one to start with.
The symptom table
This is the module’s exit test, and the mapping is the skill.
| Symptom | Layer | First probe |
|---|---|---|
| Misses by the same amount in the same direction, every trial | Calibration, or something in the rig moved | Measure bias and scatter; check the --robot.id you are running matches the one you recorded with; compare a live camera frame against a frame from the dataset |
| Misses somewhere different every trial | Data: coverage or demonstration consistency | Break the success rate down by starting condition; check the coverage grid for thin cells |
| Ignores the object entirely, or wanders off | Cameras | Look at the camera frames. Could you do the task from that view alone? |
| Works at noon, fails at dusk | Cameras and lighting; the input distribution moved | Compare frames from the two times of day side by side |
| Reaches the object confidently, then stalls or dithers in an unfamiliar state | Data: no recovery behaviour in the dataset | Record episodes that start from the state it stalls in |
| Jitters or oscillates near the target | Data: hesitant demonstrations, or undertrained | Watch your own demonstrations at that point in the task; check the training loss actually flattened |
Timeouts, sync read failures, disconnections every few episodes | Wiring and power, or USB bandwidth | LEDs, connectors, supply; then move cameras onto separate USB controllers |
| Blinking LEDs, a joint that stops holding | Wiring and power: overload or wrong supply voltage | Confirm the supply matches the build; the 7.4 V and 12 V variants are not interchangeable |
| The gripper stops gripping properly over time | Wiring and power: the gripper motor is the one that dies | LeRobot caps the gripper at 50% torque and 50% current for exactly this reason; check it, and consider that the motor is worn |
| It worked yesterday and nothing changed | Almost always the environment or a version | What is different? Light, camera position, room, the LeRobot version you upgraded |
The last row deserves its own paragraph, because “nothing changed” is nearly always false and the falseness is the diagnosis.
The failure you cannot see
Your policy learned from images. Everything in those images is part of what it learned, including the parts you were not thinking about: the colour of the light, where the shadows fell, the mug that happened to be on the table, the exact angle of the camera on its clamp.
Wider than the screen; scroll it sideways.
When evaluation conditions drift outside the region your dataset covered, the policy fails, and the failure looks exactly like a bad policy. There is no error message, no exception, no log line saying the light changed. This is why “works at noon, fails at dusk” is a canonical symptom and not a joke: a policy trained at two in the afternoon next to a window has learned that afternoon’s light as part of the task.
The practical defences are unglamorous and they work.
- Record all episodes for a dataset in one session, under artificial light you control. Diffuse and consistent, and no moving shadows. LeRobot’s guidance is blunt about this: lighting matters more than resolution.
- Bolt the cameras down and mark everything. Tape, marker lines on the table, a printed ruler grid. The point is not precision; it is reproducibility between the day you record and the day you evaluate.
- Save a reference frame from each camera at recording time. Before an evaluation, put the live view next to it and nudge the camera until they match. This is the cheapest calibration procedure in this module and it costs nothing.
Do not fix two things at once
The temptation, once you have a symptom and three candidate causes, is to fix all three and move on. Resist it for one round. Change one thing, run five trials, and write down the result before changing the next.
The reason is not tidiness. It is that on this hardware several of your “fixes” will be wrong, and a batch of five changes that collectively improves nothing tells you nothing about which four to undo. Real evaluations are expensive, so the information per trial is the resource you are actually managing.
Without hardware
Simulation inverts this lesson in a useful way: you cannot meet the faults by accident, and you can inject them deliberately.
- Instead of waiting for symptoms, cause them. Move a camera two centimetres, scale the object mass by 1.5, halve the friction, change an actuator gain - one at a time, with everything else frozen.
- Measure this: whether the symptom map in this lesson predicts what you then see. That is a test of the diagnostic table that no hardware reader can run cleanly, because on a bench the faults arrive together and unlabelled.
- What you lose: roughly half the rows. Calibration drift, bus errors, mechanical slop and lighting change cannot fire in a model, so the repeats-identically versus lands-somewhere-new split is easier here than it will ever be on a bench.
Review
The symptom names the layer
Your robot is a stack of five layers: wiring and power, calibration, cameras, data, policy. Any of them produces it does not work, and the symptom is evidence about which layer rather than about how bad things are. Two properties carry almost all the information. First, is the failure repeatable and identical, or repeatable and varied? Identical means geometry, which is layers one to three. Varied means the policy genuinely does not know, which is the data layer. Second, does it correlate with anything outside the robot, such as the time of day or which port you used? Correlation with the environment means your evaluation conditions have drifted from your recording conditions. Same miss every time: go and look at the geometry. Different miss every time: go and look at the data.
Turn the feeling into two numbers
It always misses left is a feeling. Run ten trials, write down where the object actually was and where the gripper actually closed in millimetres in the same frame every time, and split the error into a bias, the average offset, and a scatter, the spread around that average. Bias much larger than scatter is geometry. Scatter much larger than bias is data. When they are comparable, fix the geometry first, because it is cheaper and because removing it changes your estimate of the scatter. This is the triage you already do with a failing test suite: a test that fails identically points at the code under test, and a test that fails one run in five points at timing or shared state. The difference is that nobody hands you the failure rate; ten trials is the minimum that makes the distinction visible at all.
Check yourself
1. Your policy misses by 3 cm in the same direction on every trial. Name the three most likely causes and the single measurement that separates them from a data problem.
The measurement is bias against scatter over about ten trials: a bias much larger than the scatter means the arm is confidently going to a consistently wrong place, which is geometry rather than learning. The three causes, in the order worth checking: the follower’s calibration differs from the calibration in force when you recorded, usually because the --robot.id differs or a collision forced a re-calibration; a camera has moved on its mount since recording; or the physical rig has shifted, meaning the arm base, the table markings, or the bin. All three produce the same signature, which is a repeatable offset with small spread.
2. Why does a camera that moved produce a systematic offset when an imitation policy has no camera pose in its pipeline at all?
Because the policy is not doing geometry, it is doing pattern completion on images. It maps pixels and joint positions directly to joint targets, so it never needs camera intrinsics or extrinsics and there is no transform that could be numerically wrong. What changes when the camera moves is what the image means: a frame that used to correspond to the cube at the left edge now corresponds to the cube in the middle. The policy responds correctly to the scene it appears to see, and does so identically every trial, which is what makes it look like a calibration bug.
3. You get a serial timeout and a failed sync read on all six motor ids. What is the first thing you do, and why is it not reading the driver source?
Look at the motors’ LEDs and the cabling. LeRobot’s own documentation says most timeout errors are physical rather than code, and the common causes are all mechanical: a 3-pin connector that is not fully seated, a power cable that fell off while you were manipulating the board, controller-board jumpers on the wrong channel, or USB bandwidth contention from two cameras on one bus. Reading driver source is expensive and, in the overwhelmingly likely case, addresses a component that is working. The two-minute physical check comes first because it is cheap and because it eliminates a whole layer.
4. Replay of a recorded episode puts the gripper somewhere the demonstration never went. What does that rule in, and what does it rule out?
It rules the policy out completely, because replay does not involve the policy. It rules in everything between the recording and the arm: calibration that has changed since the episode was captured, a different --robot.id in use now, a motor replaced or re-homed, or a mechanical change such as a slipped horn. It also tells you something serious about the dataset: if the arm no longer reproduces the recorded joint targets, every episode in that dataset now corresponds to a different physical motion than it did when captured, so retraining on it will not help until the calibration is restored or the data is re-recorded.
5. “It worked yesterday and I changed nothing.” Why is that sentence itself a diagnosis, and what are the four things to check?
Because a deterministic system that behaves differently was acted on by something outside it, so “nothing changed” is a statement about what you were watching rather than about the world. The four candidates: lighting, which is the single most common one and moves on its own with the time of day and the weather; camera position, since clamps creep and a knocked tripod looks identical from across the room; the physical rig, meaning the arm base, the table, the object; and software version, since this ecosystem moves on roughly a six-week cadence and an upgrade can rename a command or an import out from under your scripts. Compare a live camera frame against a frame saved from the recording session; that single comparison catches the first two.
6. Why should you change one thing per round even though each evaluation round costs twenty trials?
Because the expensive resource is information, not time. Five simultaneous changes that produce no net improvement leave you unable to say which of them helped, which hurt, and which did nothing, so you have spent twenty trials and bought zero attribution. One change per round produces a directly attributable result each time. On hardware this discipline matters more than in software, because you cannot re-run yesterday’s evaluation: the light, the room and the arm’s own wear have all moved on.
Do this
A triage drill on your own arm, about an hour. Do it once while nothing is broken, so the procedure is in your hands before you need it under pressure.
1. The physical pass, from memory. Without looking anything up, power the follower and check the LED chain from gripper to base. Then reseat one 3-pin connector deliberately, run a teleoperation session, and observe exactly what a wiring fault looks like from the software side. Put it back. You now know that symptom by sight.
2. Measure your own bias and scatter. Tape a printed grid to the table. Run ten trials of your current policy. Record, in millimetres, where the object was and where the gripper closed, then:
python code/reach_bias.py --csv trials.csvWrite down the two numbers and the verdict. Keep this file: re-running it after any change to the rig is the fastest way to detect that something moved.
3. Run the two bisection probes and record what each rules out. Teleoperate for one minute; replay one episode from your dataset. Write one sentence for each saying which layers it eliminated. If either fails, you have found something more valuable than a working policy.
4. Build your own reference frames. Save one frame from each camera, labelled with the date and the dataset it belongs to, next to the dataset. Then move one camera two centimetres, evaluate five trials, and measure the bias again. That number is the cost of a bumped tripod, in millimetres, on your specific rig.
5. Write your own symptom table. Take the table above and add the rows your arm has actually produced, with the cause you eventually found. After a month it will be more useful than anything in this lesson, because it will be about your hardware.
What you can now do
You can turn a vague “it does not work” into a measurement that says which layer is at fault: bias against scatter for geometry versus data, the LED chain for wiring, teleoperation and replay as bisection probes that eliminate whole layers in seconds. You can explain why a moved camera produces a repeatable offset in a pipeline that contains no camera geometry, why “it worked yesterday” is a diagnosis rather than a mystery, and why one change per evaluation round is the only way to buy attribution with expensive trials.