Hand-eye calibration, and why you are not going to do it

A pixel is a ray, so turning one into a place needs depth and the camera-to-robot transform; an imitation policy skips both by never leaving pixel space, and charges you a repeatability discipline instead.

Where you are. Your two cameras are placed, lit and clamped down. This lesson answers the question a Module 1 graduate cannot stop asking: where is that camera, in the robot’s frame?

Click on the block

The scene camera’s feed is on your screen and the block is right there, at pixel 447 across and 213 down. You have forward kinematics. You have inverse kinematics that converges. You have an arm that reaches. Send it to the block.

You cannot, and what is missing is not a library.

That pixel does not name a place. It names a direction. Every point along one straight line leaving the lens produced exactly that pixel: the block sitting 35 cm out, a block twice the size at 70 cm, the coffee ring on the wall behind. The camera did one division on the way in, and the number it divided by is not recoverable from the result.

Two things would get you a place. Some way to recover how far, and knowledge of where the camera sits relative to the robot’s base. The second has a name and a well-known procedure, and you are not going to do it.

The idea in one paragraph

Projection divides by depth, so a pixel is a ray rather than a point. Recovering a position from pixels therefore needs two ingredients: depth, and the rigid transform between camera and robot. Solving for that transform is hand-eye calibration, and a classical detect-then-plan pipeline cannot take a single step without it. An imitation policy never forms a three-dimensional point at all - it maps images and joint positions straight to joint targets - so the transform is never needed and never computed. That is not a free lunch. The geometric relationship between camera and robot is still doing work; it has been absorbed into the network weights, learned implicitly from your demonstrations. Which makes it exact for your rig, at your working depth, under your lighting, and worthless the moment any of those move.

One pixel is a wedge leaving the lens: a 4 cm block at 35 cm, a 6 cm block at 52 cm and an 8 cm block at 70 cm all fill exactly the same pixel

Wider than the screen; scroll it sideways.

The division that loses the number

A point sitting at $(x, y, z)$ in the camera’s own frame, with $z$ measured straight out of the lens, lands on the image at

u = f_x \frac{x}{z} + c_x, \qquad v = f_y \frac{y}{z} + c_y

In plain terms: sideways position divided by distance, scaled by the lens, then shifted to put the origin in the corner of the image instead of the middle. That division is the whole story. Replace the point with $(kx, ky, kz)$ for any $k$ and both fractions are unchanged, so the pixel is unchanged. One pixel corresponds to an entire ray of points, which is what the figure above shows: a 4 cm block at 35 cm and an 8 cm block at 70 cm produce footprints identical to the last pixel.

The field of view is usually the one optical number printed on the box, and the focal length follows from it. Half the image width subtends half the horizontal field of view, so $f_x = (W/2)/\tan(\theta/2)$ ; a 640-pixel-wide frame with a 60 degree field of view gives about 554 pixels, which works out to roughly 0.63 mm of table per pixel at a 35 cm working distance.

Which transform is the unknown

Suppose you solve depth somehow, with a depth camera or a known table plane. You now have a point in camera coordinates. The arm takes commands in base coordinates. Between them sits one rigid transform, and Module 1 already taught you how to compose it:

\mathbf{p}_{base} = T_{base,cam}\,\mathbf{p}_{cam}

Which transform you have to solve for depends on where you bolted the camera.

Eye to hand: the camera is clamped to the table and the unknown runs from the robot base to the camera. Eye in hand: the camera rides the wrist and the unknown runs from the gripper to the camera

Wider than the screen; scroll it sideways.

With the camera clamped to the table - eye to hand - the unknown is the base-to-camera transform, and it is the same at every instant. With the camera on the wrist - eye in hand - the unknown is the gripper-to-camera transform, which is also constant, but the base-to-camera pose now changes every step and you recover it by composing the constant with forward kinematics.

The classical solve has a pleasing shape. Fix a marker where the camera can see it, on the table for a wrist camera or on the gripper for a table-mounted one, move the arm to a dozen poses, and at each pose record two things: where forward kinematics says the gripper went, and where the camera says the marker went. Each pair of poses gives you a relative motion measured two independent ways, and the unknown transform is precisely the thing that reconciles them. Textbooks write it as $AX = XB$ , with $A$ the motion the robot reports, $B$ the motion the camera reports, and $X$ the transform you want.

Where the transform went

Now compare the two pipelines you could build on top of the same hardware.

The classical pipeline turns pixels into an object pose and then needs the base-to-camera transform before inverse kinematics can run; the imitation pipeline maps images and joint positions straight to joint targets and forms no transform at all

Wider than the screen; scroll it sideways.

The classical route builds an explicit intermediate: pixels, then an object pose in camera coordinates, then the same pose in base coordinates, then joint angles from inverse kinematics. Every stage is inspectable, every stage has units, and the whole chain stops dead without the transform.

The imitation route has no intermediate. Weights map pixels and joint positions to joint targets. Nothing inside the network is measured in metres, so there is nothing for a transform to convert.

History

In February 2025 someone opened issue #718 on the LeRobot repository, titled “Hand-Eye Calibration for LeRobot”, asking how to do it. It drew ten comments over ten months, all from other users rather than maintainers, and the first of them is the answer this lesson has been building toward: a deep-learning policy maps images to actions directly, so hand-eye calibration is not needed - but the camera pose must stay the same between recording and evaluation. Nobody added a feature. A bot marked the issue stale in December 2025 and closed it. A second issue asking the same thing was opened in April 2026 and is still open.

Hand-eye calibration appears nowhere in the SO-101 guide, the imitation-learning guide, the camera guide or the cheat sheet, and there is no command for it. Read that as a design decision rather than an oversight: the library declined to add the step because its pipeline does not have a place to put it.

What you owe instead

Calibration is a procedure you run once and a file you keep. Repeatability is a habit, and it is what this workflow substitutes.

Bolt it down and mark the table. What cannot be reproduced cannot be evaluated, and on this rig reproduction is mechanical rather than computational.

Keep reference frames. Save one frame per camera at recording time. Before an evaluation run, compare the live view against it and nudge until they agree. The reference_frame_match.py script from the previous lesson scores geometry and exposure separately, because a moved camera and a changed lamp need different fixes.

Use a marker as a re-seating aid if you take the wrist camera off. A printed AprilTag or ArUco marker with a script printing its pose live gives you a repeatable target to line up against before you retighten the mount. This is third-party practice, not first-party tooling; there is nothing in LeRobot that does it for you.

Working depth is part of the calibration you did not do

Look at the projection equation once more. Because only the ratio of sideways position to depth survives, a policy trained on demonstrations at one working distance has never been given the information needed to handle another. Put the object on a 3 cm book and every pixel it occupies changes: it sits higher in the frame, and to a camera looking down at it, it is 3 cm nearer and therefore larger. Nothing in the image says “the same block, raised”. The image says “a larger block, or a nearer one”, and the learned mapping from pixels to joint targets has never been asked that question.

The classical pipeline suffers this explicitly and legibly. Back-project a pixel with an assumed depth of 30 cm when the truth is 35 cm, and a block 12 cm off-axis is reported at 10.3 cm: a 1.7 cm error, in an object you can see perfectly. The recovered sideways position scales exactly with the depth you assumed, so a 14% depth error is a 14% lateral error.

The policy suffers the same geometry without the error message. So the discipline is simple to state: record and evaluate at the same table height, the same camera distance, and the same object size. If you want a policy that survives depth changes, that is not a calibration problem, it is a data problem, and you fix it by demonstrating at several heights.

When you would actually do it

Hand-eye calibration is not obsolete. It is just not on the path to your first policy.

You need it when	Because
You build a detect-then-plan pipeline	Every pixel has to become a base-frame point before inverse kinematics can run
You want to log where objects were, in metres	An implicit mapping cannot be printed or plotted
A planner reasons about object positions	Module 6’s LLM planner talks about places, not pixels, so something has to produce places
You train in simulation and deploy on real hardware	The rendered camera pose has to match the physical one, and matching needs both measured
You share a rig with someone else’s dataset	Their implicit transform is not yours, and nothing checks

Your first policy needs none of these, and the fastest way to a working arm is to skip all of them. Just know exactly what you skipped and what it cost, which is the whole point of this lesson.

Without hardware

Simulation hands you the one quantity hardware never gives you: the camera-to-base transform, exactly, for free.

Instead of the repeatability routine, take the exact extrinsics out of your MJCF, project a known object position into pixels, and invert the projection.
Measure this: how far along the ray the recovered position moves when you feed the inversion a depth that is 5% wrong. That is the ambiguity this lesson is about, quantified, and it is easier to see with a perfect transform than with a noisy one.
What you lose: the reason the lesson ends where it does. On hardware you skip hand-eye calibration and pay for it with a repeatability discipline. With exact extrinsics available there is nothing to skip and no discipline to buy, so the trade the lesson describes never presents itself.

Check yourself

1. Which single operation in projection turns a point into a ray, and what would you have to supply to invert it?

The division by depth. Both image coordinates are formed from a sideways position divided by $z$ , so any scaling of the whole point leaves them unchanged, which means one pixel corresponds to an entire ray. To invert it you need $z$ supplied from outside the image: a depth sensor, stereo geometry, a known object size, or a known surface such as the table plane. Intrinsics alone are not enough, because they only tell you which ray the pixel is, not where along it the point sits.

2. Your policy works reliably. Where is the camera-to-robot transform, and what can you do with it?

It is inside the network weights, learned implicitly from demonstrations that happened to be recorded on that rig. You cannot read it out, print it, verify it, or transfer it to a second robot, because it exists only as a distributed property of the mapping from pixels to joint targets, never as a pose. That is the trade: no calibration procedure to run, and no calibration artifact to inspect or reuse. It also means any check on the geometry has to be empirical, which is what the reference frames are for.

3. Eye in hand and eye to hand: which transform is unknown in each, and which one changes as the arm moves?

Eye to hand, with the camera clamped to the table, has the base-to-camera transform as the unknown, and it is constant. Eye in hand, with the camera on the wrist, has the gripper-to-camera transform as the unknown, and that is also constant, but the base-to-camera pose changes every step and is recovered by composing the constant with forward kinematics. The distinction matters because in the eye-in-hand case, every error in the joint encoders and every bit of gearbox backlash flows through forward kinematics and lands in your camera pose.

4. You put the block on a 3 cm book. Everything else is untouched, and the policy that worked yesterday now misses. Explain it, and say why training for longer is the wrong response.

Only the ratio of sideways position to depth reaches the image, so a change in working depth changes every pixel in a way the demonstrations never covered. The mapping from pixels to joint targets is valid at the depth it was learned at, and that depth is now different. It is not undertraining, so more steps on the same data cannot help: the information required is not in the dataset. The fix is either putting the block back on the table, or demonstrating at several heights so the policy has seen the variation.

5. Why does moving a camera by a couple of millimetres matter more than the number sounds like it should?

Because the relevant unit is pixels, not millimetres. At a 35 cm working distance with a 640-pixel frame, one pixel covers about 0.63 mm of table, so a 2 mm camera shift moves the entire scene by around three pixels and a 10 mm shift by around sixteen, which is 2.5% of the frame width. Every object in the image moves together, identically, and the array produced is one the policy has never seen at that object position. There is no channel by which the policy could learn that the camera moved rather than the world.

6. Name two situations where you would spend an afternoon on a real hand-eye calibration.

Any pipeline that converts pixels to a base-frame pose before commanding the arm, since inverse kinematics needs a point in the robot’s own coordinates. And anything that has to report or reason about positions in metres: logging object locations for analysis, a planner that talks about places rather than pixels, or matching a simulated camera to a real one for sim-to-real transfer. A third, if you want it: sharing a rig or a dataset with someone else, where an implicit transform baked into weights does not carry across and nothing warns you.

Do this

Two parts, roughly thirty minutes. The first runs anywhere, the second needs your rig.

1. Make the ray concrete. Before running anything, predict three numbers: how many millimetres of table one pixel covers at 35 cm on a 640-wide 60 degree camera; whether a 4 cm block at 35 cm and an 8 cm block at 70 cm are distinguishable in the image; and how far off a back-projected position lands if you assume 30 cm depth when the truth is 35 cm. Write them down, then run:

python code/pixel_ray_ambiguity.py

Then run it again with your own camera’s numbers, which are on the box or in its spec page:

python code/pixel_ray_ambiguity.py --hfov 78 --width 1280 --depth 0.30

A wider field of view buys coverage and costs millimetres per pixel. That trade is the reason a wrist camera is precise and a wide overhead camera is not.

2. Measure your own rig’s sensitivity. With the fixed camera clamped and a reference frame already saved from the previous lesson, run the comparison live:

python code/reference_frame_match.py --camera 0 \
    --reference code/reference_frames/0.png --live

Watch the baseline number with nothing touched. Then loosen the clamp, move the camera deliberately by about a centimetre, watch the geometry score, and move it back until the score returns to baseline. You have just calibrated the tripwire rather than the camera, and the number you learned is the one that tells you, before an evaluation run, whether today’s rig is yesterday’s rig. Record that threshold in your notes and edit it into the script’s verdict function.

Neither of these has been run against physical hardware by the author of this course. The first is pure arithmetic and will behave; the second is short enough to read completely before you trust it.

What you can now do

You can derive why a pixel is a ray from the projection equation, name what intrinsics and extrinsics each describe, and say which transform is unknown in an eye-in-hand and an eye-to-hand rig. You can explain precisely where the camera-to-robot relationship lives in a trained imitation policy and why that makes it exact, invisible and non-transferable. And you can run the repeatability routine that stands in for calibration here, including knowing how many pixels a millimetre of camera movement actually costs you.