Where you are. You have your own forward kinematics, Jacobian and inverse kinematics running against the SO-101 and verified against the engine. This lesson puts your hands on the result.
The claw machine
You are at the arcade. Two buttons: one drives the claw away from you, one drives it sideways. Press, watch, release, press again. When you think it is over the plush toy, you hit the drop button and gravity takes it from there.
Notice what you are not doing. You are not thinking about the gantry motors, or which one turns how far. You are thinking in the cabinet: forward, left, down. The machine converts that into motor commands somewhere you cannot see.
Notice also that it is a terrible interface, and you know exactly why. One axis at a time. The claw either moves at its one speed or it does not move. You cannot tilt it, you cannot squeeze gently, and you are judging depth through a glass panel from the wrong angle. Nobody would call it precision equipment. It is still enough to win a toy, sometimes.
Every complaint in that paragraph will apply to the thing you build in this lesson. That is not an accident, and it is the point.
The idea in one paragraph
Teleoperation is a choice about what the human commands. You could hand the operator one dial per joint, which is honest and unusable. Or you can let them steer a single point in space and make the robot’s own maths work out the joint angles, which is what an excavator, a surgical console and the arcade claw all do. The second option is worth building, because it costs about forty lines on top of the IK you already have and it turns your arm from something you script into something you drive. It is also worth building because you will feel, immediately and in your hands, why the demonstrations it produces are the wrong training data; and knowing that before Module 3 asks you to learn from demonstrations is worth more than the arm.
What your hands are connected to
The whole thing is one loop with a mutable target in the middle.
Wider than the screen; scroll it sideways.
A key press does one thing: it adds five millimetres to one component of a three-number target. Nothing else. The key handler does not touch the robot, does not call IK, and does not block. Then, every physics tick, the loop runs one damped-least-squares update toward that target, writes the result into data.ctrl, and steps the simulator by five milliseconds.
The trick that makes one step enough is the warm start. Frame begins from frame ‘s answer, which is already within a few millimetres, so a single Newton-ish step lands close. Reset the joint angles between frames and the arm would crawl.
There is one arrow deliberately missing from that diagram. Nothing carries the tool tip’s actual position back into the target. The target is wherever your keys have put it, and the arm’s real position never gets a vote. In control terms this outer loop is open; the only thing closing a loop here is the human eye watching the screen.
Which numbers the keys move
The six jog keys map onto the three world axes, at the tool tip.
Wider than the screen; scroll it sideways.
| Key | Effect | Amount |
|---|---|---|
I / K | target forward / back, | 5 mm |
J / L | target left / right, | 5 mm |
U / O | target up / down, | 5 mm |
G / H | gripper close / open | 0.1 rad |
R | target back to the tool tip’s rest position | - |
Real teach pendants offer both, and the reason is instructive: joint jog is what you use when the arm is somewhere strange and you want it out of there without a solver deciding for you.
Getting a window open
mujoco.viewer.launch_passive gives you a window that does not own the loop; your code steps the physics and calls viewer.sync(). That is the mode a course wants, because your controller stays in charge.
On macOS it comes with one rule. The rendering context has to live on the process’s main thread, so the mujoco wheel ships a launcher called mjpython that arranges exactly that. Run the script with python instead and you get this, precisely:
The viewer takes your key handler as key_callback and hands it a raw key code, which for letters is the capital ASCII value. It also has shortcuts of its own; the ones in its shortcut list are function keys, arrows and punctuation, so the letters chosen here stay clear of it. Whether holding a key auto-repeats is up to your operating system’s key-repeat settings. Tapping always works.
To see what you are steering, the loop pushes a small sphere into viewer.user_scn at the target point each frame. Without it you are aiming at an invisible dot, which is a surprisingly large difference.
What it feels like, in numbers
You cannot put a screenshot of feel into a lesson, so the script has a --replay mode: the same loop, the same key handler, a fixed sequence of key presses, and no window. It runs anywhere, including under plain python on macOS, and it prints what happened.
| Phase | Target | Tool tip | Lag while held | Lag after release |
|---|---|---|---|---|
back off, 20 taps of K | 0.291, -0.001, 0.246 | 0.291, -0.001, 0.246 | 2.1 mm | 0.20 mm |
descend, 39 taps of O | 0.291, -0.001, 0.051 | 0.291, -0.001, 0.051 | 3.8 mm | 0.21 mm |
push, 22 taps of I | 0.401, -0.001, 0.051 | 0.402, -0.001, 0.055 | 4.4 mm | 4.02 mm |
n = 1 trial · scripted keyboard session: back off, descend, push the box · 2026-08-09
The box ended 63.2 mm further out than it started. You drove a robot into an object and moved it, which is the first thing in this course that is unambiguously a physical task rather than a calculation.
Wider than the screen; scroll it sideways.
Three numbers in that table are worth separating, because they look alike and are not.
2 to 4 mm while a key is held. The target is moving at 100 mm per second and the arm is chasing it, so it trails. Tracking lag. It scales with how fast you jog.
0.2 mm half a second after release. The target stopped, the servos caught up, and what is left is the position actuator’s standing error against gravity from Lesson 2.9. It does not shrink further no matter how long you wait.
4.0 mm after release, while pressed into the box. The target is inside the box. The arm cannot get there, so the error stops shrinking and turns into force instead. Same servo, same command, completely different reason for the gap; the only way to tell them apart from the outside is that this one never goes away.
Why this is bad training data
Everything above works. It is also, as a source of demonstrations, close to the worst thing you could hand a learning algorithm, and the reasons are the arcade cabinet’s reasons.
It is a step input. Every key press teleports the target 5 mm and the arm lunges. In the scripted session the tool tip’s peak speed is 462 mm/s while the average rate you are actually commanding is 100 mm/s, and the tip is essentially stationary for 26% of the frames. A policy trained on that learns to lunge, because lunging is what the data contains.
It is one axis at a time. Ten fingers cannot press I and U in a way that produces a smooth diagonal. Human arm motion is simultaneous in all three axes and curved; keyboard motion is a staircase.
There is no orientation in it. Position-only IK leaves the wrist wherever damped least squares put it, so a demonstration recorded here has no information about how the hand was turned, which is half of what a grasp is.
There is no force in it. You cannot press gently. You can only command a place and let the servo spend whatever force it takes.
None of which makes this lesson wasted. Keyboard teleop is how you sanity-check a scene before automating it, how you find out that an object is out of reach, and how you get a feel for an arm’s speed and workspace in ten minutes. It is a debugging tool that happens to be fun. Just do not point the dataset recorder at it and expect a policy.
Check yourself
1. The loop runs one IK iteration per physics step instead of solving IK to convergence each frame. What makes that enough, and what breaks if you remove it?
The warm start. Each frame begins from the previous frame’s joint angles, which are already within a few millimetres of the answer, so one damped-least-squares step lands close and the residual error stays small and bounded. Remove it, by resetting the joint guess each frame, and a single step from a cold start barely moves, so the arm crawls. The deeper point is that a solver run continuously from its own last answer stops being a solver and becomes a controller; the target moving out from under it is a feature, not a problem.
2. Nothing in the loop feeds the tool tip’s measured position back into the target. Why is that acceptable here, and where does it stop being acceptable?
It is acceptable because a human is closing that loop by eye: you watch the screen and press keys until it looks right. It stops being acceptable the moment the human leaves. A scripted or learned controller that sets targets without ever checking the achieved position has no way to notice that the arm is stuck against an obstacle, out of reach, or fighting a joint limit; from inside the loop, all three look exactly like success.
3. The tool tip trails the target by 2 to 4 mm while a key is held, and by 0.2 mm shortly after release. Both are lag. Why do they have different sizes, and which one would jogging more slowly fix?
The 2 to 4 mm is tracking lag against a moving target: the target advances 5 mm every 50 ms and the arm is always partway through catching up. Halve the jog rate and it roughly halves. The 0.2 mm is the position actuator’s steady-state error against gravity with a target that has stopped moving; jog rate has nothing to do with it, and the only things that change it are the servo gain and the load.
4. While the tip is pressed into the box, the gap stays at 4.0 mm no matter how long you wait. Explain why that is not the same phenomenon as the 0.2 mm.
The 0.2 mm gap is a balance between the actuator’s restoring torque and gravity; the arm is free to move and simply has no reason to move further. The 4.0 mm gap is a balance between the actuator’s torque and a contact force from a solid object. The commanded joint angle is unreachable, so the error never shrinks, and the servo converts that permanent error into permanent force on the box. One is a controller at rest; the other is a controller pushing indefinitely and being stopped.
5. Why does the gripper key write data.ctrl[5] directly instead of going through the IK step?
The jaw is a child of the gripper body and moves relative to it, so opening or closing the jaw does not move the wrist and does not appear in the tool tip’s kinematic chain at all. There is nothing for IK to solve. Which also means the gripper is the one command in this loop with no position feedback of any kind: you write an angle, the object stops the jaw short, and the servo keeps pushing against it.
6. Name two properties of keyboard-teleoperated demonstrations that a policy trained on them would faithfully reproduce, and say what the SO-101’s leader arm does differently.
Any two of: step-input lunges, because each key press moves the target instantaneously and the measured peak tip speed is several times the commanded average; axis-aligned motion, because you can only press one key at a time; unspecified wrist orientation, because position-only IK never chose one; and no force modulation, because position control cannot express “gently”. The leader arm replaces all four at once, since the demonstration becomes a recording of a human arm moving continuously in every joint simultaneously, with orientation and contact force set by the person’s own hand.
Do this
Finish code/teleop_keyboard.py, which needs on_key and control. Both are short; the IK work is already done in ik_so101.py. About thirty-five minutes.
1. Prove the loop works without a window. Run python teleop_keyboard.py --replay and reproduce the table above. If the arm does not move, control is probably rebuilding q_cmd from scratch each frame instead of keeping it.
2. Drive it. Run mjpython teleop_keyboard.py on macOS, python teleop_keyboard.py elsewhere. Steer the tool tip to the box and push it off centre. Then try to line the jaws up on either side of it and close. You will probably fail, and the failure is informative: watch which of the four missing things stops you first.
3. Find the edge of the world. Hold I until the arm stops following. The target keeps going; the tip does not. Print both, and watch the tip stall just short of m while also sinking, because the only way left to gain reach is to stretch out and down. Then hold O and drive the target through the floor, and watch the same stall happen for a completely different reason.
4. Change the feel. Set STEP_M to 0.001 and then to 0.02 and jog the same distance each time. Record the lag while held in both cases. You are measuring the same tradeoff every teleoperation interface makes: fine steps track well and take forever, coarse steps are fast and overshoot. Write down which one you would want if you were about to close a gripper on something fragile.
What you can now do
You can drive the simulated SO-101 by hand, with six keys steering a point in space and your own inverse kinematics dragging the arm after it. You can read the three different gaps between the point you asked for and the point you got, and say which is tracking lag, which is servo droop and which is contact. And you can state, with numbers from your own session, why the demonstrations this interface produces are the wrong input for the imitation learning in Module 3.