Where you are. You can load the SO-101, drive it from Python, and put its gripper where you want with your own inverse kinematics. This lesson turns that arm in an empty room into a task: something to pick up, somewhere to put it, and a program that says whether it worked.
You add a bin and nothing works
Open the arm’s scene file. Add a box on the floor a comfortable-looking distance away, and a small open-topped tray to one side. It looks right on screen. Run your reaching script.
The gripper stops short. The bin is 40 cm away and the arm is about 25 cm long, which was obvious in hindsight and invisible while you were typing coordinates. Move the bin to 20 cm. Now the gripper reaches it, but only by leaning over at an angle, and your grasp script assumes a straight-down approach. Raise the wrist to clear the tray walls, and the wrist joint hits its limit before the gripper is high enough.
Eventually you get one run that works. You watch the replay and say, out loud, “that worked.”
That sentence is the second problem. You are going to run this ten thousand times.
The idea in one paragraph
A task scene is not decoration around a robot; it is the task, written down. Two things live in it. The geometry decides whether the task is possible: an object placed where the arm cannot reach in the orientation the task needs is not a hard task, it is an impossible one, and no amount of controller tuning fixes it. The success predicate decides whether an episode counted: a small piece of code that reads the simulator state and returns one boolean, with no human in the loop. Both have to be negotiated against the arm you actually have, and the honest way to do that is to ask the arm, in code, before you commit a single coordinate to the XML.
Scenery and things
Everything you add is a <body> holding one or more <geom> shapes. What decides whether a body is scenery or a thing is a single line.
<body name="cube" pos="0.22 0 0.02">
<freejoint name="cube_free"/>
<geom name="cube" type="box" size="0.02 0.02 0.02" material="cube_mat"/>
</body>Remove the <freejoint/> and that cube is welded to the world forever. The gripper will close on it, the solver will push back, and it will not move by a micron. With the free joint it gains six degrees of freedom and joins the simulation properly.
The seven-versus-six asymmetry is why you index qpos and qvel separately rather than assuming they line up. Load the finished scene and it reports itself:
nq=13 nv=12 nu=6
cube free joint: qpos[6:13] = xyz + quaternion, qvel[6:12] = linear + angularSix arm joints then the cube. Never hard-code 6: ask the model with model.jnt_qposadr[...], because the day you add a second object every hard-coded index in your code becomes silently wrong rather than loudly broken.
Ask the arm before you place anything
The arm has a shape, and the shape is not obvious from the joint limits. Rather than guess, sweep the table with the inverse kinematics you already wrote and ask, at every spot, whether a straight-down grasp pose exists at all.
Wider than the screen; scroll it sideways.
The answer for this arm and this table is 27%. Not because the arm is bad, but because a top-down grasp is four constraints at once: three for position and one that pins the approach direction vertical. Close to the base there is no way to fold; far out the arm runs out of length; and behind, the pan joint stops at 110 degrees. What is left is a crescent.
Be honest about what that map is. It is pure kinematics: it asks whether joint angles exist that put the gripper there, and it never checks whether the arm would pass through the table, fold into itself, or clip the bin wall on the way. So the crescent is an upper bound. Every spot outside it is definitely impossible; some spots inside it will still fail once physics is involved. That is the right way round for a planning tool, because it lets you rule places out cheaply and lets the episodes themselves rule the rest out honestly.
The same trick sizes the bin. Ask how high the gripper can go while still pointing straight down, at the radius where the bin will sit. On this arm, at 20 cm out, the answer is 9.0 cm, and the limit is the wrist joint, not the reach. Everything else follows from that one number.
Wider than the screen; scroll it sideways.
Carry the cube with the gripper at 7.5 cm and the cube’s underside rides at about 4.5 cm. So the bin walls have to finish below that, with margin. Mine are 3.2 cm tall, which leaves 1.3 cm of clearance. That is why the bin in this scene is a shallow tray rather than the deep crate you pictured: the tray is not a design choice, it is what the wrist joint permits.
The verdict is code
Now the second half, and the half people skip. “That worked” has to become a function.
The temptation is one line: is the cube near the bin? Run that against a few hundred episodes and it will lie to you in three different ways.
Wider than the screen; scroll it sideways.
def cube_in_bin(arm, v_tol=0.02):
p = arm.cube()
centre = arm.model.body("bin").pos
inside = np.all(np.abs(p[:2] - centre[:2]) < BIN_HALF) and p[2] < 0.05
at_rest = np.linalg.norm(arm.data.qvel[-6:]) < v_tol
return bool(inside and at_rest and not arm.touching_gripper())Three conditions, and each one is there because dropping it admits a specific fake success. Without the footprint test, a cube that bounced off the rim and landed beside the tray scores. Without the height test, a cube still clamped in the fingers directly above the bin scores, and your scripted expert learns that hovering is winning. Without the rest-and-release test, a cube caught mid-bounce on its way back out scores, and so does one you are still holding.
The “not touching the gripper” clause is worth a moment. MuJoCo hands you the full contact list every step, so asking whether any contact pairs the cube with a gripper geom is a short loop. That kind of question, cheap and exact, is the sort of thing simulation gives you and a camera never will.
The three tolerances in that function are decisions, not details. Each one was picked against a specific fake success, and each one moves your headline number when you move it.
| Tolerance | Value here | Sized against | Loosen it and you admit |
|---|---|---|---|
| footprint half-width | 5 cm | tray inner half-width, 5.5 cm | a cube balanced on the rim |
| height cut-off | 5 cm | resting cube below, carried cube above | a cube still held over the tray |
| velocity threshold | 0.02 | solver jitter below, real motion above | a cube caught mid-bounce |
The margins run in opposite directions on purpose: the footprint sits inside the tray, the height sits between the two cases it separates, and the velocity sits above the numerical noise floor. Tighten any of them and honest successes start failing; loosen any of them and the fake success in the last column starts counting.
What the scene owes the next two lessons
One more property, and it costs nothing if you build it in now: everything you might want to vary later should be addressable by name. Name the materials, name the light, name the camera, name the bin. Then a randomiser is a handful of lines that reach into model and change numbers, with no XML rewriting and no recompilation.
The keyframe matters too. mujoco.mj_resetDataKeyframe(model, data, 0) snaps the whole world back to a named starting state in microseconds, which is the reset that Lesson 2.1 said was the real prize. Put your home pose and the object’s start pose in one <key> and every episode begins identically, by construction.
Check yourself
1. Your bin is 25 cm from the base, well inside the arm’s reach, and the gripper still cannot get over it. What is the likely cause?
Reach is direction-dependent. The arm can put its gripper at that point in some orientation, but your task needs a specific one: pointing straight down, high enough to clear the walls. On this arm that combination runs out at about 9 cm of height, and the binding constraint is the wrist joint hitting its limit rather than the arm running out of length. Sweep the IK over height at that radius and you get the number in a second, instead of guessing.
2. You add a second object to the scene and your success check starts reporting nonsense. Nothing else changed. Why?
Almost certainly a hard-coded index. Adding a body with a free joint inserts seven numbers into qpos and six into qvel, shifting everything after it. Code that said data.qpos[6:13] for the cube now reads someone else’s state, silently, with no exception. Ask the model for addresses using model.jnt_qposadr and model.jnt_dofadr instead.
3. Why is “the cube is within 5 cm of the bin centre” not a success predicate?
It scores at least three things that are not successes: a cube resting on the table right beside the tray, a cube still gripped in the fingers directly above the tray, and a cube caught in mid-air on its way back out after bouncing off the rim. A predicate needs containment, height and a rest-and-released condition together. Each missing condition is a category of failure that your dataset will label as expert behaviour.
4. A colleague says the bin walls should be 8 cm so the cube cannot bounce out. What do you tell them?
That the arm cannot deliver over an 8 cm wall. The highest pose it can reach while keeping the gripper vertical is 9.0 cm at this radius, and the carried cube’s underside sits about 3 cm below the gripper site, so a wall above roughly 4 cm guarantees a collision on the way in. If bounce-out is a real problem, the fixes are a wider tray, a slower release, or releasing lower, not taller walls.
5. Why put the object’s starting pose in a keyframe rather than setting it from Python at the top of each episode?
Mostly so the reset is one call and cannot drift. mj_resetDataKeyframe restores every element of the state at once, including velocities and control values, so episode 4000 starts exactly where episode 1 did. Setting fields by hand from Python works, but it is easy to reset position and forget velocity, and the resulting bug is a slow drift in your success rate that looks like physics.
Do this
Work in code/, with the finished versions in solutions/. Budget about 40 minutes. Everything here runs on a laptop CPU; scene_check.py takes about 12 seconds.
1. Finish the scene. code/task_scene.xml gives you the arm, the table and the lights. Two TODO(you) blocks are left: a bin body made of a floor plate and four walls, and a cube body with a free joint. Add the <key> at the bottom so a reset restores both the arm pose and the cube pose.
2. Finish the reach map. In code/scene_check.py, fill in reach_map() and carry_ceiling(). Run it. You should get a crescent, a percentage in the mid-twenties, and a ceiling near 9 cm. If your crescent is a full disc, your acceptance test is ignoring the orientation residual.
3. Finish the verdict. Implement cube_in_bin() in code/so101_pick.py, then run scene_check.py again so the five detector cases execute. All five must pass. Now deliberately break it: delete the at_rest clause and rerun. Exactly one case flips, and it tells you which failure that clause was buying you.
4. Make the bin impossible, on purpose. Raise the walls to 8 cm and run python so101_pick.py. Watch it fail, then read your own carry_ceiling number and explain the failure in one sentence before you put the walls back. This is the fastest way to internalise that scene geometry is a constraint problem.
5. Move the bin somewhere unreachable (try pos="0.34 -0.30 0") and run one episode. Note what the failure looks like from the outside: no exception, no warning, just a success rate of zero. Then check that spot against your reach map. Get in the habit of doing this in the other order.
What you can now do
You can build a MuJoCo scene that is a task rather than a picture: static scenery, a free-jointed object, a target sized by the arm’s real capability instead of by eye, and a keyframe that resets all of it in one call. You can measure where a specific arm can perform a specific grasp instead of guessing, and you can write a success predicate that is three conditions rather than one, having tested each of them against a fabricated state that only fails if that condition is missing.