Where you are. You can build a task scene, drive the SO-101 with your own inverse kinematics, randomise what the scene contains, and record frames in a schema. This milestone turns all of it into two files and one number you can defend.
Ten millimetres of table
Put the cube 28 centimetres in front of the base and run the script. The gripper comes down, the jaws close on seventeen contact points, the cube goes in the bin.
Move it one centimetre further out. Same script, same code path, same seed. The gripper comes down, the fixed finger catches the near face of the cube and shoves it 12 millimetres further away, the jaws close on five contact points at a corner, and the arm lifts nothing. The cube sits there for the remaining four seconds while the robot completes the rest of the motion perfectly.
Nothing changed except ten millimetres of table. And when you later say “it works 97% of the time”, that sentence is not describing your controller. It is describing how much of the region you sampled from happens to sit on the good side of that line.
The idea in one paragraph
The milestone is two runs of the same episode function. One run evaluates: a hundred randomised cube positions, no rendering, every outcome kept and labelled, producing a success rate that is meaningless unless you also state the sampling region, the seed and the count. The other run records: episodes repeated until fifty successes are banked, images on, failures discarded, producing the dataset Module 3 trains on. The first artifact is a measurement, and the discipline is refusing to quote it without its denominator. The second is a contract, and the discipline is proving it holds before you believe it. Neither is hard to build. Both are easy to build wrongly in a way that looks completely fine.
Wider than the screen; scroll it sideways.
The number needs a box
Success here is not a property of the robot. It is the fraction of the sampling region where the scripted expert works, so the region is half the measurement.
Wider than the screen; scroll it sideways.
Sample the cube uniformly from a square of half-width around the nominal spot at , the same spot domain randomisation used, and sweep . The reference implementation gives this.
| box half-width | sampling square | success | where the failures were |
|---|---|---|---|
| 4 cm | 8 cm | 100% | none |
| 6 cm | 12 cm | 100% | none |
| 8 cm | 16 cm | 97% | 28.7 to 29.2 cm from the base |
| 10 cm | 20 cm | 94% | 13.4 to 13.8 cm, and 29.0 to 31.1 cm |
| 12 cm | 24 cm | 83% | 11.7 to 13.6 cm (eleven), and 28.8 to 32.9 cm (six) |
n = 100 trials · cube to bin · 2026-08-09
Read the last column rather than the percentage. Every failure sits at one of two radii, and neither of them is anywhere near the middle of the box. The expert does not degrade gracefully as the box widens; it keeps working perfectly over the same annulus, and the box simply starts to poke out of it at both ends.
Measured directly with a radial sweep, that annulus runs from 14 cm to 28 cm from the base along the centre line. Twenty degrees off the centre line the outer limit moves past 34 cm, which is where the sweep stopped. So the usable region is not a circle, and a radius on its own does not decide whether a cube position works: the angle matters too, and the far edge of the box is worse at its middle than at its corners.
The failures need names, not a count
“Six failures” is not a finding. There are four distinct things that go wrong in this task, they have different causes, and lumping them together is how people conclude that a physics engine is unreliable.
| Label | What you see | Usual cause |
|---|---|---|
never_lifted | the cube never rose above the table | the descent nudged it out of the jaws, or the grasp closed on a corner |
dropped | it came up and is not in the bin | friction lost mid-carry, usually with a marginal grasp to begin with |
off_target | released, landed outside the bin footprint | the carry pose or the release height is wrong, and it will fail every time |
not_settled | inside the footprint, still moving or still held | the episode ended too early; add tail time rather than loosening the detector |
The dataset is the harder deliverable
The evaluation run is one number. The recording run is a file that something else has to consume six weeks from now, and it has two problems the evaluation does not.
The first is that discarding failures moves your data.
Wider than the screen; scroll it sideways.
Fifty successes are not fifty attempts. Under a 97% expert you will attempt about 52, and the two you throw away are not random - they are precisely the positions the expert finds hard. So the demonstrations you ship come from a slightly smaller, slightly easier region than the one you measured over, and a policy trained on them meets the hard band for the first time when Module 3 evaluates it on the full box. That is not a reason to keep failed episodes; it is a reason to write down where they were.
The second problem is that a dataset can be wrong in ways that look right. Recording datasets gave you the check: reload the directory, verify every column against info.json, verify the timestamps sit on the frame grid, then replay each episode’s actions from its recorded start and assert both that the outcome matches and that the states you pass through are the recorded observations. Run it on the finished set, not on the three episodes you recorded while debugging.
Build it in five checkpoints
Do not build this as one program. Build five, each of which runs.
| Checkpoint | You are done when |
|---|---|
| 1. One episode | record_dataset.py --replay prints identical: True at the nominal cube position. |
| 2. Randomise | Ten episodes at all succeed, and the script prints each sampled position. |
| 3. Classify | Every failure at carries a label, and the label counts sum to the episode count. |
| 4. The hundred | --eval -n 100 at finishes, writes eval.json, and clears 90%. |
| 5. The fifty | --record -n 50 finishes, --check passes every assertion, and you have noted the wall clock and the bytes. |
Checkpoint 3 is the one people skip, and it is where the project stops being a demo. A labelled failure list tells you whether you have one bug or four; an unlabelled count of six tells you nothing at all.
What done means
Concretely, all of this:
python collect.py --eval -n 100 --seed 0runs to completion and prints a table that includes the sampling box, the seed and the episode count alongside the rate.- The rate is at least 90%, and every failure has a label from your
classify, not just a tally. - You can say in one sentence each why the outer-radius failures and the inner-radius failures happen. Not “the IK failed” - the inverse kinematics converges to under a tenth of a millimetre at both radii, so that answer is already ruled out.
python collect.py --record -n 50 --out pick50produces a dataset, andpython ../code/record_dataset.py --check pick50passes the schema check, the timestamp check and both halves of the replay check: the outcome and the state column.- Every episode in
pick50is a success, and each carries the seed, the cube’s placed pose and the success criterion in its metadata. - Both runs called the same
run_episode. If the rate came from one implementation and the demonstrations from another, you have measured a robot you did not ship. - A writeup of about a page: the table, the box, the failure taxonomy, and a closing paragraph answering “what would a learned policy have to do that this expert does not?”
That last paragraph is the assignment. Everything above it is the setup.
What you are actually handing to Module 3
Your expert reads arm.cube(). There is no camera in that call, no detection, no estimate; it is the simulator telling the script the answer. Every one of the fifty demonstrations is therefore a recording of a robot that already knew where the cube was.
The policy trained on them gets no such thing. It sees observation.state and a 128 by 128 image, and it has to produce the same six numbers. Everything the expert got for free - where the cube is, when to close, how far to lift - has to come out of those pixels. That gap is the entire content of Module 3, and the reason this milestone is worth four hours is that it gives you both sides of the comparison measured the same way: a 97% expert over a stated box, and fifty examples of what 97% looks like from the inside.
Check yourself
1. A colleague reports “94% success on pick and place”. What are the three things you need before that number means anything?
The sampling region, the episode count and the success criterion. The region because the rate is the fraction of that region where the expert works, and widening a box by two centimetres moved this one from 97% to 94%. The count because 94% over 16 episodes and over 1,000 are different claims. The criterion because “in the bin” can mean the cube’s centre is inside a footprint, or that plus at rest, or that plus released - and each is a different number on the same runs. The seed is a fourth, if you want anyone to reproduce it.
2. Failures appear at 13.5 cm and at 30 cm from the base, and none in between. What does that pattern rule out?
Anything stochastic. A noisy grasp or a marginal friction setting would scatter failures through the box; two tight clusters at the extremes of radius say the cause is geometric and repeatable. It does not, on its own, tell you the mechanism, and the obvious guess is wrong here: the inverse kinematics converges to under a tenth of a millimetre at 11.7 cm and at 31 cm alike. What changes with radius is the angle the wrist arrives at, so the jaws meet the cube differently at each end. The measurement to run next is a radial sweep at a fixed angle, which turns a cluster into a boundary you can quote, followed by a contact list at the moment of closing, which turns the boundary into a mechanism.
3. Why must the evaluation run and the recording run call the same episode function, rather than two similar ones?
Because otherwise the number and the data describe different robots. The moment they diverge - a different tail time, a different carry height, one of them re-seeding the inverse kinematics differently - your reported success rate stops being a property of the demonstrations you shipped. It is the same reason a benchmark harness and a production code path should not be two implementations of the same behaviour.
4. You keep only successful episodes. Name the bias this introduces and the cheapest way to record it.
The demonstrations come from the sub-region where the expert succeeds, which is smaller and easier than the region you sampled and evaluated over. A policy cloned from them has never seen the hard band, and it meets it for the first time during evaluation on the full box. The cheapest mitigation is not to keep the failures but to log where they were: the failing cube positions and their labels, alongside the dataset. It costs a few hundred bytes and it tells the next person which part of the box the data does not cover.
5. --check passes the schema and timestamp checks but the replay lands the cube 4 cm away from where the file says. Where do you look first?
At the start state first. Replay reproduces a run only if it begins from the same place, and the pose most people save is the wrong one: arm.cube() read after the settle phase is not the pose you placed, so replaying from it starts the cube a few millimetres out and the grasp inherits the error. If the start state is right, the next suspect is the action column itself. If it holds anything other than the command that was actually executed - the waypoint rather than the interpolated command, the measured angles rather than the target, or a float64 the file quietly rounded on the way in - then the trajectory you are replaying is not the trajectory that ran.
6. Your success rate is 88% and you are tempted to shrink the sampling box until it reads 95%. When is that legitimate?
When the box is genuinely part of the task specification and you say so, and when you report the change rather than quietly adopting it. Narrowing the region is a real engineering lever - it is what a fixture on a production line does - and 95% over a stated 12 cm square is an honest claim. What is not legitimate is reporting a rate measured over a small box as though it described the large one, or shrinking the box after seeing where the failures were, which is fitting your measurement to your result.
Do this
Build it. The scaffold is at project/collect.py, and there is no solution file for this one, deliberately; the whole module has been leading here.
The scaffold gives you the constants, the command-line interface and the reporting, and stubs out three things: classify, evaluate and record. It imports your own code/so101_pick.py and code/record_dataset.py rather than carrying copies, so a bug you fix in one place is fixed everywhere.
cd module-02-simulation/project
python collect.py --eval -n 100 --seed 0
python collect.py --eval -n 100 --seed 0 --width 0.12
python collect.py --record -n 50 --out pick50
python ../code/record_dataset.py --check pick50Run the evaluation at three or four box widths and build the table yourself. Yours will not match the one above exactly, and the differences are informative: a rate that is lower everywhere points at your grasp geometry, and a rate that collapses only at the widest box points at reach, which you can confirm with a radial sweep in ten lines.
Budget the recording run before you start it. At 128 by 128 with one camera, fifty episodes is about four minutes of wall clock and about 400 MB on disk on a laptop CPU. Add a second camera and both roughly double. Keep the directory out of version control.
Then write the page. The table, the box, the seed, the four failure labels with counts, and an honest paragraph on what a learned policy would have to do that your expert does not. Keep it. You will re-read it in Module 3 with a trained policy’s numbers next to it, measured over the same box, and that comparison only means something because you measured this one properly.
What you can now do
You can state a robot’s success rate as a measurement rather than a boast: over a named region, at a named seed, across a named number of episodes, judged by a criterion you can write down. You can turn a pile of failures into a small labelled taxonomy and trace two of them to geometry rather than to luck. And you can produce a demonstration dataset that survives being reloaded, checked against its own schema, and replayed through the simulator that made it - which is the difference between fifty episodes and fifty megabytes.