Where you are. You can clone demonstrations, chunk actions, generate them instead of averaging them, evaluate a policy honestly, and watch one learn from reward alone. This lesson is the choice between those, made with your own numbers instead of your taste.
A slot in a plate
A curved slot is milled into a steel plate, about a metre long and five centimetres wide. A pin on the end of a two-link arm has to travel from one end of it to the other without touching the walls. The plate is bolted down and it does not move.
Three people offer to do it. The first opens the drawing, reads the curve out of it, and writes a tracker: look a short way ahead along the curve, move that way, repeat. The second says that is far too brittle, and offers to drive the arm by hand fifty times so a network can copy what she did. The third says forget both, describe what a good run looks like and let the machine practise until it finds one.
Each of them is right about something. Only one of them is finished by lunchtime, and which one it is has almost nothing to do with how good they are at machine learning.
The idea in one paragraph
Scripting, imitation and reinforcement are not three algorithms competing on quality. They are three different things a human has to author: a rule, a set of demonstrations, or a score. The task decides which of the three you can actually produce and what producing it costs, and the algorithm is downstream of that decision. Module 0’s paradigm ladder told you this as a claim you had no way to check. You now have a laboratory, so this lesson is the same claim with your own measurements bolted on, plus one correction that only shows up once you have measured: the choice is not made once per project. It is made once per segment of a task.
Wider than the screen; scroll it sideways.
Five paradigms collapse into three artifacts, which is why the ladder from Module 0 is easier to use in this form. A hand-built perception-planning-control pipeline is still a rule, just a longer one. A vision-language-action model, which arrives in Module 5, is still demonstrations, only they are somebody else’s and there are a million of them.
The same task, three ways
The slot in the plate is groove_world.py, the arm you have been running since the covariate-shift lesson. Three approaches, one observation space, one action space, and one success rule written before anything was trained: the pin reaches the far end without ever leaving the groove.
Then the interesting axis. Each policy is scored from the start pose the demonstrations used, and again from two and four centimetres off it, because the difference between these three approaches is invisible in the demonstrated condition and enormous outside it.
| approach | you authored | it consumed | on centre | +2 cm | +4 cm |
|---|---|---|---|---|---|
| scripted | a path and a tracker | nothing | 60/60 | 60/60 | 60/60 |
| imitation, 10 demos | 10 demonstrations | 600 transitions, ~2 s | 44/60 | 35/60 | 17/60 |
| imitation, 50 demos | 50 demonstrations | 3,000 transitions, ~4 s | 60/60 | 57/60 | 53/60 |
| reinforcement | a reward function | 200,477 env steps, ~22 s | 0/60 | 0/60 | 0/60 |
Measured on a 2018 4-core laptop CPU, 60 trials per cell, one seed each
n = 60 trials · groove world, pin from A to B without leaving the band · 2026-08-09
Read the rows in order, because each one says something different.
The rule is right everywhere. Not because it is clever, but because it is not a summary of anything. Pure pursuit along a known curve is defined at every pose in the plane, including poses no demonstration ever visited, so moving the start does nothing to it. That is the actual argument for scripting, and it survives contact with the world exactly as long as the drawing does.
The clone is right where it was shown, and decays outward. Fifty demonstrations reproduce the expert perfectly on centre and lose seven trials in sixty at four centimetres out. Ten demonstrations lose forty-three. This is the covariate shift you measured earlier, now priced: the same architecture, the same trainer, the same loss, and the only difference is how much of the state space the human happened to visit.
The search never finished the task at all. Two hundred thousand environment steps, sixty-seven times the transitions the clone learned from, and not one of a hundred and eighty rollouts made it to the end.
The reward was the expensive part
The search finds most of what it will ever find in the first seventeen thousand steps and then flattens for the remaining hundred and eighty-three thousand. That is a picture worth keeping, because it is what a stalled optimisation looks like from the outside: not a crash, not an error, just a number that stops moving while the clock runs.
Worse, the plateau is the third reward function I wrote. The first scored how far along the curve the pin ever got, and bought a policy that cut the corner and flew straight to the far end through solid steel. The second scored progress over the samples that were inside the groove, and bought a policy that left the groove, flew to the far end anyway, and landed back in the band at the other side. Both were gamed within a minute of search starting. Only the third - progress made before the pin first left the band - describes the task.
Now the mirror case
Turn the task over and every answer flips. In the policy-gradient lesson you trained a cart-pole: keep a pole balanced on a cart that can only be shoved left or right.
That pair is the whole decision, and it fits in a sentence you already met in Module 0, now with numbers under it: use demonstrations when the task is easy to do and awkward to score; use a reward when the task is easy to score and awkward to do. Dragging a pin along a groove is easy to do and awkward to score. Balancing a pole is easy to score and awkward to do. Locomotion research and manipulation research read like two different fields for exactly this reason.
The questions, in the order that saves you the most work
| Ask this first | If yes | What it costs you |
|---|---|---|
| Can I remove the variation instead of tolerating it? A fixture, a jig, a better light | scripted | design work once, and a system that is deterministic, auditable and fast |
| Can I write the motion down from geometry I already have? | scripted, or a classical pipeline | it is correct only as far as the model you wrote, and it fails without warning when the world moves |
| Can I perform the task myself, at the robot’s speed, fifty times? | imitation | an afternoon of teleoperation, and a policy that decays the moment it leaves the demonstrated region |
| Can I score an attempt cheaply, thousands of times, without breaking anything? | reinforcement | a simulator to maintain, a reward that an optimiser will attack, and interaction measured in the millions |
| None of the above | somebody else’s demonstrations, fine-tuned on yours | Module 5, plus everything you inherit from a dataset you did not collect |
The order matters more than the contents. Every row you can answer yes to is cheaper to build, cheaper to debug and cheaper to certify than every row below it, so the expensive mistake is not picking wrong. It is starting at the bottom because that is the interesting part.
The unit of decision is the segment
Here is the correction that only arrives after you have measured. Almost no real task is one of these three. It is four segments and three answers.
Wider than the screen; scroll it sideways.
Reaching above the tray is free space with a known target: geometry, and nothing to perceive. Grasping the part varies in pose and shape and is trivial to demonstrate: imitation. Seating it in the fixture is a contact-rich last three millimetres that no demonstration covers well and that scores itself cleanly - it either seated or it did not - which is precisely the shape reinforcement wants. Retracting is geometry again.
Check yourself
1. The scripted tracker scored 60/60 from a start pose four centimetres off centre, where the clone trained on fifty demonstrations scored 53/60. What property of the rule produces that gap, and what would destroy it?
The rule is a function of the geometry, not a summary of visited states, so it is defined at poses no demonstration reached. Being off centre is not a special case for it. What destroys it is the geometry changing: move the plate, bend it, or make the curve unknown, and the rule is not degraded, it is simply wrong. The clone would survive a plate that moved slightly, having seen a spread of starts; the rule would not, having been told exactly one.
2. My search spent 200,477 environment steps and finished nothing, while the clone learned from 3,000 demonstrated transitions. Why is it wrong to conclude that imitation is 67 times more efficient than reinforcement?
Because those two numbers count different things and the comparison only holds for this task. The 3,000 transitions came from an expert that already existed; producing them for a task you cannot perform costs infinity. The 200,477 steps came from a deliberately simple black-box search on a five-line reward, not from a tuned algorithm. The transferable claim is narrower and more useful: on a task that is easy to demonstrate and awkward to score, demonstrations are the cheap artifact, and the ratio is large enough that no amount of tuning changes the decision.
3. A colleague proposes reinforcement learning for a bin-picking cell where parts arrive in a fixture at a known pose. Give the strongest version of the argument against, without saying that learning is overkill.
Every property they want is already available further up the ladder, and they would pay for it twice. The environment has been engineered so the variation is gone, which means a scripted motion is not merely adequate, it is deterministic, auditable, cheap to certify and fails loudly. Reinforcement adds a simulator to maintain, a reward function that an optimiser will attack, and a policy that fails at 5% for reasons nobody can name. The correct place to spend effort is keeping the fixture accurate.
4. Balancing a pole is easy to score and awkward to demonstrate; dragging a pin along a groove is the reverse. What is the underlying property that flips, and how do you test for it on a new task in five minutes?
Whether the goal is easier to recognise than to produce. Test it by trying to write both artifacts badly and seeing which one embarrasses you first. Write the reward in one line and ask what the laziest possible policy that maximises it looks like; if you can name a cheat immediately, scoring is the hard side. Then imagine teleoperating the task fifty times and ask whether you could do it well; if the answer is no, doing it is the hard side.
5. Warm-starting the search from the cloned weights made the on-centre success rate fall from 54/60 to 20/60. The reward went up. Explain both facts at once.
The reward was computed from four fixed start poses. The search maximised exactly that and succeeded, so the reward number rose. Success on centre is measured over sixty fresh starts, which the reward never looked at, so nothing protected it. Reinforcement fine-tuning optimises the distribution its reward is evaluated on; anything outside that distribution is free for it to trade away, and it will.
6. You are handed “unload the dishwasher”. Cut it into segments and assign an artifact to each, then say which segment you would build first.
Roughly: open the door and pull out the rack, which is fixed geometry and scripted; identify and grasp each item, which varies without limit and is easy to demonstrate, so imitation; place it in the right cupboard slot, which is partly geometry once the item is identified, plus a contact-rich insertion that scores itself; close up, which is scripted again. Build the grasp first, because it is the segment with the widest failure distribution and the one whose data collection has the longest lead time - and because the scripted segments will still be correct after you change your mind about everything else.
Do this
Open code/three_ways.py. Three functions are marked TODO(you): the success predicate, the reward, and the elite update inside the search. The scripted and imitation halves are already wired to code you wrote in earlier lessons.
python module-03-robot-learning/code/three_ways.py --quick # about 40 s
python module-03-robot-learning/code/three_ways.py # about 100 sBefore you run the full version, write down three predictions: the on-centre success rate for fifty demonstrations, the four-centimetre rate, and how many environment steps the search will need to match the clone. Keep them where you can see them.
Then push on it.
- Write the reward badly on purpose. Score the furthest point along the curve the pin ever reached, ignoring whether it stayed in the groove, and watch what the search hands you. This takes about ninety seconds and it is the most useful ninety seconds in the lesson.
- Run
--warm, which seeds the same search from the cloned weights. Predict first whether polishing helps, then read what actually happened to the on-centre column, then explain it in one sentence. My run is in the lesson above; if yours disagrees, the disagreement is worth more than the agreement. - Widen the reward’s start distribution.
search()scores each candidate from four fixed poses. Sample them from the same distribution the evaluation uses instead, rerun--warm, and see how much of the collapse that buys back. This is the whole content of the last gotcha, done with your hands. - Apply the table to something of your own. Pick a task you actually want a robot to do, cut it into segments, and write four lines per segment: the artifact you would author, what has to be true for it to work, how it fails, and who maintains it in a year. That last column is the one people skip, and it is the one that decides what you should have built.
The reference implementation is solutions/three_ways.py, and it prints the same table this lesson quotes.
What you can now do
You can name the three artifacts a robot task can be built from, and say which one a given task will actually let you author. You can price the choice with your own measurements rather than a preference: what the rule bought in robustness, what the demonstrations cost in coverage, what the search cost in interaction and in the two reward functions it ate on the way. You can state the ordering rule - engineer the environment, then write the rule, then demonstrate, then score - and defend starting at the top. And you can do the thing that separates a working system from an interesting one: cut a task into segments and answer the question separately for each, instead of picking a paradigm and making the whole task fit it.