Scoring a robot instead of showing it

Reinforcement learning replaces the demonstration with a score, which buys you tasks nobody can demonstrate and charges you a simulator, a hundred million failures, and a specification the optimiser will read against you.

Where you are. You can train an imitation policy, chunk its actions, and evaluate it honestly enough to publish. Everything so far started from a recording of a human doing the task. This lesson is about what you do when no such recording can exist.

Try to puppeteer a trot

You have a leader-follower rig. You have spent Module 2 recording demonstrations by moving one arm and watching the other copy it, and it works: fifty episodes, ten minutes of your time, a policy.

Now somebody wheels in a quadruped and asks for a trot.

Take the twelve joints. A trot means diagonal pairs of legs swing together while the other diagonal is on the ground, both at around two steps a second, with the foot landing under the hip, the knee absorbing the impact, and the body’s roll cancelled out by the shoulders. You have to produce all twelve joint angles, correctly phased, fifty times a second, while the robot’s actual balance depends on what you did forty milliseconds ago.

You cannot do it. Nobody can. There is no leader rig with twelve arms and no human with the bandwidth.

Now notice what you can do. You can watch the robot and say whether it went the way you asked, whether it stayed upright, and whether it burned an absurd amount of current doing so. You cannot produce the behaviour, but you can put a number on it.

That asymmetry is the whole reason the second half of this module exists.

The idea in one paragraph

Reinforcement learning replaces the demonstration with a score. You define a reward: a single number, computed from the robot’s state at each instant, that says how well things are going right now. The robot then acts, gets scored, and adjusts so that acting that way again earns more. Nobody ever shows it the answer. In exchange for not needing a demonstrator, you pay three prices, all steep. It has to actually try, which means falling over a hundred million times, which means a simulator. It only ever optimises the number you wrote, which makes that number a specification and the optimiser an adversarial reader of it. And it can only learn from rewards it manages to stumble into, so a task whose reward only arrives at the very end teaches nothing at all.

Two ways to specify a task: demonstrate it and fit the demonstrations, or write a score and let a search find behaviour that scores well

Wider than the screen; scroll it sideways.

The vocabulary, once

Five words, then we can talk about the real problems.

The return is written with a discount factor $\gamma$ , a number just under one:

G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots = \sum_{k=0}^{\infty} \gamma^k\, r_{t+k}

In words: add up all future reward, but count reward that arrives later slightly less. At $\gamma = 0.99$ a reward a hundred steps away is worth about a third of the same reward right now. Two reasons for it, one honest and one practical. The honest one is that the far future genuinely is less certain. The practical one is that without it the sum need not converge, and the arithmetic falls apart.

The thing being optimised is the expected return of the whole episode:

J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\!\left[\,\sum_{t=0}^{T} \gamma^t r_t \right]

Read $\tau \sim \pi_\theta$ as “episodes generated by running this policy”. That subscript is the entire difficulty of the field in three symbols: the thing you are optimising over is also the thing that generates the data you optimise it with. Change the policy and the dataset changes underneath you. Nothing in supervised learning behaves like that.

The reinforcement learning loop: the policy sends an action, the environment returns the next state and a reward that you wrote

Wider than the screen; scroll it sideways.

What it costs

Here is the number that decides everything downstream. A quadruped locomotion policy typically takes on the order of one hundred million environment steps to train.

Put that on a real robot. At a 50 Hz control rate, a hundred million steps is two million seconds, which is twenty-three days of continuous motion - assuming the robot never falls, never needs a reset, and never breaks. It will fall roughly every second for the first hour.

Compare it with what you already have. An imitation policy for an arm needs about fifty demonstrations, which is about ten minutes of teleoperation. That is not a small difference in degree. It is the difference between an afternoon and a research programme, and it is why reinforcement learning for robots is almost always reinforcement learning in simulation.

The reward is a specification

You will write the reward function. It is code, it is short, and it is the least tested code in the entire system. It is also the only place where you get to say what the task is.

Run reward_hacking.py. It puts one point on a line, one unit from a target, gives it a small controller, and searches the entire controller space twice: once scored by the reward you meant, once by a reward that looks helpful.

The reward you meant is $-\lvert x \rvert$ at every step: be near the target. The helpful-looking one pays for closing the gap and, kindly, never charges for opening it. In code that is max(0, |x_prev| - |x|). Engineers write this constantly, to give a sparse task something to climb.

The winning controller under each of two reward functions: the honest one converges on the target, the progress-only one oscillates between the wall and the target and banks three times the available progress

The search under the honest reward lands on the target and stays there. The search under the progress reward finds an oscillator: run in, collect the payment, drift back out for free, run in again. On my run it banked 3.01 units of progress toward a target 1.0 units away, ending 1.96 units away from it. Under the reward it was given, that is a threefold improvement.

You cannot learn from a reward you never receive

The other failure has nothing to do with the reward being wrong. It is the reward being right and unreachable.

Suppose you reward a robot arm with $+1$ when a cube is in the box and $0$ otherwise, which is a perfectly correct specification. A randomly initialised policy flails. It will never, in any number of attempts, happen to reach, grasp, lift, transport and release. The reward is zero for every episode, every gradient is zero, and the policy learns nothing at all. This is the exploration problem, and it is why sparse-reward manipulation from scratch is close to hopeless while sparse-reward locomotion is merely difficult: a flailing quadruped moves somewhere, so there is always a gradient.

The two escapes are worth naming now because they shape the whole field. The first is shaping the reward into something dense that guides the search, which is what invites the gaming you just watched. The second is starting from demonstrations and using reinforcement learning only to improve on them, which is where the interesting 2026 systems live.

Why legs and hands land on different sides

Both halves of this module solve the same problem - produce actions from observations - and the choice between them is not about which algorithm is better. It is about which specification you can actually write.

	Locomotion	Manipulation
Can a human demonstrate it?	No. Twelve coordinated joints at 50 Hz is beyond a teleoperator.	Yes, and easily. Move the leader arm.
Can you write the score?	Yes, and it is almost obvious: track the commanded velocity, stay upright, do not waste power.	Badly. “The towel is folded” has no clean formula, and “the cube is in the box” is unreachable by chance.
Is failure cheap?	In simulation, yes, and the physics that matters is rigid-body contact, which simulators are good at.	Less so. What matters is grip, slip and deformation, which simulators are worse at.
So the field uses	reinforcement learning in simulation, then transfer	imitation learning from demonstrations

Check yourself

1. Why does the subscript in

\mathbb{E}_{\tau \sim \pi_\theta}

make this a fundamentally harder optimisation than supervised learning?

Because the distribution being averaged over depends on the parameters being optimised. In supervised learning the dataset is fixed: you can evaluate the loss, take a step, and the data has not moved. Here, changing $\theta$ changes which episodes you will ever see, so the objective is defined over a moving target and any data you already collected describes a policy that no longer exists.

2. A colleague reports “our policy needed 200 million steps, which is a lot of data”. What do you need to ask before that sentence means anything?

Whether those steps were simulated or real. Two hundred million simulated steps is a modest GPU job. Two hundred million real steps at 50 Hz is roughly forty-six days of unbroken robot motion and does not happen. The unit changes the claim from routine to impossible.

3. You reward a walking robot with the distance travelled since the last step, clamped at zero below so it is never punished for going backwards. What behaviour should you expect, and why?

Oscillation. A forward-backward cycle earns on the forward half and costs nothing on the backward half, so the return grows without bound with no net progress. It is the same bug as reward_hacking.py’s progress term. The fix is to let the term go negative, or to reward absolute position rather than change in position.

4. Why is sparse-reward locomotion merely hard while sparse-reward manipulation is close to hopeless?

Exploration. A randomly initialised quadruped still twitches, falls and slides, so it accumulates some displacement, and any reward tied to displacement is non-zero for almost every episode; there is a gradient from the first minute. A randomly initialised arm will essentially never execute the reach-grasp-lift-transport-release sequence by chance, so a reward that only pays on success is zero on every episode and every gradient is zero. You cannot climb a signal you never receive.

5. Your quadruped’s reward is velocity tracking alone, with no other terms. It learns to hit the commanded velocity. Name two behaviours that satisfy this reward perfectly and that you would refuse to deploy.

Anything that moves at the right speed while being unusable: dragging itself along on its belly, or vibrating every joint at maximum torque to shuffle forward. Both track the velocity exactly. Neither is a gait. Every penalty term in a real locomotion reward exists to rule out one of these, which is why the reward is a weighted sum rather than a single term.

6. Give a task where a human can demonstrate it and you can score it cleanly, and say which method you would use.

Reaching a known pose in free space, for instance. You can teleoperate it and you can score it with the distance to the target. The answer is that you should use neither: it is inverse kinematics plus a trajectory, which you wrote in Module 1, and it will be more reliable and more debuggable than any learned policy. When both specifications are easy, that is usually a sign the task has a closed-form solution. Lesson 3.18 makes this argument properly.

Do this

1. Watch a reward get gamed, then fix it. Fill in the two reward functions in code/reward_hacking.py and run it. It takes about fifteen seconds.

Then change reward_progress so it cannot be farmed, and re-run. There are at least three fixes: drop the max(0, ...) so backward motion is charged for; add a small per-step penalty proportional to distance; or pay only for the best distance achieved so far, so a round trip earns nothing the second time. Try one, and check that the winning controller under your new reward now actually lands on the target.

2. Do the arithmetic that decides the method. For a task you care about, write down four lines: how long one human demonstration takes, how many you would need, how long one simulated step takes, and how many steps a from-scratch policy would need. If the second product is smaller than the first, you have an imitation problem. This is a two-minute calculation and it will save you a month.

3. Try to break your own reward on paper. Take any reward function you have written or can imagine and spend five minutes as the adversary: find a cycle that pays, a way to stop the episode early to avoid a penalty, or a degenerate pose that scores well. You will find one. Doing this before the twelve-hour training run rather than after is the entire skill.

What you can now do

You can state the reinforcement learning problem in its own terms - state, action, reward, policy, return - and explain why the fact that the policy generates its own data makes it harder than any supervised problem. You can do the arithmetic that separates a simulated sample budget from a real one, and use it to decide which method a task wants. You have watched a search game a reasonable-looking reward function on your own machine, and you can name the two failure modes that will consume most of your reinforcement learning time: a specification that is wrong, and a specification that is right but unreachable.

Next, how the gradient of that objective is even computed, given that no gradient can pass through the physical world.