From a score to a gradient, and why PPO clips

You cannot differentiate through the physical world, so policy gradients make the actions that led to good scores more likely, and everything after that is fighting the noise in that estimate.

Where you are. You know what reinforcement learning is asking for: maximise the expected return of episodes generated by your own policy. This lesson is how that turns into a gradient you can hand to an optimiser, when nothing in the loop is differentiable.

The knob and the scoreboard

You ship a change to a recommendation system on a Monday. There is no test that says whether it was a good change, and there is no way to write one. What comes back, a week later, is revenue.

You cannot differentiate revenue with respect to your code. There is no chain rule that runs through a million users’ Tuesday. All you have is a knob you turned and a score that came back.

So you do the only thing available. You try a batch of variants, you note which ones scored above average, and you shift your next batch toward those. Nobody told you the right answer. You inferred a direction from a scoreboard.

That is the entire idea behind every algorithm in this lesson. The robot version has one extra convenience: you know exactly which knob you turned, because you sampled the action.

The idea in one paragraph

No gradient can pass through physics. The simulator is not differentiable in any way you want to rely on, and the real robot certainly is not. What saves you is that the policy is a probability distribution over actions, and that part is differentiable. So instead of asking “how should the action change to raise the score”, you ask “how should the probability of the action I already took change”. The answer turns out to be beautifully simple: increase the log-probability of every action taken, weighted by how well things went after it. That is the policy gradient, and it is unbiased and correct. It is also extremely noisy, and the next thirty years of the field, up to and including PPO, are almost entirely about reducing that noise without breaking the correctness.

Three sampled trajectories scored after the fact: the good one has the probability of its actions pushed up, the bad one pushed down, the ordinary one barely moved

Wider than the screen; scroll it sideways.

The one derivation worth doing

We want the gradient of $J(\theta) = \mathbb{E}_{x \sim p_\theta}[f(x)]$ where $f$ is a black box - the world - and $p_\theta$ is something we can differentiate. Write the expectation as an integral and push the gradient inside:

\nabla_\theta \mathbb{E}_{x \sim p_\theta}[f(x)] = \nabla_\theta \!\int p_\theta(x)\, f(x)\, dx = \int \nabla_\theta p_\theta(x)\, f(x)\, dx

Now the trick, and it is one line. Since $\nabla_\theta \log p_\theta = \nabla_\theta p_\theta / p_\theta$ , we can write $\nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta$ . Substituting turns the integral back into an expectation:

\nabla_\theta \mathbb{E}_{x \sim p_\theta}[f(x)] = \int p_\theta(x)\, \nabla_\theta \log p_\theta(x)\, f(x)\, dx = \mathbb{E}_{x \sim p_\theta}\!\left[\nabla_\theta \log p_\theta(x)\, f(x)\right]

The left side needs the derivative of the world. The right side needs only the derivative of your own network, evaluated on samples the world handed back. That swap is the whole reason any of this is computable.

Apply it to trajectories. The probability of an episode factorises into the physics (which does not depend on $\theta$ ) times the policy’s choices (which do), so every term that mentions physics differentiates to zero and drops out. What survives is:

\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\!\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t)\; G_t \right]

In words: for every action you took, nudge its log-probability up in proportion to the return that followed it. Good episode, everything in it becomes more likely. Bad episode, everything in it becomes less likely. That is REINFORCE, and in PyTorch it is one line - a weighted negative log-likelihood, which is to say cross-entropy with the returns as the weights.

The estimate is correct and nearly useless

Unbiased does not mean usable. The estimator is an average over sampled episodes, and if that average swings wildly from batch to batch, you are taking gradient steps in a direction that is mostly noise.

Two fixes, in the order they matter.

Drop the past. The formula above credits every action with the whole episode’s return. But an action at $t = 40$ cannot have caused reward collected at $t = 10$ . Including it adds noise without adding information, and it can be dropped without introducing bias, which is why we write $G_t$ (the return still to come) rather than $R(\tau)$ (the whole episode).

Grade on a curve. “This action led to a return of 92” is not useful on its own; what matters is whether 92 is better than this state usually pays. Subtract a baseline $b(s_t)$ that depends only on the state. Because it does not depend on the action, it cancels in expectation and the estimator stays unbiased. The natural choice is the value function $V(s_t)$ , the expected return from that state, and the difference gets its own name:

A_t = G_t - V(s_t)

These are not equal in size, and the difference is worth measuring rather than believing. Run python policy_gradient.py variance, which freezes one policy and takes thirty-two independent gradient estimates from it three different ways. On my laptop:

Weight on each log-probability	Total variance of the gradient
$R(\tau)$ , the whole episode return	48.32
$G_t$ , the return still to come	3.59
$A_t = G_t - V(s_t)$	2.81

All three estimate the same gradient. Dropping the past cut the noise by 13.5 times; the learned baseline cut it by a further 1.3. That ordering is not what most write-ups imply, and it is worth carrying: the causality fix is nearly free and does most of the work, and the baseline is the refinement.

Learn $V$ with a second network trained by regression onto the observed returns, and you have an actor-critic method: the actor picks actions, the critic predicts how good states are, and the actor’s updates are graded against the critic’s predictions.

The data expires the moment you use it

Here is the constraint that shapes PPO. Every expectation above is over $\tau \sim \pi_\theta$ : episodes generated by this policy. Take one gradient step and the policy changes, so the batch you just collected describes a policy that no longer exists.

A batch collected by one policy stays informative only near that policy, so a large update walks out of the region where its own data means anything

Wider than the screen; scroll it sideways.

Being strictly on-policy is enormously wasteful: you pay for a whole batch of simulation and spend it on a single gradient step. The obvious fix is to take several steps per batch. The obvious fix does not work, because after a few steps the batch is describing a policy you have left behind, and you are now confidently optimising a stale objective.

You can correct for the drift with an importance ratio - how much more likely the new policy makes each action that the old one actually took:

r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}

That correction is exact in theory and unstable in practice, because a ratio can blow up and one enormous term can dominate a whole batch.

PPO is a rate limiter

Its answer is not to correct the drift more cleverly. It is to make drift stop paying:

L^{\text{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\Big(r_t(\theta)\,A_t,\;\; \text{clip}\big(r_t(\theta),\, 1-\epsilon,\, 1+\epsilon\big)\, A_t\Big)\right]

with $\epsilon$ typically 0.2. Read it in two cases and it stops being cryptic.

The clipped objective as the policy drifts: it flattens once the action's probability has moved more than 20 percent in the helpful direction, so the gradient there is zero

If the action beat expectations ( $A_t > 0$ ), the objective rises as you make it more likely - up to $1 + \epsilon$ , and then it goes flat. Beyond a 20% increase there is nothing more to gain, so the gradient is zero and the update stops pushing. If the action fell short ( $A_t < 0$ ), the mirror image: you are paid for suppressing it, but only down to $1 - \epsilon$ . The min is what makes it conservative in both directions - it always takes the less favourable of the clipped and unclipped values.

What that is worth, measured

Fill in the three update rules in code/policy_gradient.py and run them on the cart-pole. Same network, same data budget per iteration, sixteen complete episodes per batch; the only difference is the loss.

Four update rules on one cart-pole task, one seed each: the unclipped version reaches the ceiling and then collapses, while PPO holds it

Three seeds each, counting environment steps to first reach a mean episode length of 195:

Update rule	Steps to solve, three seeds	Median
REINFORCE	70k, 86k, 78k	78k
plus a value baseline	88k, 59k, 62k	62k
PPO	50k, 82k, 47k	50k
PPO’s reuse, clip removed	52k, 24k, 70k	52k

Look at those honestly, with lesson 3.12 in mind. The medians are ordered the way the theory says. The ranges overlap almost completely, and at three seeds this table cannot support “PPO is faster than REINFORCE here”. On a task this easy, it probably is not by much.

The clip’s value shows up somewhere else entirely: not in how fast you arrive, but in whether you stay. Run python policy_gradient.py stability, which keeps training past the finish line.

	worst mean episode length after solving
PPO	191, 174, 190
the same reuse, unclipped	20, 21, 153

Two of the three unclipped seeds reached the 200-step ceiling and then fell off it back to about 20, which is where an untrained policy sits. They reused the batch four times, walked far enough that the batch no longer described them, and destroyed a working policy. PPO’s worst moment across three seeds was 174.

The other half of the family

PPO is on-policy. The other branch keeps a replay buffer and learns a Q-function, so old experience stays useful; SAC is the representative, and it adds one idea worth knowing: alongside reward it maximises the policy’s entropy, which is to say it is explicitly paid to stay as random as it can while still doing well. That keeps exploration alive instead of collapsing early onto whatever worked first.

The practical split follows directly. In simulation, samples are nearly free and you want stability and parallelism, so locomotion uses PPO. On a real robot, every sample costs wall-clock and wear, so real-robot reinforcement learning uses SAC-family methods for their sample efficiency.

One naming note if you go looking in LeRobot: as of the v0.6.0 release (July 2026) the policy type sac became gaussian_actor under a rebuilt reinforcement learning API, so any tutorial passing --policy.type=sac predates that. It is a small thing and a good illustration of why the resources page carries dates.

Check yourself

1. Why can the policy gradient be computed at all, when the simulator is not differentiable?

Because the gradient never touches the simulator. The log-derivative identity rewrites the gradient of an expectation as an expectation of $\nabla_\theta \log \pi_\theta(a \mid s)$ weighted by a scalar score. The only derivative taken is of the policy network’s own output distribution, evaluated at actions that were already sampled; the environment contributes numbers, not gradients.

2. Subtracting

V(s_t)

changes every weight in the update. Why does it not bias the gradient?

Because the baseline depends only on the state, not on the action that was chosen. In expectation, the extra term is $V(s_t)$ times the expected value of $\nabla_\theta \log \pi_\theta(a \mid s_t)$ , and that expectation is zero for any distribution that sums to one. So the mean of the estimator is unchanged, and only the spread shrinks. If your baseline peeked at the action, this argument would fail and the gradient would be biased.

3. In the measured variance table, dropping the past helped 13.5 times and the baseline a further 1.3. What does that ordering tell you about where to look first when a run is noisy?

That the cheap structural fix beats the learned one. Crediting an action with reward collected before it was taken injects pure noise, and removing it costs nothing and needs no extra network. Check that you are using returns-to-go before you go tuning a critic. The baseline is worth having, but on this task it was a refinement on top of a much bigger win.

4. Why is a clipped objective necessary at all, when the importance ratio already corrects for the policy having moved?

The correction is unbiased but not bounded. As the new policy diverges from the old, ratios for rare actions can become enormous, one sample can dominate the batch, and the variance of the estimate explodes. Clipping does not try to correct further; it removes the incentive to move that far in the first place, so the ratios stay near one and the correction stays well behaved.

5. Two of three unclipped seeds solved the task and then collapsed to an episode length of about 20. Give the mechanism.

Each batch was reused for four epochs of minibatch updates. Without a clip, nothing stops those updates from moving the policy a long way, and by the later epochs the advantages and log-probabilities in the batch describe a policy that has already been left behind. The optimiser then confidently takes a large step on a stale objective, which can land far from anything the data supports, and the policy that worked is gone.

6. The medians in the sample-efficiency table are ordered as the theory predicts, but the ranges overlap. What is the correct thing to write in a report?

That three seeds do not separate these methods on this task, and quote the ranges rather than the medians. The honest sentence is something like “across three seeds, REINFORCE solved between 70k and 86k steps and PPO between 47k and 82k; the difference is not resolved at this sample size”. Reporting only the medians would turn seed noise into a claim, which is the exact failure lesson 3.12 is about.

Do this

1. Implement the three updates. In code/policy_gradient.py, fill in update_reinforce, update_baseline and update_ppo, then run each. Each takes roughly twenty to sixty seconds on a laptop CPU. Watch the printed solve point move.

2. Measure the variance yourself. Fill in the three grad_vector calls in gradient_variance and run python policy_gradient.py variance. Your absolute numbers will differ from mine; the ratios should not, and if the baseline makes things worse you have a bug in the detach.

3. Break it on purpose. Run python policy_gradient.py noclip, then python policy_gradient.py stability. Then go further: set CLIP_EPS = 1.0, which is so wide it may as well not exist, and epochs=16 in update_ppo. Note how many seeds still hold the ceiling. This is the cheapest possible demonstration of why a trust region exists, and it takes four minutes.

4. Predict, then check. Before running anything, write down which of the four you expect to solve the task in the fewest environment steps, and which to end up with the highest final score. On my machine those are not the same method. Getting this prediction wrong is more instructive than getting it right.

What you can now do

You can derive the policy gradient from the log-derivative identity and say exactly why it sidesteps the non-differentiable world. You can explain what returns-to-go, a baseline and an advantage each remove from the estimate, and you have measured how much each is worth on your own machine. You can read PPO’s clipped objective and describe it as what it is - a rate limiter that makes reusing a batch safe - and you have watched a policy that reused data without one solve a task and then destroy itself.

Next, what this looks like at a hundred million steps, on a robot with legs.