Covariate shift: why the clone falls apart

A policy generates the states it is later judged on, so being slightly wrong changes the question it is asked next, and the error feeds itself.

Where you are. You can clone demonstrations into a network and score it on held-out frames. This lesson runs one, which is the only way to find out that the score was never the point.

One number, and forty out of forty becomes zero out of forty

A cloned policy. Fifty demonstrations, held-out loss of 0.0009, which is about as clean a fit as you will ever see. Forty fresh episodes: it completes all forty.

Now change one number and run the same forty episodes again. Not the learning rate, not the width, not the amount of data, not the seed. The number is how long you let it run: sixty steps, which is how long a demonstration lasted, becomes 240 steps, which is what a robot does when nobody stops it.

Zero out of forty.

The median episode ends at step 81. By then the arm has torn the pin out of the slot it was following. Left running to step 239, the joint angles are around 189 radians away from anything in the training set, which is thirty full turns of a joint that spent every demonstration inside a range of two radians. The arm is spinning.

Nothing about the policy changed between those two runs. Everything about what it was asked changed, and it did the asking itself.

The idea in one paragraph

Supervised learning rests on one assumption: the data you train on and the data you are tested on come from the same distribution. A policy breaks that assumption by construction, because it acts, and its actions produce the observations it will be tested on next. Train on the states an expert visits, deploy into the states the policy visits, and those are different sets the instant the policy makes its first small mistake. Worse, the difference feeds itself: a small error moves the robot slightly off the demonstrated path, where it has less supervision, so the next error is larger, which moves it further. This is covariate shift, and it is the reason imitation learning is a subject rather than a fit call.

A held-out set that being wrong cannot change, against a policy whose action rewrites the next observation it is asked about

Wider than the screen; scroll it sideways.

A world you can act in

The SO-101 dataset cannot show you any of this, because a recording has no opinion about what happens when the policy is wrong. So this lesson uses the smallest world that can act back.

The groove task: a two-link arm drags a pin along a curved slot, seeing only its own two joint angles

Wider than the screen; scroll it sideways.

A two-link planar arm, the one from Module 1, dragging a pin along a curved groove milled into a plate. The observation is the two joint angles. The action is how far to turn each joint. The pin must get from A to B, and if it leaves the groove it jams and the episode is over.

The demonstrator is a scripted expert: it finds the nearest point on the groove, looks a little way ahead, and steers there. It is the pure-pursuit tracker from Module 1, and the important thing about it is that it is a rule. It is correct at every pose in the plane, including poses no demonstration ever visited, so it never runs out of road. Forty out of forty at 240 steps, and forty out of forty even when it is started four centimetres off centre.

Fifty demonstrations, 60 steps each, 3,000 rows. Exactly the shape of the last lesson, small enough to see.

Sixty steps, then 240

how long it runs	scripted expert	cloned policy
60 steps, the length of a demonstration	40/40	40/40
240 steps	40/40	0/40

Both rows are the same policy on the same task. The clone is not merely imperfect at 240 steps, it is never once successful, and the median run is over by step 81.

Nothing here is even off-distribution in an exotic sense. Every demonstration ends with the pin parked at B and the expert commanding almost nothing, so holding still at B is the single most heavily represented situation in the training set. The expert holds it indefinitely. The clone holds it for about twenty steps and then wanders off.

It is not underfitting, and it is not the data

The two reflexes are to train longer and to record more. Both were measured.

Training longer, on the same fifty demonstrations:

epochs	held-out loss	median steps survived
100	0.003899	112
200	0.000907	81
400	0.000802	72
800	0.000533	98

Eight times the training reduces held-out loss sevenfold and does nothing whatsoever for the robot. More demonstrations, three seeds each:

demonstrations	held-out loss	success at 240 steps
10	0.02144	0.75 ± 0.04
25	0.00224	0.33 ± 0.47
50	0.00127	0.00 ± 0.00
100	0.00097	0.00 ± 0.00
200	0.00112	0.33 ± 0.47

Twenty times the data does not fix it, and the column is not even monotone. Be careful how much you read into the middle rows: a spread of 0.47 across three seeds means one seed succeeded and two failed completely, which is not a measurement of anything. What the table does support is the negative claim, and the negative claim is the point. The supervised metric improves. The task does not.

The experiment that settles it

All of the above is circumstantial. Here is the direct measurement, and it takes one network and one metric.

Take the trained clone. Ask it what to do at every state the expert visits, and measure how far its answer is from the expert’s. Then ask it what to do at every state it visits itself, and measure exactly the same thing. Same network. Same question. Same units. Only the states differ.

The same network, asked about the expert's states and about its own

where the question comes from	action error
states the expert visits	0.000496 rad
states the clone visits	0.002911 rad
over the last ten steps of an episode	0.000063 against 0.006976 rad

Six times worse on average, and by the end of an episode a factor of 112. The network has not changed between those two columns. It is simply being asked about a part of the world that it was never shown, and it answers badly there, and its answers are what put it there.

Watch the gap open up over a single episode:

Distance from the states a policy visits to the nearest state it was trained on

step	expert	clone
30	0.00000	0.00381
60	0.00000	0.01436
90	0.00000	0.06594
120	0.00000	0.29126
180	0.00000	7.60433
239	0.00000	189.43341

Distance to the nearest training state, in radians. The expert stays at zero because it is a rule that reproduces its own data. The clone doubles its distance roughly every thirty steps until the numbers stop meaning anything.

The demonstrated states, and where the clone actually goes

Why it compounds instead of just being wrong

The mechanism is worth stating precisely, because the intuition “errors add up” is not quite it.

Errors adding up would be linear. What happens here is that each error changes the question. A policy with per-step error $\epsilon$ on the expert’s states makes one small mistake and arrives somewhere the expert never was. At that state the policy’s error rate is no longer $\epsilon$ - nobody ever told it what to do there - so the next mistake is bigger, and it lands somewhere further out still.

Each error moves the policy off the demonstrated tube, where the next error is larger

Wider than the screen; scroll it sideways.

Interactive: nudge the pin a few millimetres off the groove and three policies leave from that same start. The scripted expert recomputes pure pursuit from wherever it is, so it steers back. The cloned policy replays the action recorded at the nearest demonstrated state, so being slightly off buys it a slightly worse answer, which buys it a larger offset, which buys it a worse answer again. The third policy has expert labels on off-path states and recovers, which is the whole of the next lesson. Static version: with no nudge at all the clone still drifts out of the groove, because its command on the data is nearly right rather than exactly right; a nudge of a few millimetres cuts its life by more than half, and the same nudge costs the expert nothing.

The formal version is the founding result of this corner of the field. For a horizon of $T$ steps, behaviour cloning has a worst-case cost bound that grows like

O(T^{2}\epsilon)

quadratic in the horizon, while a policy trained on the states it will actually visit gets

O(T\epsilon)

which is linear. In words: doubling how long you run a cloned policy can quadruple how badly it does, and that extra factor of $T$ is entirely the cost of being tested on your own mistakes.

What this means for the policy you trained last lesson

The SO-101 clone from the last lesson has this problem. You cannot see it, because you have no way to run it, and that is the most uncomfortable sentence in this module.

Two consequences follow immediately, and they change how you read every robot-learning result including your own.

A held-out loss is not evidence that a policy works. It is evidence about frames drawn from the expert’s distribution. It is a necessary condition and a weak one, and this lesson has just shown a case where making it seven times better changed nothing.

Episode length is part of the claim. “It works” from a policy evaluated over sixty steps and “it works” from a policy evaluated over 240 are different statements, and the demo you saw online was almost certainly the first one. When you read a success rate, look for how long each trial ran and what counted as the end.

The rest of this module is a sequence of answers to this lesson. The next one gives the policy data where it actually goes. After that, action chunking cuts the number of chances the policy gets to drift, and generative policies stop it drifting for a different reason again.

Review

One number turns forty out of forty into zero

A cloned policy, fifty demonstrations, a held-out loss of about one ten-thousandth, which is as clean a fit as you will ever see. Forty fresh episodes and it completes all forty. Now change one number and run the same forty again. Not the learning rate, not the width, not the amount of data, not the seed: how long you let it run. Sixty steps, the length of a demonstration, becomes two hundred and forty, which is what a robot does when nobody stops it. Zero out of forty. The median episode is over by step eighty-one, and left running the joint angles end up about one hundred and eighty-nine radians from anything in the training set, thirty full turns of a joint that spent every demonstration inside a range of two. Evaluated over the horizon its demonstrations happened to have, the clone looks finished.

The policy writes its own exam

Supervised learning rests on one assumption: the data you train on and the data you are tested on come from the same distribution. A policy breaks that assumption by construction, because it acts, and its actions produce the observations it will be tested on next. Train on the states an expert visits, deploy into the states the policy visits, and those are different sets the instant the policy makes its first small mistake. Worse, the difference feeds itself. A small error moves the robot slightly off the demonstrated path, where it has less supervision, so the next error is larger, which moves it further still. That is covariate shift. Nothing about the task moved; the set of questions moved. And it is not exotic. Every demonstration ends with the pin parked at the target, so holding still there is the most heavily represented situation in the whole training set, and the clone holds it about twenty steps before wandering off.

Not underfitting, and not the data

The two reflexes are to train longer and to record more, and both were measured. Eight times the training cut the held-out loss sevenfold and did nothing whatever for the robot. Twenty times the data did not fix it either, and the column is not even monotone. The supervised metric improves and the task does not. Then notice which policy survives best in that table: the ten-demonstration one, whose held-out loss is twenty times worse than the best. It is blurrier, less confident and moves less decisively, and on this task those are survival traits. Whenever a worse fit gives you a better robot, you are not looking at an optimisation problem, and no amount of training or data will remove it.

Check yourself

1. The clone scores 40/40 at sixty steps and 0/40 at 240 steps, with no retraining in between. Where did the extra failures come from?

From states the clone put itself in. Sixty steps is the length of a demonstration, so up to that point the policy is still roughly inside the region the demonstrations covered. Past it, the only thing generating states is the policy’s own slightly-wrong actions, each one moving it a little further from anything it was trained on, where its actions get worse still. The task never changed and the network never changed; the distribution of states it was asked about did.

2. Training eight times longer improved held-out loss from 0.0039 to 0.00053 and left median survival at roughly a hundred steps. Explain why that is not a contradiction.

Because held-out loss measures accuracy on states drawn from the expert’s distribution, and the policy fails on states drawn from its own. Optimising harder makes it more accurate on the first set, which the second set barely intersects by the time failure happens. You can drive the training objective to zero and still have said nothing about behaviour off the demonstrated path, because there was never any training signal there to optimise.

3. Design the measurement that distinguishes “the network underfits” from “the network is being asked the wrong questions”, using only a trained policy and an expert.

Evaluate the same network on two sets of states: those the expert visits, and those the policy visits when you roll it out. Compare its action error against the expert on each. If the two errors are similar and both large, the network underfits. If the error on expert states is small and the error on its own states is much larger, the fit is fine and the distribution has moved. Measured here: 0.000496 rad against 0.002911 rad on average, and a factor of 112 by the end of an episode.

4. Holding still at B is the most heavily represented situation in the training set, since every demonstration ends there. Why is that where the clone fails?

Because “heavily represented” is not the same as “correctly executed forever”. The clone’s command at B is very nearly zero but not exactly zero, so it creeps. A pose slightly past B is a pose no demonstration contains, and the clone’s answer there is an extrapolation, which creeps a little more. The expert has no such problem because it recomputes its rule from the geometry every step, so it re-derives “stay put” from wherever it happens to be. This is compounding error with no disturbance at all: just a residual and enough time.

5. Why is the bound quadratic in the horizon rather than linear, and what would make it linear?

Linear is what you get if each step contributes an error of $\epsilon$ independently. Here each mistake also moves the robot into a region where the error rate is no longer $\epsilon$ , so the horizon appears twice: once for the number of chances to be wrong, and once because being wrong earlier makes you worse later. Training on the state distribution the policy actually induces removes the second factor, which is what DAgger does and what makes it $O(T\epsilon)$ .

6. A colleague reports 90 percent success on a manipulation task. Name three things you now need to know before the number means anything.

How long each trial ran and what ended it, since a policy that looks perfect over a demonstration-length episode can be at zero over four times that. What the starting conditions were, and whether they were the demonstrated ones or varied. And how many trials the 90 percent is over, because at ten trials that is nine successes and one failure. The last of those has a whole lesson of its own in evaluation methodology.

Do this

code/groove_world.py is given to you complete. code/covariate_shift.py has three TODO(you) markers: the two-horizon comparison, the same-network probe, and the distance-to-data measurement.

python groove_world.py            # check the expert first: it should be perfect
python covariate_shift.py         # 60 steps against 240 steps
python covariate_shift.py --probe # expert's states against its own
python covariate_shift.py --drift # how far off the data it gets
python covariate_shift.py --demos # does more data fix it?

Each of the first four takes well under a minute on a laptop CPU; --demos retrains fifteen times and took about two minutes here. When the TODOs are right you will reproduce this lesson’s numbers: 40/40 then 0/40, 0.000496 against 0.002911 radians, and a nearest-training-state distance that reaches 189 radians.

Then three experiments that are more interesting than the ones above:

Find the horizon where it breaks. Evaluate at 60, 90, 120, 180 and 240 steps and plot success against horizon. You are measuring the shape of the $T^{2}$ argument on your own policy. Note where the curve turns over, and compare that step number against where the distance-to-data plot starts climbing.
Nudge it and watch the two policies diverge. evaluate(policy, offset=d) starts the pin d metres off the groove centre. Sweep d from 0 to 0.04 for the expert and for the clone, at both horizons. At sixty steps both are perfect at every offset, which is worth seeing for yourself before you trust any “it is robust” claim made over a short episode.
Give the expert a bad day. Add a small random disturbance to the expert’s own executed action inside collect, so the demonstrations wobble, and clone that instead. Your fit will get worse and your held-out loss will get worse. Predict what happens to survival before you run it, then run it. What you find is the whole of the next lesson, and finding it yourself is much better than being told.

What you can now do

You can explain why a policy is tested on a distribution it created, and why that makes a held-out loss a weak claim rather than a strong one. You can run the measurement that separates underfitting from distribution shift using one trained network and an expert, and read the answer off two numbers. You can say why training longer and recording more demonstrations both fail to fix it, why the horizon enters the error bound twice, and what to ask about any success rate somebody quotes you.