Behaviour cloning from scratch

Behaviour cloning is supervised learning on a table of observation and action rows, and the whole algorithm is one call to fit.

Where you are. You can read a demonstration dataset and build policies that need no training at all. This lesson fits one, on the same 50 episodes and the same split, so the numbers land in the same table.

Delete a column and shuffle the rows

Put the pick-and-place dataset on screen. Eleven thousand nine hundred and thirty-nine rows. Six columns of what the arm reported about itself. Six columns of what the human’s hand told it to do a thirtieth of a second later. And one more column, saying which of the fifty demonstrations each row came from.

Delete that last column.

Now shuffle the rows.

Nothing breaks. The pairs are still pairs. Every row still says “when the arm looked like this, the human did that.” You can hand the whole shuffled pile to any regression tool ever written and it will happily fit it.

Cover the word robot and what is left on screen is X and y.

That shuffle is the best idea in this lesson and the worst one. It is why the next hour of your life is easy. It is also, precisely and entirely, the thing that will break in the lesson after this one.

The idea in one paragraph

Behaviour cloning is that shuffle taken seriously. You treat the demonstrations as a supervised dataset, fit a function that maps observation to action by minimising squared error, and call the fitted function a policy. There is no simulator, no reward, no exploration, and no robot in the loop while it trains. It is the same procedure you would use to predict house prices, pointed at a robot arm. That sounds like an insult and it is not: behaviour cloning is the strongest baseline in robot learning, it is what every method in this module is a variation on, and on a laptop with fifty demonstrations it takes about four seconds.

Behaviour cloning turns ordered episodes into an unordered bag of rows, then fits a network to them

Wider than the screen; scroll it sideways.

The network is the boring part

Six numbers in, six numbers out, two hidden layers of 256 units, a ReLU between them. That is the whole architecture and it is deliberately unremarkable, because in imitation learning the architecture is rarely what decides your outcome.

Two details in it are worth stating out loud.

No activation on the output layer. The network’s output is a joint angle. A tanh or a sigmoid there would squash every prediction into a fixed range, and the arm would quietly lose the parts of its travel that fall outside it. Leave the last layer linear.

Mean squared error against what the human commanded. Written out, you are searching for the weights $\theta$ that solve

\theta^{*} = \arg\min_{\theta}\ \frac{1}{N}\sum_{i=1}^{N} \bigl\lVert \pi_\theta(o_i) - a_i \bigr\rVert^{2}

which says, in words: over all $N$ recorded frames, make the action the network produces from observation $o_i$ as close as possible to the action $a_i$ the human actually sent, where “close” means squared distance summed over the six joints.

Two things that must be right

Everything else about this fit is forgiving. These two are not.

Standardise both sides, using training-set statistics only. The six joints do not live on the same scale in this dataset: shoulder pan is commanded across a span of 181 units, the gripper across 33 - the numbers the audit in lesson 3.1 printed. Squared error adds those columns together, so an unscaled fit spends its capacity on the joints that happen to have large numbers and treats the gripper as rounding error. Subtract the mean and divide by the standard deviation, computed on the fitting episodes alone, and every joint gets an equal vote.

Split by episode, never by frame. At 30 frames per second, two neighbouring frames are almost the same picture with almost the same command. Shuffle frames and then split, and near-duplicates of your test rows are sitting in the training set. The error collapses and means nothing.

Splitting frames at random leaks near-duplicates across the line; splitting whole episodes does not

Wider than the screen; scroll it sideways.

Forty episodes to fit, ten held out, and the line falls between whole demonstrations. That is 9,180 rows for fitting and 2,759 to be scored on.

Run it

Thirty epochs, batch size 128, Adam at $10^{-3}$ . On this laptop CPU, with no GPU of any kind, it finishes in about four seconds.

Held-out action error while the clone trains, against the two untrained policies from the last lesson

Held-out error, in the same units as the last lesson, where one unit is roughly one percent of a joint’s calibrated travel:

policy	held-out action error
always command the average pose	23.893
nearest neighbour	4.884
command the pose you are already in	3.020
behaviour cloning	2.108

It beats everything, including the do-nothing baseline that embarrassed all three untrained policies last lesson. It does so from 69,126 weights instead of 9,180 stored rows, it answers in about 0.05 ms per call on this machine - three times faster than the nearest-neighbour lookup it replaces - and that cost stays flat however many demonstrations you add.

The cloned command beside the human's, over one held-out episode

Where it wins, and where it loses

The average hides the interesting part. Per joint:

joint	do nothing	cloned
shoulder pan	5.110	3.019
shoulder lift	3.172	2.651
elbow flex	3.596	1.999
wrist flex	1.811	2.049
wrist roll	2.183	1.284
gripper	2.250	1.648

The clone wins biggest on the joints that move most, which is the sensible place to win. And on wrist flex it loses: 2.049 against 1.811 for a policy that does nothing at all.

Capacity is not what is limiting you

The instinct after seeing 2.108 is to make the network bigger. Measured, at the same thirty epochs:

hidden width	parameters	held-out error
32	1,478	2.417
64	4,998	2.297
128	18,182	2.463
256	69,126	2.108
512	269,318	2.008

One hundred and eighty times the parameters buys seventeen percent of the error, and the curve is not even monotone. Training longer is worse than useless: at 120 epochs the training loss has fallen from 0.0208 to 0.0142 while the held-out error has climbed from 2.108 to 2.398.

What you have, and what you have not

You have a function that reproduces a human’s commands on episodes it has never seen, better than any baseline available without training, in four seconds, on a laptop.

You do not have a robot that picks up bricks, and nothing in this lesson could have told you whether you did. Every number here is a claim about frames: given this observation, how close was the command. Task success is a claim about episodes, and the two come apart badly - the last lesson already showed you a policy that scores well by sitting perfectly still.

There is a harder problem underneath. To measure task success you have to run the policy, and this dataset is a recording. A recording has no opinion about what happens when the policy is slightly wrong, because in a recording the policy never acts. Getting an opinion requires a world you can act in, which is what the next lesson goes and gets.

Check yourself

1. Why is there no activation function on the network’s last layer, when there are ReLUs between the hidden ones?

Because the output is a joint angle and must be free to take any value the arm can reach. A tanh or sigmoid bounds the output to a fixed interval, so any commanded pose outside that interval becomes unreachable no matter how much data you have, and the failure looks like a policy that mysteriously refuses to extend fully. The ReLUs between hidden layers serve a different purpose: without a nonlinearity somewhere, the whole stack collapses to a single linear map.

2. You standardise the observations using the mean and standard deviation of the entire dataset rather than the fitting episodes only. What exactly has gone wrong, and how would you notice?

The held-out episodes have influenced the numbers that shaped the fit, so the model has seen a summary of data it is about to be scored on. That is leakage, and it makes the reported error a little better than the truth. You would probably never notice: the effect is small, it moves the number in the flattering direction, and nothing errors. The only defence is to compute statistics on the training split as a matter of habit.

3. Behaviour cloning scores 2.108 and a policy that never moves scores 3.020. Give a reason that gap could be smaller than it looks.

Because both are frame-level averages, and being close on every individual frame is nearly free at 30 frames per second. The arm moves so little between frames that “repeat the current pose” is almost right every time, which is why it scores 3.020 at all. Neither number says anything about whether an episode ends with the brick in the box. Only rollouts measure that, and this dataset cannot be rolled out.

4. On wrist flex the clone is worse than doing nothing. Is the fit broken?

No. The loss is a sum over all six joints, so the optimiser trades accuracy between them, and wrist flex barely moves in this task - “stay put” is already a strong prediction there. Giving up a little on the cheapest joint to gain more on the shoulder lowers the total, which is exactly what it was asked to do. The lesson is about reporting rather than fitting: a single averaged number hid a joint where your policy is beaten by a constant.

5. Going from 1,478 parameters to 269,318 improves held-out error from 2.417 to 2.008. What does that tell you about where to spend your next hour?

Not on the architecture. A 180-fold increase in capacity buying 17 percent means the model is not what is limiting this fit; the data is, and so is the fact that you are measuring the wrong thing. The next hour is better spent on more or better-covering demonstrations, on a different observation space, or on getting a way to measure task success at all. This is the general shape of imitation learning, and it is why the field talks about data engines more than it talks about layers.

6. What single property of the dataset made the shuffle in the hook harmless, and where does that property stop holding?

That every row is a complete, self-contained training example: observation on one side, the action the human took on the other, with nothing else needed to make sense of it. It stops holding the moment the policy runs, because then the observation in one row is a consequence of the action in the previous one. Training never has to care about that ordering. Deployment is nothing but that ordering.

Do this

code/bc_train.py has four TODO(you) markers: the standardiser, the network, the training step, and the error in real units. code/demos.py from lesson 3.1 does the reading.

This is the first exercise in the course that needs a neural-network library, so install one now. It is the last new dependency this module asks for, and every training lesson after this one uses it.

pip install torch              # once, for this and every training lesson after it
python bc_train.py             # train, score, per-joint breakdown
python bc_train.py --capacity  # does a bigger network help?

The first takes a handful of seconds on a laptop CPU once the dataset is cached. When the four TODOs are right you will reproduce this lesson’s numbers exactly: 2.108 overall, 3.019 on shoulder pan, and wrist flex losing to the do-nothing baseline.

Then three things worth doing yourself:

Break the normalisation on purpose. Delete the standardisation and train on raw units. Predict what happens to the gripper column before you look, then look. This is the fastest way to make the scaling argument stop being advice and start being something you have seen.
Find where it overfits. Run 5, 10, 30, 60 and 120 epochs and plot both the training loss and the held-out error. The crossing point moves with the network width, so find it for two widths rather than one and note that nothing in the training loss would have told you.
Predict, then measure, on your own data. Run it against the demonstrations you recorded in Module 2 with --path. Your Module 2 expert is a script, so it is perfectly consistent, where the human on the SO-101 dataset is not. Write down whether you expect a lower error before you run it. Then hold on to the answer, because whether a flawless demonstrator is easier or harder to clone is exactly the question the next lesson takes apart.

What you can now do

You can train a neural network policy on real demonstration data, standardise it correctly, split it honestly, and beat every untrained baseline on held-out frames. You can read a per-joint breakdown and spot the joint where your average was hiding a loss, and you can say why more parameters and more epochs are not the next thing to try. Most importantly you can state what the number you produced does and does not claim: it is a statement about frames, made by a policy that has never once acted.