Two right answers, averaged into a wrong one

A squared-error loss asks the network for the average of everything the demonstrators did, and when they did two different valid things, the average is a third thing that fails.

Where you are. You can train a chunked behaviour-cloning policy and you know why chunking slows down drift. This lesson is the other way plain regression breaks, and it is the one that more data makes worse.

Somebody left a trolley in the aisle

You are in a supermarket. Halfway down the dairy aisle, someone has abandoned a trolley, dead centre.

Watch two shoppers. The first drifts left, passes it, carries on to the milk. The second drifts right, passes it, carries on to the milk. Both were completely correct. Neither hesitated.

Now build a rule from what you saw. You have two recordings: trolley three metres ahead, went left and trolley three metres ahead, went right. Average them. The rule says: walk straight ahead.

That rule walks into the trolley. It is the one thing neither shopper did, and the only thing that fails.

Collect a thousand shoppers instead of two. Five hundred go left, five hundred go right. The average is still straight ahead - and now you are very confident about it.

Two demonstrations reach the same goal by going round opposite sides of an obstacle, while the average of their actions is a straight line into it

Wider than the screen; scroll it sideways.

The idea in one paragraph

When you train a policy with a squared-error loss you are not asking it to imitate a demonstrator. You are asking it, at every observation, for the average of everything any demonstrator ever did there. If the demonstrators agreed, the average is what they did and everything is fine. If they genuinely disagreed - two valid routes, two valid grasps, two valid orderings - the average is a point between the things they did, and there is no reason at all for that point to be a valid action. It usually is not. This is not noise, it is not a bug in your training loop, and it does not go away with more demonstrations. It is a property of the loss function, and it is the single clearest reason the field moved away from plain regression.

What a squared-error loss actually asks for

Write down what training minimises. For an observation $o$ where the demonstrated actions follow some distribution $p(a \mid o)$ , the loss is

\mathcal{L}(f) = \mathbb{E}_{a \sim p(a \mid o)}\left[\lVert a - f(o) \rVert^2\right]

Expand the square, differentiate with respect to $f(o)$ , set the derivative to zero, and one line falls out:

f^{*}(o) = \mathbb{E}\left[a \mid o\right]

The best possible network under this loss - infinite capacity, infinite data, perfect optimisation - outputs the conditional mean of the demonstrated actions. That is not an approximation or a failure mode. It is the exact thing the objective was asking for.

Why the average lands in the gap

Picture the distribution of sideways motion at the trolley. One peak at “half a metre left”, one at “half a metre right”, nothing in between, because nobody drove into the trolley. The mean of that distribution is zero.

Zero is not a rare action or an unlikely one. It has zero probability. The conditional mean has landed in a region the data explicitly says is empty, and the network has done its job perfectly to get there.

Switching to an absolute-error loss does not rescue you. Minimising $\lVert a - f(o) \rVert_1$ gives you the conditional median rather than the mean, which is more robust to a few wild outliers - a real benefit, and why several policies use it. For a distribution with two symmetric peaks the median sits between them too.

Where the second mode comes from

Multimodality is not exotic. It appears whenever the thing that decided the action is not in the observation.

Source	What it looks like in your data
Symmetric solutions	Elbow up or elbow down for the same hand position; left or right round an obstacle
Free ordering	Two blocks to stack and no reason to prefer one first
Grasp choice	Mug by the handle, or by the rim
Operator style	Fast and loose on Tuesday, careful on Friday; two people sharing a teleoperation rig
Timing	When to start closing the gripper, given the same approach
Unrecorded intent	The demonstrator had already decided where to go. The camera never saw the decision

That last row generalises the rest. The demonstrator’s action was a function of the observation and something else - a plan, a habit, a coin flip - and you only wrote down the observation.

Prove it on the arm you already have

You do not need a robot for this. The 2-link arm from Module 1 has a two-to-one inverse: for almost every reachable point, there is an elbow-up joint configuration and an elbow-down one, and both put the gripper in exactly the same place.

So train the same small network three times. Once forwards, joint angles to position, which has one right answer per input. Once backwards, position to joint angles, which has two. And once backwards on data filtered to a single elbow branch, which has one again. Same architecture, same optimiser, same 2,000 gradient steps.

Two exact inverse-kinematics solutions for one target and the arm pose you get by averaging their joint angles, which lands 0.62 metres away

Real example

Measured on a 2018 four-core laptop CPU, held out on 4,000 unseen targets. “Miss” is where the gripper actually lands when you drive the arm to the predicted angles.

What the network learns	Held-out error	Median miss	Worst 10%
angles to position	0.0004	0.024 m	0.040 m
position to angles, both elbows	0.2336	0.325 m	0.776 m
position to angles, one branch only	0.0001	0.009 m	0.026 m

The arm is 1.7 m long, so the two-branch version misses by about a fifth of the robot’s entire reach. Deleting one branch drops the error by a factor of two thousand.

The third row is the control, and it is the part that makes this an experiment rather than an anecdote. Nothing changed except that the second right answer was removed from the data. Same capacity, same optimiser, same budget. So “the network is too small” and “you did not train long enough” are both ruled out, without argument.

Training loss for the same network on the same map, with and without a second elbow branch in the data: one converges, the other flattens

Take one target and do the arithmetic by hand. For $(0.9, 0.6)$ the two exact solutions are $\theta = (-0.094, +1.801)$ and $\theta = (+1.270, -1.801)$ radians. Both reach the target to the last decimal place. Their average is $(+0.588, 0.000)$ , an arm stretched perfectly straight, whose gripper sits 0.618 m away from where you asked. Average two correct answers, get a third answer that is not correct.

Spotting it in your own data

Three checks, cheapest first.

Look for near-duplicate observations. Find pairs of frames whose observations are close and whose actions are far apart. If that set is large, you have modes.
Plot the actions in a bin. Take all frames whose observation falls in a narrow band and histogram one action dimension. One lump is fine; two lumps with a gap is the problem.
Run the branch-deletion control. Filter the data to one convention - one side, one grasp, one ordering - and retrain. If the loss suddenly converges, the modes were the cause.

The residual tells you too. A well-behaved regression leaves residuals scattered around zero. A policy averaging two modes leaves residuals in two clumps either side of zero, and the mean of the residual is near zero while nothing is near zero.

Three ways out

Remove the ambiguity from the data. Record the hidden variable and feed it in: a language instruction, a goal image, a target side. Or impose a convention when collecting - always elbow up, always left, always the red block first. This is the cheapest fix and it is badly underused. If your task genuinely has a preferred answer, say so in the dataset rather than asking the model to invent one.

Let a latent variable absorb it. Give the network a small extra input $z$ that is inferred during training from the action the demonstrator actually took. The decoder is then predicting “the action, given the observation and which style this was”, which is single-valued again. That is the conditional variational autoencoder inside ACT, and it is why ACT is not naive regression.

Learn to sample instead of to summarise. Stop predicting a single action and learn the whole distribution $p(a \mid o)$ , then draw one sample from it per decision. Both peaks stay where they are, the empty middle stays empty, and the policy commits to one valid route instead of splitting the difference. That is Diffusion Policy, and it is the next lesson.

Check yourself

1. Under a squared-error loss, what exactly is the best possible network asked to output at an observation

o

The conditional mean $\mathbb{E}[a \mid o]$ of the demonstrated actions at that observation. This is not an artefact of limited capacity or a short training run. It is the exact minimiser of the loss, so a bigger network trained for longer approaches it faster.

2. Your success rate has stalled at 30%. You collect fifty more episodes of the same task from the same operator. What do you predict, and why?

No improvement, and possibly a small regression. Extra data reduces the error in estimating the conditional mean, but the conditional mean is the thing making you fail. You are converging more confidently onto an action nobody demonstrated. Before collecting more, bin the observations and look at the spread of actions inside a bin.

3. Why is the single-branch run in the arm experiment the load-bearing part, rather than the two-branch failure itself?

Because it is the control. A failure on its own is consistent with a network that is too small, a learning rate that is wrong, or a bug. The one-branch run uses the identical network, optimiser and step budget on the identical map, and converges to a held-out error two thousand times lower. The only thing that changed was the number of correct answers per input, so that is the cause.

4. Give one source of multimodality that would survive even if every demonstration were collected by the same person on the same day with a perfectly repeatable setup.

Genuine symmetry in the task. Elbow up and elbow down are both exact solutions for the same gripper position, and a bimanual task where either hand can pick up the object has two solutions no amount of operator discipline removes. The same is true of free ordering: with two blocks to stack and no stated preference, both orders are correct. Symmetry is a property of the task, not of the demonstrator.

5. Someone proposes switching from mean squared error to mean absolute error to fix this. What do they actually get, and does it help?

They get the conditional median instead of the conditional mean. That is genuinely more robust to a handful of badly-recorded outlier frames, which is a real reason policies use it. It is not a fix for multimodality: for two symmetric peaks the median lands between them, in the same empty region the mean did. Changing which summary statistic you regress does not stop you regressing a summary statistic.

6. In the supermarket, what single change to the observation would remove the multimodality entirely?

Record the shopper’s chosen side and include it in the observation - “trolley ahead, passing on the left”. Then each observation has exactly one demonstrated action again and plain regression works. This is the first of the three fixes, and on real tasks it takes the form of a language instruction, a goal image or an explicit target, added to the dataset when the demonstration is recorded rather than reconstructed afterwards.

Do this

Run the experiment, then break it on purpose. About fifteen minutes, all on CPU.

cd module-03-robot-learning/solutions
python mean_of_two_modes.py

The script prints the three-row table above, then the two exact solutions for one target and the pose you get by averaging them, then a histogram of the second joint angle for every training sample that lands near one target - two spikes, nothing between them. Each row is 2,000 gradient steps on a small network, about 6 seconds on an idle 2018 laptop CPU. On a busy machine expect several times that: the same three rows took 25 to 120 seconds each while other work was running.

Then work through the starter in code/mean_of_two_modes.py, which leaves three pieces for you:

Fill in the elbow filter. sample_arm(..., elbow=+1) should keep only configurations with a positive second joint angle. Confirm the held-out error collapses.
Find the modes without knowing they exist. Fill in mode_histogram: take a target position, find every training sample within 3 cm of it, and histogram the second joint angle. You should see two spikes and an empty middle. This is the diagnostic you run on real robot data, where nobody tells you the modes are there.
Sweep the branch mixture. Fill in mixture_sweep and run python mean_of_two_modes.py --sweep, which builds datasets that are 100%, 90%, 75% and 50% one branch and reports the miss distance for each. Predict the shape before you look: is a 10% minority branch a rounding error or a real cost? Whatever you find is the instinct to carry into real data collection, because one operator having an off day is exactly a small minority branch.

What you can now do

You can state what a squared-error loss is really optimising and derive it in one line. You can recognise a multimodal demonstration set from symmetry, ordering, style or unrecorded intent, and you can prove multimodality is the cause of a stalled policy with a branch-deletion control rather than guessing at capacity. And you can name the three fixes - put the hidden variable in the observation, absorb it in a latent, or learn to sample the distribution - which is exactly the fork the next two lessons take.