What a policy actually is

A policy is one function from what the robot can see to what it should command, called thirty times a second with no memory in between.

Where you are. You can audit a demonstration dataset and say what is in it. This lesson is about the thing that consumes it, before any of it is trained, so that “the robot learned it” stops being a sentence about magic.

The dumbest thing that could possibly work

Take one recorded demonstration. Two hundred and three rows, each row six numbers the arm reported about itself and six numbers the human sent it. Put them in a dictionary, keyed by the six it reported.

table = {tuple(observed): commanded for observed, commanded in episode}

Now stand the robot up, read its joints, and look the reading up.

KeyError.

So put forty demonstrations in the table instead - 9,180 rows - and try it against ten demonstrations the table has never seen. Two thousand seven hundred and fifty-nine attempts. It finds a key 45 times. More than 98 percent of the time your robot brain has nothing whatsoever to say, because 86.10779 is not 86.10778 and floating-point sensor readings do not repeat.

That failure is worth more than a working demo, because every clever thing in the rest of this module is a fix for it.

The idea in one paragraph

A policy is a function. One argument in, one value out: what the robot can currently observe, and what it should command right now. That is the whole object. It has no memory between calls, no plan it is executing and no idea what it is doing; it gets called thirty times a second and answers each call from nothing. What separates a famous robot policy from your dictionary is only how it fills the gaps between the observations you recorded. A dictionary fills them not at all. Nearest neighbour fills them crudely, and has to carry every row you ever recorded to do it. A neural network compresses the table into a fixed number of weights that answer in fixed time, for observations nobody ever recorded. “The robot learned it” means a function was fitted to a table. It does not mean anything more than that.

A policy drawn as a function: observation in, action out, thirty times a second

Wider than the screen; scroll it sideways.

Writing it down

The notation is worth ten seconds because it appears in every paper you will read.

\pi_\theta : \mathcal{O} \rightarrow \mathcal{A}

In words: the policy $\pi$ maps the space of possible observations $\mathcal{O}$ to the space of possible actions $\mathcal{A}$ . The subscript $\theta$ is the parameters - the weights - and it is there to say that the policy is one specific member of a whole family of functions, picked out by numbers you are going to fit. Change $\theta$ and you have a different policy with the same shape.

Some policies return a distribution over actions rather than a single action, written

\pi_\theta(a \mid o)

and read as “the probability of taking action $a$ given observation $o$ ”. You would want that whenever more than one action is genuinely correct, which turns out to be most interesting tasks and is the whole subject of the multimodality lesson. For now, take the simple version: one observation in, one action out.

What actually goes in, and what comes out

For the SO-101 pick-and-place dataset, the observation is three things: observation.state, the six joint angles from the encoders; two 480 by 640 camera frames; and the task sentence, “pink lego brick into the transparent box”. The action is six numbers, and here is the part worth pausing on - they are target joint angles, the exact same six numbers the human’s leader arm was sending. Not torques, not velocities, not “move 3 cm left”.

That is not an accident of this dataset. It falls out of the servo you opened in the actuator lesson: a position-controlled servo accepts an angle and nothing else, so an angle is the only thing a policy for this arm can emit.

Action space	What one call means	Where it hurts
Absolute joint targets	“be at these six angles”	Ties the policy to one arm’s calibration, and a single bad prediction is a full-speed lunge to a wrong pose.
Joint deltas	“move each joint by this much”	Errors accumulate: the policy never states where it wants to be, so drift has nothing to correct against.
End-effector pose	“put the gripper here, facing this way”	Needs inverse kinematics in the loop, and inherits every singularity from the IK lesson.

The observation is not the state

Here is the distinction that most robot failures eventually reduce to.

What the world contains against what the observation carries

Wider than the screen; scroll it sideways.

The state of the world includes how hard the gripper is squeezing, whether the brick is genuinely held or merely touching, what the friction is today, and what the operator was intending. The observation is six joint angles and two images. A policy is a function of the observation. It has never once been a function of the state.

This is the exteroception problem from the first lesson of the course, arriving in its final form. Proprioception is cheap and honest, six numbers accurate to a fraction of a degree. Everything else has to be inferred from pixels, and everything hard lives there.

Policies you can build without training

You can build a working policy right now, with numpy and no gradients. There are only three ways to fill the gaps between recorded observations, and two of them need no fitting at all.

Three ways to answer the same question: exact lookup, nearest key, and a fitted function

Wider than the screen; scroll it sideways.

The dictionary is the one from the hook: 45 hits in 2,759 tries.

The nearest-neighbour policy is one line different and always answers:

\pi_{\text{NN}}(o) = a_{j^{*}}, \qquad j^{*} = \arg\min_{i}\;\lVert o - o_{i} \rVert

which says: find the recorded observation closest to the one you were handed, and repeat whatever the human did there. To score it, measure how far its command sits from the command the human actually gave, averaged over every held-out frame and every joint:

\mathrm{MAE} = \frac{1}{N D} \sum_{i=1}^{N} \sum_{d=1}^{D} \left\lvert \hat{a}_{i,d} - a_{i,d} \right\rvert

One unit is roughly one percent of a joint’s travel. Two more policies go in the table purely to keep you honest: one that always commands the average pose, and one that commands the pose it is already in. Measured on ten held-out episodes, with forty in the table:

policy	held-out action error
exact dictionary, falling back to the mean when it misses	23.61
always command the average pose	23.89
nearest neighbour	4.88
nearest neighbour, distances scaled by each joint’s spread	4.75
command the pose you are already in	3.02

Read that last row again. The best number on the table belongs to a policy that ignores the dataset entirely and commands the joint angles it is already at - a robot that sits perfectly still, forever.

More table, better answers, up to a point

Nearest neighbour has one honest virtue: it gets better as you feed it more demonstrations, and you can watch it happen.

Held-out action error of the nearest-neighbour policy against the number of demonstrations in its table

One demonstration gives an error of about 12.7, and the spread across which demonstration you happened to pick is enormous. Five demonstrations halves it. Then the curve flattens: going from five to forty, an eightfold increase in data and in memory, buys about 1.3 units. That is the shape of a method that memorises rather than generalises, and it is the argument for the next lesson in one picture.

The budget nobody mentions

A policy is not just a function; it is a function with a deadline. At 30 frames per second the loop is 33 milliseconds long, and the policy has to answer inside it, every time, forever.

The nearest-neighbour policy takes about 0.16 milliseconds per call against 9,180 rows, measured on a four-core laptop from 2018; the script prints the figure it measured on yours, and under load it drifts upward, so treat it as an order of magnitude rather than a benchmark. Comfortable - until you notice the cost is linear in the size of your dataset. Ten thousand episodes instead of forty is 250 times the rows, which is about forty milliseconds, which is a control cycle missed on every call. A fitted network is the opposite: its cost is set by the weights, not by the data, so it answers in the same time whether it learned from fifty episodes or fifty thousand.

What a policy is not

Three things it does not have, all of which people assume it does.

It has no memory. Each call is answered from the observation alone. If the task genuinely needs history - “have I already tried this grasp?” - the history has to be handed in as part of the observation, or the policy has to commit to a whole sequence at once. That second option is action chunking, and it is the single most important architectural idea in this module.

It has no plan. Nothing inside it represents “first approach, then grasp, then lift”. The apparent plan is an emergent property of running a stateless function in a world that keeps changing underneath it, which is also why the whole structure collapses so completely when the robot ends up somewhere the demonstrations never went. That is the covariate-shift lesson, two ahead.

And it has no notion of success. The function was fitted to match a human’s commands, not to achieve an outcome. Nothing in it knows the brick is supposed to end up in the box. Wanting an outcome instead of an imitation is what reinforcement learning is for, and it costs you a great deal to get it.

Check yourself

1. Your dictionary policy finds a key for 1.6 percent of held-out frames but 100 percent of the frames already in its table. What is that gap, in one word, and why is it the central problem of this module?

Generalisation. The table is a perfect record of what happened and a near-useless guide to what to do next, because the situations the robot meets are never bit-identical to the ones you recorded. Every technique in this module - fitting a function, chunking actions, generating action distributions - is a way of answering for observations that are not in the table.

2. Why can a policy for the SO-101 only emit target joint angles, and what would you need to change to emit forces instead?

Because the servo accepts an angle and nothing else; its position loop closes inside the sealed package, and your code never sees the current. To command force you would need a different actuator - a low gear ratio so motor current honestly reflects output torque, a joint torque sensor, or a series elastic element - and then a different action space, and then a new dataset, because the demonstrations you recorded are logs of angle commands and contain no force information at all.

3. A policy that commands the pose it is already in scores the lowest action error of any policy tested here. Why is that not a contradiction, and what does it tell you about validation loss?

Because at 30 frames per second the arm barely moves between frames, so “stay where you are” is nearly correct at every single frame while being completely wrong across an episode. Per-frame error rewards being locally close and is blind to whether the trajectory goes anywhere. Validation loss is a useful training signal and a worthless success claim; only rollouts measure the task.

4. Two world states produce identical observations and require different actions. Name the two ways to fix it, and the one thing that will definitely not fix it.

Change the observation - add a sensor or camera view that distinguishes the two states - or give the policy access to time by feeding it a short history of frames, so “what was happening a moment ago” becomes observable. What will not fix it is more capacity or more data: the function is being asked to return two different values for the same input, and no function does that. A policy that outputs a distribution can at least represent the ambiguity honestly, which is a different thing from resolving it.

5. Nearest neighbour answers in well under a millisecond against 9,180 rows, comfortably inside a 33 ms budget. Why is a neural network still the right answer?

Because the cost is linear in the dataset and the dataset is the thing you intend to grow. Two orders of magnitude more demonstrations turns 0.16 ms into 16 ms, half the budget gone on the lookup alone, and another order of magnitude puts it past the deadline entirely. A fitted network’s inference cost is set by its weights, so it is constant in the amount of data it learned from - and unlike nearest neighbour it interpolates smoothly instead of snapping to whichever single recorded frame happened to be closest.

6. The scaling curve flattens hard after about five demonstrations. Does that mean more demonstrations are a waste?

No - it means more demonstrations of the same thing are close to a waste, for this particular memorising policy. The curve measures held-out frames drawn from the same narrow setup, so extra episodes mostly add near-duplicates. Demonstrations that add new situations, a brick somewhere it has never been, a different starting pose, move a different curve entirely: the one measured by task success under conditions you have not tested yet.

Do this

Build all four policies. code/lookup_policy.py has three TODO(you) markers: the dictionary lookup, the nearest-neighbour search, and the evaluation loop. code/demos.py from the last lesson does the reading.

python lookup_policy.py            # the four policies, on 40 train / 10 held-out episodes
python lookup_policy.py --sweep    # error against table size, 3 random subsets each

The first takes a few seconds, the sweep about half a minute, both on a laptop CPU. When the three TODOs are right you will reproduce every number in this lesson: 45 exact hits out of 2,759, nearest neighbour at 4.88, the do-nothing baseline at 3.02.

Then push on it:

Beat the do-nothing baseline. You are allowed anything except training. The obvious move is to average the actions of the $k$ nearest recorded observations instead of taking one; try $k = 5$ and $k = 20$ . Note what happens to the error and then ask yourself whether a smoother command is actually a better one, or just a closer one.
Break the split on purpose. Change split so that train and test are sampled from shuffled frames rather than whole episodes, and rerun. The error will collapse. It should: neighbouring frames are almost identical, so a frame-level split puts the answer in the table. This is the most common way to publish a robotics number that means nothing, and it is worth seeing your own harness do it once.
Run it against your Module 2 recording with --path. Predict the dictionary’s hit rate before you run it, and write the prediction down: your expert is deterministic, but the cube starts somewhere different in every episode, so it is not obvious which way this goes. Then check. The deeper question - whether copying a flawless scripted expert is easier or harder to deploy than copying a wobbly human - is what the covariate-shift lesson is about.

What you can now do

You can say precisely what a policy is: one function, from an observation space you chose to an action space your hardware forced on you, called on a fixed deadline with no memory and no plan. You can write it as $\pi_\theta$ and say what the subscript means. You can build three of them without training anything, measure them against each other on held-out episodes, and explain why the best-scoring one is a robot that never moves. And you can name the two properties a fitted function buys you over a lookup table - constant inference cost, and a smooth answer for observations nobody ever recorded - which is the entire motivation for the next lesson.