What pretraining actually buys a robot

Where you are. You have trained ACT on demonstrations you collected yourself, evaluated it with a harness that does not flatter it, and watched it work on a real SO-101. This lesson says what a pretrained model would add to that policy, and what it would not.

The blue one

There is a policy on your desk that works. Fifty demonstrations, a red block, a bowl, eight successes out of ten. You are pleased with it, and you should be.

Someone picks up a blue block and puts it where the red one was.

Sit with what happens next, because it is not a question about accuracy. Your policy takes camera pixels in and puts joint angles out. There is no input where the word “blue” would go. There is no input for “the one on the left”, or for “no, you dropped it, try again”. And there is no representation inside the network of the block as a thing that is separate from the particular pixels it happened to be made of.

It will still do something. What it does is whatever a function fitted to red-block pixels does when you hand it blue-block pixels, which is nobody’s idea of a plan.

The obvious fix is fifty demonstrations of the blue block. That works. It also does not scale, because there is always another colour, another lighting condition, another table. The other fix is to start from a model that knew what a block was before it ever saw your robot.

The idea in one paragraph

A policy trained only on your demonstrations has learned exactly one thing: what your joint angles did when your cameras saw pixels like these. It carries no notion that the object is a block, that “blue” names a colour, that a bowl is a container, or that the block sitting on the floor means something has gone wrong. A foundation model arrives already carrying all of that, because it was trained on internet-scale images and text long before anyone bolted a robot to it. Fine-tuning on your fifty demonstrations then teaches it only the last mile: how your arm moves. So what pretraining buys is not accuracy on the demonstrations you collected. A small task-specific policy is often just as good there, and far cheaper. It buys behaviour in the situations you never collected.

Three things a demonstration cannot contain

Your Module 4 dataset is a good dataset. It still cannot express these, in principle, no matter how many episodes you add.

A request. Behaviour cloning fits a mapping from observation to action. If every episode in the dataset performs the same task, the task is a constant, and constants do not appear as inputs. Your policy does not ignore language; it has no port for it.

An object you never showed. The network’s features are whatever separated your successful frames from each other. Nothing forced it to learn “block” as a category, so nothing guarantees it treats an unseen block as one.

The knowledge that things went wrong. This is the subtle one and the most valuable. A demonstration dataset collected by a competent teleoperator is a dataset of things going well. The policy therefore has no model of correct, so it cannot detect incorrect. It reproduces the distribution of motions it was shown, including in states no demonstration ever visited, where the right answer is usually “stop and re-approach” and the fitted answer is usually “keep going”.

That gap has a mechanism, and it is worth saying out loud: a model with a strong prior about how the world normally looks can tell that the current scene is not one of the normal ones. Detecting “this is wrong” is upstream of doing anything about it, and a policy fitted only to successful frames never learned what wrong looks like.

So what is a foundation model, concretely

Strip the marketing and there are three ingredients.

1. A backbone pretrained on web-scale images and text. This is an ordinary vision-language model of the kind you have used: images and words in, tokens out. It supplies the semantics.
2. Robot data on top. Demonstrations from many robots and many tasks, used to teach the model to emit actions instead of, or alongside, words.
3. One set of weights for many tasks. Not one checkpoint per task. That is what makes the word “foundation” mean anything: the same weights are the starting point for work nobody had in mind when they were trained.

The ledger, honestly

Here is the trade as measured, not as advertised. All figures below were checked on 9 August 2026 and every one of them comes from a real-robot evaluation, not simulation.

Look at what that figure does and does not say. Out of distribution, pi0.5 is still the best model on the rig, but it fell from 0.54 to 0.35. SmolVLA did not fall at all, and started far lower. ACT started respectably and lost more than half. “Generalizes better” and “is better” are two different rankings, and a benchmark that reports only the in-distribution column hides the more interesting one.

The scale is doing the work, not the acronym

This is the single most useful correction to make early, because the marketing runs the other way.

Keep this near the front of your mind for the rest of the module. Every architecture you are about to meet is a different answer to “how do I attach actions to a big pretrained model”, and the differences between them are real and worth understanding. But the first-order effect is how big and how well-trained that pretrained model was.

There is one more thing to hold: the two independent studies above disagree with the model authors. SmolVLA’s own paper reports 78.3% against ACT’s 48.3% on similar hardware with 50 demonstrations per task. The independent benchmark reports a tie. Different tasks, different operators, different amounts of effort spent tuning each baseline, and no way to reconcile them from the outside. When you find yourself weighing an author-reported number against an independent one, weigh the independent one. Lessons later in this module take that apart properly.

Check yourself

Do this

Twenty minutes, no GPU.

1. Write your own variation ladder. Take the task you recorded in Module 4 and write five rungs, each a single change from the last: same object same place, same object moved 10 cm, a different-coloured instance of the same object, a different object of the same class, and the task selected by a spoken instruction with a distractor present. This ladder is your out-of-distribution axis, and you will run it for real when you benchmark later in this module.

2. Register a prediction, before you have any data. For each rung, write down what you expect your existing ACT policy to score, and what you expect a fine-tuned SmolVLA to score. Put a date on it and do not look at it again until the benchmarking lesson. A prediction made after seeing the result is worth nothing, and this is the cheapest possible way to find out whether your intuitions about pretraining are calibrated.

3. Look at the shape of the published evidence. Run this against the numbers reported by VLA-REPLICA in May 2026, which are pasted in as data rather than computed, so it is honest about being a reading exercise and not a measurement.

# In-distribution and out-of-distribution average success, 10 tasks,
# 50 demonstrations per task, one SO-101 rig. VLA-REPLICA, May 2026.
results = {
    "ACT":     (0.18, 0.075),
    "DiT-D":   (0.16, 0.05),
    "DiT-F":   (0.12, 0.025),
    "SmolVLA": (0.26, 0.30),
    "X-VLA":   (0.14, 0.075),
    "pi0":     (0.34, 0.30),
    "pi0.5":   (0.54, 0.35),
}

print(f"{'model':9} {'in':>6} {'out':>6} {'kept':>7}")
for name, (in_dist, out_dist) in sorted(
    results.items(), key=lambda kv: kv[1][1] / kv[1][0], reverse=True
):
    print(f"{name:9} {in_dist:6.3f} {out_dist:6.3f} {out_dist / in_dist:6.0%}")

Then answer two questions in writing. Which model would you ship if the deployment environment were exactly your lab bench, and which if it were somebody else’s kitchen? And: the three worst rows on retention are the three methods with no pretrained backbone at all. Is that the explanation, or is it a coincidence of which tasks were chosen? You cannot settle it from this table, and noticing that you cannot is the skill.

What you can now do

You can state what a pretrained backbone adds to a robot policy - a language input that works, robustness to scene changes, and the ability to notice that something went wrong - and what it does not add, which is precision, grasp reliability, or cheap iteration. You can read an in-distribution and an out-of-distribution number as two different rankings. And you can say why a small VLA can tie a non-pretrained policy without that being a paradox.