Where you are. You know what pretraining buys and what it does not. This lesson is where that capability came from, told as four inventions rather than as a story about models getting bigger.
Pick up the extinct animal
Put a few objects on a table in front of a robot arm. A soda can, a banana, a small plastic dinosaur. Now type an instruction that names none of them: pick up the extinct animal.
The arm picks up the dinosaur.
Stop and ask where that came from, because it did not come from the robot. No teleoperator ever recorded a demonstration labelled “extinct animal”. The word “extinct” does not appear anywhere in any robot dataset, and neither does the fact that dinosaurs are extinct or that a small plastic one counts as a dinosaur. Every part of that instruction was understood somewhere other than in the robot data, and then somehow it reached the motors.
Demonstrations of roughly this shape are what made RT-2 famous in mid-2023, and the mechanism behind them is a single design decision that a software engineer will find almost anticlimactic. Getting to it took two years and three other inventions.
The idea in one paragraph
The path from a task-specific robot network to something you can download and give instructions to is not one long scaling curve. It is four separable ideas, arriving between late 2022 and mid-2024, each of which you can remove and then name exactly what breaks. One network for many tasks, with actions written as discrete tokens. Co-training on web data, so that the semantics the model learned from the internet survive contact with robot data instead of being fine-tuned away. Pooling data across different robot bodies, which turned out to be a data-format problem rather than a modelling one. And releasing the weights, which is what turned the field from four labs into everybody. Everything since 2024 is refinement of the action head and the training recipe on top of those four.
Wider than the screen; scroll it sideways.
One network, many tasks
Before this, a robot learning setup meant one model per task. RT-1, from Google in December 2022, put a single transformer behind many tasks at once: one set of weights, trained by behaviour cloning on a large in-house dataset of teleoperated demonstrations collected on a fleet of mobile manipulators.
The design decision that carried forward is how it emitted actions. Each dimension of the action - each joint delta, the gripper - was discretized into bins, and each bin was treated as a token in a vocabulary. Predicting an action became predicting a sequence of tokens, exactly like predicting a sentence.
Put the actions in the same vocabulary as the words
RT-2, from Google DeepMind in July 2023, was built on two existing vision-language backbones: PaLM-E at 12 billion parameters and PaLI-X at 55 billion. Both were already trained on web images and text before any of this started.
The trick is one sentence long, and it is worth memorising in exactly this form: robot actions are written as text tokens and mixed into the model’s own training set, so it learns to emit “1 128 91 241 5 101 127 217” the same way it learns to emit words. At inference, that string of numbers is de-tokenized back into a change in end-effector pose plus a gripper command.
That choice is what carries the semantics through, and the mechanism is worth being precise about, because the usual summary gets it wrong. It is not that the model reads the internet and then a separate robot module acts. It is that the same next-token predictor that learned from web images and text is trained on a mixture that includes robot trajectories written as text. Because web data stays in the mixture, the web knowledge is never fine-tuned away. That is why “extinct animal” survives the trip from a caption on the internet to a wrist rotation.
RT-2 reported roughly twice the generalisation of RT-1 and comparable baselines, and about three times the performance on evaluations designed to probe capabilities that were never demonstrated. Those are the authors’ own numbers, published on the project site in July 2023.
Wider than the screen; scroll it sideways.
Pool everybody’s robots
By late 2023 the bottleneck was obvious. Web data was abundant; robot data was not, and every lab’s robot data was locked to that lab’s robot.
Open X-Embodiment, published in October 2023 by 21 institutions, was an act of plumbing rather than modelling. Data from 22 different robot bodies was converted into one standardised format and pooled: 527 skills across 160,266 tasks.
| What was pooled | Count |
|---|---|
| Contributing institutions | 21 |
| Distinct robot embodiments | 22 |
| Skills | 527 |
| Tasks | 160,266 |
The empirical claim was that existing methods, with barely any modification, get better when trained on the pool. RT-1-X reported roughly 50% higher success than the original state-of-the-art method each collaborating lab had contributed, evaluated on that lab’s own robot. RT-2-X reported about three times the generalisation of a model trained only on the embodiment it was evaluated on.
The genuinely hard part was the format. One robot reports joint angles in degrees across six axes at 30 Hz; another reports seven axes in radians at 15 Hz; a third commands end-effector velocity. Pooling them means choosing a shared action space that all of them can be mapped into, and every such choice throws something away. You will build the smallest version of that map yourself in the exercise, because feeling where the information goes is worth more than reading that it does.
Give the weights away
OpenVLA arrived in June 2024 from a collaboration across Stanford, UC Berkeley, the Toyota Research Institute, Google DeepMind, MIT and Physical Intelligence. Seven billion parameters. A Llama-2 language backbone. A vision encoder that fuses features from two pretrained vision models, DINOv2 and SigLIP. Trained on 970,000 demonstrations drawn from the Open X-Embodiment pool.
Wider than the screen; scroll it sideways.
The reported results were good: comparable to RT-2-X on Google robot evaluations and substantially better on BridgeData V2, at roughly one eighth the size, 7 billion parameters against 55 billion. But the score is not the contribution.
The contribution is that the weights, the training code, the data recipe and a documented low-rank fine-tuning path were all published together, and that the model runs in bfloat16 on a 16 GB GPU. Before June 2024, working on VLAs required being inside a frontier lab. After it, it required a graphics card.
What each step bought
Remove any one of the four and something specific stops working. This table is the lesson compressed, and it is what you should be able to reproduce from memory.
| Step | Year | The one thing it added | Remove it and |
|---|---|---|---|
| RT-1 | 2022 | One transformer, many tasks, actions as discrete tokens | you are back to one model per task, and there is no format in which a language model could emit an action |
| RT-2 | 2023 | Robot actions written into the model’s own vocabulary, co-trained with web data | the model can act but cannot generalise semantically; web knowledge gets fine-tuned away |
| Open X-Embodiment | 2023 | 22 robot bodies pooled into one standardised dataset | every model is trained on one lab’s robot, and there is not enough robot data anywhere to matter |
| OpenVLA | 2024 | Open weights, code, data recipe and a fine-tuning path | the work stays inside four labs and none of this module is possible on your hardware |
Everything after mid-2024 - and that is most of what this module covers - refines two things on top of that base: how the action is represented and emitted, which is the next lesson, and how the training recipe mixes web data, robot data and corrections, which is the lesson on pi0.5.
Check yourself
1. A robot has never seen a demonstration mentioning “extinct”. Trace the path by which the word reaches a wrist rotation.
The word is understood by the vision-language backbone, which learned it from web text and images long before any robot data arrived. That understanding lives in the same network that emits actions, because actions were written as tokens in the same vocabulary and trained with the same next-token objective. So the instruction conditions the same hidden state that produces the action tokens, and those tokens are then de-tokenized into a pose change. There is no separate robot module; there is one model with a broader training mixture.
2. Why is co-training on web data different from fine-tuning a vision-language model on robot data, and why does the difference matter?
Fine-tuning trains only on the new data, so the representation drifts toward the narrow distribution it is now being asked to fit and the web knowledge degrades. Co-training keeps web data in the mixture throughout, so the semantic capability is continuously reinforced while the action capability is acquired. It matters because the web knowledge is the entire reason to start from a pretrained model. Fine-tune it away and you have paid for a large model and kept none of the benefit.
3. Open X-Embodiment is often described as a dataset rather than a research contribution. What is the actual technical difficulty?
Choosing a shared representation that 22 incompatible robots can all be mapped into. They differ in number of joints, units, control rate, camera placement, and even in what an action means: joint position for one robot, end-effector velocity for another. Any shared action space is a lossy projection of all of them, and the same numeric output means a different physical motion on each machine. The modelling was nearly unchanged; the format was the work.
4. RT-1-X reported about 50% higher success than each contributing lab’s own best method. Why is that number weaker evidence than it sounds?
Because it is an average of improvements measured against many different baselines of varying quality, not against one consistent reference. A large improvement over a weak baseline and a small improvement over a strong one average into a figure that describes neither. The direction - cross-embodiment data helps - is what survives; treat the magnitude as unquantified.
5. OpenVLA scored comparably to a model roughly eight times its size. Why is that not the reason it mattered?
Because scores get beaten. Within two years a 450-million-parameter model was averaging 87.3% on LIBERO against OpenVLA’s 76.5%. What did not get undone is that OpenVLA published the weights, the training code, the data recipe and a low-rank fine-tuning path, and fitted into 16 GB. That changed who was allowed to work in the field, and no later result reverses it.
6. You have a 6-joint arm reporting degrees and a 7-joint arm reporting radians. A pooled model emits 0.5 on its first dimension. What has the shared action space cost you?
The number is meaningless without knowing which robot is plugged in: on a joint spanning -90 to +90 degrees it is +45 degrees, and on one spanning -2.8 to +2.8 radians it is +1.4 radians, which is about 80 degrees. The pooled space also has to be wide enough for the widest arm, so narrower robots carry padded dimensions that the model still spends capacity predicting. Both are real losses, and both were considered worth paying for the data.
Do this
About thirty minutes.
1. Build the smallest possible cross-embodiment map. This is a toy version of the Open X-Embodiment problem, not the real format, and that is the point: the real one is more involved and loses information in the same places.
import numpy as np
# Two robots that will never agree on anything.
ROBOTS = {
"arm-A": {"dof": 6, "unit": "deg", "hz": 30,
"lo": np.array([-90, -60, -60, -90, -90, -180.0]),
"hi": np.array([90, 60, 60, 90, 90, 180.0])},
"arm-B": {"dof": 7, "unit": "rad", "hz": 15,
"lo": np.full(7, -2.8), "hi": np.full(7, 2.8)},
}
SHARED_DOF = 8 # wide enough for the widest arm in the pool, plus a gripper slot
def to_shared(action, spec):
"""One robot's native action, mapped into the pooled space."""
scaled = 2 * (action - spec["lo"]) / (spec["hi"] - spec["lo"]) - 1
padded = np.zeros(SHARED_DOF)
padded[: spec["dof"]] = scaled
return padded
def from_shared(vec, spec):
"""And back out, on whichever robot is actually plugged in."""
return (vec[: spec["dof"]] + 1) / 2 * (spec["hi"] - spec["lo"]) + spec["lo"]
emitted = np.full(SHARED_DOF, 0.5) # one vector out of one model
for name, spec in ROBOTS.items():
native = from_shared(emitted, spec)
print(f"{name}: joint 0 -> {native[0]:7.2f} {spec['unit']} "
f"dims used {spec['dof']}/{SHARED_DOF} "
f"control period {1000 / spec['hz']:.0f} ms")
for name, spec in ROBOTS.items(): # the map must at least round-trip
a = np.random.default_rng(0).uniform(spec["lo"], spec["hi"])
err = np.abs(from_shared(to_shared(a, spec), spec) - a).max()
print(f"{name}: round-trip max error {err:.2e} {spec['unit']}")The first block prints 45 degrees for one arm and 1.4 radians, about 80 degrees, for the other, from the same emitted number. Then extend it: add a third robot that commands end-effector velocity rather than joint position, and try to fit it into the same eight dimensions. Write one paragraph on what you had to give up. That paragraph is the Open X-Embodiment contribution, felt rather than read.
2. Draw OpenVLA from memory. Close this page and sketch four blocks with arrows: what goes in, what fuses with what, what the backbone is, and what comes out. Then check it against the figure above. The module’s exit test asks you to do this for three harder architectures, and this is the one they are all variations of.
3. Practise the triage question once. Find any VLA release announcement from the last three months and answer two things about it in writing: who ran the evaluation, and what the baseline was and who trained it. If you cannot answer either from the announcement itself, that is your finding.
What you can now do
You can name the four inventions that produced a downloadable, instructable robot policy, and say what specifically breaks if you remove any one of them. You can explain why co-training preserves web semantics where fine-tuning destroys them, why pooling robot data is a format problem rather than a modelling one, and why the number attached to a cross-lab improvement is weaker evidence than its direction. And you can sketch OpenVLA’s four blocks without looking.