Where you are. You have a real SO-101, a dataset you collected yourself, and enough architecture to diagram three frontier models. This lesson turns those episodes into fine-tuning input, and freezes them so the experiment two lessons from now is worth publishing.
Open one frame
Load one episode from your Module 4 recording and stop on a single frame.
You know most of what is in there. Two camera images, one from above and one on the wrist. Six joint angles read off the follower arm’s encoders. Six commanded targets, which is what the leader arm was doing at that instant and therefore what a policy is supposed to reproduce. A timestamp, an episode index, a frame index.
There is one more field. It holds a short piece of text, and unless you were deliberate while recording it probably holds nothing at all, or a single word you typed to get past a prompt.
ACT never opened that field. Every model in this module opens it first.
The idea in one paragraph
Preparing data for a foundation model is not a conversion job. Your frames are already the right shape: these models consume the same images, the same joint state and the same action targets your imitation-learning policy consumed. What they add is one sentence per episode saying what the robot was doing, plus a set of mechanical contracts - frame rate, camera names, units, normalisation statistics - that must hold identically at training time and at deployment time, or the arm will move smoothly and confidently to the wrong place. Then comes the part that costs nothing and gets skipped most often. You freeze the dataset. The moment you re-record one episode between two training runs, the comparison you were building stops being a comparison and becomes an anecdote.
Wider than the screen; scroll it sideways.
The column ACT ignored
ACT has no input for words. It maps pixels and joint angles to an action chunk, and everything it knows about the task is baked into the weights of one checkpoint trained for that one task. Ask it to do something else and there is no place to put the request.
A vision-language-action model has that place. The instruction string goes through the language half of the backbone, becomes tokens sitting in the same attention stack as the image tokens, and conditions every action the model emits.
This buys you two different things and it is worth keeping them apart.
The first is free and applies even if you never change the sentence. The backbone was pretrained on web images and text, so cube and bin are not arbitrary symbols to it; they arrive attached to a great deal of prior knowledge about what cubes and bins look like. That is the whole bet of this module, and you get it whether or not your dataset has more than one instruction in it.
The second only exists if you build it. If every episode in your dataset carries the same sentence, the model has no evidence that the sentence matters, and at evaluation time you cannot test whether it is listening. To test language you need at least two instructions that select different behaviour in the same scene. A red cube and a blue cube on the table, and two instructions naming one each, is the smallest version of that experiment and it is the one worth running.
The mechanical contract
The rest of dataset preparation is bookkeeping, and every item on the list has bitten somebody.
| What | Why it matters | The failure if you get it wrong |
|---|---|---|
| Frame rate | Chunk length is counted in frames, so the same chunk covers half as much time at 60 fps as at 30 | The policy predicts motion that is twice as fast or half as fast as the demonstrations |
| Camera keys | The policy config maps dataset keys such as observation.images.top onto model inputs by name | A silently missing view, and a policy trained on one camera when you thought it had two |
| Camera count | Every extra view costs tokens and memory in the backbone | A run that fits on your GPU with two cameras and will not start with three |
| State and action dimension | Taken from your dataset config; the base checkpoint was trained with its own | Shape errors at load time if they disagree, which is the good case because it is loud |
| Units and sign | Degrees or radians, absolute targets or deltas, whichever Module 4 recorded | Motion in the right shape at the wrong scale |
| Normalization statistics | Computed from your dataset, stored with the checkpoint, applied in reverse at deployment | Silence, then wrong motion |
The last row is the one that deserves a picture, because it is the only entry on that table that fails without any error message at all.
Wider than the screen; scroll it sideways.
How much, and of what kind
The SmolVLA authors’ real-world results come from 50 demonstrations per task, collected as 10 trajectories from each of 5 starting positions. The independent SO-101 benchmark from Hiroshima used 100 per task. Fifty is the number to design around: it is roughly a 30 to 60 minute teleoperation session for a simple pick-and-place, and it is enough to say something.
What matters more than the count is what varies across those 50. The model will generalise along the axes your data varies and nowhere else. If the cube starts in the same place every time, you have 50 copies of one demonstration and a policy that has learned an open-loop trajectory with extra steps.
Freeze it
Now the discipline that makes the next two lessons worth doing.
Wider than the screen; scroll it sideways.
Locking it means three concrete things. Push the dataset to the Hub and note the exact revision, so “the dataset” names a specific commit rather than a moving branch. Take a hash of the local copy so you can prove nothing shifted underneath you. Write down, in prose, what you pruned and why, because “I removed the bad ones” is a decision that will affect both models and a reader is entitled to know what you considered bad.
Prune before you freeze, prune once, and apply the same rule to every episode. Deleting the demonstrations where your teleoperation was clumsy is legitimate data hygiene. Deleting the ones where the policy later failed is not; you would be tuning the training set to the test.
The test set is not a slice of your dataframe
One habit from ordinary machine learning does not survive contact with a robot. There is no held-out split here that means anything, because the quantity you care about is not per-frame prediction error - it is whether the arm completes the task, which can only be measured by running the arm.
So the held-out thing is physical. Before you train anything, write a numbered list of 20 initial conditions: where the cube sits, how it is oriented, what else is on the table, where the lighting is coming from. Never demonstrate those exact setups. That file is your test set, and every policy you evaluate for the rest of this module gets the same 20 rows in the same order.
Writing it now, before you have a model, is what stops it from drifting toward the conditions your model happens to like.
Without hardware
Everything in this lesson is a property of files, so it transfers whole. The freeze is actually easier to defend on this path.
- Instead of your Module 4 recordings, pin a public revision of
lerobot/svla_so101_pickplace, which carries the instruction pink lego brick into the transparent box on all 50 episodes, plus the simulated dataset from your Module 4 milestone. A revision you do not control cannot be quietly re-recorded, which is the failure the freeze exists to prevent. - Instead of a two-instruction recording session, get the second sentence from a second
dataset.
lerobot/svla_so100_stackingcarries Put the red cube on top of the blue cube. on 56 episodes at the same 30 fps and the same six-dimensional state. For a controlled language test, build the two-coloured-cube scene in MuJoCo instead, where you can write both instructions against one scene. - Measure this: the same audit of one frame - keys, shapes, dtypes, ranges - on the
downloaded set. The camera keys are
observation.images.upandobservation.images.side, which is not what your Module 4 config expects, and finding that yourself is the exercise. - The test set becomes 20 numbered simulator initial conditions from a seed never used in training, plus a held-out episode split of the real dataset for the offline half.
Check yourself
1. Your dataset has 50 episodes and every one carries the instruction “pick up the cube”. What have you gained from the language input, and what have you not?
You have gained the pretrained backbone’s prior: the words pick, up and cube arrive attached to what the model learned from web images and text, so the visual features it brings to your table are not starting from zero. You have not gained any tested language conditioning, because there is no evidence in your data that changing the sentence should change the behaviour, and no way to evaluate whether the model is reading it at all. To test that, you need at least two instructions that select different actions in the same scene.
2. You prune eight clumsy episodes after computing normalisation statistics, then train. What happens, and how would you notice?
The model normalises inputs and de-normalises outputs with numbers describing a 50-episode dataset it was never shown. Nothing raises an error, because the statistics are just numbers of the right shape. You notice it as systematically offset motion: the arm moves in roughly the right shape but consistently short, long or biased in one joint. It looks exactly like a calibration or camera-extrinsics bug, which is why it costs an evening. Recompute statistics as the last step before training and never in between.
3. Why is it not legitimate to add ten more demonstrations for your VLA fine-tune after seeing that your ACT baseline did better?
Because the experiment you set out to run holds data constant and varies the policy. Adding data to one arm of the comparison changes two things at once, so whatever number comes out cannot be attributed to either. It is also the specific direction of bias that flatters the result you were hoping for, which is why the freeze exists as a rule rather than a preference. If you genuinely want more data, re-collect for both, re-freeze, and rerun both training runs.
4. Your Module 4 recording ran at 30 fps. A colleague’s identical rig recorded at 60. You fine-tune with the same chunk length. What differs?
The chunk covers half as much wall-clock time on the 60 fps data, so at the same chunk length the policy is committing to half as much future motion per prediction. Everything downstream shifts: how far ahead the model is effectively planning, how often it re-observes the world, and how a given latency budget maps onto control. Frame rate is not a storage setting; it is the unit that chunk length is denominated in.
5. What is in your test set, where does it live, and why must it be written before training?
It is a numbered list of 20 physical initial conditions - object position and orientation, distractors, lighting - stored as a text file in the repository, and never demonstrated. It has to be written first because a test set written afterwards drifts toward the setups your model handles well, in a way that feels like fairness at the time. Fixing it in advance is what lets you say the 20 trials measured something you did not choose after seeing the answers.
Do this
About ninety minutes, most of it teleoperation.
1. Fix the instruction. Decide the exact sentence for your task and write it into every episode’s task field. If you are running the language test, decide two sentences and which episodes get which. Write both strings into notes/05-task-strings.md verbatim, because these are the strings your evaluation must use.
2. Audit one frame. Load a single frame and print the keys, the shapes, the dtypes and the value ranges of the state and action arrays. Confirm the camera keys are the ones your policy config expects, confirm the state and action dimensions are 6, and confirm the units match what you believe you recorded. Five minutes here saves the whole failure mode above.
3. Write the test set. notes/05-initial-conditions.md, 20 numbered rows, each one specific enough that someone else could set up your table from it. No condition may appear in a demonstration.
4. Freeze. Push to the Hub, note the revision, then fingerprint the local copy:
"""Write a one-line fingerprint of a dataset directory. Standard library only."""
import hashlib
import sys
from pathlib import Path
root = Path(sys.argv[1]).expanduser()
digest = hashlib.sha256()
files = sorted(path for path in root.rglob("*") if path.is_file())
for path in files:
digest.update(str(path.relative_to(root)).encode())
digest.update(hashlib.sha256(path.read_bytes()).digest())
print(f"{len(files)} files")
print(f"sha256 {digest.hexdigest()}")Run it, paste the two lines into notes/05-dataset-freeze.md alongside the Hub revision and the date, and run it once more after both training runs are finished. If the hash changed, one of your numbers is not what you think it is.
What you can now do
You can take a dataset you recorded for imitation learning and make it fine-tuning input for a vision-language-action model: name the task in words, keep that naming consistent, and check the frame rate, camera keys, dimensions and units that the policy config depends on. You can explain why normalisation statistics are a contract shared between training and deployment and why breaking it fails silently. And you can freeze a dataset - Hub revision, content hash, written pruning rules - so that the comparison you run in lesson 12 measures the policy and nothing else.