Where you are. Your dataset is frozen, instructions and all. This is the flagship experiment of the course: taking a model pretrained on other people’s robots and pointing it at your table.
What else shipped on Tuesday
A caching layer went out on a Tuesday. By Friday the dashboard shows page loads down by a third, and the write-up says the cache did it.
Ask the one question that settles it. What else shipped on Tuesday? Two config changes and a dependency bump, probably; nobody wrote it down. The improvement is real and it is now permanently unattributable, because there is no version of that week without the cache in it and there never will be.
Tonight you do the same thing to yourself. You start the run before dinner, and when you come back the loss has fallen by more than an order of magnitude: one of those beautiful clean decays that makes you feel briefly competent about machine learning. You copy the checkpoint onto the robot and it moves, smoothly and confidently, and picks the cube up or does not.
Either way, the sentence you will want to write is the pretraining did that. You cannot write it from one run. The only week in which the missing half of the comparison can still be built - the same fifty episodes, the same lighting, the same arm, random weights instead of pretrained ones - is this one, while all of it is still on the desk.
The command is one line. The experiment is two runs and a handful of checkpoints, and none of what you want to know is decided by the curve.
The idea in one paragraph
Fine-tuning a small vision-language-action model on your own data is genuinely cheap: 450 million parameters, 50 demonstrations, a few hours on a graphics card you probably own, and the model arrives already knowing what arms and cubes and grippers are. What it does not know is your table, your lighting, your gripper’s particular way of missing. The fine-tune is what teaches it those. The hard part is not making the training run work; it is that the only signal the training loop produces is a loss that falls smoothly whether or not the resulting policy can do the task, so the run has to be built to emit several checkpoints and the real measurement happens on the robot. Alongside it you run one extra training run that differs in exactly one thing - starting from random weights instead of the pretrained ones - because that difference is the only honest answer to “what did the pretraining actually buy me?”
What a fine-tune actually changes
Wider than the screen; scroll it sideways.
SmolVLA is 450M parameters total, of which roughly 100M is the action expert. The backbone is SmolVLM2-500M: a SigLIP vision encoder feeding a SmolLM2 language decoder. Three deliberate economies keep it small enough to be yours - visual tokens are cut to 64 per frame, only about half the backbone’s layers are used, and the action expert runs at 75% of the backbone’s hidden size.
The action expert is the part that emits motion, using flow matching, which lesson 3 covers properly. For today one sentence is enough: it starts from noise and integrates a learned velocity field forward a few steps to arrive at a chunk of real-valued actions.
Both the backbone and the action expert load from the base checkpoint. The state and action dimensions come from your dataset config, and on an SO-101 with six joints they generally match what the base was trained with, which is a large part of why this specific fine-tune goes so well: the pretraining corpus was mostly SO-100 arms, so the model has seen hardware very much like yours.
The command, and the four knobs that matter
Training is one invocation of LeRobot’s trainer, pointed at the base checkpoint and your frozen dataset.
lerobot-train \
--policy.path=lerobot/smolvla_base \
--dataset.repo_id=<your-username>/<your-dataset> \
--batch_size=8 \
--steps=20000 \
--save_freq=2500 \
--seed=1000 \
--output_dir=outputs/smolvla-yourtask-run-a| Knob | What it does | How to set it |
|---|---|---|
batch_size | The dominant lever on memory | Start at the largest that fits, drop it if you hit an out-of-memory error. The LeRobot docs demonstrate batch sizes as small as 4 |
steps | How long you train | 20,000 is the figure the SmolVLA docs use as a reference point. Not a law; a starting bet |
save_freq | How often a checkpoint lands on disk | Often enough that you have three or four to evaluate. This is the setting this lesson exists to argue for |
| gradient checkpointing | Trades compute for memory by recomputing activations rather than storing them | Turn it on when batch size alone will not fit; expect a slower run, not a worse model |
Half precision (bf16) is the other standard memory lever and is usually on by default in these recipes. Check rather than assume.
What it costs, as of August 2026
The practical boundary is a 24GB consumer card. Below it you fine-tune small models and run inference on mid-sized ones. Above it, at 40 to 80GB, the 3B-class models in lesson 14 become fine-tunable. Nobody outside a funded lab pretrains anything.
The loss is not the metric
Here is why the beautiful curve lied.
Wider than the screen; scroll it sideways.
That figure is a shape, not a measurement - your own numbers go in it. But the shape is the point, and it comes from three separate facts.
The loss is an average over the dataset, so it is dominated by the easy frames. Most frames in a pick-and-place demonstration are the arm travelling through free space, which is trivially predictable. The frames that decide success - the last centimetre of the approach, the moment of contact, the grasp - are a small minority, and their contribution can shrink while the average falls.
The loss is measured on demonstrations, so it never sees the states the policy actually visits. This is covariate shift, the same problem behaviour cloning had in Module 3, and it does not go away because the model got bigger. A tiny error puts the arm somewhere no demonstration ever went, and the loss on the demonstrations has nothing to say about what happens next.
The loss is per-frame, so it cannot see compounding. Task success is a long conjunction of things that all have to work; per-frame error is an average. Those two quantities are related, loosely, and not monotonically.
The practical protocol is small. Save every 2,500 steps. Take three checkpoints spread across the run - somewhere near a quarter, half and all of it - and run each one on the robot for a modest number of trials, perhaps 10, purely to rank them. Then take the winner into the full evaluation in lesson 12 with the real trial count. Ranking and measuring are different jobs and it is fine to be cheap about the first one.
The run that answers “what did pretraining buy?”
The exit test for this module asks you to say, quantitatively and from your own numbers, what the pretraining gave you. There is exactly one way to know.
Wider than the screen; scroll it sideways.
The authors measured this for themselves and reported real SO-100 success going from 51.7% without community-dataset pretraining to 78.3% with it, a gap of 26.6 points on 50 demonstrations per task. That is their number on their tasks, and it is one of the cleanest published pieces of evidence that internet-scale-ish pretraining reaches all the way down to a $300 arm. Yours will differ. Yours is the one you can defend.
Write down enough to repeat it
Both runs and every checkpoint need a record: the dataset revision and hash from lesson 10, the exact command, the seed, the git commit of your own repository, the LeRobot version, the GPU, and the wall-clock. Three of those will look like bureaucracy until the moment somebody asks why your two runs disagree.
Measure your own wall-clock and put it in the record. The four-hour figure above is somebody else’s documentation, and the honest version of this course is the one where the numbers in your writeup came off your own machine.
Without hardware
The fine-tune itself never needed a robot. What needs one is deciding which checkpoint won.
- Instead of your own episodes, fine-tune from
lerobot/smolvla_baseon the pinned public dataset. Its shape is what SmolVLA expects already: six-dimensional state and action, two cameras, 30 fps, one instruction per episode. - Instead of ten trials per checkpoint on the arm, rank checkpoints twice. Closed loop in the Module 2 scene on your simulated task, and offline on held-out real episodes by action-prediction error over a full chunk.
- Measure this: both rankings, and whether they agree. Where they disagree is the most interesting thing available on this path, and it is a measurement the hardware reader usually cannot afford to make.
- Run B is unchanged. The no-pretrained-weights ablation is the cleanest result on either path, because it holds the data, the recipe and the seed fixed and varies one thing.
Check yourself
1. Your training loss falls smoothly for 20,000 steps. What have you learned about the policy?
Almost nothing about whether it can do the task. The loss is an average over demonstration frames, so it is dominated by easy free-space motion rather than the contact-critical frames that decide success; it is measured only on states the demonstrations visited, so it says nothing about the states the policy drifts into once it makes a small error; and it is per-frame, so it cannot see the compounding of a long conjunction of steps. A smoothly falling loss tells you the optimizer is working. Nothing more.
2. Why save checkpoints every 2,500 steps rather than keeping only the final one?
Because checkpoint selection is a real decision and the training loop cannot make it for you. On-robot success is not monotonic in training steps, so the last checkpoint is not reliably the best one, and you have no cheap proxy for finding the best. Several checkpoints on disk turns an unanswerable question into a small, affordable experiment: rank three of them with a handful of trials each, then measure the winner properly.
3. You have 22GB of VRAM and the run dies with an out-of-memory error. Name two levers, and say which costs you accuracy.
Reduce the batch size, and turn on gradient checkpointing. Neither one costs accuracy directly. Gradient checkpointing recomputes activations instead of storing them, so you pay in wall-clock and get the same model. A smaller batch changes the optimization dynamics rather than the capacity, and may want a lower learning rate or more steps to reach the same place, but it is not a quality ceiling. The lever that genuinely costs you is training a smaller model or on less data, and neither is on this list.
4. Describe the only experiment that answers “what did pretraining buy me?” and name what must be identical between its two arms.
Two training runs differing in exactly one thing: starting weights. One from lerobot/smolvla_base, one from random initialisation. Identical between them: the dataset down to the frozen hash, the number of steps, the batch size, the seed, the checkpoint-selection procedure, and the evaluation protocol including the same list of initial conditions and the same trial count. The gap in on-robot success is the answer. If anything else differs, the gap is attributable to two causes and you cannot separate them.
5. Fifty episodes against a pretraining corpus of 10.6 million frames. Why is the fine-tune worth anything at all?
Because the two runs are teaching different things. Pretraining taught the model what arms, grippers, cubes and instructions are, using other people’s robots. Your 50 episodes do not repeat any of that; they specify which of the things it already understands you want, on your table, in your lighting, with your gripper’s particular geometry. The small dataset is doing targeting, not teaching, and targeting is a much smaller job than the parameter count suggests.
6. Why should you distrust the “four hours on an A100” figure even though it comes from the model’s own documentation?
Because it is a reference point from a setup that is not yours: different batch size, different dataset size and frame rate, different data-loading throughput, possibly a different release of the training code. It is useful for deciding whether to rent by the hour or the day, and useless as a claim in your writeup. The version of this number worth publishing is the one your own run printed, on your own hardware, which is why the wall-clock goes in the record.
Do this
One evening plus one overnight run.
1. Run A, the real one. Fine-tune from lerobot/smolvla_base on your frozen dataset, with checkpoints saved often enough to give you at least four. Record the command, seed, versions, GPU and wall-clock in notes/05-run-a.md as you go, not afterwards.
2. Rank three checkpoints. Pick roughly a quarter, half and all of the run. Ten trials each from the top of your initial-conditions list, scored strictly. Write the three numbers down. If the last checkpoint is not the best, you have just learned the lesson first-hand and should say so in your writeup.
3. Run B, the ablation. Same recipe, same data, same seed, no pretrained weights. Rank its checkpoints the same way. Expect it to be worse; note by how much, and note whether it is worse in a different way rather than only by degree.
4. Watch a failure properly. Take your best checkpoint, record one failed trial on video, and step through it. Was the reach wrong, or was the reach right and the grasp slipped, or did it grasp and then loop? Those are three different problems and only one of them is about the model. You will build this into a taxonomy in the next lesson.
Keep both checkpoints. Lesson 12 evaluates them against your ACT baseline, and lesson 15 puts the winner on hardware that has to hold a control loop.
What you can now do
You can fine-tune a 450M vision-language-action model on 50 episodes you collected yourself, on a consumer GPU, and say what each knob in the command trades away. You can explain why the training loss falls smoothly whether or not the policy works, and name the three reasons - the average is dominated by easy frames, it never sees off-distribution states, and it cannot see compounding. You can select a checkpoint by the only signal that counts, and you can run the one-variable ablation that turns “pretraining helps” into a number measured on your own arm.