Training on real data

Where you are. You have a real dataset recorded by your own hand, verified in the visualiser and replayed on the arm. This lesson turns it into a policy. The pipeline is the one you built in Module 3; what changes is that the data is yours, and so are the mistakes in it.

A number going down in one room

Your training run is on a rented GPU. There is a curve on your screen and it is going down, smoothly, the way a loss curve is supposed to. It has been going down for two hours.

In the other room, on the desk, sits an arm that has never executed a single action from this model.

Nothing on your screen can tell you whether that arm will pick up the brick. Not the loss, not the gradient norm, not the learning rate. You have a number that measures how well the model reproduces actions you already recorded, and the question you actually care about is what happens when the model produces an action nobody recorded, in a state your demonstrations never reached.

Everything in this lesson is about keeping those two things separate in your head.

The idea in one paragraph

Training a policy on your own data is a short arithmetic problem followed by a long wait. Your dataset is a number of frames; that number divided by the batch size is one epoch; the documented recipe is five to ten epochs, not the hundreds of thousands of raw steps that get quoted out of context. One configuration default will hurt you - checkpoints are only useful if you save enough of them to evaluate more than the last one - and one widely-repeated fix for a second will hurt you differently, by aborting the run, because the flag it names does not exist on the policy you are training. And the loss the run reports is a measure of how well the model imitates frames it has seen, which is related to, but not the same as, whether the arm completes the task. The only ground truth in this module is a success rate measured on the robot.

Your dataset is a number of frames

Start here, because every other decision hangs off it.

LeRobot’s own worked example: 50 episodes at 30 frames per second lasting 30 seconds each is $45{,}000$ frames; at a batch size of 8 that is about $5{,}625$ steps per epoch, so five epochs is roughly $28{,}000$ steps and ten epochs is about $56{,}000$.

How long to train

The documented guidance is “typical total: 5-10 epochs”, with these starting bands per policy: ACT at batch 8 to 16 for 30,000 to 80,000 steps, and “usually converges under 50k for single-task”; Diffusion Policy 80,000 to 150,000; SmolVLA 30,000 to 80,000 at batch 4 to 8.

Those two rules only meet at the top of the range for a fifty-episode dataset. Five epochs of 45,000 frames at batch 8 lands at 28,000 steps, just under ACT’s suggested floor; ten epochs lands at 56,000, comfortably inside it. That is arithmetic on two published numbers rather than a measurement, but the practical reading is clear enough: on a first small dataset, take the upper end of the epoch range.

Which policy you can train at all is mostly a memory question. LeRobot publishes a profiling run, one update step each, with a caveat you should carry with the numbers:

The decision rule that follows from the table: under 8 GB, which includes most laptops, a 3060 and an M-series Mac, train act. At 12 to 16 GB you can add smolvla, or run act and diffusion with a bigger batch. Above 24 GB anything fits, and act is still often the best return for a single-task grasp-and-place. On CPU alone, do not train locally at all.

Wall-clock time is the question everyone asks and the one this course cannot answer for you, because there is no GPU here and no run to time. What exists is a shape to check yours against. One practitioner report - a single source, unnamed here because the course has no link to give you, and not re-verified - puts an ACT model of about 52 million parameters at roughly four hours on a 12 GB RTX 3080. The only thing that makes it worth repeating is that it is broadly consistent with LeRobot’s own measured 84 ms per update once you add dataloading and a larger batch, and that consistency is the part you can check. Treat it as an order of magnitude, not a promise. If your run is heading for forty hours, something is wrong with the batch size or the device; if it finishes in ten minutes, check that it really did the steps you asked for.

Fifty-two million is the right size to compare your own run against, and it is not the 80 million you will see quoted for ACT elsewhere, including in this course. Both figures are correct: the paper counts a seven-layer decoder, LeRobot’s default builds one layer, and that difference is worth about 32 million parameters. Lesson 3.7 opens with why.

The scheduler advice that does not apply to you, and the one default that does

The scheduler is sized for someone else’s run - if your policy has one at all. LeRobot’s agent guide tells you that the default schedulers use scheduler_decay_steps=30_000, sized for long runs, and that a short run therefore never anneals; the fix it gives is --policy.scheduler_decay_steps=<your steps>.

Two things about that advice, both checked against LeRobot 0.6.1’s source rather than its prose.

So the practical rule for this module is short: on ACT, do not pass the flag. On a VLA later in the course, pass it if you like, and read the log line either way.

Checkpoints are your only way to compare. Set --save_freq to something that gives you five or six checkpoints across the run rather than one at the end, because the checkpoint with the lowest training loss is not reliably the one that performs best on the arm, and you cannot discover that after the fact if you only kept the last one.

The command

lerobot-train \
    --dataset.repo_id=${HF_USER}/so101_brick_v1 \
    --policy.type=act \
    --output_dir=outputs/train/act_brick_v1 \
    --job_name=act_brick_v1 \
    --policy.device=cuda \
    --batch_size=8 \
    --steps=56000 \
    --save_freq=10000 \
    --wandb.enable=true \
    --policy.repo_id=${HF_USER}/act_brick_v1

--policy.device takes cuda, mps or cpu. To resume an interrupted run, pass --config_path=<checkpoint path or Hub repository id> --resume=true.

One thing does carry over unchanged from Module 3: the evaluation harness you wrote there. The protocol is the same, the twenty-trial floor is the same, and the discipline about reporting variance is the same. What changes is that a trial now costs a physical reset instead of a simulator call, which makes the temptation to run five trials and call it a result much stronger. Resist it in exactly the same way. One naming trap while you are in the command list: lerobot-eval still exists, and it is for simulated environments rather than a physical arm. That is structural rather than a matter of convention - EvalPipelineConfig declares env as a required field, so the command cannot be run without naming a simulator. The physical arm gets its own command in the next lesson.

Without hardware

This lesson transfers almost intact, because training never touched the robot.

• Instead of your own repository id, point the run at the pinned public dataset or at your simulated one. lerobot/svla_so101_pickplace reports six-dimensional state and action at 30 fps, so every configuration constraint in this lesson applies unchanged.
• Measure this: the same frames-batches-epochs arithmetic, on 11,939 frames rather than on a count you produced. Then do it again for your simulated dataset and note how differently the two size out.
• What you lose: nothing in this lesson, and one thing after it. The loss curve is equally untrustworthy on both paths; what differs is that the hardware reader finds out on an arm and you find out in a rollout whose physics you also wrote.

Check yourself

Do this

Twenty minutes of setting up, then a wait.

1. Compute your own numbers. Use the average length of your kept episodes, not the value you passed to --dataset.episode_time_s:

python code/train_plan.py --episodes 60 --seconds 25 --batch 8 --epochs 10

It prints frames, steps per epoch, total steps, a checkpoint frequency, and whether your run lands inside the documented band for the policy you chose. It emits the training command, and includes --policy.scheduler_decay_steps only for the policies whose config actually declares it, which ACT does not.

2. Sanity-check the numbers by hand once. Multiply, divide, and confirm the script’s arithmetic agrees with yours. You want the relationship in your fingers, because you will be re-running it every time the dataset grows.

3. Launch, with --wandb.enable=true. Watch the first hundred steps to confirm the loss is falling and memory is stable, then stop watching. The curve has nothing else to tell you.

4. Write down what you cannot see. Before the run finishes, note in your task specification file: the exact LeRobot version, the policy type, the batch size, the step count, and which dataset revision. When the twenty-trial number comes back, that block is the difference between an experiment and an anecdote.

5. Keep the checkpoints. You will need at least three of them for the next lesson.

What you can now do

You can size a training run from your own dataset instead of copying a step count out of a blog post, convert freely between frames, steps and epochs, and pick a policy your hardware can actually hold. You can set the two flags that stop a short run from being wasted, and you can look at a falling loss curve and correctly conclude that you have learned almost nothing yet about whether the robot works.