Where you are. You can build a dataset, clone it, chunk it, generate actions instead of averaging them, and score a policy honestly. This is where those become one artifact you can hand to somebody else.
The question you cannot answer yet
Someone on your team is about to spend two weeks collecting demonstrations on a real arm. Before they start, they ask you one question: ACT or Diffusion Policy?
You have read both papers. You have trained both. You have a video of each doing the task, and two loss curves that went down. And there is nothing defensible you can say, because the two videos were recorded from different starting positions, the loss curves are in different units, and you never ran either policy more than a handful of times.
Notice what is missing. It is not more training. It is that “better” was never a property of the algorithms in the first place. It is a property of a task, a data budget, a trial count and a protocol, and you have not fixed any of the four.
The idea in one paragraph
This milestone is not “train two policies”. It is producing a comparison that survives someone else re-running it. Two tasks with deliberately different shapes, two policy families, two data budgets, three seeds each, one ablation per policy, every cell scored on the identical seeded scenes through the harness you built in the eval-harness lesson, and a writeup in which no rate appears without its denominator and its interval. Thirty training runs, fifteen hundred trials, four log files. The deliverable is the writeup, and the sentence in it you will be proudest of is the one that says which of your comparisons your trial count could not settle.
Wider than the screen; scroll it sideways.
Two tasks, chosen to disagree
Task A has one right answer everywhere. The groove world from the covariate-shift lesson: at any pose, exactly one direction takes the pin further along the slot. Task B has two. The corridor from the multimodal-demonstrations lesson: every demonstration goes round the obstacle, half of them left and half right, and the observation never says which.
That pairing is the point of running two tasks rather than one. The entire argument for a generative action head is that averaging two valid answers produces an invalid one, so a policy that only ever sees one valid answer per state should not benefit from it. If your ranking is the same on both tasks, one of the two results is telling you something is wrong with your setup.
Five rules, and why each one is a rule
| Rule | Why it is a rule |
|---|---|
| Matched budget: identical gradient steps, batch size and learning-rate schedule for both policies | An unmatched comparison measures how long you were willing to wait. If you shorten a run, shorten the schedule with it - LeRobot’s own guide warns that cutting --steps without cutting --policy.scheduler_decay_steps leaves the learning rate never decaying. |
| The same demonstrations inside a (task, budget, seed) triple | Otherwise the data budget stops being a controlled variable and becomes a lottery over which episodes each policy drew. |
| Every cell faces the identical seeded scenes, in a rotating order | Pairing removes scene difficulty from the comparison; interleaving spreads any drift across the session over every cell instead of the last one. Both come free from paired_schedule. |
| The protocol is frozen and fingerprinted before trial one | The success predicate is the softest joint in the whole apparatus, and the harness refuses to append trials from an edited protocol. That refusal is the feature. |
| Three seeds per arm, and the seed spread reported separately from the interval | Trial-to-trial variance shrinks with trials; seed-to-seed variance does not shrink with anything. Quoting one number from one seed describes a checkpoint while implying a method. |
Two paths to the same protocol
The starter is path-agnostic on purpose. You implement two functions, train_policy and run_trial, and everything else - the matrix, the pairing, the log, the summary, the done-gate - is already written and never learns which path you took.
The laptop path trains the module’s own chunked heads on the two sim tasks. Measured here on a 2018 4-core laptop CPU, on task B as it ships: a 213k-parameter ACT-style CVAE and a 141k-parameter diffusion head each train in 15 to 20 seconds at 2,500 steps, and fifty rollouts of either take under a second, because the corridor rollout is batched over episodes. Thirty cells is minutes, not hours. This path is the default, and it is genuinely enough to answer the question the project asks.
The rented-GPU path runs real LeRobot on real datasets: your Module 2 recording as task A, lerobot/pusht as task B. It is the path that transfers directly to Module 4, and it costs money.
Do not mix paths in one log. Whichever you pick, run one cell end to end first, time it, multiply by thirty, and decide before committing. That measurement is itself part of the writeup.
The ablations
One knob each, moved on task A at fifty demonstrations, with three seeds, sharing scenes with the baseline they modify.
ACT: the chunk size. The chunking lesson measured what predicting many actions from one observation buys and what it costs in reactivity. Move it in the direction you expect to hurt and see whether the harness can detect the difference at all.
Diffusion: the observation space. One observation step or two. This is the cheapest possible test of whether your task is actually Markov in the observation you chose, and the answer is often embarrassing.
An ablation is a paired comparison against its own baseline, which is why both arms live in one log with one protocol. Running the ablation in a separate file against separate scenes would throw away the only thing that makes a six-run experiment worth running.
The curve that matters more than the winner
Every cell is trained twice, at ten demonstrations and at fifty. That doubling is the part of this project you will still be using in two years.
Wider than the screen; scroll it sideways.
Before you run anything, work out what you will be able to claim. Fifty trials at an observed 80% gives a 95% Wilson interval of 67% to 89%, twenty-two points wide. Pool three seeds into a hundred and fifty trials and the same 80% narrows to 73% to 86%, about thirteen points. So a ten-point gap between arms is arguable at pooled scale and invisible per seed, and a five-point gap is not claimable at any trial count this project can afford. Write those two numbers down now; they decide what your writeup is allowed to say.
Done means done
Done is not a feeling. python act_vs_diffusion.py --check reads the logs and the writeup and prints a pass or fail line per criterion, exiting non-zero while anything is unmet:
- four log files, one per task and data budget, each with every cell the matrix asks for;
- at least fifty trials per cell, and every trial index run by every cell in its log, so the pairing is intact;
- one protocol fingerprint per file, unchanged from the first line to the last;
- every failed trial carrying a declared failure tag, not a null;
writeup.mdpresent, containing all seven required sections, at least one trial count and at least one interval.
The gate is deliberately mechanical and deliberately incomplete. It cannot tell whether your success predicate is honest, whether your budgets were really matched, or whether you wrote the writeup before or after you saw the numbers. Those stay yours.
What the writeup must report
Seven sections, and the awkward ones are the point.
| Section | What it must contain |
|---|---|
| What I ran | Path, hardware, wall-clock, library version, both protocol fingerprints, the factors you varied and the factors you froze |
| Results | Per-arm rate with and its Wilson interval, per-arm seed spread quoted separately, and the paired McNemar comparison |
| The data-scaling curve | The 10-against-50 table for all four task-and-policy pairs, and which of the three shapes you got |
| The ablations | What you moved, by how much, and whether the change survives its interval |
| What I could not separate | Every pair whose intervals overlap, named. This section is what makes the rest believable |
| Failure modes | The tag histogram per arm, and what each tag would take to fix |
| What I would do next | One paragraph, and it should follow from the failure histogram rather than from taste |
Check yourself
1. Why do the ablation arms share a log file with the baseline they modify, instead of getting their own?
Because an ablation is only meaningful as a paired comparison against the unmodified policy on identical scenes. Sharing a log means sharing one frozen protocol and one seeded scene sequence, so the difference you measure is the knob and not the draw. Separate files would mean separate scene draws, and you would be back to comparing two independent samples, which needs far more trials to say the same thing.
2. At fifty trials per cell you observe ACT at 84% and Diffusion at 78% on task B. What may you write?
That you cannot separate them. At the interval around 80% is roughly twenty-two points wide, so a six-point gap is inside the noise, and the paired McNemar test on the disagreements is what decides it rather than the two rates. Pooling three seeds to narrows the interval to about thirteen points, which still does not settle six. The honest sentence names both rates, the trial count, and the fact that the difference is not resolvable at this budget.
3. Your matrix’s four cells on task A all sit above 95%, and nothing distinguishes any arm from any other. Has the project failed?
No, it has produced a ceiling, which is a finding with a specific consequence: task A cannot discriminate between these policies, so every claim about the ranking has to come from task B or from a harder version of task A. The productive response is to make task A harder - widen the start distribution, narrow the groove, add disturbance - not to report a tie as if it were a measurement of similarity.
4. Why does the starter refuse to run at all until SUCCESS_RULE and FAILURE_TAGS are filled in?
Because a predicate written after you have watched a few rollouts is fitted to the policy you happen to have, and it is the cheapest way in the whole project to publish a number that means nothing. Freezing it first is a discipline that fails at eleven at night, so the file makes it a precondition instead. It is the same move the harness makes by hashing the protocol: the thing you must not omit becomes the thing the code will not let you omit.
5. You get the “flat” scaling shape on both tasks: fifty demonstrations score no better than ten. Give two very different explanations and one experiment that separates them.
Either coverage saturated - the extra forty episodes were near-duplicates of the first ten, so they added frames but no new situations - or the policy is not data-limited at all, and something else, capacity, budget or the observation space, is the binding constraint. Separate them by auditing coverage the way the data-engine lesson did and then adding ten deliberately different episodes rather than forty more of the same: if the curve moves on the diverse ten, it was coverage.
6. Why is “no solution file” the right call for this project specifically?
Because the deliverable is a measurement of your own setup, and a reference implementation would become the answer key rather than a check. Every number here depends on your success predicate, your factor ranges and your matched budget, so a shipped result would be a different experiment wearing your name. What can be shipped honestly is the bookkeeping and the gate, and both are in the starter.
Do this
The starter is project/act_vs_diffusion.py. There is no solution file, deliberately.
1. Declare the protocol. Fill in SUCCESS_RULE, FAILURE_TAGS and FACTORS for both tasks. Do this before you write a line of the adapters. python act_vs_diffusion.py --plan should then list thirty cells.
2. Write the two adapters. train_policy(cell) and run_trial(handle, cell, conditions, seed). Nothing else in the file should need editing; if it does, your interface is wrong, and it will be wrong again in Module 5 when a foundation model arrives.
3. Time one cell, then commit. Run a single cell end to end. Multiply by thirty. On the rented path, decide with that number in front of you.
4. Run the matrix. --train, then --eval. It resumes after a crash, so let it.
5. Read it before you write it. --report and --scaling. Look at the failure histograms first and the rates second. The histogram usually contains the actionable finding even when the rates cannot be separated.
6. Write the writeup, then run --check until it exits zero. --writeup emits the skeleton with every number already filled from the log, so the only thing you supply is the prose. Nothing in the numbers section should be typed by hand.
7. Publish it. A blog post, a gist, a README, wherever. The test is not whether it is impressive; it is whether a stranger with your two datasets could reproduce your table and get the same answer.
Two extensions, if you want them. Add a third seed budget - twenty-five demonstrations - and see whether the curve is straight between ten and fifty; it usually is not. And run the whole matrix again with the eval-scene distribution widened past the demonstrated one, which turns the project into a generalisation study and is the single most informative extra afternoon available here.
What you can now do
You have run a controlled comparison of the two workhorse imitation-learning algorithms: two tasks with different structure, two data budgets, three seeds, matched budgets, paired scenes and a frozen protocol. You can state which policy won where, quote the interval that says whether the gap is real, report the seed spread beside it, and name the comparisons your trial count could not settle. You have a data-scaling curve that tells you where the next two weeks of effort should go, and a writeup that a stranger could reproduce. That artifact, not the checkpoints, is what you take into Module 4 when the arm on your desk starts producing data of its own.