Where you are. You have fine-tuned a vision-language-action model, run it head to head against a task-specific policy on one task, and learned to read somebody else’s benchmark without being fooled by it. This is where you produce the benchmark and publish it. Two acceptance criteria are stated below, one per path from Module 4 lesson 0; the design, the protocol and the honesty requirements are shared.
Two tables that disagree
Open the SmolVLA paper to its real-robot table. On a low-cost arm, 50 demonstrations per task, the small vision-language-action model scores 78.3% and a task-specific policy trained from scratch scores 48.3%. Thirty points. Clear.
Now open the independent SO-101 benchmark from Hiroshima University. Same class of arm, more demonstrations per task, four tasks, four policies. SmolVLA 32.5%. The task-specific policy 33.75%. The thirty-point win is a statistical tie, with the sign facing the other way.
Nobody lied. Different tasks, different operators, different tuning effort, different everything. And no authority is going to resolve it for you, because the answer is not a property of the two architectures. It is a property of a task, a dataset, an arm and an operator.
You own a bench, a dataset and an operator. The bench is an SO-101 on a table or a scene file on a laptop, and either way it is yours and nobody else has run this comparison on it. Which means you are one of a fairly small number of people who can settle this question for your own bench, and the settling costs about three evenings.
The idea in one paragraph
Collect one dataset. Train two or three policies on it, each the way its own authors recommend, with no advantage handed to any of them. Evaluate every policy under one protocol, with the trial order randomised so your own fatigue does not correlate with which model is being tested, and with enough trials that the intervals mean something. Then publish the whole thing: the table, the intervals, a failure taxonomy, the cost of each training run in hours and dollars, and a closing section naming what you would not conclude. The deliverable is not a winner. The deliverable is a document somebody else could disagree with precisely, which is a rarer artefact in this field than any model.
The design
Wider than the screen; scroll it sideways.
Three things are held fixed and one is deliberately not.
Fixed: the data. One recording session per task, one dataset, one split. Every policy sees the same demonstrations. If you collect extra episodes because a model is struggling, you have ended the experiment.
Fixed: the evaluation. One scene specification, one reset procedure, one timeout, one success criterion written down before any policy is trained. The criterion goes in a text file, not in your head.
Fixed: the operator. You. Which is a problem, and the protocol section below is mostly about containing it.
Not fixed: the training recipe. Each policy trains the way its authors recommend, with their published hyperparameters. This is the one asymmetry you want: the question is “which approach works on my bench”, not “which architecture wins under a shared learning rate”.
The task ladder
Four tasks, ordered along the axis where the difference is expected to appear. This ordering is the experiment; a flat set of four similar tasks would produce four similar numbers and teach you nothing.
Wider than the screen; scroll it sideways.
| # | Task | What it adds | Why it is on the ladder |
|---|---|---|---|
| 1 | Pick the block, place it in the cup. Fixed start pose. | nothing | The task-specific policy’s home turf. If it does not win here, something is wrong with your training, not with the field. |
| 2 | Same, object position randomised across a marked 15 by 15 cm region. | spatial variation | Separates “memorised a trajectory” from “learned a servo behaviour”. |
| 3 | “Pick the red block” with a blue and a green block also on the table. | language grounding | The cliff. A policy with no language input cannot know which block is meant; the best it can do is guess. |
| 4 | Block into cup, then cup onto the coaster. | horizon | Two chained sub-goals, so a mid-sequence error has somewhere to compound to. |
For tasks 1 to 3, also define an out-of-distribution variant: an unseen object colour, a different table surface, the camera moved a few centimetres. Change exactly one thing per variant and write down which. Out-of-distribution is where pretraining shows up, and a comparison reporting only in-distribution numbers reports the boring half.
Write your prediction down before you run
Before the first training job, write a paragraph predicting the result and commit it with a timestamp. This is the only defence against what otherwise happens: you form the conclusion during the evaluation, then find support for it.
Here is the prediction the August 2026 evidence supports, which you can adopt or argue with:
- On tasks 1 and 2, ACT and SmolVLA come out within a few points of each other, and the interval will not separate them.
- On task 3, ACT is near zero and both VLAs are not. This is the single clearest expected effect in the whole experiment.
- On task 4, everything is poor. Long-horizon on cheap hardware is unsolved.
- Out of distribution, ACT loses roughly half its success rate and SmolVLA stays roughly flat.
- If you run a 3B-class model such as pi0.5, it beats both, by enough to survive the intervals.
- Recovery rate ranks the models more cleanly than success rate does.
- Nothing averages above about 60%. The best model on the independent SO-101 benchmark averaged 56.25%, and that is the state of the art on hardware like yours.
If your result contradicts the prediction, that is the most interesting sentence in your writeup, and it is also the moment to check the boring explanations first. Was one model trained for fewer steps? Did the camera move between two evaluation blocks? Did the arm’s calibration drift over three hours of trials?
The per-trial protocol
Success rates on real robots are produced by a tired person doing the same thing four hundred times, and that person is the largest uncontrolled variable in the experiment.
Wider than the screen; scroll it sideways.
Six rules, each of which exists because of a specific way the number gets corrupted.
- Randomise and interleave the trial order across policies. Never run all of one model’s trials in one block. If you do, the difference between models is confounded with the time of evening, the arm’s temperature, and how much you have learned about resetting the scene.
- Reset from a written specification, not from memory. Tape marks on the table, a photograph of the correct starting scene, an explicit tolerance. Scene drift over three hours is real and it is directional.
- One timeout, applied to everyone. Sixty seconds is a reasonable start. A policy that would have succeeded after two minutes fails; write the timeout in the report.
- Score against the written criterion, not against your impression. “Block fully inside the cup, cup upright, arm clear” is a criterion. “Looked good” is not.
- Record a failure mode from a fixed list, every time. The list you inherit from the literature works well: went to the wrong object or place, grasped and dropped or slipped, retried the same motion in a loop until the timeout, other.
- Log the row before starting the next trial. Batching the scoring at the end of a block means scoring from memory.
How many trials you can afford
Do this arithmetic before you start, not after you are tired.
Twenty trials per cell is the field standard, and the field standard is thin. For a rate near 50%, the 95% interval is
Report intervals with the Wilson method instead, because it stays sane at the edges where your table will actually live. Zero successes out of twenty gives a Wilson interval of 0 to about 16%, which is how to write down “ACT scored 0% on the language task”: not never, but below roughly one in six, at this budget. The starter file computes it and its check command verifies that number, so you can run it rather than believe it.
The budget, concretely, for three policies:
| Line | Count | Time |
|---|---|---|
| Demonstrations, 4 tasks at 50 each | 200 episodes | 2 to 4 hours of teleoperation |
| In-distribution trials, 3 policies x 4 tasks x 20 | 240 | about 6 hours |
| Out-of-distribution trials, 3 policies x 3 tasks x 20 | 180 | about 4.5 hours |
| Total real-robot trials | 420 | about 10.5 hours, across three evenings |
If that is too much, cut policies before trials. Two policies at 20 trials beats three at 10: at ten trials the interval is about 31 points and nothing separates from anything.
On the simulation path this arithmetic inverts. A trial is a few seconds of compute and a reset is a function call, so the budget is a wall-clock question about a machine rather than about you: 4,200 trials at five seconds each is under six hours, unattended, overnight. Which means the field standard of 20 is not a constraint you inherited, it is a number you would have to choose, and choosing it would leave every interesting comparison inside the interval. Run 200 per cell.
Cost is a column, not a footnote
The comparison people actually need includes what each result cost, and here the small model does unexpectedly well.
| Policy | Fine-tune hardware | Rough time | Rough cost |
|---|---|---|---|
| ACT, per task | any consumer GPU | tens of minutes | free on hardware you own |
| SmolVLA 450M | single consumer GPU; community reports on a 24 GB card | about 4 hours for 20k steps on an A100 | roughly $5 to $10 rented, or free overnight on your own card |
| pi0.5 low-rank fine-tune | over 22.5 GB, so a 24 GB card fits with essentially no headroom | slow on a consumer card | about $3.40 for a 10-hour rented 4090, or about $13 on an A100 and much faster |
| pi0.5 full fine-tune | over 70 GB, so an 80 GB accelerator | a day | $50 to $85 |
Two honesty notes. The wall-clock figures come from vendor documentation and community reports, not a controlled measurement, so measure your own and report yours instead; that measurement is part of the deliverable. Rental prices are from aggregator sites as of mid-2026, easily 30% off, and move monthly.
The line that matters for a reader: pretraining is not on this table at all. SmolVLA, the smallest serious model in the comparison, reports roughly 30,000 GPU-hours for the project that produced it. Nobody outside a funded lab does that, which is exactly why an open checkpoint you can fine-tune for the price of a sandwich is the interesting object.
What the writeup must contain
Seven sections. This is the artefact, and it is what the module has been building towards.
- Setup. Arm, cameras, lighting, mounting. A parts list with prices. Photographs of the rig and of a correct reset for each task.
- Data. Episodes per task, who teleoperated, over how many sessions, and the dataset published if you can publish it.
- Training. Per policy: checkpoint, steps, batch size, hardware, wall clock, and the recipe source. Explicitly: one policy instance per task for the task-specific baseline.
- Protocol. Trial count per cell, timeout, success criterion verbatim, reset procedure, randomisation seed, and the failure-mode list.
- Results. The table, with Wilson intervals on every cell. In-distribution and out-of-distribution side by side. Recovery rate. Dominant failure mode per policy. Cost.
- What surprised you. Including the prediction you wrote beforehand, quoted, and where it was wrong.
- What you would not conclude. The most valuable section and the one everybody omits.
That last section is the assignment. It reads something like: four tasks, one arm, one room, one operator; the intervals do not separate these two models; nothing here transfers to bimanual or contact-rich work; and the language-conditioned gap is the only difference large enough to survive the trial budget.
The two acceptance criteria
The design above is shared. What differs is the trial budget, and therefore what the intervals are allowed to leave unresolved.
| Hardware path | Simulation path | |
|---|---|---|
| The ladder | four tasks on your bench | the same four tasks built in the Module 2 scene; all four are constructible, including the language cliff with coloured distractors |
| Demonstrations | 200 episodes, 50 per task, one operator | 200 episodes teleoperated by hand through the keyboard rig, 50 per task, plus a pinned public real SO-101 dataset |
| Policies | ACT per task, a SmolVLA fine-tune, optionally a π₀-class model | the same three, with the no-pretraining ablation mandatory rather than optional |
| Trials per cell | 20 | 200 |
| Total scored trials | 280 to 420 | 4,200, which is unattended compute rather than three evenings of your life |
| Interval per cell | about ±22 points | about ±7 points |
| What that obliges | say plainly when 20 trials did not separate two policies | resolve the ten-point gaps, because at this budget nothing is hiding behind the interval |
| Second surface | none needed; the bench is the ground truth | offline chunk-level action error for all three policies on held-out episodes of the pinned real dataset, plus the rank correlation between the two surfaces |
| Out of distribution | move a camera, change a surface, use an unseen colour | the same, with the axes listed explicitly, because you can only perturb what you coded |
| Frozen | dataset revision and hash | dataset revision and hash, plus a scene-file hash per cell and a statement that no scene changed between cells |
Both paths also produce: the prediction written first and quoted, a results table with Wilson intervals in and out of distribution, recovery rate, dominant failure mode per policy, cost as a column, and a “what I would not conclude” section a hostile reader could not add to.
Check yourself
1. Why is randomising the trial order across policies a correctness requirement rather than a nicety?
Because the operator is part of the apparatus and drifts. Over three hours you get better at resetting the scene, the arm warms up, the light changes, and your scoring gets looser or stricter. Run all of one model’s trials in a single block and every one of those drifts is perfectly confounded with the model identity, so a real ten-point difference in the table might be entirely the difference between 7pm and 10pm. Interleaving spreads the drift evenly across policies, converting a systematic bias into noise, which the intervals already account for.
2. A single ACT is trained across all four tasks and loses badly. What have you measured?
That you disabled the baseline. ACT is designed as a single-task policy with no language conditioning, so one instance covering four tasks has no way to know which task it is being asked for; on the language task it cannot even guess informedly. The resulting number says nothing about ACT versus a generalist, only about what happens when you use a method outside its design. Train one instance per task, following its authors’ recipe, and state it in the writeup so a reader does not have to wonder.
3. Your table shows SmolVLA 45% and ACT 35% on task 2, with 20 trials each. What do you write?
That the two are not separated at this trial count. Ten points of difference sits well inside the roughly 20 to 22 point interval on each cell, so the honest sentence is “45% against 35%, intervals overlapping, no separation at n=20”. If the direction matters to you, the fix is more trials rather than more confidence: to resolve a ten-point gap you need a few hundred trials per cell, which is why almost no real-robot paper resolves ten-point gaps either.
4. ACT scores 0 out of 20 on the language-conditioned task. Why is “ACT cannot do language” a stronger claim than that number alone supports, and why is the number still informative?
Zero out of twenty gives a Wilson interval of about 0 to 16%, so the number alone establishes an upper bound rather than an impossibility. What makes the conclusion strong is not the number but the mechanism: ACT has no language input, so nothing in the model can distinguish “pick the red block” from “pick the blue block”, and its expected success is whatever guessing among three distractors yields. The number is consistent with the mechanism and the mechanism is the argument. Report both, and be careful to claim only that the observed rate was below roughly one in six.
5. Both models score 40% on a task, but one recovers from 30% of its failed first attempts and the other from 5%. Which would you deploy, and why is that not visible in the success column?
The one that recovers, in almost any real setting, because its failures are more often retryable than terminal and because it has some notion that the attempt went wrong. The success column cannot show this: it collapses “failed and knew it and tried again but ran out of time” and “failed and confidently carried nothing to the bin” into the same zero. Recovery rate needs the first-attempt outcome recorded separately, which is a protocol decision made before the first trial and impossible to add afterwards.
6. Why is “what I would not conclude” the most valuable section, given that it adds no results?
Because it is the section that makes the rest of the document usable by someone else. Every result here is scoped to four tasks, one arm, one room, one operator and one trial budget, and a reader who cannot see those boundaries will over-extend your numbers exactly the way the marketing you spent this module learning to discount does. Writing the boundaries yourself also forces you to notice which of your own conclusions were resting on differences smaller than the intervals, which is usually at least one of them.
Do this
Build it. The scaffold is at project/compare.py, and there is no solution file, deliberately - a solution to this project would be my results on my bench, which is precisely the thing that is not transferable.
The scaffold gives you a complete trial planner with randomised interleaving, a complete operator-in-the-loop recording loop that appends one row per trial to CSV, and a correct Wilson interval with a self-check. It leaves you three things: the task definitions, including the success criterion you must write yourself; the adapter that invokes your own policies, since that depends on your inference stack; and the definition of recovery for your tasks, which blocks the report until you have decided what it means.
On the simulation path the same scaffold applies with the operator prompt replaced by your simulator’s success predicate, which is the one place a machine may score a trial: the predicate was written before training and cannot drift the way a tired human judgement does. Raise --trials to 200, keep the interleaving, and run the second evaluation surface as a separate script that reports offline chunk error and the rank correlation against your closed-loop ordering.
Four commands drive the whole project:
python compare.py check
python compare.py plan --seed 7 --trials 20
python compare.py record --seed 7 --trials 20 --out results.csv
python compare.py report --in results.csvRun check first: it verifies the interval arithmetic against two worked cases, so the statistics in your writeup are something you tested rather than something you trusted.
Then write the page. Publish it with the dataset, the CSV and the seed. Somebody will disagree with it, and because you published the protocol they will be able to disagree precisely, which is the entire point.
What you can now do
You can design a matched comparison between a task-specific policy and a pretrained foundation model on a bench you own, physical or simulated: one dataset, one protocol, per-method recipes, a task ladder built so the interesting difference has somewhere to appear, and a trial budget you costed before spending. You can run hundreds or thousands of scored trials without letting your own drift contaminate the result, report every cell with an interval, explain a difference through a failure taxonomy and a recovery rate rather than a single number, and write down the boundaries of what you measured, including which bench produced it. That document is the artefact that makes the rest of this course legible to somebody else.