Benchmarking your fine-tune against ACT

Where you are. You have an ACT baseline from Module 4 and a SmolVLA checkpoint fine-tuned on the same frozen episodes. This lesson turns those two artifacts into a number somebody else can believe.

The hour between the two halves

You start at seven with the fine-tuned model. Twenty trials, the cube reset by hand onto its cross between each one, and you write down twelve.

Then you load ACT and run twenty more, and by the time you finish, three things have moved that you never chose to move. The sun has come off the window and the desk is darker than it was. The shoulder servo has been working for the best part of an hour and is warm enough to sag a little on the way down. And your own hand has got quicker and looser at putting the cube back, so the placements in the second half of the evening sit further out than the ones in the first.

ACT gets nine.

Every one of those forty trials was honest, and each was scored exactly as you said you would score it. What you have is still not a comparison, because the policy was not the only thing that differed between the first twenty and the second twenty - it was just the only difference you were paying attention to.

The idea in one paragraph

A comparison is a protocol, not a number. Two success rates are only evidence about the policies if everything else was held fixed and written down first: the same frozen dataset, the same task, the same physical starting conditions in the same order, the same person resetting the scene, the same definition of success decided before anyone ran anything. Even then, twenty trials per policy is a small sample, and the interval around a success rate at that count is wide enough to swallow most of the differences people publish. The way out is not more confidence, it is more honesty: report the interval, run the trials in pairs so the comparison is within-condition rather than between-evenings, look at how the policies fail rather than only how often, and then run the second evaluation - the one where you change the scene - because that is where the pretraining shows up and where the interesting gap actually lives.

Fix everything except the policy

Your Module 3 eval harness already does the mechanics. What this lesson adds is the list of things that must be identical, and the discipline of writing them down before the first trial.

What twenty trials can and cannot tell you

Twelve out of twenty is 60%, and the 95% interval around it runs from 39% to 78%. Nine out of twenty is 45%, with an interval from 26% to 66%. Those two ranges overlap across more than half their width. The correct sentence is not “SmolVLA beat ACT by fifteen points”. It is “both policies are somewhere between about a quarter and about three quarters, and this experiment did not separate them”.

Pairing helps, and it is worth understanding exactly how much. Because both policies attempted the same twenty setups, the trials where both succeeded and the trials where both failed tell you nothing about which is better; only the disagreements do. If seven conditions went to the VLA and four to ACT, the question becomes whether a seven-four split of eleven disagreements is surprising, and the answer is no: it happens more than half the time by chance alone.

Compute both, in fifteen lines with nothing installed:

from math import comb, sqrt


def wilson(successes, trials, z=1.96):
    """95% confidence interval for a success rate."""
    rate = successes / trials
    denom = 1 + z * z / trials
    centre = (rate + z * z / (2 * trials)) / denom
    half = z * sqrt(rate * (1 - rate) / trials + z * z / (4 * trials * trials)) / denom
    return centre - half, centre + half


def paired_p(a_only, b_only):
    """Two-sided exact p for matched pairs, from the two disagreement counts."""
    total = a_only + b_only
    if total == 0:
        return 1.0
    smaller = min(a_only, b_only)
    tail = sum(comb(total, i) for i in range(smaller + 1)) / 2 ** total
    return min(1.0, 2 * tail)


print(wilson(12, 20))     # (0.387, 0.781)
print(wilson(9, 20))      # (0.258, 0.658)
print(paired_p(7, 4))     # 0.549

Run it on your own counts. A seven-four split gives 0.549. Even a nine-two split only reaches 0.065, and you need something like eight disagreements to one before twenty trials produce a p-value under 0.05. To pull the interval in to about plus or minus ten points at mid-range you need roughly a hundred trials per policy, and even then 45% against 60% still touches.

Success rate hides the mechanism

Two policies at 50% can be 50% in completely different ways, and the difference matters more for what you do next than the rate does.

Record every trial and label each failure. Four categories cover most of what an SO-101 does: reached the wrong place (a targeting or localisation failure), grasp slipped (a contact failure), stuck in a loop (the policy repeating an approach forever), and wrong object or wrong bin (a language or selection failure). Then count one more thing: after a failed attempt, did the policy try again and succeed?

The mechanism behind the recovery gap is the one sentence to remember from this module. A pretrained model has a prior about what the world should look like, so it can notice that the world does not look like that and try something else. ACT has learned a distribution over trajectories and has no representation of “that went wrong”.

The second evaluation, where the gap actually lives

Now change the scene. New object colour, a different shape, a distractor on the table, a starting position outside the range you demonstrated. Twenty more trials each, same protocol, same interleaving.

Those numbers come from VLA-REPLICA, an independent 2026 study on SO-101 hardware with 50 demonstrations per task - the same arm and the same demonstration count this course uses. On the trained scene the gap between ACT and SmolVLA is small. Change the scene and ACT falls from 0.18 to 0.075, losing more than half of what it had, while SmolVLA goes from 0.26 to 0.30, which is to say it does not fall at all. π0.5, roughly seven times the size, leads on both and still loses about a third of what it had.

Whose number is it

One more habit before you publish. SmolVLA’s own paper reports 78.3% against ACT’s 48.3% on real SO-100 hardware with 50 demonstrations per task. The independent Hiroshima benchmark reports SmolVLA at 32.5% against ACT at 33.75%, on similar hardware with more demonstrations.

Both are honest. Different tasks, different operators, different amounts of effort spent tuning each side. But they point in different directions, and the one that should move you more is the one run by people with nothing to gain. When a lab publishes results for its own model, halve your prior; when an outside group replicates on hardware like yours, that is the number worth planning around.

Without hardware

• Instead of forty physical trials, run two evaluation surfaces and report both. Closed loop in the Module 2 scene, 200 trials per cell with interleaved policy order and a Wilson interval. Offline on held-out episodes of the pinned real dataset, reporting action error over a full chunk rather than per step.
• Measure this: the rank correlation between the two surfaces across your checkpoints. An offline metric that reorders your policies is a finding, and it is the concrete version of the warning this lesson gives about loss curves.
• What you lose: out-of-distribution as a real thing. You can move a simulated camera, change a colour or add a distractor, but only along axes you wrote, so a flat out-of-distribution result on this path means your randomisation was narrow at least as often as it means the policy generalised.

Review

Check yourself

Do this

Two evenings, forty trials each, and the writeup.

1. Write the protocol first. One page in notes/05-eval-protocol.md: the success criterion in one unambiguous sentence, the 20 initial conditions from lesson 10, the instruction string verbatim, the interleaving order, who is resetting, and the checkpoint identifiers for both policies. Sign and date it before the first trial.

2. Run the in-distribution evaluation. Forty trials, interleaved, uncut video running throughout. Log every trial as a row: condition number, policy, outcome, failure category, whether it recovered.

3. Run the out-of-distribution evaluation. Twenty more conditions, deliberately outside what you demonstrated: a different coloured object, a distractor, a starting position at the edge of the workspace. Same protocol.

4. Compute and report. Run the snippet above on your counts. Produce one table with, for each policy and each evaluation: successes out of trials, the interval, the recovery rate, and the failure breakdown. Then write the paragraph you would be comfortable defending if one of the model’s authors read it.

5. Publish the failures. Cut a reel of the failed trials alongside the successes. It costs you nothing, it is the part practitioners actually watch, and a comparison without one reads as marketing regardless of how careful the numbers were.

What you can now do

You can run a comparison that survives scrutiny: matched data, a written protocol, matched initial conditions, interleaved trials and a success criterion fixed in advance. You can put a confidence interval on a success rate and say plainly when twenty trials did not separate two policies. You can read a failure taxonomy and a recovery rate as the mechanism behind the numbers, and you can run the out-of-distribution evaluation that shows what pretraining is actually for. And you can tell an author-reported result from an independent one, which is the single most useful reflex in a field this loud.