What it takes to say a policy works

Where you are. You can train ACT and Diffusion Policy on your own demonstrations. This lesson is about the sentence you write afterwards, which is the part everyone else will actually read.

Friday’s number and Monday’s number

Six o’clock on a Friday. The checkpoint finishes, you run it ten times on cube-to-bin, and nine of the ten work. You screen-record the good one, write 90% success, and post it.

Monday morning, someone else runs the same checkpoint on the same rig. Six work.

Nothing changed. Same weights, same camera, same cube, same lighting. The number moved thirty points, and neither of you did anything wrong. Ten is a very small number, and what you measured on Friday was never the thing you wrote down.

The idea in one paragraph

A success rate is not a score, it is an estimate. You are guessing at the bias of a coin from a handful of flips, and every such guess comes with a range of biases consistent with what you saw. At ten trials that range is roughly forty-five points wide, which is wider than nearly any policy difference worth arguing about. So the bare number is not a claim. The claim is the number, the trial count under it, the interval around it, and the protocol that produced all three. Writing less than that is not brevity; it is publishing something that will not survive being rerun.

A success rate is a coin, not a score

Each trial either works or it does not, and the trials are independent given a fixed policy and a fixed reset distribution. That makes the count of successes a binomial quantity, and the thing you want, the policy’s true success rate, is the binomial’s unknown parameter. You never observe it. You observe $k$ successes out of $n$, and you infer.

The standard way to state what you can infer is a confidence interval: the range of true rates that would have made your observation unsurprising. For a proportion, use the Wilson score interval,

$\text{interval} = \frac{\hat{p} + \frac{z^{2}}{2n}}{1 + \frac{z^{2}}{n}} \;\pm\; \frac{z}{1 + \frac{z^{2}}{n}}\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^{2}}{4n^{2}}}$

where $\hat{p} = k/n$ is what you observed and $z = 1.96$ for 95% confidence. In words: shrink the observed rate a little towards one half, by an amount that matters only when $n$ is small, then put a band around it whose width falls roughly as $1/\sqrt{n}$. That last part is the whole economics of evaluation. Halving the width costs four times the trials.

The claim the field makes most often

Now put two policies side by side. Yours got 90%, the baseline got 80%, ten trials each. That is the single most common shape of claim in robot learning, and it is one trial of difference.

Run the exact test on that pair and the p-value is 1.000. Not 0.06, not “trending”. One. At twenty trials it is 0.661, at fifty 0.262, at a hundred 0.073. The gap only becomes something you can state at around two hundred trials per policy, and that is not an accident of these particular numbers; it is what the arithmetic of proportions costs.

Two spreads, not one

There is a second source of variation that trial counts do nothing about. Retrain the same architecture on the same data with a different random seed and you get a different policy, sometimes a materially different one. That variation lives between training runs, not between trials.

Conflating them is the standard mistake. Twenty trials on one seed tells you about that one policy, tightly, and nothing at all about the recipe. Report both: at least three training seeds per configuration, and quote the seed-to-seed spread separately from the trial-to-trial interval. If the seeds disagree more than the trials do, your headline number is a property of a lucky initialisation.

Looking early is a decision

Here is the failure with the widest blast radius. You planned twenty trials, you are at trial fourteen, the new policy is ahead, and you stop.

That is not laziness. It feels like efficiency, and it silently breaks the test. Every extra look is another chance for noise to cross your threshold, so the rate at which you declare a winner between two policies that are genuinely identical climbs with the number of looks.

Those bars are measured, not asserted: two simulated policies both truly at 70%, forty trials each, a hundred thousand simulated experiments. Fixing $n$ in advance and testing once declares a winner 2.8% of the time. Checking the running score after every trial and stopping at the first significant result declares one 11.0% of the time. Same data, same test, four times the false wins, purely from when you looked. (The honest bar lands below 5% rather than on it because the exact test is conservative on small counts. Compare the bars against each other, not against the line.)

What you sampled is part of the claim

A policy does not have one success rate. It has a success rate over a distribution of initial conditions, and you chose that distribution. Cube always at the same spot, one lighting condition, one background: the number is real, and it describes a much smaller world than the sentence implies.

So list the factors you varied, with ranges, and list the ones you froze. The frozen list is the more informative one.

Who ran it, and whether they knew

The last piece is social rather than statistical. You configured the baseline, you know which policy is running, and you decide when a marginal trial counts as a success. None of that requires bad faith to bias a result.

You will not have seven institutions. You can still interleave the policies instead of running all of A then all of B, hide which is which from whoever scores the trial, and pick your checkpoint on a held-out split rather than on the test rollouts. That last one matters more than it sounds: choosing the checkpoint that scored best on your evaluation set turns the evaluation into part of training, and the number stops being an estimate of anything.

Everything above collapses into one discipline: decide the whole protocol before the first trial exists, and treat any later change as the start of a new experiment rather than an edit to this one.

Review

Check yourself

Do this

Run the arithmetic yourself. Open code/trial_math.py, fill in the three functions marked TODO(you) (the Wilson interval, the sample-size formula, and the peeking loop), then:

python module-03-robot-learning/code/trial_math.py

It takes about ten seconds and prints three blocks. Before you look at the second one, write down your guess for how many trials it takes to separate 90% from 80% at 80% power. Then answer three questions in your lab notes:

1. Your last training run’s success rate: what is its 95% interval? Would it separate from a baseline five points below it?
2. At the trial count you can actually afford on your rig, what is the smallest gap you could honestly claim?
3. Pick a robot-learning demo video or paper you find convincing. Find its trial count. If it is not stated, that absence is itself the finding.

The reference implementation is in solutions/trial_math.py, with assertions that pin the numbers quoted in this lesson.

What you can now do

You can turn a raw success count into a claim: an interval instead of a point, a stated $n$, a seed spread reported separately from the trial spread, and an explicit list of what varied and what was frozen. You can recognise the two ways an honest-looking comparison goes wrong, by peeking and by narrow sampling, and you can say what it would cost in trials to make the comparison you want. The next lesson turns this discipline into code, so that none of it depends on you remembering.