Building an eval harness you can trust

An eval harness is a recording device for the protocol, not a loop that counts successes.

Where you are. You know what an honest success rate has to carry. This lesson builds the machine that produces one, so that none of it depends on you remembering at eleven at night.

The best thing you ever cooked

You made something excellent once. Somebody at the table asks for the recipe, you say yes of course, and then you sit down to write it and find that you cannot.

You know it was good. You do not know how long it was in the oven, because you were watching the dish rather than the clock. You do not know whether you used the flour from the tin or the flour from the bag. You changed the heat twice and could not say when. What survived that evening is one word, good, and one word is not a recipe.

So you write it from memory anyway, cook it again on Sunday, and it comes out fine. Not the same. Six things differ and nothing in front of you can say which one mattered, so the honest note to hand over would have to read this is roughly what I think I did.

Nothing about the first evening went wrong. It simply was not written down while it was happening, and there is no later moment at which it can be.

The idea in one paragraph

An eval harness is not a loop that counts successes. It is a recording device for the protocol. Three properties carry all the weight: the protocol is declared and frozen before the first trial runs, every trial is appended to a log along with the exact initial condition that produced it, and every number you report is computed from that log rather than typed. The runner, the printing, the plots are all conveniences you could rewrite in an afternoon. Those three properties are what make a result you can still defend three months later, and they are what you are building here.

The harness in four stages: a frozen protocol, a runner, an append-only trial log, and a summariser

Wider than the screen; scroll it sideways.

The record is the product

Start from the end. What does a reader, or you in three months, need in order to believe the number and to reproduce it? That list is the schema.

Field	Why it is not optional
`protocol_id`	Ties the trial to the frozen protocol; without it, two experiments merge silently
`policy`	The thing under test, named the same way in every trial
`index` and `seed`	The pairing key and the recipe for the scene; together they make the trial replayable
`conditions`	The sampled factor values, written down rather than recomputed later from a seed you might change
`success`, `failure`	The outcome and, when it failed, which of your declared tags applies
`seconds`, `stamp`	Catches drift over a session, and tells you the order things actually ran in

The point of conditions deserves a sentence, because it looks redundant next to seed. It is redundant right up until you change how conditions are sampled, at which point every old seed silently means a different scene and the old log becomes a fiction. Record the derived value, not only the recipe.

Freeze the protocol, and let the code enforce it

The previous lesson said: decide everything before the first trial. That is a discipline, and disciplines fail at eleven at night. So make it mechanical. Put the protocol in a frozen dataclass, hash it, and stamp the hash on every line of the log:

def fingerprint(self) -> str:
    blob = json.dumps(asdict(self), sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:12]

Opening a log whose header carries a different fingerprint raises. Not a warning: an exception, with a message saying that changing the protocol starts a new file.

The log itself is JSONL, one JSON object per line, opened in append mode. That buys three things at once: a crash halfway through 60 rollouts loses nothing, the run resumes by reading which (policy, index) pairs are already present, and you cannot quietly rewrite a trial that came out badly, because the only operation the file supports is adding to the end.

Give both policies the same scenes

Now the part that changes what your numbers are worth. When you compare two policies, do not let them face different initial conditions. Sample the scene from the trial index, and hand the identical scene to every policy.

Six seeded scenes across a session, each run by both policies, with the order alternating

Wider than the screen; scroll it sideways.

Two things are happening in that picture. Pairing means scene difficulty stops being noise: if one seeded cube pose is genuinely hard, it is hard for both policies and drops out of the comparison. Interleaving means the running order rotates, so anything that drifts across a session, a warming servo, a slipping camera mount, your own fading attention, contaminates every policy equally instead of landing on whichever ran last.

The measured payoff, from the harness evaluating its own mock robot over 1,000 simulated experiments at 20 trials per policy: the spread of the estimated gap fell from 0.144 to 0.110, and the chance of correctly ranking the better policy first rose from 78% to 86%. That is a 1.30x reduction in spread, not the dramatic factor you might hope for, because a lot of the noise here is the per-trial coin flip rather than the scene. Pairing pays in proportion to how much of your variance the scene explains, and it costs nothing, so take it.

Pairing also changes which test applies. With paired trials the informative events are the disagreements: the scenes one policy solved and the other did not. Scenes both solved, or both failed, tell you about the task rather than the policies. Counting only disagreements is McNemar’s test, and it is why the bookkeeping is worth it.

Compute the summary, never type it

The last stage reads the log and produces the claim. Two things it must get right.

First, use the Wilson interval from the last lesson, not the textbook $\hat{p} \pm z\sqrt{\hat{p}(1-\hat{p})/n}$ that everybody writes from memory. That formula is not merely loose at small $n$ , it is wrong in a way that flatters you.

Measured coverage of a nominal 95% interval, textbook against Wilson, at a true success rate of 90%

At ten trials and a true rate of 90%, the textbook interval contains the truth 65.1% of the time while claiming 95%. Wilson manages 92.9% in the same setting. Run python solutions/eval_harness.py --coverage and you get the whole table, including the 70% case where the textbook version behaves fine, which is exactly why the problem is easy to miss: it breaks in the high-success regime, and that is where every robot policy claim lives.

Second, report the failure tags. Here is what the harness prints for its built-in demo, thirty scenes, both policies, mock rollouts:

Policy	Successes	Rate	95% Wilson	Top failure tag
act	24 / 30	80%	63% to 90%	grasp_angle, 5 of 6
diffusion	20 / 30	67%	49% to 81%	perception, 9 of 10

The demo run: both policies on the same thirty seeded scenes

n = 30 trials · cube to bin, mock rollouts · 2026-08-09

Thirteen points apart, and the paired comparison says: both succeeded on 18 scenes, both failed on 4, act alone on 6, diffusion alone on 2. Eight disagreements, McNemar exact p = 0.289. You cannot call the rate difference at this trial count, and the harness says so rather than letting you round it up into a headline.

But look at the last column, which costs nothing extra and is not a rate at all. One policy is failing on cube rotation, the other on lighting. Those are different bugs with different fixes, and the failure histogram identified them from the same thirty trials that could not separate the success rates.

The one thing the harness will not do

It will not decide what counts as success. That predicate is yours, it lives in the protocol as a sentence you wrote, and it is the softest joint in the whole apparatus.

Check yourself

1. Why does the trial record store the sampled conditions when it already stores the seed?

Because the seed is only a recipe, and the function that turns seeds into scenes will change. The moment you widen a factor range or add a factor, every old seed means a different scene and the old log silently describes an experiment that never happened. Storing the derived condition makes the record self-contained.

2. What breaks if fingerprint() serialises the protocol without sort_keys=True?

Two protocols identical in content can hash differently depending on key order, so the gate that refuses a mismatched append stops firing reliably. You would still get an id on every line, which makes it look like the safety is there. Canonical serialisation is what makes a content hash mean anything.

3. Both policies solve 18 of the same 30 scenes and both fail 4. Why do those 22 scenes carry no information about which policy is better?

They measure the task, not the difference. A scene both policies solve tells you it was easy; one both fail tells you it was hard. Only the 8 scenes where they disagree can distinguish them, which is why the paired test counts those and asks whether the split between them looks like a fair coin.

4. At

n = 10

and a true rate of 90%, the textbook interval covered the truth 65% of the time. Why does its error land specifically in the high-success regime?

Because it centres on $\hat{p}$ and scales its width by $\sqrt{\hat{p}(1-\hat{p})}$ , which collapses as $\hat{p}$ approaches 1. Observe 10 out of 10 and it reports a zero-width interval at 100%, claiming certainty from ten flips. Wilson pulls the centre towards one half and keeps a floor on the width, so it survives the boundary. High success rates are exactly where robot policy claims sit, which is why this matters here more than in most places.

5. Why append-only JSONL rather than writing one JSON file at the end of the run?

Three reasons, all practical. A crash at trial 47 of 60 loses nothing and the run resumes from the log. There is no in-memory state to corrupt or forget to flush. And a file whose only operation is append cannot be quietly edited to remove a trial that went badly, which turns a discipline into a property of the format.

6. Your harness reports 80% against 67% with p = 0.289, and the failure tags differ sharply. What do you write up?

Both, in that order of honesty: the rates with their intervals and the statement that 30 trials cannot separate them, followed by the failure breakdown, which is where the actionable finding is. The rate comparison needs more trials to become a claim; the failure modes are already a claim, and they tell you what to fix.

Do this

Open code/eval_harness.py and fill in the four functions marked TODO(you): Protocol.fingerprint, paired_schedule, summarise and compare. The append-only log, the protocol gate, the mock robot and the printing are already written. Then:

python module-03-robot-learning/code/eval_harness.py            # the demo report card
python module-03-robot-learning/code/eval_harness.py --pairing  # what identical scenes buy
python module-03-robot-learning/code/eval_harness.py --coverage # Wilson against the textbook

The self-check at the bottom asserts that the protocol gate fires, that every policy at a given index gets the same seed, and that the running order rotates. Three follow-ups worth doing:

head -3 runs/demo.jsonl. Read a trial record and confirm you could replay that exact scene from it alone.
Raise n_trials to 60 in DEMO and run again. The harness refuses to touch the old log, because the protocol changed. Delete the file, rerun, and see whether 60 trials separates the two policies. (It does not. That is the lesson from 3.12 arriving with numbers attached.)
Replace mock_rollout with a function that drives your simulator from the module’s earlier lessons. Nothing else in the file should need to change. If it does, the harness is not policy-agnostic yet, and fixing that now is much cheaper than fixing it in the hardware module.

The reference implementation is in solutions/eval_harness.py.

What you can now do

You have a harness that freezes a protocol and hashes it, appends every trial with the initial condition that produced it, resumes after a crash, gives both policies identical scenes in an alternating order, and computes success rates with intervals, failure histograms and a paired comparison from the log rather than from your memory. It is policy-agnostic, so the same file carries you through the hardware and foundation-model modules. It is also what makes the module project a measurement rather than an anecdote.