Where you are. You can run the data-quality loop and debug a real arm by symptom. This lesson is about what you hand to other people when it finally works.
Twelve seconds of somebody else’s robot
You have watched this video. Everyone has.
A small arm reaches for a coloured block against a clean white desk. It grips, lifts, swings across, and drops the block into a bowl. The clip is twelve seconds long. There is a caption saying which model it is, and often a number: 90%.
Now try to answer three questions about it. How many times did they run this? Was that clip the first attempt or the ninth? If you set the same arm up on your desk tonight, would it work?
You cannot answer any of them, and neither can anybody else, which means the twelve seconds carry no information about whether the robot works. They only tell you that it worked once, while a camera was pointed at it. There is nothing dishonest about the clip. It is simply not a result, and the difference between a clip and a result is a couple of hours of work that almost nobody does.
The idea in one paragraph
A hardware result is a claim about a distribution: this policy does this task, this often, under these conditions. Publishing it means shipping enough for a stranger to check the claim, which turns out to be five artifacts: the dataset, the checkpoint, the evaluation protocol, an uncut video of every trial, and a reel of the failures. Each one answers a specific question a sceptical reader has, and the one everybody omits, the failure reel, is the one that makes the rest believable. Alongside those, the number itself needs an interval, because twenty trials is a much weaker measurement than it feels like, and quoting a bare percentage from twenty trials overstates what you know by roughly twenty percentage points in each direction.
Wider than the screen; scroll it sideways.
Your number has an interval, and it is wide
Say you ran 20 trials and 14 worked. The headline is 70%. What does 70% actually license you to say?
Not much. A 95% interval around 14 out of 20 runs from about 48% to 85%. Your policy might genuinely work five times in six, or it might work slightly less than half the time, and twenty trials cannot separate those two worlds.
Wider than the screen; scroll it sideways.
That figure is the single most useful thing in this lesson. Every bar reports the same 70%. Only the trial count differs, and the meaning of the claim changes completely down the column.
The reason is arithmetic you already have an intuition for from load testing: uncertainty falls with the square root of the sample. The rough half-width of the interval around a rate measured over trials is
where is a constant set by how confident you want to be, 1.96 for the conventional 95%. Read the whole expression as: your uncertainty shrinks only with the square root of the trial count. At and it comes to about 20 percentage points either side, and halving that takes four times the trials. code/eval_stats.py computes it properly, and its plan subcommand answers the practical question directly: at a true rate near 70%, you need about 78 trials for a 20-point-wide interval and about 320 for a 10-point-wide one.
None of this means twenty trials is worthless. It means twenty trials establishes an order of magnitude, and you should say so. “70% (14/20, 95% CI 48-85%)” is a sentence a stranger can use. “90% success” with no denominator is a sentence they cannot.
Write the protocol before the trials, not after
The evaluation protocol is the artifact that makes everything else meaningful, and it costs ten minutes. Four things:
| What to state | Why it decides the number |
|---|---|
| The initial-condition distribution | The rate applies only to conditions your setup can produce. “Cube anywhere in a 30 by 20 cm zone, random yaw” is a claim. “Cube on the desk” is not. |
| The success criterion, in one sentence | Written afterwards, it drifts toward whatever the robot did. Written before, it is a test. |
| The trial count, and how trials were selected | Consecutive, in one sitting, no re-runs. Anything else needs saying. |
| What you voided and why | A trial where the bowl slid off the table is legitimately void. A trial the robot failed is not. Publish both counts. |
Video etiquette
The video is not decoration. It is the evidence that the count is real.
- Uncut, all trials, in order. One continuous recording of the whole session, including resets, is better than a montage. A viewer can count.
- Real time, or the speed labelled on screen. Sped-up footage hides dithering, which is exactly the failure mode people most want to see.
- A visible trial counter, even if it is you saying the number out loud.
- A separate failure reel. Every failure, tagged with the bucket you sorted it into.
Wider than the screen; scroll it sideways.
The failure reel feels like the thing that makes you look bad, and it does the opposite. A published result with fourteen successes and six failures on screen is checkable and therefore credible. A published result with one clip is not checkable at all, so a reader has only your reputation to go on, and you do not have one yet. The failure reel is also, in practice, the part of your post that other people learn the most from, because everyone’s robot fails and almost nobody shows how.
What actually goes on the Hub
The mechanics are short. LeRobot pushes the dataset for you during recording with --dataset.push_to_hub, and pushes the trained policy with --policy.repo_id during training. Locally, datasets land under ~/.cache/huggingface/lerobot/{repo-id}. Before you publish anything, view the dataset in the Hub’s dataset viewer space; LeRobot’s own guidance is to always do this before training, and it is equally the right last check before publishing.
The model card is where reproducibility lives or dies. Put in it: the exact LeRobot version, the policy type and training steps, the camera models and their positions, the lighting, the episode count, and a link to the dataset. Two of those matter more than they look.
The writeup that travels
The post people actually read is not a demo, it is a story about a curve. What you tried, what happened, what you changed, what happened next. The three-attempt shape from the data-quality lesson is a good spine: the first policy that did nothing, the change that was not a model change, the coverage round that finally moved it.
Report the numbers that made you look bad along the way. “0% on the first fifty episodes” is more useful to a reader than the final figure, because it is the number they are staring at right now, and the reason they are reading you at all.
Without hardware
The five artifacts are the same five, and the honesty requirement is higher rather than lower, because a simulated result is easier to produce and easier to over-read.
- Instead of the rig photograph, publish the scene file itself with a content hash, plus the simulator version. Those are your camera positions.
- Instead of the uncut bench video, publish an uncut screen recording of the scored trials in the order the harness dealt them, plus the seed that reproduces them.
- Measure this: the same rate, the same interval, the same failure histogram - and put the venue in the headline sentence rather than in the methods section. “82% over 200 MuJoCo trials, seed 7” is a result. “82%” is a claim about a bench you do not have.
Check yourself
1. Your policy succeeded on 14 of 20 trials. Write the sentence you would publish, and say what it is you are refusing to claim.
Something like: “70% success (14/20 trials, 95% CI 48-85%), with the cube placed uniformly across a 30 by 20 cm zone under fixed artificial light.” You are refusing to claim the policy works 70% of the time, because twenty trials do not distinguish a true rate of 50% from one of 85%. You are also refusing to claim anything at all outside the stated placement zone and lighting. What you are claiming is that the rate is somewhere in that band under those conditions, which is a real and checkable statement.
2. Why does the textbook interval fail exactly when your result is perfect, and what does that tell you about evaluation code in general?
Because the width is proportional to the square root of , and at that term is zero, so the interval collapses to a point. Twenty perfect trials become a claim of certainty, which is obviously wrong: a policy that truly works 90% of the time produces 20 out of 20 reasonably often. The Wilson interval handles it because it does not centre on the observed rate; it reports roughly 84% to 100% for 20 out of 20. The general lesson is that evaluation code deserves the same scepticism as the policy: check it against the extremes, where a plausible-looking formula gives an obviously silly answer.
3. Which of the five artifacts is most often missing, and why does its absence undermine the other four?
The failure reel. Without it, a reader has a success rate they cannot audit, a video that shows only successes, and no way to tell whether the failures were near-misses or the arm not moving at all. Those two possibilities imply completely different next steps and completely different levels of maturity, and the number alone cannot distinguish them. Publishing the failures makes the count verifiable, tells readers where the limits are, and signals that you were measuring rather than demonstrating. It is also the part with the most transferable content, since everyone’s robot fails in a small number of shapes.
4. You decide mid-evaluation that a trial where the gripper nudged the cube into the bowl counts as a success. What is wrong with that, and what is the right way to handle the case?
The criterion is now a function of what you saw, so it will bend toward whatever the robot happens to do, and the resulting rate is not a measurement of anything. The right handling: score the trial against the sentence you wrote beforehand, whatever that sentence implies, and finish the run. If the criterion turns out to be ambiguous, note the ambiguity, publish the run under the original criterion, and rewrite the sentence for the next run. A criterion you improve between runs is science. A criterion you improve during a run is a screenshot.
5. Someone shows you an SO-101 result claiming 90% success. What three questions tell you almost everything about whether to believe it?
How many trials, and were they consecutive? What was the initial-condition distribution, and was the object placed the same way every time? Can I see the failures? The first bounds the uncertainty: 9 out of 10 has an interval running from roughly 60% to 98%, which is a very different claim from 90%. The second establishes whether the number covers a real distribution or a single rehearsed setup. The third checks that failures were counted at all, since a result with no failures shown at 90% claimed is a result where somebody chose which trials to report.
6. Why pin the LeRobot version in your writeup when you could just say “latest”?
Because “latest” names a different thing every few weeks, and this ecosystem has been shipping breaking changes on roughly a six-week cadence. Within one month in 2026, the documented way to deploy a policy to a real arm changed from lerobot-record --policy.path to lerobot-rollout, and a core module was renamed in a way that breaks user imports. Anyone following your post six months later on a newer release will hit errors that have nothing to do with your work and will reasonably conclude your work does not reproduce. A pinned version costs one line and preserves the value of everything else you wrote.
Do this
Publish one result properly. Two to three hours, and most of it is recording rather than writing.
1. Write the protocol first. Four sentences: the initial-condition distribution, the success criterion, the trial count, and what will make a trial void. Commit this to your repository before you run anything, so the timestamp shows it came first.
2. Run the trials in one sitting with the camera on the whole time. Start the recording before trial one and stop it after the last one. Say the trial number out loud each time. Reset between trials the same way every time.
3. Get the number with its interval.
python code/eval_stats.py rate 14 20Substitute your own counts. Copy the sentence it prints; it is already in publishable form.
4. Cut two videos from the one recording. A trimmed but uncut run of all trials in order, and a failure reel of just the failures with your bucket tag on each. Neither is sped up unless the caption says so.
5. Push the dataset and the checkpoint, and write the model card. LeRobot version, policy type, training steps, episode count, camera models and positions, lighting, and a photograph of the rig.
6. Write the post around the curve, not the demo. What you tried, what the first policy did, what you changed, what it did next. Lead with the interval, not the point estimate. Include the numbers from the attempts that failed.
What you can now do
You can publish a hardware result that a stranger could check: a success rate carrying its own interval, an evaluation protocol written before the trials, an uncut video of every attempt, a failure reel, and a dataset and checkpoint pinned to a specific library version. You can explain why twenty trials establishes an order of magnitude rather than a value, why a perfect score is not a certain one, and why the failures are the part of the post that makes the successes believable.