Where you are. You have fine-tuned a vision-language-action model, benchmarked it against a task-specific policy on your own arm, and read enough architectures to draw three of them from memory. This lesson is the part of the module that does not expire.
A Tuesday, and a video
A forty-second clip lands in your feed. A humanoid folding laundry, or a bimanual rig pulling an espresso shot, cut every three seconds, music underneath. One quote-tweet says robotics is solved. The next says it is cherry-picked and the robot is teleoperated. Both accounts have more followers than you.
You watch it twice. Then you have a question neither of them has answered, and it is the only question that costs you anything: should anything on my bench change this week?
Almost nothing you need to answer that is in the video. What you need is dull, textual, and available in under an hour: a repository that either contains weights or does not, a licence file, a table with a trial count in it, and a method section that either exists or does not.
The idea in one paragraph
Triage a release in three passes, in this order, and stop early whenever a pass settles it. Pass one: what can I actually get? Weights, a paper, or a video - these are three different kinds of object and only one of them is a tool. Pass two: what licence covers the weights specifically? Not the repository; the weights. Pass three: who ran the evaluation, and with how many trials? Everything after that is arithmetic on the numbers they published and a decision about your own stack. The models in this module will be replaced within a year. These three passes will not, because they are questions about evidence rather than about any particular architecture.
Pass one: which bucket
Sort every release into one of three buckets before you form any opinion at all. Most of the confusion in this field collapses the moment you do this.
Wider than the screen; scroll it sideways.
Here is the sort as of August 2026, which is exactly the part of this lesson that ages fastest:
| Bucket | Models, August 2026 |
|---|---|
| Downloadable | SmolVLA, OpenVLA, pi0 / pi0-FAST / pi0.5, GR00T N1.7, Wall-OSS-0.5, and - on secondary coverage I did not verify - Cosmos 3 |
| Published | pi*0.6, pi0.7, Gemini Robotics 1.5 and 2 |
| Demonstrated | Figure Helix 02 |
A release in bucket three cannot change your stack this week no matter how impressive it is, because there is nothing to run and nothing to check. That is not a criticism of the work; it is a statement about what you can do with it on a Tuesday. Bucket two can change your plans. Only bucket one can change your bench.
Pass two: which licence, specifically
This pass takes ninety seconds and catches more errors than any other step, because the answer is routinely reported wrong by people who are not being careless.
The same caution applies to models this course likes. The openpi repository carries Apache-2.0 for its code; I could not find an explicit weights-licence statement while researching this module, so if you are building anything commercial on a pi0-family checkpoint, go and read the licence text yourself rather than inheriting my assumption. Same for the SmolVLA checkpoints: the ecosystem is Apache-2.0 and the Hub copies are tagged that way, and I still did not confirm the field directly. Two unverified licences in a module that recommends both models is an honest statement of where I stopped, and it is exactly the kind of gap this pass exists to close.
Pass three: who measured
Wider than the screen; scroll it sideways.
The rule that falls out: when the lab that built the model also designed the benchmark and ran the trials, halve your prior. Not zero - the ablations in a good paper are often the most useful part of it - but halve it.
There is a sharper version of this inside material you have already worked through in benchmarking against ACT. The SmolVLA authors report 78.3% against a task-specific policy’s 48.3% on a real SO-100 with 50 demonstrations per task. The independent SO-101 benchmark, with 100 demonstrations per task, measured SmolVLA at 32.5% against 33.75% for the same class of baseline. A thirty-point win becomes a statistical tie. Nobody lied. The tasks differed, the operators differed, the effort spent tuning each side differed. That contradiction is the reason a number carries the name of whoever produced it, forever, in your notes.
Reading the numbers
Once a release survives three passes, you are reading a table. Five questions, in rough order of how often they change the conclusion.
How many trials per cell? Real-robot evaluation is expensive, so numbers routinely come from twenty trials. Twenty is not many. For a success rate near 50%, the 95% interval from the normal approximation is
Averaging four tasks gets you to and an interval of roughly points, which is better and still loose. Apply it to the independent SO-101 benchmark and see what survives.
Wider than the screen; scroll it sideways.
The surviving conclusion is not the four-way ranking the table implies. It is: the 3B-class model beats the task-specific policy and the 450M model, which are tied with each other, and the two large models cannot be separated at this trial count. That is a smaller claim than the table looks like, and it is the one you can defend.
A related trap sits at the bottom of the table. A cell reading 0% does not mean impossible. Nought out of twenty is consistent with a true success rate as high as about 16%, and the authors of that benchmark say so explicitly: read a zero as “not observed under this budget.”
Was the baseline trained fairly? A task-specific policy trained across ten tasks at once, with no language input, compared against a language-conditioned generalist, is not a fair fight; it is a category error dressed as a comparison. When I researched this module I hit exactly this on one otherwise excellent independent benchmark and could not determine from the main text whether the baselines were trained per-task or across all tasks. That single unresolved detail decides whether the paper refutes the “a small task-specific policy sometimes wins” position or simply fails to test it. Ask whether each baseline was trained the way its own authors recommend.
In-distribution or out-of-distribution? In-distribution numbers cluster; every method looks similar. Out-of-distribution is where pretraining actually shows up, and it is where the same benchmark family found a task-specific policy dropping from 0.18 to 0.075 while a small VLA went from 0.26 to 0.30 - a collapse against a flat line. A paper that reports only in-distribution results is hiding the interesting half.
Is the protocol reproducible? Released demonstration datasets, published task definitions, a parts list. One independent SO-101 benchmark published a rig bill of materials totalling about $1,050 - arm, depth camera, webcam, light box - and checked reproducibility across independently built copies. That is the standard to hold others to, and it is achievable by a person with a desk.
Success rate only, or failure taxonomy and recovery? Aggregate success hides the mechanism. On that same SO-101 benchmark the recovery rate - the fraction of failed attempts where the policy noticed and tried again - was 6.45% for the task-specific policy against 30.77% for pi0.5. That gap is arguably more decision-relevant than the success gap, because it tells you why: a pretrained model has some notion of what the scene should look like, so it can tell that something went wrong.
Claims to actively distrust
- A polished video with cuts. The number of takes is never disclosed. A blog claiming a long autonomous sequence with no resets is describing one specific take.
- “Matches human teleoperators.” Always find the reference class, and quote it rather than paraphrasing. One 2026 release compared its success rate to expert teleoperators - the ones who had recorded the training data - driving an unfamiliar robot zero-shot. That is a more interesting reference than “beginners” and still not a benchmark: no trial count, no protocol, and a comparison group chosen by the party reporting the win. The failure mode here runs both ways, and paraphrase causes both.
- A baseline the authors trained themselves, with unpublished hyperparameters.
- Percentage improvements with no absolute numbers. “+61%” from what to what?
- “Zero-shot” that turns out to mean zero-shot on a close variant of something in the pretraining mix.
- Benchmarks the releasing lab designed. Shared benchmarks are imperfect and at least everyone is standing on the same floor.
- Scaling laws derived from one lab’s proprietary corpus. Interesting, unreproducible, and not a law.
The decision, which is the actual output
Triage that ends in an opinion has failed. It ends in one of four actions, and you write down which and why.
| Verdict | Trigger | What you do |
|---|---|---|
| Ignore | demonstrated bucket, or a capability you do not need | nothing; note the date so you do not re-triage it in a month |
| Bookmark | published but not downloadable, or downloadable with a licence you cannot use | one line in a log, with the specific event that would change it |
| Bench it | downloadable, licence works, someone outside the authoring lab has run it on hardware like yours | schedule the fine-tune; it is a day, not a quarter |
| Re-plan | it invalidates a design assumption, not just a number | rewrite the plan before touching code |
The strongest single predictor that a model will work for you is not its benchmark score. It is: has someone outside the authoring lab fine-tuned it on hardware like yours and published the result? As of August 2026, on the SO-101, the models that pass that test are pi0.5, SmolVLA, Wall-X, and - by Google’s own model card - Gemini Robotics On-Device 2.
Review
Three passes, in this order
Triage a release in three passes and stop early whenever a pass settles it. Pass one: what can I actually get? Weights, a paper, or a video are three different kinds of object and only one of them is a tool. Pass two: what licence covers the weights specifically, not the repository? Pass three: who ran the evaluation, and with how many trials? Everything after that is arithmetic on the numbers they published and a decision about your own stack. Every model named in a course like this will be superseded within a year. These three passes will not be, because they are questions about evidence rather than about any particular architecture.
You already run this procedure, on dependencies
Before adopting a library you check whether it is actually released or just announced, what the licence permits, who else runs it in production, and what breaks if it disappears. You do not read the landing page and adopt. Model releases arrive with far better production values than a package README and far worse evidence, so the same reflex applies; it just has to fire against a launch video instead of a badge row. The output of the triage is a decision with the reason written down: ignore it, bookmark it, bench it, or re-plan around it. Writing the reason is what makes the decision reviewable later, when the release everyone was excited about turns out to have been a video.
Check yourself
1. A release drops with weights, a paper, and results showing it beating everything on a benchmark the lab designed. What is your next move, and what is it not?
Pass one and two are satisfied, so go to pass three: find out whether anyone outside that lab has run it, ideally on hardware resembling yours. If nobody has, it is a bookmark rather than a bench-it, however good the numbers look, because a benchmark designed by the releasing lab tells you the model works on the tasks they chose to show. What it is not is a re-plan. Nothing about a self-reported score on a self-designed benchmark should reorganise your work.
2. Two models are reported at 56% and 51% on a real-robot benchmark with twenty trials per task across four tasks. Which one is better?
Neither, on this evidence. Eighty trials gives a 95% interval of roughly eleven points on each number, so intervals of 45 to 67 and 40 to 62 overlap across most of their range. The honest statement is that the benchmark cannot separate them. Ranking them anyway is how a five-point gap becomes a sentence in a blog post and then a procurement decision.
3. A benchmark reports 0% for one model on one task. What does that tell you?
That the model was not observed to succeed within the trial budget. With twenty trials, zero successes is statistically consistent with a true rate as high as about 16%, so it is a weak upper bound rather than a demonstration of impossibility. It is still informative in context - a model at 0% while others are at 55% on the same task is telling you something real about a capability gap - but the number itself is “below roughly one in six”, not “never”.
4. Why is the weights licence a separate question from the repository licence, and what is the concrete example?
Because they are separately chosen by the publisher and frequently differ. GR00T N1.7 ships Apache-2.0 code with weights under the bespoke NVIDIA Open Model License Agreement; the licence permits commercial use, but it is not the OSI-approved licence that several write-ups reported. Anyone who read the LICENSE file and stopped there has the wrong answer to the only question that governs shipping. The code licence tells you what you can do with the training scripts; the weights licence tells you what you can do with the model.
5. A vendor benchmark shows a model beating pi0.5 by 17.5 points. An independent lab measures the same model 5 points behind pi0.5. How do you write this in your notes?
Both, with attribution, scope and metric, and neither as “the” number. Something like: “vendor-reported +17.5 on average task progress, their own 15-task suite; independent SO-101 benchmark, 4 tasks, n=20 per cell, measured -5 on binary success rate.” The attribution and the metric are the content, and here they differ: partial-credit progress and binary success are not the same quantity, so the two numbers do not even contradict each other in the way the framing suggests. Averaging them, picking the favourable one, or picking the sceptical one all discard the actual finding, which is that the ranking depends on who chose the tasks and what counted as a win. If the independent result is on hardware like yours, it is the one that predicts your bench.
6. Why does the triage end in one of four verdicts rather than in a summary?
Because a summary costs nothing to produce and commits you to nothing, which is how the same release gets re-triaged three times. Ignore, bookmark, bench, re-plan are actions with different costs and different trigger conditions, and writing one down with a date turns a stream of announcements into a log you can audit. It also makes the bookmark verdict useful: a bookmark with a named trigger event is a plan, while a bookmark without one is a browser tab.
Do this
About an hour, and repeat it monthly.
1. Triage one real release, on a timer. Pick anything announced in the last month. Set sixty minutes. Produce exactly five lines: bucket; weights licence with the URL you read it at; who ran the evaluation and the trials per cell; whether out-of-distribution results exist; verdict from the four. If you cannot fill a line, write “not stated” - that is a finding, not a gap in your work.
2. Do the interval arithmetic yourself once. Six lines of Python, so the number stops being something you accepted from a lesson:
from math import sqrt
def ci95(p, n):
return 1.96 * sqrt(p * (1 - p) / n)
for n in (10, 20, 50, 80, 200, 500):
print(f"n={n:>3} 95% CI at p=0.5: +/- {100 * ci95(0.5, n):.1f} points")Run it and read the last column. Then find the trial count in the most recent robotics paper you read. Most real-robot evaluations sit in the top two rows.
3. Start frontier-log.md. One row per release: date, name, bucket, licence, verdict, and the specific event that would change the verdict. Six months of this is worth more than any curated list, because it is a record of your own judgement and you can go back and see which calls were wrong.
What you can now do
You can take any model announcement and, in about an hour, sort it into downloadable, published or demonstrated; find the licence that actually governs the weights; identify who ran the evaluation and how many trials stand behind each cell; compute whether the reported ranking survives its own error bars; and commit to one of four verdicts with a written trigger for revisiting it. That procedure is the durable part of this module. Everything named in it is not.