Where you are. You have a SmolVLA fine-tuned on your own SO-101 data, an ACT baseline trained on the same episodes, and numbers for both from the same eval harness. This lesson is the case the launch posts never cover: what to conclude, and what to claim, when the foundation model does not win.
Catch yourself typing the query
Both rows of the log are in. ACT: seven out of twenty. The fine-tuned VLA: seven out of twenty.
Watch what you do next, because this is the part worth catching. You do not write the tie down. You open a browser and type a search with its own answer already inside it: smolvla outperforms act so-101.
It works, the way that kind of search always works. There is a 2026 paper reporting this exact comparison, on this class of arm, at this demonstration count, where the VLA wins by thirty points. Cite it, mention that your own run was noisy, move on.
Now type the query the other way round. There is also a 2026 paper, on the exact arm you own, with more demonstrations per task, where the VLA does not win at all.
Both were published. Neither is fraudulent. And the thing that was about to decide which one you believed was a sentence you typed before you had read either of them.
The idea in one paragraph
A pretrained vision-language-action model is not automatically better than a small policy trained from scratch on your task. What pretraining buys shows up on axes a success-rate column hides: the model still works when you swap the object, it still works when the instruction is worded differently, and it notices that a grasp failed instead of continuing serenely to the drop-off point. Scale is doing much of that work, not the letters V, L and A: a 450M model does not reliably beat a well-trained task-specific policy on the task it was trained for - one independent study has them tied there and another has the small VLA ahead - while a 3B one wins clearly on both. And on cheap hardware everybody is still bad, because the failure that dominates is not “picked the wrong object”, it is “the object slipped out of the gripper”.
Somebody already ran your experiment
In June 2026 two researchers at Hiroshima University published Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis (arXiv:2606.08881) - a benchmark on the same arm this course builds on. Four tasks. 100 teleoperated demonstrations per task, collected on the target hardware with randomised object poses. Four policies, each fine-tuned per task under identical data splits, each following its own authors’ recipe with only the interface adapted. Twenty evaluation episodes per model-task pair, 320 real-world episodes in total. The demonstration datasets were released.
That protocol is worth reading twice, because it is the thing being bought. Nobody at Hiroshima built any of the four models, so nobody there had a reason to tune one harder than another.
[Success rate, percent, on a low-cost SO-101 arm]
| Model | Pen Transfer | Color Sorting | Object Packing | Pen Placement | Average |
|---|---|---|---|---|---|
| ACT, trained from scratch | 75 | 0 | 10 | 50 | 33.75 |
| SmolVLA, 450M | 70 | 5 | 10 | 45 | 32.5 |
| Wall-X | 95 | 0 | 30 | 80 | 51.25 |
| π₀.₅, about 3B | 95 | 10 | 55 | 65 | 56.25 |
n = 20 trials · 4 SO-101 tasks, 100 demos each, per-task fine-tuning · 2026-06
Read the SmolVLA row against the ACT row. A pretrained vision-language-action model, fine-tuned on a hundred demonstrations of each task, came in one and a quarter points behind a policy trained from scratch on the same data. The paper’s own authors say the two are indistinguishable.
Read the π₀.₅ row against the same baseline and the picture inverts. Twenty-two and a half points is not a rounding artifact. Between those two rows sits the entire honest version of the pitch: pretraining helps, at 3B, on this hardware, and not at 450M.
Which means “a VLA beat ACT” is not a claim about architecture at all. On this benchmark the 450M VLA tied and the 3B VLA won by a mile, and what changed between those two rows is how much model and how much data went in before your demonstrations ever arrived.
Read the denominator before you read the table
Twenty trials per cell is small, and that matters more than any individual number in the table. The half-width of a 95% interval around a success rate measured over trials is
which says two things in plain language: uncertainty shrinks with the square root of how many times you tried, so four times the trials buys half the error bar, and it is widest when the rate sits near one half. Put and in and it comes out at 0.219.
The zeroes deserve the same caution in the other direction. ACT scoring 0% on Color Sorting over twenty trials does not mean the task is impossible for ACT; it means no success was observed inside a twenty-trial budget. That is a weaker statement, and the paper says so.
What pretraining actually bought
The success-rate table is the least interesting table in that paper. Two others say more.
The first is the failure taxonomy: for every failed trial, what went wrong. ACT’s failures were overwhelmingly state mismatch - the arm was somewhere the demonstrations never put it, and it kept going anyway. That appeared in 98.11% of ACT’s failed trials and in 45.71% of π₀.₅’s. That gap is the semantic grounding of a pretrained backbone showing up as a measurable quantity rather than a slogan.
The second is recovery: after something goes wrong mid-episode, does the policy do anything about it? ACT recovered 6.45% of the time. SmolVLA, 3.23%. Wall-X, 20.51%. π₀.₅, 30.77%.
Wider than the screen; scroll it sideways.
The mechanism is worth stating in one sentence, because it is what you will say when someone asks you why any of this pretraining business matters: a pretrained model has some notion of what the scene should look like, so it can notice that the scene does not look like that. ACT has learned a distribution over trajectories. Nothing in it represents “that went wrong”.
And then the humbling part. Grasp instability and repetition loops appear in over 90% of the failed trials of every model in that study, π₀.₅ included. The dominant failure on low-cost hardware is not semantic. It is a servo with backlash closing on a smooth object under a camera that cannot see the contact. No foundation model released so far fixes that.
The cliff
The second independent study is the one that reframes the whole argument. VLA-REPLICA (arXiv:2605.20774, May 2026) is a reproducibility-focused benchmark on an SO-101 rig with a published parts list at roughly $1,050 - a $200 arm, a $425 depth camera, a $14 webcam, a $153 light box and $216 of objects - covering ten tasks with 50 demonstrations per task, which is the number this course’s flagship experiment uses. Fifty in-distribution and forty out-of-distribution test scenes, and the rig rebuilt independently to check that the results held.
They reported two sets of numbers: on the objects and phrasings the policies trained on, and on new ones.
| Method | Familiar objects | New objects and counts |
|---|---|---|
| ACT | 0.18 | 0.075 |
| SmolVLA | 0.26 | 0.30 |
| π₀ | 0.34 | 0.30 |
| π₀.₅ | 0.54 | 0.35 |
Wider than the screen; scroll it sideways.
ACT loses 58% of its performance the moment the objects change. SmolVLA does not move. That flat line is the clearest published picture of what internet-scale pretraining transfers to a two-hundred-dollar arm, and note that it is invisible in the in-distribution column, where SmolVLA and ACT are eight points apart on numbers this small.
The same two models, two verdicts
Now put the independent SO-101 result beside the number SmolVLA’s own authors published: on real hardware, 50 demonstrations per task, SmolVLA 78.3% against ACT’s 48.3%.
Wider than the screen; scroll it sideways.
Different tasks, different operators, different amounts of effort spent tuning each side. Nobody has to be lying for both to be true. But the two results are not equal evidence, and the asymmetry is structural rather than moral: the SmolVLA team chose the tasks, chose the baseline’s hyperparameters, and chose which results to publish, and they are a talented group who know their own model better than they know ACT. The Hiroshima group chose nothing that favoured any of the four.
What you can defensibly say
Five statements survive the August 2026 evidence. They are what you should be able to defend in a comment thread.
- On a narrow, fixed, single-task setup with clean demonstrations and no language conditioning, ACT is competitive and extremely cheap. It trains in tens of minutes on a consumer GPU and needs no pretrained checkpoint.
- Add language-conditioned selection, distractors, object variation or multi-step sequencing, and ACT falls off a cliff while pretrained models do not. Color Sorting at 0%, and 0.18 dropping to 0.075, are the same finding twice.
- The advantage tracks the size and data of the pretrained backbone rather than the acronym - but state this carefully, because the two studies do not agree about the 450M model. On the Hiroshima tasks SmolVLA and ACT are a point apart, which is a tie. On VLA-REPLICA’s tasks SmolVLA is ahead of ACT in distribution (0.26 against 0.18) and four times ahead out of it (0.30 against 0.075). So “450M ties” is one study’s finding, not the field’s, and the safe version is: at 3B the advantage is large enough to survive both studies; at 450M it depends on whose tasks you run, and it is much clearer out of distribution than in it. If you catch yourself reaching for the crisp “450M ties, 3B wins”, notice that you are quoting the study that produced the tidier story.
- Recovery is the underrated axis, and success rate hides it. A six-point recovery rate against a thirty-one-point one is arguably a bigger practical difference than the success-rate gap, because it decides whether the robot needs a human standing next to it.
- Nobody is good at this hardware yet. The best model on the independent SO-101 benchmark averages 56%, and grasp instability dominates every model’s failures. A foundation model does not solve your SO-101.
Check yourself
1. Your fine-tuned VLA scores 8 out of 20 and your ACT baseline scores 7 out of 20 on the same task. What can you honestly claim?
That you measured no difference. The 95% interval around either rate at twenty trials is roughly plus or minus 21 points, and the gap is 5. The honest write-up is “20 trials each, 8 and 7 successes, indistinguishable at this sample size”, plus a statement of what you would need to separate them: either far more trials, or a comparison on an axis where the effect is larger, such as swapping the object or rewording the instruction. Reporting “the VLA edged ahead” would be a claim your own data cannot support.
2. Why is recovery rate often more decision-relevant than success rate?
Because it decides whether the robot can be left alone. Two policies with the same success rate behave completely differently in a real deployment if one of them abandons the episode on first contact failure and the other tries again. In the independent SO-101 study ACT recovered from 6.45% of its mid-episode failures and π₀.₅ from 30.77%. The mechanism is that a pretrained model carries some expectation of what the scene should look like, so a mismatch is detectable; a from-scratch policy has learned a distribution over trajectories with nothing in it that represents “this went wrong”.
3. The SmolVLA paper reports SmolVLA beating ACT by 30 points on real hardware. An independent lab reports them tied. How do you weight the two, and what does the disagreement itself teach?
Weight the independent result higher, not because the authors are dishonest but because of who chose what. The authoring lab picked the tasks, tuned both sides, and decided which results to publish, and it knows its own model far better than it knows the baseline. The independent lab picked nothing that favoured any entrant and published its datasets. What the disagreement teaches is that a single real-robot success rate is not a property of a model at all - it is a property of a model on a task set, tuned by a particular person. That is why the only number you should fully trust about your own robot is one you measured on it.
4. A company reports its model beating π₀.₅ by 17.5 points on its own 15-task suite. An independent lab measures it 5 points behind. Which one is wrong?
Probably neither. They are measurements over different task sets, and success rates do not transfer between task sets. The correct reading is that the company’s number tells you the model is strong on tasks its builders considered representative, and the independent number tells you what happens on tasks nobody chose with this model in mind. The second is what you care about, because your task was also not chosen with this model in mind.
5. ACT scored 75% on Pen Transfer and 0% on Color Sorting in the same study, with the same training budget. What single property of the task explains the gap?
Whether the instruction has to be understood. Pen Transfer is a fixed-scene, short, precise pick-and-place: the mapping from pixels to trajectory is nearly a function, and behaviour cloning learns functions well. Color Sorting requires selecting a target based on a language-expressed property, and ACT has no mechanism for grounding a word in the scene, so it does something plausible-looking and wrong every time. The moment a task requires the policy to be told which thing, an architecture with no language pathway is not a weak choice, it is the wrong one.
6. Without running more trials than you can afford, what result would convince you that pretraining bought you something on your own rig?
Change the condition rather than the count. Hold the trial budget fixed and evaluate both policies on objects you never demonstrated with: a different colour, a different shape, a different starting position, an instruction phrased in words that are not in your dataset. The published effect there is large - one baseline losing 58% of its performance while the pretrained model stayed flat - and large effects are the only kind a twenty-trial budget can see. Measuring a big difference on a hard condition beats measuring a small difference on an easy one.
Do this
1. Compute the noise yourself. Save this and run it; it needs nothing but the standard library.
import math
def half_width(p, n):
"""95% confidence half-width for a success rate p measured over n trials."""
return 1.96 * math.sqrt(p * (1 - p) / n)
for label, p, n in [
("one cell of a paper, 20 trials", 0.50, 20),
("a 4-task average, 80 trials", 0.50, 80),
("your own run, 20 trials", 0.35, 20),
]:
print(f"{label}: +/- {100 * half_width(p, n):.1f} points")
def gap_survives(p1, n1, p2, n2):
"""Is the distance between two measured rates bigger than the noise in it?"""
noise = 1.96 * math.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
verdict = "real" if abs(p1 - p2) > noise else "not measurable"
return f"gap {100 * abs(p1 - p2):.1f} vs noise {100 * noise:.1f} -> {verdict}"
print("pi0.5 vs ACT ", gap_survives(0.5625, 80, 0.3375, 80))
print("SmolVLA vs ACT ", gap_survives(0.3250, 80, 0.3375, 80))It prints plus or minus 21.9 points for a single twenty-trial cell, and it prints not measurable for the SmolVLA-against-ACT comparison that half the internet would have reported as a win. Now feed in your own two rates and your own trial count.
2. Rewrite your claim. Take the one-line conclusion you wrote after benchmarking your fine-tune against ACT and rewrite it in this shape: “n trials each, x and y successes, [gap real / not measurable at this sample size]. The condition I varied was ___. The condition I did not vary was ___.” Almost every honest robot-learning result fits that sentence, and almost no marketing claim does.
3. Classify your own tasks. List every task in your eval set and mark each as in-distribution or out-of-distribution relative to your demonstrations. If every row says in-distribution, you have not yet run the experiment that would show pretraining doing anything, and that is the run to do next: same policies, same trial count, new objects.
What you can now do
You can read a robot-learning benchmark the way its protocol allows rather than the way its abstract invites, compute whether a reported gap survives its own sample size, tell an author-reported result from an independent one and say why the difference matters, name the two axes where pretraining actually shows up - out-of-distribution performance and recovery - and write up your own ACT-versus-VLA comparison in a form that holds up when a stranger checks it.