Where you are. You can plan an openpi run and you know a network hop has appeared inside your control loop. This lesson is about the clock: why the model is always too slow, what covers for it, and where the model should physically live.
The arm never stops moving
Watch a VLA-driven arm do a pick and place. The motion is continuous. No stutter, no pause to think, no visible seam between one decision and the next.
Now time one forward pass of the model driving it. Say it comes back at 200 milliseconds, which would not be surprising for a multi-billion-parameter model on a card you own. The controller underneath wants a fresh command every twenty.
So nine out of ten ticks - more, usually - have no new answer behind them. Something is feeding the servos during those ticks, and it is not the model. The smoothness you are watching is not the model keeping up. It is the model having answered a question you asked a while ago, in more detail than you asked for.
The idea in one paragraph
Deploying a robot policy is a scheduling problem, not a modelling one. The forward pass takes longer than one control period - under the best published conditions barely longer, on hardware you can afford an order of magnitude longer - and every system in this module reconciles the two the same way: predict a chunk of future actions in one pass, execute it while the next chunk is being computed, and let the slow part and the fast part run on different clocks. Where the model physically runs then sets your latency floor, and on-robot, same-network and cloud are three genuinely different systems rather than three configurations of one. The published latency numbers in this field are thin - the fastest one anybody has printed for a 3B model is 35.9 Hz, on the most expensive accelerator in common use, with every knob turned down - so the number that decides your design is one you will have to measure.
The gap, drawn to scale
Take the best published figure. NVIDIA reports GR00T N1.7 running at 35.9 Hz on an H100 with TensorRT, at four denoising steps, with a single camera view.
Wider than the screen; scroll it sideways.
Every qualifier in that sentence is load-bearing. An H100 is a datacentre card. TensorRT is a compilation step, not a model.eval(). Four denoising steps is the diffusion action head run at its cheapest setting. One camera view means the wrist camera or the scene camera, not both.
And after all that, 35.9 Hz is 27.9 milliseconds per pass, which is longer than the 20 millisecond period of a 50 Hz control loop. The best number in the field, under the friendliest conditions anyone has published, still does not fit inside one tick.
What fills the gap
The answer is the same one ACT gave you: predict many steps at once. π₀ emits 50 future actions per forward pass. The robot host holds them and plays them out one per tick while the next request is in flight.
That trade is worth stating precisely, because it is the design tension in every system in this module. By construction, every action in a chunk after the first is open-loop with respect to the world: it was chosen from an observation the robot has since moved past. A longer chunk hides more latency and reacts to less. A shorter chunk reacts more and demands a faster model. Fifty actions is the number π₀ landed on, and it is a choice, not a law.
Synchronous is leaving half the throughput on the floor
The naive schedule is: ask, wait, move, ask again. During every wait the arm is stationary, because the chunk it was executing has run out and the next one has not arrived.
Wider than the screen; scroll it sideways.
The SmolVLA team measured this on their own setup and reported roughly 30% faster task completion - 9.7 seconds against 13.75 - and about twice the throughput compared with running synchronously. Their number, their tasks, so treat the magnitude as indicative. The direction is not in doubt, because the mechanism is arithmetic: if inference and execution can overlap, the loop’s period is the longer of the two rather than their sum.
The knobs that actually move latency
Four things change inference time by a lot, and none of them is “buy a bigger card”.
| Knob | What it does | Where it shows up |
|---|---|---|
| Denoising or integration steps | how many passes the action head makes per chunk | GR00T’s published figure is at four denoising steps |
| Camera views | each view is another image encoded every pass | GR00T’s published figure uses one view |
| Visual tokens per frame | how much of the image survives into the transformer | SmolVLA cuts to 64 per frame, from around 1024 |
| Backbone depth | how much of the language model you actually run | SmolVLA runs half of its VLM’s layers |
The action representation is the fifth knob, and it is decided before you ever load the model. Flow matching integrates a smooth velocity field in a handful of steps, which is exactly why π₀ can support 50 Hz control. π₀-FAST predicts discrete action tokens autoregressively instead, which trains around five times faster and decodes significantly slower. Same family, same lab, opposite ends of the trade, and you pick one at checkpoint-selection time.
Where the model lives
Wider than the screen; scroll it sideways.
On the robot. Inference is the cheap regime - GR00T runs on a single 16 GB card, and NVIDIA lists Jetson AGX Thor and Orin modules alongside desktop cards for it. SmolVLA is small enough that its authors claim it runs on a CPU. There is no network leg to fail mid-grasp, and nothing to go wrong when your home connection does. The cost is a hard ceiling on model size, and the fact that fine-tuning still happens elsewhere: GR00T wants 40 GB or more to train, which no edge module has.
On a workstation on the same network. One hop, on a switch you own, to a card with 24 GB or more. This is the sane default for course work: you can run models the robot could never hold, and the latency added is small and yours to measure.
In the cloud. Correct for training, poor for a closed loop. A rented 80 GB card is the only way to fully fine-tune a 3B model, and it is a bad place to serve one from, because a round trip you do not control has landed inside a real-time system.
Clock separation gets more extreme the more physics a robot has. Figure’s Helix 02 is described as three tiers - a semantic layer that “reasons slowly” at a rate Figure does not publish, a visuomotor transformer at 200 Hz, and a 10-million-parameter whole-body controller at 1 kHz - and the reason is not model design fashion. Balance needs kilohertz because falling over is fast; language does not, because sentences are slow. Your SO-101 has two tiers rather than three, because the servos handle their own position loop and the arm cannot fall over. Lesson 8 draws all three architectures side by side; the point here is only that the rate of each tier is set by the physics it answers to, not by the model that fills it.
Without hardware
The measurement harness in this lesson already runs on a laptop; what is missing is the machine at the far end of the wire.
- Instead of the stand-in, time your actual fine-tuned checkpoint: warm up, then 60 calls, and report p50, p95 and worst.
- Measure this: the chunk length your simulated control rate requires at that latency, and the fraction of a chunk consumed by one forward pass.
- What you lose: the two latencies that are properties of hardware - the serial bus and the camera pipeline. State them as assumptions with a source rather than as measurements, and say in the writeup that your end-to-end budget is missing its two physical terms.
Check yourself
1. Your policy takes 200 ms per forward pass and your arm wants a command every 20 ms. Why does the arm not stutter?
Because each forward pass returns a chunk of many future actions rather than one, and the robot host plays them out locally at 50 Hz while the next request is in flight. The model is answering roughly five times a second; the arm is being commanded fifty times a second; the chunk is the bridge. It stutters only if the chunk runs out before the next reply lands, which is why you fire the next request early rather than when the buffer is empty.
2. A longer action chunk hides more latency. What does it cost?
Feedback. Every action in a chunk after the first was chosen from an observation the robot has already moved past, so it is executed open-loop with respect to whatever has happened since. A long chunk means the policy commits further into a future it cannot see, and it is exactly the wrong choice for contact-rich or unpredictable moments. Chunk length is a dial between latency tolerance and reactivity, and the right setting depends on how fast your scene changes, not on what the paper used.
3. GR00T N1.7 is reported at 35.9 Hz. Why can you not carry that number over to your own deployment?
Because it names a configuration, not a model. It is an H100 with TensorRT compilation, four denoising steps and a single camera view. Change the card, skip the compilation step, add a wrist camera, or raise the denoising steps for quality, and the number moves - each of those independently. It is also a vendor figure with no independent replication. The transferable content is the shape: even under the friendliest published conditions, one forward pass is longer than one 50 Hz control period.
4. Why report p95 latency instead of the mean?
Because the mean does not stall the arm. A loop survives on its worst common case: if one pass in twenty takes three times as long, that pass is the one that empties the chunk buffer and leaves the controller repeating a stale command mid-grasp. Sizing the chunk against the mean means designing for the case that never hurts you. Report p50 for a sense of the typical cost, p95 for the design, and the worst sample you saw so nobody is surprised later.
5. You have a Jetson-class module on the robot and a 24 GB workstation in the same room. Where do you serve the policy from, and what would change your answer?
For course work, the workstation: one hop you control, and enough memory to run models the module cannot hold. Move to the on-robot module when the robot has to leave the network, when a dropped link during a grasp is unacceptable, or when the model is small enough that the memory ceiling stops binding. Either way, fine-tuning happens somewhere else - the training memory floor for a 3B model is far above any edge module - so the choice is only about serving.
6. π₀ and π₀-FAST come from the same lab and share a backbone. Why would the choice between them change your deployment before you write any code?
Because they represent actions differently, and that decides decoding cost. π₀ generates a continuous chunk by integrating a flow field in a small fixed number of steps, which is fast at inference and supports a 50 Hz loop. π₀-FAST generates discrete action tokens autoregressively, which trains around five times faster and decodes significantly slower. If your bottleneck is training budget, one of them is right; if your bottleneck is the control loop, the other is. It is a scheduling decision wearing an architecture costume.
Do this
1. Measure a policy the way you would measure a service. This runs on any laptop in about twelve seconds, with a stand-in for the model, so you can build the harness before you have the GPU.
import math, random, time
random.seed(0)
def fake_forward_pass():
"""Stand-in for a policy: slow, with the long tail real ones have."""
delay = random.gauss(0.180, 0.020)
if random.random() < 0.05:
delay += 0.25 # the occasional stall you must plan for
time.sleep(max(delay, 0.0))
def measure(call, warmup=5, trials=60):
for _ in range(warmup):
call() # never time a cold start
samples = []
for _ in range(trials):
t0 = time.perf_counter()
call()
samples.append(time.perf_counter() - t0)
samples.sort()
return samples[len(samples) // 2], samples[int(0.95 * len(samples)) - 1], samples[-1]
p50, p95, worst = measure(fake_forward_pass)
print(f"p50 {1000*p50:.0f} ms p95 {1000*p95:.0f} ms worst {1000*worst:.0f} ms")
CONTROL_HZ, CHUNK = 50, 50
for name, latency in [("median", p50), ("p95", p95), ("worst", worst)]:
ticks = math.ceil(latency * CONTROL_HZ)
print(f"{name:<7} {ticks:>3} of the {CHUNK} actions in a chunk are spent waiting")It prints a median near 180 ms, a p95 near 230 ms, and a worst case past 450 ms - and it tells you that at the worst case roughly half a 50-action chunk is consumed just waiting for the next one. Now swap fake_forward_pass for your real policy call and re-run it on the machine that will actually serve.
2. Add the network leg. Ping your policy server from the robot host a few hundred times and take the p95 round trip, not the average. Add it to your p95 inference time. That sum, times your control frequency, is the number of actions a chunk must cover before the arm runs dry. Compare it to 50.
3. Record the configuration next to the number. Card, precision, compilation on or off, number of camera views, denoising or integration steps, image resolution, batch size. A latency measurement without those is not reproducible, and a latency measurement with them is the thing this field is currently missing.
What you can now do
You can explain why every VLA predicts a chunk instead of a step, size that chunk against a measured p95 latency rather than a paper’s default, name the four settings that move inference time and the action-representation choice that fixes its ceiling, choose between an on-robot, same-network and cloud deployment with a reason attached to each, and report a latency figure with enough configuration next to it that somebody else could get the same number.