Where you are. You have a planner that calls skills, a monitor that judges them, and a way to score the whole thing. This lesson puts numbers on how long each part takes, and shows what you do about the parts that are slow.
The call where everyone talks over everyone
You have been on this call. There is about half a second of lag. Someone asks a question, you start answering, they start repeating the question because they think you did not hear, you both stop, you both start again with an apology.
Nobody’s connection got worse. Everyone is acting on a picture of the conversation that is half a second old, and the pictures disagree.
Notice how it gets fixed, because the fix is never “get lower latency”. You cannot. It gets fixed by changing the protocol. People start taking turns. Somebody says “go ahead”. Silence stops meaning “your turn” and starts meaning “they are thinking”. The lag is exactly what it was; it stopped mattering.
Now put that on your desk. The arm is moving at 20 cm/s and the planner takes 1.2 seconds to answer. If the arm stops to listen, every step of the task begins with a lurch. If it does not stop, the planner is reasoning about an arm that has already travelled 24 cm past the picture it was given.
The idea in one paragraph
An agentic robot runs on three clocks at once: a planner at fractions of a hertz, a learned policy at tens to hundreds of hertz, and a joint controller at hundreds to a thousand. Each tier is one to two orders of magnitude slower than the one below it, and no amount of engineering closes those gaps, because they come from different physics: matrix multiplies on a large model, matrix multiplies on a small one, and a current loop in a motor driver. So the design job is never “make the planner fast”. It is to arrange that every tier always has something to execute while the tier above it thinks. Get that right and a three-second planner is invisible. Get it wrong and a three-second planner is three seconds of a robot standing still, thirty times a task.
Wider than the screen; scroll it sideways.
Three clocks
| Tier | What runs here | One decision takes | Rate |
|---|---|---|---|
| Planner | the LLM or VLM: what to do next, did it work, replan | 0.3 to 3 s | 0.2 to 2 Hz |
| Policy | the learned visuomotor model emitting action chunks | 3 to 50 ms | 10 to 200 Hz |
| Controller | joint servo loops, torque limits, safety stops | 1 to 5 ms | 200 to 1000 Hz |
Three tiers, not two. The two-brain framing is a useful story and it hides the tier that actually keeps the robot safe. Collapsing the policy and the controller into one layer is the usual way this architecture gets described, and it matters here because they have completely different failure modes: a slow policy makes the robot clumsy, a slow controller makes it dangerous.
The policy tier is the one you can measure
End-to-end measurements for a 2.7B-parameter vision-language-action model, running in BF16, from one published benchmark sweep. These are somebody else’s numbers on somebody else’s model, so treat the ratios as the durable content and re-measure the absolutes on your own stack:
| Hardware | Latency | Achievable rate |
|---|---|---|
| Jetson Thor (on the robot) | 52.6 ms | 19 Hz |
| RTX 4090 (a desk beside the robot) | 31.1 ms | 32 Hz |
| A100 | 16.2 ms | 62 Hz |
| H100 | 6.2 ms | 163 Hz |
| B100 | 3.2 ms | 314 Hz |
Model size moves it as hard as hardware does. On the same B100, going from 2.7B to 9.1B parameters takes you from 3.2 ms to 13.6 ms, and an 81B version lands at 104 ms, which is 9.6 Hz. That is a planner-tier rate on policy-tier hardware, and it tells you why nobody ships an 80B action model as the fast brain.
One more measurement, because it changes what you buy. Breaking the 3.2 ms down on that B100: vision encoding 0.40 ms, the language backbone 1.87 ms, action prediction 0.91 ms. The first two are compute-bound. The action head is memory-bound, at about 54 floating-point operations per byte moved. Practical consequence: a faster GPU helps the part that is already fast and barely touches the part that is not.
Where the GPU sits beats which GPU it is
If inference runs off the robot, add the round trip:
| Link | Added latency |
|---|---|
| 10 gigabit Ethernet | +3.3 ms |
| WiFi 7 | +8.4 ms |
| 5G | +27.8 ms |
| A distant cloud region | +100 ms and up |
The planner tier is the one you cannot measure
This is the honest gap in the current literature, and you should know it is a gap rather than assume you failed to find the number.
No frontier lab publishes per-call latency for an embodied planner. What is documented is the shape of the dial rather than its readings. Google’s robotics API exposes a thinking level and recommends the middle setting as the balance between latency and accuracy, which is an explicit statement that reasoning quality is bought with time. Video into the planner is capped at one frame per second. The bidirectional streaming endpoint is described as letting the model reason about the next step while the robot keeps moving, rather than stopping to think, which is the same protocol change as the video call.
So the 0.3 to 3 second figure in the table above is an engineering estimate, not a published measurement. Treat it as a starting budget, then replace it with your own numbers on day one: send the same request a hundred times and record the median and the 95th percentile. The tail is what your design has to survive, not the median.
Three ways to hide the delay
1. Blocking skill calls, at the planner tier
The single most valuable field in Google’s shipped robotics tool schema is "behavior": "BLOCKING" on a tool definition. It tells the model that the call will not return until the robot has physically finished.
What that removes is polling. A tool that returns the moment the command is accepted leaves the model with a question it can only answer by asking again: is it done yet? Every one of those is a full round trip. Worse, it is a decision made about a robot that is still moving, which is how you get a command collision: the model, unsure whether the arm arrived, issues the next skill on top of the one still running.
With a blocking call the model’s waiting is free. One round trip per skill, the robot moving for all of it, and the planner physically cannot interleave a decision into the middle of a motion because it has not been given a turn. That is the turn-taking fix from the video call, written as one field in a JSON schema.
Wider than the screen; scroll it sideways.
What blocking does not do is close the gap between skills. The model still spends its full latency choosing the next skill after the last one reports back, and the arm is genuinely idle for that. Sizing skills so that gap is small is the rule at the end of this lesson.
2. Asynchronous action chunking, at the policy tier
A policy does not emit one action, it emits a chunk of them: half a second of future motion. The obvious optimisation is to execute the current chunk while computing the next. Done naively it breaks in two ways, and both have names worth knowing because you will see them on a plot before you see them in a log.
- Action waiting. The chunk runs out before the next one arrives. The arm stops mid-motion, then resumes. Looks like stutter.
- Action jumping. The new chunk was computed from an observation taken before the last few actions executed, so it disagrees with what the arm just did. The command jumps. Looks like a twitch, and on a stiff arm it sounds like one too.
3. Real-time chunking, which is the good fix
Physical Intelligence’s answer is the cleanest idea in this lesson: treat the seam between chunks as an inpainting problem.
You know how long inference will take. So you know which actions from the current chunk are guaranteed to have executed by the time the new chunk arrives. Freeze those. Ask the model to generate the rest of the new chunk conditioned on that frozen prefix, exactly the way an image model fills a masked region so that it matches its surroundings. The new chunk is now consistent with the motion already committed, so there is nothing to jump from.
Wider than the screen; scroll it sideways.
It is an inference-time change with no retraining, and it works on any diffusion or flow-based policy. Which is the useful pattern to notice: two of the three mechanisms on this page are protocol changes rather than model changes.
The rule this leaves you with
If a planner call costs seconds and the skill it chooses takes seconds to execute, the fraction of wall-clock time the robot spends doing nothing is
| Planner latency | Skill 0.5 s | Skill 2 s | Skill 5 s | Skill 10 s |
|---|---|---|---|---|
| 0.3 s | 38% idle | 13% | 6% | 3% |
| 1.0 s | 67% idle | 33% | 17% | 9% |
| 3.0 s | 86% idle | 60% | 38% | 23% |
Two ways to fix a skill that is too short. Batch it into a larger one, so open_gripper and move_to and close_gripper become pick(object). Or push the decision down a tier, so the policy makes it at 30 Hz instead of the planner making it at 1 Hz. Both are the same move: take the decision off the slow clock.
Without hardware
Steps 1 to 3 need only a planner endpoint and your own skill layer, so they run unchanged.
- Instead of a policy on an arm, log the commanded joint positions through a chunk boundary in your simulated rollout and plot them. The stutter at the seam is a property of chunking, not of servos, so it is visible here.
- Measure this: your own median planner latency, your mean skill duration, and the idle fraction that falls out of them. All three are real numbers on this path.
- What you lose: the two physical terms. Serial-bus round trip and camera pipeline latency cannot be measured without the hardware, so carry them as cited assumptions and say so where the budget is stated.
Check yourself
1. Your planner takes 1.2 s per call and your skills average 0.8 s. What fraction of the run is the robot idle, and name two fixes that do not involve a faster model.
About 60%, since . First fix: make the skills bigger, so one planner call buys more motion. Merging three 0.8 s primitives into one 2.4 s skill drops the idle fraction to 33%. Second fix: make the calls blocking, so the model’s thinking time overlaps a motion that is already running rather than sitting between motions. A third, if the planner genuinely needs to decide often, is to move the decision down to the policy tier where it costs 30 ms instead of 1.2 s.
2. Why is an 80M-parameter model the fast half of a two-tier system rather than a 7B one?
Because the budget is time, not accuracy. A 200 Hz control rate leaves 5 ms per decision, and a 7B model does not fit in 5 ms on hardware you can bolt to a robot: measured latencies for a 2.7B model run 53 ms on an embedded board and 31 ms on a desktop GPU. Parameters are spent where there is time to spend them, which is the tier above. The fast tier gets the smallest model that can turn the slow tier’s instruction into smooth motion.
3. Someone benchmarks two inference servers and reports 3.2 ms and 6.2 ms per action. What is the first question to ask?
Where the network is. A 3 ms difference in raw inference is smaller than the link: WiFi 7 adds about 8.4 ms and 5G about 27.8 ms, so the slower server on Ethernet beats the faster one over WiFi by a wide margin. The second question is model size, since going from 2.7B to 9.1B parameters on identical hardware cost more than a 4x in latency. Raw inference time is often the smallest term in the end-to-end budget.
4. Explain action jumping to someone who has not seen it, and say why freezing a prefix fixes it.
The policy emits half a second of future motion at a time. While the arm executes chunk one, the model computes chunk two from an observation taken at the start of the computation. By the time chunk two arrives, the arm has moved on, so chunk two’s first action assumes a position the arm has already left, and the command discontinuously jumps. Freezing the prefix removes the disagreement: you compute which actions will certainly have executed during the inference window, hold those fixed, and generate the remainder conditioned on them, so the new chunk is constructed to continue the motion rather than to replace it.
5. Why is "behavior": "BLOCKING" on a tool definition a latency mechanism rather than a convenience?
Because it moves the model off the critical path by construction. A non-blocking call returns immediately, so the planner must poll or guess when the motion finished, and every model call it spends doing that is a call during which nothing is moving. A blocking call returns when the physical action completes, which means the entire model round trip happened while the robot was busy. The planner’s latency stops being time the robot loses and becomes time the robot was going to spend anyway.
6. A colleague proposes exposing set_joint_angles(j1..j6) as a tool so the model has fine-grained control. What goes wrong?
The model ends up inside the control loop, three orders of magnitude below the rate the transport can sustain. A tool round trip costs hundreds of milliseconds to seconds; joint control needs 1 to 5 ms. The arm will move in visible, unsafe steps, and no amount of prompt work will help because the limit is the protocol. Worse, it moves safety enforcement above the network, so a dropped connection mid-motion has no recovery path. Expose skills whose duration exceeds a planner round trip, and keep joint-level commands behind them.
Do this
Measure your own budget instead of using mine. About an hour.
1. Time your planner honestly. Send the same tool-use request 100 times with a representative image attached, and record the full round trip from your process, not the server-reported time. Report the median and the 95th percentile. Then repeat with the reasoning effort turned up and again with an extra image. You now have three numbers and a sense of which knob costs what.
2. Time your skills. Instrument the skill layer to log wall-clock duration per call. Sort them. Any skill shorter than your median planner latency is a design bug: it will spend more time being authorised than being executed.
3. Compute the idle fraction for your own stack. Use with your measured median and your mean skill duration . Write the percentage down. If it is above 25%, decide now whether you are merging skills or moving decisions down a tier, and write that decision next to the number.
4. Watch a chunk seam. If you have a policy from the earlier modules, log the commanded joint positions at full rate through a chunk boundary and plot them. Look for the stutter and the step. Then compute how many actions your inference latency covers at your control rate. That count is the prefix a real-time chunking scheme would freeze, and seeing it as a specific number of timesteps makes the mechanism concrete.
What you can now do
You can name the three clocks in an agentic robot, state the period of each, and explain why the gaps between them are physical rather than engineering debt. You can budget a policy from published inference measurements, add the network tax, and say why where the GPU sits usually matters more than which GPU it is. You can size a skill so the robot is never idle waiting for permission, compute the idle fraction to prove it, and name the three mechanisms that hide inference latency: blocking calls at the top, asynchronous chunking in the middle, and prefix-frozen inpainting at the seam.