Where you are. You can name the three tiers and say what crosses each boundary. This lesson takes the top tier apart, asks where the second goes, and turns the answer into three rules you will use for the rest of the module.
The same order, twice
You give the robot the same instruction on two mornings.
Monday: one camera view, a short prompt, the model told to think as little as it can. The answer comes back in under half a second, and it is wrong. It reaches for the pen nearest the camera instead of the one you asked for.
Tuesday: three camera views, a longer prompt that lists the available skills properly, thinking turned up. The answer comes back after two and a half seconds. It is right.
Both are legitimate configurations of the same system. Between them sits five times the wall clock, and there is no third setting that gives you Tuesday’s answer at Monday’s speed.
That is the constraint. It does not go away with a faster chip, and this lesson is about building around a number you do not get to choose.
The idea in one paragraph
A planner call is slow for four reasons that add up rather than trade against each other: it is a large autoregressive model, it thinks before it answers, images are expensive in tokens, and it usually lives at the other end of a network. Together, budget something like 0.3 to 3 seconds per call, which is 0.2 to 2 Hz. That is slower than almost anything a robot does, and you cannot engineer it away. So the design runs in the other direction. Every skill is built to take longer than one planner call, every skill call blocks until the robot has physically finished, and no loop that has to correct anything is ever closed through the planner.
Where the second goes
| The cost | Why it is there | Your lever |
|---|---|---|
| Looking | every camera frame becomes hundreds or thousands of tokens before the model has read a word of your prompt | fewer views, smaller images, a lower frame rate |
| Thinking | reasoning models generate tokens you never see before they generate the answer | the thinking-level dial, if the API exposes one |
| Answering | output tokens come out one at a time, each one a full forward pass | ask for a short tool call, not an essay |
| Travelling | the model is almost never on the robot | co-locate it, or accept the tax |
The important word in that table is add. Turning thinking down does not make the pictures cheaper. Cropping the images does not make the decoding faster. Each row is a separate cost with a separate lever, and a latency problem you have not decomposed will get “fixed” by pulling the wrong one.
Google’s robotics streaming API is the clearest published example of the levers being real. It exposes a thinking-level setting whose documentation recommends medium as the balance point between latency and accuracy, and it caps video input at one frame per second. Both are latency decisions made visible in the interface.
Why a faster accelerator does not fix it
It is tempting to file this under “hardware will solve it”, because at the tier below, hardware really does. A 2.7B-parameter vision-language-action policy runs end to end in about 31 milliseconds on an RTX 4090 and about 53 milliseconds on a Jetson Thor. Move to a datacentre-class accelerator and the same model finishes in single-digit milliseconds.
The planner does not scale that way, for two reasons. It is one to two orders of magnitude larger, and it does not do one forward pass. It generates a token, then another, then another, hundreds of times, each one a full pass through the network with the previous tokens as context. Scale is the smaller half of the problem; serialisation is the larger one. Even in the same family, pushing a policy from 2.7B to 81B parameters takes a single forward pass from about 3 milliseconds to about 104 milliseconds on the same top-end hardware, and a planner call is many such passes back to back.
Slow is a fit, not a defect
Look at what the planner is actually for: split a goal into steps, order them, choose which skill to call next, judge whether the last one worked, notice that the plan is no longer valid, decide to ask a human. Every item on that list is naturally a once-every-few-seconds decision. None of them gets better by happening thirty times a second.
The mismatch only appears when you ask the planner to do something that belongs lower down. A planner asked to sequence a task is well matched to its clock. A planner asked to nudge a gripper is two orders of magnitude too slow, and no prompt engineering repairs that.
What the latency forces
Skill calls block
Here is a tool declaration from a shipping robotics API, with the load-bearing line in the middle:
{
"name": "navigate",
"description": "Navigate the robot to a named waypoint.",
"behavior": "BLOCKING",
"parameters": {
"type": "OBJECT",
"properties": {"name": {"type": "STRING"}},
"required": ["name"]
}
}BLOCKING means the model waits for the robot to physically finish before it is given the result. The loop is: the model emits a tool call, your code runs it against the robot SDK, you send back a response carrying the matching call id, and the session stays open while the model picks the next action from what it just learned.
Skills must outlive a call
If the planner call for the next skill is issued the moment the current skill starts, then one cycle lasts as long as whichever of the two is slower, and the fraction of the time the robot is actually working is
which is a plain way of saying: once a skill lasts longer than a planner call, the planner is free. Below that line, every millisecond the model spends thinking is a millisecond the arm spends still.
Wider than the screen; scroll it sideways.
This is a design rule about how you carve up skills, not a runtime tuning knob, which is why it belongs here rather than in a performance appendix.
| Candidate skill | Roughly | Verdict |
|---|---|---|
move_joint(3, +0.02) | 30 ms | a policy action wearing a tool’s clothes; the planner cannot possibly steer at this rate |
grasp(object="red mug") | 2-4 s | right: longer than a call, short enough that a failure is caught early |
tidy_the_desk() | 3 min | too coarse: nothing to steer, and the first evidence of trouble arrives at the end |
No loop closes through the planner
Wider than the screen; scroll it sideways.
Take any loop that corrects an error and ask how long one turn takes. Closed at the policy tier it turns every 33 milliseconds, and a mug sliding at 0.3 metres per second moves a centimetre between glances. Closed through the planner it turns every two seconds, the same mug moves sixty centimetres, and the correction that finally arrives was computed from a photograph of somewhere the mug used to be.
Overlapping the thinking with the acting
The frontier version of this idea is to keep the model’s session open and stream, so it can reason about the next step while the robot is still executing the current one instead of stopping to think between steps. Google describes its embodied-reasoning model as running over a bidirectional streaming endpoint for exactly that reason.
Check yourself
1. Name the four independent costs inside a planner call, and give the lever for each.
Looking, thinking, answering and travelling. Looking is image tokens, and the lever is fewer views, smaller images or a lower frame rate. Thinking is the reasoning tokens generated before the answer, and the lever is the thinking-level setting. Answering is serial token decoding, and the lever is asking for a compact tool call rather than prose. Travelling is the network, and the lever is co-locating the model or accepting the tax. They add rather than trade, so pulling one lever leaves the other three costs exactly where they were.
2. A policy of 2.7B parameters runs in about 31 ms. Why is a planner not simply “the same thing, ten times bigger, so 310 ms”?
Because the planner generates its answer one token at a time, and each token is a full forward pass. Size accounts for part of the gap; serialisation accounts for more. A few hundred output tokens means a few hundred passes back to back, and reasoning models add a further block of tokens that are generated and never shown. That is why the planner’s cost is quoted in seconds while the policy’s is quoted in milliseconds, and why the ratio will not collapse with the next generation of accelerators.
3. Why does "behavior": "BLOCKING" make the system faster rather than slower?
It does not make anything faster. It hides the latency by overlapping it with motion that was going to happen anyway. Because the call does not return until the robot has physically finished, the model’s thinking time is spent while the arm is moving, so it costs nothing on the wall clock. The alternative, returning immediately and letting the model poll, puts the model’s latency in series with the motion and forces you to invent your own progress protocol.
4. Your skills average 400 ms and your planner call takes 1.2 s. What fraction of the time is the robot working, and what are your two options?
About a third, since the cycle is paced by the slower of the two and the robot only works for 400 ms of every 1.2 s. Either fold several skills into one larger skill so the unit of work outlives a planner call, or push the decision downwards so the policy makes it without asking. Which one you choose depends on whether the planner’s judgement was actually needed between those small steps. If it was not, it was never a skill boundary.
5. Streaming lets the planner think about step n+1 while step n executes. What does that buy, and what does it cost?
It buys the removal of a stop-and-think gap between every pair of steps, so the robot moves continuously. It costs freshness: the plan for step n+1 was computed from observations taken before step n finished, so it describes a world that no longer exists by the time it is applied. The more overlap, the staler the state the planner reasoned from, and stale state is what makes replanning fail silently rather than loudly.
Do this
Twenty minutes. Part 1 uses tooling you already run.
1. Measure your own planner latency. Take any agent stack you already have wired up, with any MCP server attached. Time twenty end-to-end round trips from prompt to tool call, at your default settings, with one image attached and with none. Report the median and the slowest of the twenty, not the mean; the tail is what the robot experiences.
You now have your own number in place of this lesson’s 0.3-3 second estimate. Every design rule below uses your number.
2. Carve the skills. Run this with your measured value.
PLANNER_S = 1.2 # replace with your own median
def busy_fraction(skill_s, plan_s, overlapped=True):
cycle = max(skill_s, plan_s) if overlapped else skill_s + plan_s
return skill_s / cycle
for skill_s in (0.2, 0.5, 1.0, 2.0, 4.0, 8.0):
over = busy_fraction(skill_s, PLANNER_S)
seq = busy_fraction(skill_s, PLANNER_S, overlapped=False)
print(f"{skill_s:4.1f} s skill overlapped {100 * over:3.0f}% sequential {100 * seq:3.0f}%")Find the shortest skill duration that keeps the robot busy at least 90% of the time when overlapped. That number is the floor on skill granularity for your stack, and it is the answer you will bring to the next lesson.
3. Re-carve a bad skill list. Here is a plausible-looking set of tools somebody would write on day one: open_gripper(), close_gripper(), move_to(x, y, z), move_up(cm), look(). Using your floor from part 2, mark each one as too fine, right, or too coarse, then rewrite the list so every entry outlives a planner call. Keep your rewrite. You will implement it two lessons from now.
What you can now do
You can account for a planner call’s latency in four independent parts and name the lever for each, explain why the gap between planner and policy timing will not close with better hardware, and state honestly which parts of the latency budget are published and which are estimates. You can apply the three rules that latency forces: block the call so the motion hides the thinking, size every skill to outlive one call, and keep every correcting loop below the planner. And you know the price of overlapping thought with motion, which is the staleness the rest of this module has to handle.