Where you are. You have built a planner over learned skills once already. This lesson is about what it takes to make that a specialism rather than a demo, and it starts with the one property of physical tools that has no software equivalent.
The call that timed out
Your planner issued place(mug, shelf). Thirty seconds later, no response.
In every system you have shipped, you know the drill. Check whether the write landed. Look for the idempotency key. Retry, or surface it, or reconcile on the next pass. The call either happened or it did not, and with a little care you can find out which.
Now stand in the room. Right now the mug is in exactly one of these states: on the shelf; still in the gripper, held two centimetres above the shelf; on the floor; or wedged against the shelf edge with a servo drawing stall current and warming up. The planner knows none of that. The transport told it nothing, because the transport was never carrying that information.
The state is in the room. The only way to read it is to look, with a sensor, on purpose, as a deliberate step you wrote.
That gap, between what the call returned and what the world is, is the entire track.
The idea in one paragraph
Agentic robotics is the discipline of connecting a slow, fallible reasoner to a fast, irreversible actuator without lying to either one. Almost all of your tool-use architecture transfers directly: schemas, capability scoping, timeouts, structured errors, tracing, approval gates, evaluation harnesses. What does not transfer is the assumption underneath all of it, which is that a call either happened or did not, and that a failed call left no residue. Physical calls always leave residue. So the interface you design is not a function signature; it is a contract with a precondition that perception checks, a postcondition that perception verifies, and an abort path that says what the machine does with the object it is currently holding. Get that contract right and the agent layer becomes reliable. Get it wrong and you have built a very sophisticated way to knock things over.
What transfers, unchanged
More than you would expect, which is the reason this track compounds so hard for you.
| Thing you already do | Holds here? |
|---|---|
| Typed tool schemas, narrow arguments, no free-text side channels | Yes, and the pressure toward narrow arguments is stronger |
| Capability scoping: this agent may do these things, not those | Yes, and now it has a physical blast radius |
| Timeouts and cancellation as first-class design | Yes, with a new question: what does cancel mean mid-motion |
| Structured errors over prose errors | Yes, and the taxonomy changes shape |
| Tracing every call with inputs, outputs and timing | Yes, and the trace now needs video attached |
| Evaluation over many trials rather than one demo | Yes, and it is the part most people skip |
| Human approval gates on dangerous operations | Yes, and “dangerous” is now literal |
The four things that do not transfer
Idempotency is a perception problem. In software you achieve it with a key and a lookup. Here, “has this already happened” is a question about the room, answered by a camera and a classifier that is sometimes wrong. Every idempotent skill needs a check that is itself a perception task with its own failure rate.
Failure leaves residue. A failed HTTP call leaves your process exactly as it was. A failed grasp leaves an object somewhere new, possibly on the floor, possibly still in the gripper. Recovery is not a code path around the call; it is another skill, with its own preconditions, that has to run before the retry is even meaningful.
Partial success is the normal case. The mug reached the shelf and is resting at an angle against a book. Was that success? Your postcondition has to answer, and if it answers by asking a vision-language model, that answer is a probability wearing a boolean’s clothes.
The two ends run at different rates and neither can wait for the other. Your reasoner takes a second or more. The policy underneath runs tens of times per second and cannot pause for a decision. So the reasoner never sits inside the loop; it sets goals between loop runs. That split is the architecture, and skill calls are the only place the two layers touch.
Wider than the screen; scroll it sideways.
The contract, written out
The failure surface, side by side
Wider than the screen; scroll it sideways.
This is the diagram to keep. Everything distinctive about the track falls out of the right-hand column: recovery skills, perception-backed postconditions, abort semantics, retry budgets, and the reason an agentic robotics system needs an evaluation harness rather than a demo video.
What the track actually contains
Six to eight weeks, roughly in this order.
- Harden the skill layer. Every skill gets a precondition, a postcondition, a timeout, a defined abort behaviour and a structured error taxonomy. This alone usually doubles the reliability of a system that already worked once.
- Add recovery as first-class skills.
release_safely,retreat_to_home,reacquire_object. The planner selects them like anything else, which means the failure taxonomy has to map onto them. - Multi-step tasks with verification between steps. Three to five steps is enough to expose everything: error compounding, stale world models, and the moment the planner confidently proceeds from a step that silently failed.
- An evaluation harness at the task level. Success rate over many trials, broken down by condition and by which step failed. This is the same discipline you applied to policies, moved one layer up.
- Latency and reliability numbers, published. Time from goal to first motion, planner overhead per step, and end-to-end success. Nobody publishes these, which is precisely why yours will be read.
- Optionally, a second embodiment. Running the same skill interface on a different robot is the cleanest possible proof that you built an interface rather than a script.
The honest catch
This is a thin category, and thin cuts both ways. There are few people who genuinely hold both halves, which is the advantage. There are also few teams hiring specifically for it, which means your work has to be legible to people hiring for something adjacent.
The practical consequence is an ordering rule. Lead with the robotics. The end-to-end system, the real hardware, the success-rate table, the failure analysis. Then the agent layer as the distinctive second exhibit. Leading with the agent layer reads, to a robotics hiring manager, as someone who found a way to avoid the robot, and that impression is very hard to reverse in a forty-five minute screen.
There is a second honesty point. Some teams building large vision-language-action models run learned policies straight to joint commands at tens of hertz and involve no classical planner at all, and a few practitioners treat the whole middleware layer as an implementation detail beneath an agent. I have seen this asserted more often than demonstrated, so hold it loosely. Either way it argues the same thing: your value is in the interface design and the verification, not in loyalty to any particular stack.
Check yourself
1. A skill call times out. Why is “retry” a worse default here than in any distributed system you have built?
Because a failed physical call leaves residue. The object may be on the floor, still in the gripper, or wedged somewhere with a motor stalled against it, and the transport carried none of that back. A retry issued without re-reading the world is a second real attempt from an unknown starting state, which is how a recoverable failure becomes a broken object. The correct default is: check the postcondition with a sensor, classify what actually happened, run a recovery skill if needed, re-check the precondition, and only then consider retrying.
2. What replaces the idempotency key?
A perception check. “Has this already happened” becomes a question about the room, answered by a sensor and a model that is sometimes wrong. The consequence is that idempotency stops being a guarantee and becomes a measurement with a false-positive and false-negative rate, so it belongs in your evaluation harness rather than in your assumptions.
3. Why can the reasoner never sit inside the control loop, and what is the interface between them?
Because the loop needs a decision every few tens of milliseconds and the reasoner takes a second or more; putting one inside the other means missing every deadline. They connect through skill calls: the slow layer sets a goal between loop runs, the fast layer executes it closed-loop, and a terminal result comes back. That is why the skill contract carries all the weight in this architecture, and it is why a long-running-operation shape with goal, feedback, result and cancel keeps being reinvented.
4. Your skill is cancelled mid-motion while holding a mug. Name three defensible abort behaviours and what each one costs.
Freeze in place: safest for the object, but the arm now occupies the workspace and may be stalled against something, burning current. Retreat to a known pose while still holding: recovers a clean state for the planner, but sweeps whatever is held through space that was never checked for obstacles. Open the gripper: guarantees the arm is free and the object is not, which is correct for an emergency and destructive for a mug. The point is that there is no default, so the contract has to name one per skill.
5. Why should the agent layer be your second exhibit rather than your first?
Because a robotics hiring manager reading an agent-first portfolio infers that you avoided the hard physical work, and that inference is expensive to reverse in a short screen. Lead with the end-to-end system on real hardware and a success-rate table, which establishes that you can make a robot work. The agent layer then reads as unusual depth on top of demonstrated competence, which is what it actually is.
Do this
Two to three hours. It produces the artifact that this track is built on.
1. Write the contract for one skill. Take the single skill from your agentic-robotics work that failed most often. In a file, write: the arguments, the precondition as something a sensor can evaluate, the postcondition the same way, the timeout, the abort behaviour, the retry budget with a reason, and an error taxonomy of at least four named failure kinds. No prose descriptions that a camera could not check.
2. Injure it deliberately. Run the skill twenty times with faults you introduce yourself: move the object after the precondition check, occlude the camera at the moment of the postcondition check, cancel mid-motion, and unplug or disable one sensor. Record what the system did each time, not what it should have done.
3. Count. Produce a small table: failure kind, how many times it occurred, whether the postcondition detected it, and whether recovery restored a state where a retry was meaningful. The undetected-failure count is the number that matters, because an undetected failure is the one that propagates into the next step of a multi-step task.
4. Write three sentences on the failure your contract did not anticipate, because there will be one, and it is the beginning of your capstone.
What you can now do
You can state precisely which parts of your tool-use architecture survive contact with a physical robot and which do not, specify a skill as a contract whose preconditions and postconditions a sensor can actually evaluate, name three abort behaviours and the cost of each, explain why retries need a fresh perception check rather than a counter, and say why this track is a strong second exhibit and a weak first one.