Where you are. You can design a skill and make it report honestly. Every one of those skills takes an argument that came from perception, and this lesson is about how much that argument is worth. It is the weakest link in the stack, and it fails in the one way your error handling cannot see.
Ask a model what is on the desk
Point a camera at your desk and ask a good vision-language model what it sees. It will tell you: a black mug, a pen, a spiral notebook, a laptop, a coil of cable. It is right. It will describe the scene better than most people would.
Now ask it where the mug is, in a form the arm can use.
You get a point. Not a pose: a point, in image coordinates, normalised, with the vertical axis first. To turn that into somewhere the arm can go you need the camera’s intrinsics, a depth value or an assumption about the table plane, and the hand-eye calibration between camera and robot base. Three conversions, each contributing error, none of them the model’s problem.
Then move the desk lamp. The mug is matte black, the new lighting kills the highlight the detector was keying on, and the detector returns nothing. The vision-language model, asked the same question in words, still says confidently that there is a mug on the desk. So your planner calls pick on an object identifier that no longer exists, or worse, on the identifier that now belongs to the pen.
No exception was raised anywhere in that paragraph.
The idea in one paragraph
Grounding is the step that turns a word into something the robot can act on. It has three jobs that people usually collapse into one: refer, deciding which physical thing the word mug means; locate, producing a position in a frame the arm uses; and persist, deciding whether the thing you see now is the same thing you saw a second ago. Most tools do one of the three well. All three degrade sharply when the lighting, background or object appearance moves away from what the detector was trained on, and they degrade by returning a confident wrong answer rather than an error. That is why grounding is the most dangerous component in an agentic robot: a wrong detection does not produce a failed plan, it produces a perfectly sensible plan about a world that is not there.
The chain from a word to a pose
Wider than the screen; scroll it sideways.
Say the model hands you a detection in the format the current robotics APIs use: a list of {"point": [y, x], "label": "..."}, with coordinates normalised to a 0 to 1000 range and the vertical axis first. To get from there to a pixel:
which just says: scale each normalised coordinate by the image width or height it belongs to. Then the pixel becomes a ray through the camera using the intrinsics, the ray becomes a 3D point using either a depth reading or an assumption that the object sits on a known table plane, and the 3D point becomes a pose in the robot base frame through the hand-eye transform you calibrated in Module 4. That last step is exactly the transform chain from Module 1, composed the same way and breaking the same way if you compose it backwards.
How badly it degrades
Open-vocabulary detectors are evaluated on clean benchmark images and deployed on your desk. The gap has been measured, and it is not small. Open-Vocabulary Object Detectors: Robustness Challenges under Distribution Shifts (ECCV Workshops 2024, arXiv:2405.14874) ran three detectors against shifted versions of COCO. On natural distribution shifts, the kind you get from a change in weather, style or rendering, and on adversarially altered backgrounds:
| Model | Clean benchmark | Natural shift | Adversarial background |
|---|---|---|---|
| OWL-ViT | 26.4 | 16.0 (down 39%) | down 46% |
| YOLO-World | 39.3 | 23.4 (down 40%) | down 51% |
| Grounding DINO | 48.4 | 41.5 (down 14%) | down 14% |
The columns are not directly comparable to each other, since the two shift benchmarks use different clean baselines, but the relative drops within each row are the point. The study’s own conclusion is blunt: no model it evaluated maintains acceptable performance across all shift categories, and using one in a real environment means adapting it to that environment. Grounding DINO is roughly three times more robust than the alternatives and still loses a seventh of its accuracy to a background change. Note the date on that paper. Detectors have moved since 2024; the shape of the failure has not, and the discipline it argues for is what you are meant to carry away rather than the specific numbers.
The two silent failures
Wider than the screen; scroll it sideways.
Spurious rejection. The object is right there, clearly visible to you, and the detector returns nothing for it. OWL-ViT in particular is reported to do this often. The planner’s behaviour is then either to plan around an object it cannot see, or to loop: look, report no mug, look again, report no mug, while a mug sits in the middle of every frame. This one is at least visible to a human watching the run.
Confident misassociation. The label lands on the wrong object. Now the planner writes a completely sensible plan and executes it perfectly, on the wrong thing. Every precondition you wrote in the previous lesson passes, because the object identifier exists and something is at that position. Every postcondition passes too: something is in the gripper. The system reports full success and has put your pen on the shelf.
This is also why failure attribution is so hard in practice. One published breakdown of 301 failed long-horizon agent tasks attributed 46% to execution, 28% to task planning and 26% to memory. Grounding does not appear as a category, and it cannot: a plan written about a misidentified object looks like a planning failure, and a grasp aimed two centimetres off looks like an execution failure. The cause sits upstream of the bucket it lands in.
Building for a detector that lies
None of this is a reason to avoid open-vocabulary perception. It is a reason to build the loop around it differently. Five things pay for themselves immediately.
Perception mints identifiers; the planner only refers to them. This is the rule from the previous lesson, and grounding is why it exists. look() returns detections with identifiers, scores and positions, and every skill takes an identifier. A stale identifier gives you a clean lookup failure. An invented name gives you a motion.
Return the uncertainty, and let a skill refuse on it. If the top two candidates for mug score 0.42 and 0.39, that is not a detection, it is a coin flip. A skill that refuses ambiguous references and says why is worth far more than one that picks the higher score. This is one of the few places a request for human input genuinely earns its latency.
Ground twice. Detect when you plan, then detect again immediately before you act.
Wider than the screen; scroll it sideways.
The gap between planning and acting is seconds, and seconds are long enough for a human to move the mug, for the arm to nudge it, or for the lighting to change. Re-grounding costs one detector call, tens of milliseconds, and it converts a whole class of silent failures into a clean precondition refusal.
Close the vocabulary where you can. Open vocabulary is a research capability and a deployment liability. If your desk has six kinds of object, a detector adapted to those six on images from your own camera beats a general model on every axis that matters to you. The generality was never the goal; it was a way of avoiding data collection, and by Module 4 you already know how to collect data.
Prefer physical evidence for physical questions. Whether something is in the gripper is answered by the gripper, not by a camera looking at the gripper. Push every question you can down to the cheapest sensor that can settle it.
Check yourself
1. Name the three jobs grounding has to do, and give a failure of each.
Refer: deciding which physical object the word means. It fails when the label lands on the pen instead of the mug. Locate: producing a position in a frame the arm can use. It fails when the axis order or the plane assumption is wrong and the pose is plausible but displaced. Persist: deciding whether this is the same object as a moment ago. It fails when identifiers churn between frames, so the identifier the planner is holding now points at a different physical object.
2. A detection arrives with the normalised pair 412 then 733, on a 1280 by 720 image. What are the pixel coordinates, and what is the trap?
The vertical axis comes first, so 412 is the row and 733 is the column. That gives a horizontal pixel of and a vertical pixel of . The trap is that reading the pair in the other order also produces a valid pixel inside the image, so nothing raises an error; the arm just goes to a mirrored location. Encode the axis order in your variable names so the convention cannot be silently reinterpreted.
3. Why is confident misassociation more dangerous than spurious rejection, given that both are silent?
Spurious rejection is at least visible in behaviour: the robot either fails to act or loops reporting it cannot find an object that is plainly in frame, and a human watching notices in seconds. Misassociation produces a valid identifier at a real position, so the plan is coherent, the preconditions pass, the grasp succeeds, the postconditions pass, and the system reports success while having manipulated the wrong object. Every safeguard downstream of grounding is satisfied by it.
4. Your detector reports 39 mAP on a public benchmark. What does that tell you about your desk?
Very little on its own. Measured drops on distribution-shifted versions of the same benchmark run from about 14% to over 50% depending on the model and the kind of shift, so a benchmark score in the high thirties is consistent with performance in the low twenties in your room. The only number that predicts your system’s behaviour is one you measured on your own objects, camera and lighting, which is the same evaluation discipline you applied to policies in Module 3.
5. What does re-grounding immediately before acting actually buy, given you already grounded at plan time?
It converts stale-world failures into clean precondition refusals. Several seconds pass between planning and acting, which is enough for a person to move the object, the arm to nudge it, or the lighting to change. A re-detection costs tens of milliseconds and either confirms the object is where the plan assumed or reports that it is not, before any motion happens. Without it, the arm executes a correct motion toward a position that stopped being true, and the failure surfaces later as a grasp that missed.
6. Why does grounding rarely appear in published failure-attribution breakdowns?
Because it is upstream of every category those breakdowns use. A plan written about a misidentified object is recorded as a planning failure, since the plan was wrong. A grasp aimed at a position that was two centimetres off is recorded as an execution failure, since the grasp missed. The grounding error is the cause in both cases and the label in neither, which is why attribution numbers systematically understate it.
Do this
About forty minutes, standard library only. You are building the perception half of the loop you have been assembling since lesson 4.
1. A detector that is wrong the way real ones are wrong. Write detect(query, condition) over a fixed ground-truth scene of five objects. Under condition="clean" it returns all five with high scores. Under condition="shifted" it drops each object with probability 0.35 and, with probability 0.15, attaches the queried label to a different object with a score above 0.8. Never raise an exception.
2. Mint stable identifiers, and prove they are not stable. Write look() to wrap detect and assign an object_id per detection. Call it twice on the identical scene and print both id lists. If you assigned ids by enumeration order, you have just demonstrated identifier churn: the same physical mug is obj_1 and then obj_2, and every identifier the planner is holding now points somewhere else. Fix it by matching detections across calls on position, within a tolerance, and re-run.
3. Measure the silent failure rate. Run 100 grounded picks under condition="shifted", where the pick succeeds if the identifier the planner used still maps to the object the human asked for. Print three counts: correct picks, refusals, and wrong-object picks that reported success. That third number is the one this lesson exists for, and it should be uncomfortable.
4. Add an ambiguity refusal. Make look return the top two candidates with scores. Have pick refuse when the gap between them is under 0.1, returning both candidates and asking which was meant. Re-run the 100 trials. Wrong-object picks should fall and refusals should rise; write down the exchange rate, because that is the tuning knob you will argue about for the rest of the module.
5. Ground twice. Add a disturb() that moves one object between planning and acting. Run 100 trials grounding once, then 100 re-grounding immediately before the grasp, and compare wrong-object picks in each. Then take the whole harness forward: the failure taxonomy and monitoring lessons are built on distinguishing exactly these outcomes from a skill that simply failed.
What you can now do
You can separate the three jobs grounding has to do and say which of them a given tool actually performs. You can trace a detection from a normalised image point through intrinsics, depth and the hand-eye transform into the base frame, and name the error each step adds. You can state what an open-vocabulary detector’s benchmark number does and does not predict about your own room. And you can build a planner loop that assumes its perception is sometimes confidently wrong: identifiers minted by perception, uncertainty returned rather than hidden, refusal on ambiguity, and a second grounding pass immediately before anything moves.