Where you are. You know what a robot is, what its sensors actually return, and the five ways one can be programmed. This lesson explains why the field’s centre of gravity moved from hand-written code to learned models, and which parts of a robot did not move at all.
Write the mug detector
Someone puts a camera over a table, sets a white mug on it, and gives you one job: return where the handle is. You have an afternoon.
You take it, because it is not hard. Threshold for bright pixels, take the largest blob, fit an ellipse to the rim, find the lump on one side. By four o’clock it works every time, in a millisecond, and you are quietly pleased.
Then someone puts a white tablecloth on the table.
So you add a rule: the blob has to be brighter than what surrounds it. Fixed. Then the mug is glass and there is no blob at all. Then there are two mugs. Then afternoon sun blows half the rim out to white. Then a cereal box hides everything but the handle, and “find the rim first” turns out to be load-bearing.
Six weeks later you have four hundred lines of thresholds. Every rule was correct when you wrote it, and every rule covers exactly one situation. You can no longer touch any of them: the fix for the glass mug quietly re-broke the tablecloth.
Nobody there was a bad engineer. The approach ran out of room. This lesson is about what ran out, and why working harder would not help.
The idea in one paragraph
Classical robotics works beautifully whenever you can specify the world in advance. A factory specifies it physically: fixtures hold the part, the lamp never moves, the part is always the same part. Then hand-written code is not a compromise, it is the correct answer: fast, deterministic, auditable, cheap. When the world is not specified, the number of distinct situations stops being a list you can enumerate and becomes a product of independent factors, while each rule you write covers one region of it. Learning is the bet that you can buy coverage from data instead of from cases. It paid off for perception and for deciding what to do next, and did not pay off underneath.
That factory did not buy its reliability with code. It bought it with steel.
Take the fixture away and the deleted factors come back. All of them, at once.
Cases multiply, rules add
Count the ways a tabletop scene can differ: twenty objects, fifty reachable positions and orientations each can sit in, six lighting conditions, eight background surfaces, four amounts of the object hidden behind something.
Those factors are independent, so the number of distinct scenes is their product: , which is 192,000. Add one more five-setting factor, say how greasy the object is, and you are near a million. Rules do not multiply. They add. Each if covers one region, and that region is small precisely because the condition firing it is specific.
Wider than the screen; scroll it sideways.
The tail has no end
Sort situations by how often they occur and you get a tall head and a very long, flat tail: a few common cases, then an enormous number of rare ones. Rare in aggregate is not rare. It is where nearly all your failures live.
This is why reliability gets more expensive as it improves. As a shape rather than a measurement: the first 80% of an arm task is an afternoon’s work, 95% is a month, and 99% has consumed multi-year efforts at well-funded companies. Hand-written rules are the worst instrument for that, because they charge you per case and the tail is made of cases nobody thought of. Rules are not inelegant. Their payment model is wrong.
You have watched this argument before
Machine translation and speech recognition went the same way. Vision held out longest, because hand-designed feature detectors such as SIFT and HOG were genuinely excellent. Then in 2012 AlexNet, a convolutional network, won the ImageNet benchmark with 15.3% top-five error against 26.2% for the best entry that was not a deep network, and the field switched over within two years. The features were no longer written; they were learned from a million labelled photographs.
Robot perception is a vision problem wearing a hat. It fell the same way.
Where robotics is not like spam filtering
Every win above ran on nearly free data: spam corpora labelled by users clicking a button, ImageNet built from photographs that already existed, translation data sitting aligned in parliamentary proceedings.
Robot data is not like that. A training example pairs observations with the actions actually taken, and those actions exist nowhere on the internet. Video of a person chopping an onion has no joint angles and no gripper commands; it shows a policy’s outcome, not the policy. The data must be produced deliberately: a human on a teleoperation rig, in real time, one episode at a time, with a robot that can drop the onion and break itself.
That is the no-undo property from the first lesson reappearing as economics. In supervised learning, being wrong costs one gradient step; on hardware it costs a gripper or a working day. Two responses follow, and they shape the next several modules. Simulation makes failure free: if the physics is close enough, a policy fails a billion times overnight, then transfers. Imitation makes each episode cheap enough to record and skips exploration, because a human demonstrates the behaviour instead of the robot finding it.
What classical robotics kept
Read none of this as classical robotics losing. The argument applies wherever the specification could never be written down, and a robot has whole layers where the specification is a physics equation. There, hand-written code is correct.
Where the gripper ends up given the joint angles is exact arithmetic, and a network approximating it is strictly worse than the closed form. A Kalman filter fusing an inertial sensor with wheel odometry beats a learned estimator, because its model is true. And the safety layer has to be code an auditor can check: ISO 10218, the standard covering industrial robots and their integration, is written around behaviour you can bound, which a sampled network is not. Its 2025 revision absorbed the collaborative-operation requirements that used to live in the separate ISO/TS 15066, so older references to that document now point at something withdrawn.
| Layer | Who owns it now | Why |
|---|---|---|
| Task-level behaviour | Learned | Open-ended situations, impossible to specify. |
| Perception | Learned, decisively | Hand-designed features lost around 2012. |
| Motion generation | Contested | Planners win with a map; policies win in clutter. |
| Servo control | Classical | Deterministic, hundreds of times a second, exact maths. |
| Safety and limits | Classical, deliberately | You certify only code you can read. |
Wider than the screen; scroll it sideways.
Which leaves one question open
“Learning” is doing an enormous amount of work in that sentence. It covers at least four things with different data needs and failure modes: learning from demonstrations, from a reward signal, from a learned predictor of what happens next that you can plan against, and from pretraining one large model across many robots before fine-tuning on yours. Choosing between them comes back to three questions you have met: can you engineer the environment, can you write down a reward, can you demonstrate the task. The next lesson turns those into a map.
Check yourself
1. A colleague says the last team gave up too early. Answer in terms of how two quantities scale.
Situations grow multiplicatively, as a product of independent factors. Rules grow additively, one small region each. Worse, they interact: rules give up to conflicting pairs, so each new rule costs more than the last. Effort is not the binding constraint; the shape of the two curves is.
2. A factory arm has run scripted motion for eleven years with no perception at all. Why is that not a counterexample?
That reliability was bought with hardware, not code. Fixtures, fixed lighting and one known part deleted the factors that would otherwise multiply, leaving something genuinely specifiable. A learned policy there would be strictly worse.
3. What makes robot data expensive compared to spam or ImageNet data, and what are the two responses?
Robot data needs actions, not just observations, and actions exist nowhere on the internet: video shows a policy’s outcome, never its commands. Producing it means a human teleoperating a real robot in real time, where mistakes cost hardware and hours. Simulation makes failure free; imitation makes each episode cheap enough to record.
4. You swap a hand-written grasp heuristic for a learned policy. Where did the long tail go, and what does failure look like?
Into the dataset. The case you never wrote a rule for is now the case you never recorded. Instead of falling through an if, the policy acts confidently and wrongly on an unfamiliar input. And since its actions determine what it sees next, errors compound.
5. Which layers should you refuse to replace with a network, and what property do they share?
Kinematics and dynamics, the inner control loop, state estimation with a trustworthy model, and the safety layer. The shared property: the specification is writable, an exact equation or a bound you must guarantee. Learning is for layers where nobody could write it.
Do this
Two things, forty minutes.
-
Build the failing detector yourself. Take ten phone photos of one object: different positions, two or three lighting conditions, one on a background close to the object’s own colour, one partly hidden. Write a Python detector returning the object’s centre, tuning it on three photos, then run it on the other seven. Log two numbers: thresholds added, and which working photo each new one broke. Stop at thirty minutes, estimate how many rules ten out of ten would need, then ask whether you believe it.
-
Do the multiplication for a real task. Pick something you would want a robot to do in your home. List the factors that can vary, guess how many settings each has, multiply them out. Then ask the question that decides your approach: how many demonstrations would you record?
What you can now do
You can explain why hand-written perception and behaviour stopped scaling: situations multiply while rules add and interact. You can place the long tail in the right layer, say why robot data costs more than spam data, and defend the layers where hand-written code is still correct.