The hard part of robotics isn’t the robot.
A modern arm is a well-understood piece of engineering: motors, gears, encoders, a bus. You can buy one. What’s hard is that the world doesn’t hold still, the robot only knows what its sensors tell it, and every action it takes changes the situation it then has to reason about.
That’s why the field keeps restarting. Each era found a method that worked beautifully inside its assumptions and fell over outside them. This page is the stack those attempts built, and the honest state of the argument now.
If frames, quaternions and the Jacobian are already familiar to you, start at Simulation and skip the geometry.
What a robot actually is
Strip away the science fiction and a robot is a loop.
Wider than the screen; scroll it sideways.
Everything in this subject is a way of filling in one of those boxes. And the loop’s rate silently decides which techniques are even available to you: a method needing 200 milliseconds per decision is unusable in a loop that has to close in 10.
The machine itself
Wider than the screen; scroll it sideways.
Links are the rigid pieces. Joints are where they move relative to each other, rotating, mostly. Actuators turn electricity into motion at those joints. Encoders report where each joint actually is.
The count of independently controllable joints is the arm’s degrees of freedom, and it decides what poses are reachable at all. With 6 you get any position and any orientation within reach. Fewer, and some orientations are simply unavailable.
What sensing actually returns
This is where intuition misleads people most.
The gap between raw numbers and useful belief is most of the work. It’s also why a demo that runs perfectly in a lab falls over in a kitchen: the numbers changed, and nothing in the pipeline was built to notice.
Why the field kept restarting
Robotics has had several confident eras, and each one ended the same way.
Wider than the screen; scroll it sideways.
Industrial automation worked magnificently by removing the uncertainty: bolt the part in a jig, repeat the same motion a million times. It still runs most of manufacturing. It also can’t pick up a crumpled cloth, because nothing about it was ever about perception.
The next attempt was to program the intelligence explicitly: enumerate the cases, write the rules. That worked in the lab and collapsed in the world.
Wider than the screen; scroll it sideways.
Why does the rate of the control loop constrain which methods you can use?
Because every method has a cost per decision. If the loop has to close every 10 milliseconds to keep the arm stable, anything taking 200 milliseconds can’t sit inside it. That’s why the field ends up with layered architectures: something slow deciding what to do, something fast keeping the machine steady while it happens.
Where things are, and how they move
Before any learning, there’s geometry. This part is the vocabulary the rest of the subject is written in.
Frames: every position is relative to something
“The mug is at (0.3, 0.1, 0.05)” isn’t a fact until you say in which frame. Most confusing robotics bugs are one frame mistaken for another.
Transforms compose, and the order isn’t negotiable
A transform carries a pose from one frame into the next: a rotation and a shift, packed into one object so they can be chained.
Wider than the screen; scroll it sideways.
Chaining transforms is function composition. Base to shoulder, shoulder to elbow, elbow to wrist, wrist to hand. Multiply them in order and you have the hand’s pose in the base frame.
Three angles, and where the map tears
The obvious way to write a 3D rotation is three angles: roll, pitch, yaw. It’s readable, it’s what everyone reaches for first, and it has a defect that isn’t obvious until it bites.
Wider than the screen; scroll it sideways.
The fix is a four-number representation called a quaternion. It’s harder to read and it has no tears anywhere, which is why essentially all production robotics carries orientation as quaternions and converts to angles only for display.
Wider than the screen; scroll it sideways.
Forward and inverse: the two directions
Forward kinematics goes from joint angles to hand pose. It’s a straightforward chain of transforms, and it always has exactly one answer.
Wider than the screen; scroll it sideways.
Inverse kinematics goes the other way: given a pose you want the hand to reach, what angles get it there? That question may have one answer, several(elbow up or elbow down), or none at all if the point is out of reach.
The Jacobian: if I nudge this joint, where does the hand go?
Between forward and inverse sits the most useful object in the subject.
Wider than the screen; scroll it sideways.
It answers the practical question directly: to move the hand this way, which joints and how much? That makes it the engine inside most inverse-kinematics solvers, and the reason they’re iterative. Take a step, recompute, repeat.
Singularities: where the arm loses a direction
Wider than the screen; scroll it sideways.
At certain configurations(an arm stretched straight out, typically), two joints end up contributing the same motion, and some direction becomes unreachable no matter what the joints do.
Control: measuring the mistake instead of predicting it
Commanding a joint to an angle doesn’t put it there. Gravity, friction and load all interfere. So instead of predicting the right command, you measure the error and correct continuously.
Wider than the screen; scroll it sideways.
Inverse kinematics can have several answers. Why is that a practical problem?
Because you have to choose, and the choices aren’t equivalent. Elbow-up and elbow-down may both put the hand in the right place while one collides with the table, passes through the workspace boundary, or leaves the arm near a configuration where it loses a direction of movement entirely. The solver gives you geometry; something else has to supply judgement.
Simulation: making the mistakes for free
A simulator is where you break things that would cost money to break, and where the gap between your model of the world and the world becomes measurable.
A physics engine is a state machine you advance
You describe bodies, joints and contacts; the engine integrates forward a small step at a time. Nothing happens except when you advance it. That’s a feature: the run is reproducible, and a reproducible failure can be debugged.
You decide what the robot may see
Wider than the screen; scroll it sideways.
Everything the controller wants becomes a single command per joint, which is where simulation and hardware finally agree. And on the sensing side you choose the cameras, their placement and their resolution, which means you can accidentally give a simulated robot information no real one would have.
Contact: the parameters you didn’t write
Grasping is where simulators are least trustworthy, and it’s worth knowing why.
Wider than the screen; scroll it sideways.
Free-space motion is well-determined physics. The moment two surfaces touch, the result depends on friction, stiffness, damping and the solver’s own settings: numbers you probably never chose, inherited from whatever model file you started from.
Planning: the arm becomes a point
A neat reframing. Instead of thinking about an arm sweeping through space, represent every possible configuration as a single point in a space of joint angles. Obstacles become forbidden regions, and planning a motion becomes finding a path between two points.
Wider than the screen; scroll it sideways.
Throwing darts at a space you can’t draw
Configuration space for a 6-joint arm has 6 dimensions, and the forbidden regions have no tidy description. They’re whatever shape the obstacles happen to project into it. You can’t draw that, and you certainly can’t search it exhaustively.
Wider than the screen; scroll it sideways.
So the working algorithms give up on completeness and sample instead: pick random configurations, keep the ones that are collision-free, connect them into a tree that grows towards the goal. Run it twice and you get two different valid paths.
The gap, and what to do about it
A policy trained purely in simulation usually fails on hardware, because the simulator is wrong in ways nobody enumerated: friction, delay, mass, lighting.
Wider than the screen; scroll it sideways.
Why does randomising a simulator’s parameters help more than measuring them precisely?
Because you can’t enumerate everything that’s wrong. Measuring friction exactly still leaves delay, flex, wear and lighting unmodelled. A policy trained across a distribution of worlds learns to depend on features that survive the variation, and that’s what transfers. A policy tuned to one very accurate simulator can depend on details that are true only there.
Learning a skill from demonstrations
Writing the rule works when you can state the rule. For “pick up that crumpled cloth”, nobody can. So you demonstrate instead.
A policy is a function
Behaviour cloning is the simplest version: record a human doing the task, then train a network to reproduce the human’s action given the same observation. Supervised learning, with robot data.
Wider than the screen; scroll it sideways.
Why the clone falls apart
It works in training and fails on the robot, for a reason worth understanding precisely.
Wider than the screen; scroll it sideways.
This is called covariate shift, and the fixes are all versions of showing the policy how to recover: demonstrate corrections, add data where it actually goes wrong, or let it act and have an expert label the situations it reaches.
Two right answers, averaged into a wrong one
A second failure, and a subtler one. Ask 10 people to pick up a mug and some go left of the obstacle, some right. Train on all of them and the network learns the average: straight into the obstacle.
Deciding less often
A policy that outputs one action per observation has to be right 30 or 50 times a second, and each decision is an independent chance to wobble.
Wider than the screen; scroll it sideways.
Two things improve at once. The motion smooths out, because a chunk is internally consistent in a way that independent per-step predictions aren’t. And the compounding problem above weakens, because there are simply fewer decision points at which to go wrong.
Saying a policy works
Why can a policy score well in training and fail immediately on the robot?
Because training measures its predictions against demonstrated situations, while the robot puts it in situations its own errors created. Nothing in the training score reflects what happens once small mistakes compound and carry it off the distribution it learned. Evaluation has to be on the robot, acting, not on a held-out slice of the demonstrations.
Real hardware, and the data it gives you
Everything above can be done in simulation. This is where it stops being free.
Wider than the screen; scroll it sideways.
An arm is a chain of servos on a shared bus, each with an identity, each needing calibration so that “90 degrees” means the same thing on two different arms. None of this is intellectually deep and all of it will consume an afternoon.
The demonstration is the label
On real hardware, data quality stops being an abstraction. Your demonstrations are your training labels, so a hesitant, inconsistent or rehearsed demonstration teaches exactly that.
Stop isn’t the same as hold
Wider than the screen; scroll it sideways.
A safety detail that catches everyone once. Cutting torque to a motor doesn’t freeze the arm; it relaxes it, and gravity takes over. “Stop” and “hold position” are different commands with opposite effects on a loaded arm, and discovering that with a payload in the gripper is a memorable way to learn it.
The same class of surprise runs through the hardware module: settings that survive a power cycle versus settings that don’t, limits enforced in the servo versus limits enforced in your code. None of it is conceptually hard and all of it is the difference between a working bench and a broken one.
The loop that matters
The useful skill isn’t collecting data once. It’s knowing where the next 20 episodes should go: watch what the policy fails at, demonstrate that, retrain, measure. That loop is the job.
Why might 20 demonstrations outperform 200?
Because a policy learns whatever is consistent in the data, including artefacts. Train on 200 near-identical episodes and you teach one narrow trajectory and nothing about recovery. Train on 20 deliberately varied ones, with different starting positions, different approaches, and corrections included, and you cover more of the situations the policy will actually meet.
Foundation models: one model for many tasks
The current frontier, and the part of the subject with the widest gap between what’s demonstrated and what’s announced.
What pretraining buys
Wider than the screen; scroll it sideways.
The bet is that a model pretrained on enormous quantities of general data already understands most of what a mug is, so teaching it your task requires far less robot data than starting from nothing.
Four ideas, not one scaling curve
It’s tempting to read the last few years as one graph going up. It is not. The current designs are four separable ideas that arrived at different times and can be adopted independently.
Wider than the screen; scroll it sideways.
Roughly: treat actions as another language the model can emit; borrow a vision-language model’s understanding rather than learning the world from robot data alone; pool data across many different robots so one model sees more situations than any single lab could collect; and generate a chunk of actions rather than one, which is the same idea from the previous part arriving in a larger model.
The clock problem
Wider than the screen; scroll it sideways.
A large model can’t run in a fast control loop; it’s too slow. So real systems split by timescale: something big and slow decides what to do, something small and fast keeps the machine steady in between. That split recurs at every level of the stack, and where you put the seam is a genuine design decision.
Where the model actually runs
Wider than the screen; scroll it sideways.
The headline models are large, and a robot has a power budget and a latency budget. So a practical question sits under every announcement: does this run on the robot, on a machine beside it, or in a datacentre, and what happens to the arm when the network hiccups?
Reading the field honestly
Why doesn’t a large model simply replace the control layer beneath it?
Timing. A model taking 100 milliseconds or more per decision can’t sit inside a loop that has to close in 10. It can decide what to do and even emit a short sequence of actions, but something fast has to keep the machine stable between its decisions. That’s an architectural constraint, not a temporary engineering limitation.
Planning over learned skills
If one model can’t do everything, the alternative is composition: a deliberate planner directing fast reactive policies.
Wider than the screen; scroll it sideways.
Grounding: connecting a word to a thing
A planner says “pick up the red mug”. Something has to decide which pixels are the red mug, and whether there’s exactly one.
Wider than the screen; scroll it sideways.
The interesting engineering is the seam
Three things a plan does when it dies: the skill fails outright, the skill reports success while having achieved nothing, or the world changes underneath a plan that’s still executing. Those need three different responses, and conflating them is how a robot ends up confidently repeating an action that can’t work.
Three clocks that must never wait on each other
Wider than the screen; scroll it sideways.
The planner thinks in seconds, the policy acts in tens of milliseconds, and the safety layer has to react in single milliseconds. Those are three different clocks, and the architecture’s real job is keeping them decoupled.
Why does a skill need to report honestly rather than optimistically?
Because the planner’s next decision assumes the previous step’s postcondition holds. A skill that reports success without achieving it doesn’t cause one failure; it causes every subsequent step to be planned against a world that doesn’t exist. An honest failure is recoverable, a dishonest success is not.
Where the field is going
Wider than the screen; scroll it sideways.
Robotics isn’t one job. Manipulation, locomotion, perception and fleet infrastructure pull apart into different daily work, different literature and different hardware, and the honest advice is to pick on evidence about what you actually enjoy, not on which has the best demo reel this quarter.
Two things are worth carrying away whichever direction you go. The layered architecture isn’t going away: something slow deciding, something fast stabilising, whatever the models look like. And evidence discipline is the scarce skill: the ability to say precisely what was measured, over how many trials, under what conditions, and what it doesn’t show.
That last one is rarer than it should be, and it travels.