Robotics, from scratch

A robot is a machine that runs one loop, forever, against a world that will not cooperate.

Robots fill factories and not kitchens because factories were arranged for them, and most eras of robotics stalled at the edge of the arrangement.

A servo is a motor, a gearbox, an encoder and a small controller in one sealed box, and the only thing you are allowed to ask it for is an angle.

Every sensor hands you an array with a unit, a shape and a rate; none of them hand you a fact about the world.

Five ways to write the decide step of a robot's loop, from a hardcoded list of poses to a fine-tuned foundation model.

Hand-written rules cover one case each while real situations multiply, so perception and high-level behaviour went to data; the layers underneath did not.

Four axes that place every robotics paper, product and module of this course.

Who has actually put robots to work, who has announced it, and the four questions that separate the two.

The route through eight modules, the five blocks that make up a working week, and how to use an AI assistant so the understanding ends up in your head.

A position is never a property of an object; it is a reading taken against something else, and the something else has to be said out loud.

Two named frames, one point, and the single line of arithmetic that turns either description into the other.

A 2D rotation matrix is nothing but the two places the unit axes end up, written side by side as columns.

The three basic 3D rotations are the same 2D matrix you already derived, each acting in its own pair of axes, and stacking them never leaves the set called SO(3).

Two rotations about different axes land somewhere different depending on which one you do first, and the gap between the two answers is itself a rotation you can measure.

Three angles are the most readable way to describe an orientation and the least safe way to compute with one, because the description breaks down at places where the rotation itself is perfectly ordinary.

Four numbers on a unit sphere describe any orientation with no seams, which is why every robotics stack stores rotation this way and interpolates it with slerp.

Packing a rotation and a translation into a single 4x4, so applying, chaining and undoing a pose all become plain matrix multiplication.

Stringing poses together from world to base to joint to hand, and using the subscripts as a type check on the order.

Forward kinematics is a pure function from joint angles to where the hand is, and that determinism is the reason a robot can repeat anything at all.

Two links, two joints, one plane: the smallest machine that still has every hard problem in robot kinematics inside it.

The set of poses an arm can reach is a ring with a hole in it, joint limits carve pieces out of it, and reaching a point is a different question from reaching it facing the right way.

A sensitivity table with one column per joint, saying which way and how fast the hand moves when that joint alone turns.

Poses where the Jacobian's columns line up, one direction of hand motion disappears, and any solver that keeps asking for it commands joint speeds no motor can deliver.

Running forward kinematics backwards: the closed form for a two-link arm, why it always has two answers, and the damped iteration that works when no closed form exists.

Why a perfectly computed command still misses, and why looking at the result and correcting is a cheaper thing to require than knowing the world.

Proportional action moves the joint and leaves it sagging, derivative action stops the ringing, integral action closes the last gap; each term earned separately on a gravity-loaded joint.

A controller never asks whether the setpoint was reasonable, so you have to feed it a sequence of reachable setpoints instead of one impossible jump.

Assemble eighteen lessons of primitives into an arm that picks a block and bins it, then measure the exact amount of perception error that destroys it.

A simulator is a staging environment for physics: attempts cost nothing, reset is a function call, and every number in it is a guess you eventually pay for.

Every step, a simulator guesses where everything will be, finds every place that guess is illegal, and solves one system for the forces that make it legal again.

One pip command puts a physics engine on your laptop; the only real decision left is whether you want pixels in an array or a window you can drag, because on macOS those two things need different launchers.

MuJoCo's scene format nests bodies inside bodies, and that nesting is the kinematic chain you built by hand in Module 1, written as indentation.

The SO-101 model file is the complete specification of the arm, and four minutes of reading gets you its degrees of freedom, its limits, its servo gains and one thing it can never do.

Two objects and one function: mjModel holds everything that never changes, mjData holds everything that does, and mj_step turns one into the next.

The actuator type decides what the number in data.ctrl means, and choosing position control turns force from something you command into something you can only watch - for free in simulation, and not at all on the real arm.

A MuJoCo model ships with no sensors and perfect knowledge, so every number a policy is allowed to read is a line you wrote in the XML - and the easiest numbers to reach for are the ones no hardware can produce.

Point the transform maths you wrote in Module 1 at the SO-101, diff it against MuJoCo's own answer, and find out that the geometry agrees to the last bit while the physics does not.

Six keys steer a point in space, one IK step per physics tick drags the arm after it, and the resulting demonstrations are exactly as bad as the interface that produced them.

A task scene is a specification: its geometry decides whether the task is possible at all, and its success predicate decides whether an episode counted.

A grasp is a sustained force balance across a dozen contact points, governed by friction, priority and cone settings that mostly came from somebody else's XML.

Every pose of an arm is one point in a space with one axis per joint, obstacles in the world become forbidden regions in that space, and planning is finding a route through what is left.

Free space becomes impossible to enumerate at about four joints, so planners stop examining it and start sampling it, which buys an answer in milliseconds and gives up on ever saying no.

A planner needs a model and hands you a route you can check before anything moves; a policy needs demonstrations and works where the model runs out, and no real system picks only one.

Randomisation is judged by the spread of the data it produces, not by the success rate, and the evidence on what actually transfers is narrower than the folklore around it.

A dataset is a contract with a model you have not written yet, and almost everything that breaks it is decided in the fifteen lines that write one frame.

Two artifacts and one honest number: a success rate that names the region it was measured over, and a demonstration dataset that reproduces itself when you play it back.

A demonstration dataset is not a pile of recordings; it is the specification of everything the robot will ever be competent at, written one episode at a time.

A policy is one function from what the robot can see to what it should command, called thirty times a second with no memory in between.

Behaviour cloning is supervised learning on a table of observation and action rows, and the whole algorithm is one call to fit.

A policy generates the states it is later judged on, so being slightly wrong changes the question it is asked next, and the error feeds itself.

Every fix for covariate shift is the same move - get expert labels on states the policy will actually visit - and they differ only in how well they guess where those are.

Predicting a run of future actions from a single observation cuts the number of decision points, and the number of decision points is what compounding error is quadratic in.

ACT is a transformer that turns one observation into the next hundred actions, trained as a conditional autoencoder so it can absorb the variation between demonstrations.

A real training run on real SO-101 demonstrations, with measured wall-clock numbers from a 2018 laptop and an honest account of what that laptop cannot do.

A squared-error loss asks the network for the average of everything the demonstrators did, and when they did two different valid things, the average is a third thing that fails.

Instead of predicting one action, learn to turn noise into an action, and every roll of the dice lands on a different valid answer instead of the average of all of them.

Train three heads on one dataset at one budget, roll them out, and discover that the policy with the worse validation loss is the one that reaches the goal.

A success rate is an estimate with a denominator, an uncertainty and a protocol, and all three belong in the claim.

An eval harness is a recording device for the protocol, not a loop that counts successes.

Reinforcement learning replaces the demonstration with a score, which buys you tasks nobody can demonstrate and charges you a simulator, a hundred million failures, and a specification the optimiser will read against you.

You cannot differentiate through the physical world, so policy gradients make the actions that led to good scores more likely, and everything after that is fighting the noise in that estimate.

Nobody writes the gait: it falls out of a weighted sum of penalties, a velocity command in the observation, and enough parallel simulation that a hundred million failures fit in an afternoon.

The way to survive the gap between simulator and robot is not a more accurate simulator; it is a family of deliberately wrong ones wide enough that reality is just another member.

The three paradigms are three different things a human has to author, and the task decides which one you can actually produce.

Two policies, two tasks, two data budgets and three seeds, scored on identical scenes, written up so a stranger could re-run it.

A real arm hands you two separate things at once, real data and a closed loop, and without one you can still buy each of them separately.

The SO-101 is a published bill of materials rather than a product, and every line in that list is a decision you are now responsible for.

The build is mechanically easy and has an ordering constraint that nothing on the parts enforces, so every expensive assembly mistake is a gate you walked past.

Six motors share one wire, the ID is the primary key on that wire, and every motor ships with the same one.

Calibration is the map from raw encoder counts to joint angles, and its real job is agreement between everything that quotes those numbers rather than accuracy against the physical world.

One joint, five degrees, read back: the smallest command that exercises every link in the chain from your Python process to a gear tooth, and gives each failure its own signature.

Force on this arm is not commanded, it is capped in registers, and the cap that would protect everything except the gripper ships switched off.

Leader-follower teleoperation copies six joint angles in one direction, tens of times a second, and nothing the follower touches ever travels back to your hand.

Every second you teleoperate writes thirty rows of training data, so the quality of a demonstration is not whether it worked but whether it matches your other demonstrations.

The camera rig is not equipment sitting next to the robot; it is the observation space, and anything the cameras fail to resolve is unlearnable however good the model is.

A pixel is a ray, so turning one into a place needs depth and the camera-to-robot transform; an imitation policy skips both by never leaving pixel space, and charges you a repeatability discipline instead.

A task is not a sentence, it is the set of scenes that sentence has to work in, and your first one should be small enough that sixty demonstrations cover it.

Sixty demonstrations is an afternoon of deliberate repetition, and the most important key on the keyboard is the one that throws an episode away.

Training on your own fifty-odd episodes is mostly arithmetic - frames, batch, epochs - plus one widely-quoted scheduler flag that will abort an ACT run, and one number that does not mean what you want it to.

The first autonomous run is a supervised experiment with a short duration, a motion clamp, and a hand near the power plug - and cutting the power does not stop the arm, it drops it.

A trained policy is a compressed copy of your demonstrations, so improving it is a measurement problem: watch it fail, sort the failures, and aim the next episodes at the biggest bucket.

A failure that repeats identically is a geometry problem; a failure that lands somewhere new each time is a data problem, and the two live in different parts of the building.

A robot result nobody can audit is a screenshot, and the five artifacts that make yours auditable cost an hour between them.

Pick a task you can demonstrate ten times out of ten, take it round the data loop until it clears the bar, and prove it with an uncut scored run under a protocol you wrote first.

A policy trained on your demonstrations knows your demonstrations. A pretrained model arrives already knowing what a block is, and that is a different kind of thing to own.

Between 2022 and 2024 four separable inventions turned a task-specific robot network into something you can download and talk to, and each one is worth naming on its own.

There are two ways to get a number out of a neural network, pick it from a list or produce it directly, and that choice sets your training cost, your control rate and how much language ability survives.

Different questions in robotics have deadlines set by physics rather than by taste, and no single network runs well at both ends of the range.

Bolting a motor system onto a language model damages the language model, and the π-family's published answer to that problem is two objectives, one forward pass, one deliberate gradient cut.

Two more ways to cut the fast-slow line, and the difference that matters is not the architecture but what crosses the cut.

The most impressive robot system in public view is also the one you can check least, and holding both of those facts at once is a skill worth building deliberately.

Every VLA is a pretrained backbone, a fast action head, and a seam between them. Only three things vary, and this lesson is the drawing you should be able to reproduce from a blank page.

450 million parameters, pretrained on 481 community datasets from other people's spare rooms, and small enough that the whole loop closes on a GPU you already own.

A vision-language-action model eats exactly what ACT ate, plus a sentence, and the whole experiment depends on freezing the result before anyone trains anything.

The command is one line and the loss curve is beautiful and neither of them tells you whether the policy works, so you design the run around checkpoints you can put on the arm.

Twelve out of twenty against nine out of twenty is not a result, and knowing exactly why is the difference between a blog post people trust and one they scroll past.

A pretrained VLA is not automatically better than a small policy trained from scratch, and the published evidence as of August 2026 says so out loud.

Running a frontier open VLA is a plumbing problem with a hard memory floor, a licence you have to read yourself, and a network hop sitting inside your control loop.

The forward pass is slower than the control period, always, and every deployment in this field is a scheme for hiding that.

A world model predicts what happens next instead of what to do next, and as of August 2026 it earns its keep as a data generator and a perception backbone rather than as a replacement for your physics engine.

Every model named in this module will be superseded; the procedure for deciding whether a new one matters to your bench will not be.

One frozen dataset, three policies, one protocol, several hundred scored trials, and a public writeup that reports the result you did not want.

An agentic robot is three loops nested by rate, and the only interesting design question is what representation crosses the boundary between them.

A planner call costs a second or two for four reasons that add rather than trade, and that one number decides how you carve up every skill in the system.

A trained policy is a function with no beginning, no end and no opinion about whether it should be running; a skill is that policy plus a contract, and the contract is all of the work.

A skill is a contract offered to a model that cannot see your code, cannot feel your robot, and will take every name you choose literally.

A skill that reports success because the motion finished is the most expensive lie in the stack, because the planner believes it and never checks again.

Turning the word mug into something an arm can reach for takes three separate steps, every one of them fails silently, and a confident wrong answer looks exactly like a correct one.

The protocol is the part you already know. What is new is that this tool is slow, cannot be retried for free, fails physically, and can return success while its postcondition is false.

A plan is a hypothesis about a world you looked at once, and the only job of the loop around it is to keep the model's picture of the desk matched to the desk.

A failed run has three possible authors - the skill, the plan, or the world model - and each one wants a different repair, so working out which broke comes before deciding what to do about it.

Retry is one rung on a ladder of six, it is correct for exactly one failure class, and the thing that picks the rung is a monitor that watches the world rather than the log.

A model's refusal is a property of text, and a robot's harm is a joint trajectory, so the only layer that can actually say no is the skill layer, in code the model cannot argue with.

Two honest numbers can describe the same ten runs and differ by more than fifty points, because partial credit and full-task success are not the same measurement.

The planner is a hundred times slower than the policy and the policy is ten times slower than the controller, and the whole design problem is keeping each tier busy while the one above it thinks.

Build the full agentic loop against a mock desk with three failure dials, then measure which recovery mechanism buys back which failure class.

Three tracks, four criteria, and nine months of evidence you already collected about which one is actually yours.

Everything you know about tool-use architecture holds, right up to the point where a failed call leaves the world changed.

The track where the physics is genuinely harder, the bill arrives in GPU hours and broken hardware, and almost none of your background compounds.

The least glamorous track, the most employable one, and the only one where nearly all of your existing background transfers without translation.

Robots holding accounts, paying each other for metered resources, and proving what they did to someone who was not there, treated as early and mostly unbuilt.

You built a working robot without typing ros2 once. Here is exactly how much of it you owe the industry, and how much of what you will find written about it is already dead.

A robot failure cannot be reproduced, so the recording is the only evidence that will ever exist. Learn the format deeply and the viewers loosely.

A capstone is an argument with a robot attached, and the argument has to be settled before the first commit.

The version of your capstone that works with you absent, which is the only version most people will ever see.

The piece that gets read is the page somebody was already looking for, and in robotics the missing page is almost always the one with the failures in it.

A year of work becomes a position when a stranger can check it in seven minutes without you in the room.