Milestone: scripted pick and place, and the width of its error budget

Assemble eighteen lessons of primitives into an arm that picks a block and bins it, then measure the exact amount of perception error that destroys it.

Where you are. You own every primitive this module set out to teach: frames, rotations, transforms, forward kinematics, the Jacobian, inverse kinematics, feedback control and trajectories. This is where they become one machine that moves, and where you find out how much error that machine can survive.

Set the mug down with your eyes shut

Put a mug in your hand and a coaster about a forearm away on the desk. Look at the coaster. Fix where it is. Now close your eyes and set the mug down on it.

You will hit it. Try again, and have someone nudge the coaster three or four millimetres between the moment you look and the moment you reach. You will still hit it, because the mug’s base is wider than the coaster is off. Now have them slide it three centimetres. You set the mug down on bare desk.

Here is the part worth noticing. In all three attempts your arm did the same thing, with the same confidence, and it felt the same doing it. Nothing about the third attempt told you it had failed. You looked once, committed to a number, and then executed against that number no matter what the world did afterwards.

That is the robot you are about to build. It works, genuinely and repeatably. It is also the reason the rest of this course exists, and the real deliverable of this project is not the demo. It is the measurement of how far the coaster can move before your arm stops finding it.

The idea in one paragraph

The milestone bolts your own primitives into one program: a two-link arm is told where a block is, drives its hand there through a sequence of poses, closes a notional gripper, carries the block to a bin, and goes home. No learning anywhere, and no second look at the block after the first reading. This is paradigm one, scripted motion, built honestly out of code you wrote. It will work almost perfectly. Then you will corrupt the one number it depends on, the reported position of the block, and watch a system that never failed slide to roughly one success in ten. The width of the error band it can absorb before that happens is a number, you can predict it in closed form, and measuring it is the point of the exercise.

What you are building

The milestone pick-and-place pipeline: an eight-state machine from idle to home, the four steps every transition runs underneath, and the evaluation sweep that measures where scripted motion breaks

Wider than the screen; scroll it sideways.

Eight states, in a line: idle, hover above the block, descend onto it, grasp, lift, carry to the bin, release, home. Each state names its successor. Underneath, every one of those transitions runs the same four steps. Compute a target pose for the hand. Solve inverse kinematics for the joint angles that reach it. Interpolate from the current joint angles to those. Step the animation one frame at a time so the motion is visible rather than instantaneous.

That is the whole architecture. Every piece of it is code you already wrote, doing the job it was written for. The gripper is the one piece of pretend: a boolean flag, plus drawing the block attached to the hand while the flag is true.

Five rules, and why each one is there

Rule	Why it is a rule
Import your own `transforms.py`, `fk_2link.py` and `ik_2link.py`. No robotics libraries.	The maths in your files has to be load-bearing. The first time a chain composes in the wrong order the arm swings somewhere absurd, and you fix it in your code rather than in a library’s issue tracker.
Seed every IK call from the current joint angles.	A two-link arm reaches most points two ways, elbow up and elbow down. Damped least squares returns whichever branch the seed falls into, so a fixed seed lets the elbow flip between waypoints.
Interpolate in joint space. Never teleport.	A pose is not a motion. The interesting failures live between waypoints, and you cannot see them if the arm jumps.
The grasp counts only within 1 cm and 10 degrees of the block.	This tolerance is the experiment. It is the width of the mug’s base.
Every run prints a structured log: state, duration, IK iterations, final placement error.	The writeup needs numbers, and numbers you did not print are numbers you do not have.

Joint space is not a straight line

One consequence of rule three is worth measuring before it surprises you, because it is the root of the whole trajectory-planning problem.

Interpolating linearly between two sets of joint angles does not move the hand in a straight line. Each joint sweeps at a constant rate, and the hand traces whatever curve that produces. On the short legs it barely matters: the 15 cm descent from hover onto the block bows off the vertical by 2.8 mm, which you would not see. On the long carry leg it matters a great deal. In my reference run the straight-line distance between the two hover poses was 1.73 m, and joint-space interpolation bowed the hand 34 cm away from that line, travelling 2.63 m of actual arc to cover it.

Nothing is wrong. This is what joint-space interpolation is. But it means that “the straight line between these two points is clear” tells you nothing about whether the motion is clear, which is why obstacle avoidance and Cartesian-space paths are their own subject.

Build it in five checkpoints

Do not build this as one program. Build five, each of which runs.

Checkpoint	You are done when
1. Scene and renderer	The arm waves sinusoidally, smoothly, with the block and the bin drawn in the right places.
2. `goto(target_xy, duration_s)`	The arm visits four corners of its workspace on command, with no jumps and no elbow flips.
3. The state machine	A full pick and place succeeds on the default block position, and the log prints eight state transitions.
4. Randomise and evaluate	`--headless -n 100` runs without rendering and prints a success rate and mean placement error.
5. The noise sweep	`--noise` corrupts the reported block position, and you have five success rates and a plot.

Checkpoint 4 is where the arithmetic of the module pays out. The arm reaches 1.7 m and the far corner of the block box sits at 1.581 m, so every block is comfortably reachable with 11.9 cm of margin. A correct implementation therefore sits at or extremely close to 100% here; my reference implementation was 10,000 for 10,000. If yours is at 97%, the missing 3% is a bug in your own code, and worth every minute of finding.

The hover pose spends that margin, and you are the one who picks the hover height. Above the far corner of the box, hovering 15 cm leaves 2.9 cm of reach to spare, and hovering 20 cm puts the target 2.9 mm outside the workspace, where no solution exists at all. The crossover is at 19.5 cm. Choose 20 cm because it looks tidier, and you have bought a rare failure that fires only in one corner of the sampling region, which is the most irritating class of bug there is.

If your failures do cluster near that corner, you are looking at the conditioning problem from singularities inside your own program. At the centre of the box, moving the hand 1 cm in the stiffest direction costs about 0.9 degrees of joint motion. At the far corner it costs 2.0 degrees, because the arm is straighter there and the Jacobian’s smaller singular value has fallen from 0.62 to 0.28. Add a hover height on top and that number keeps climbing.

What done means

Concretely, all of this:

python pick_and_place.py runs the animated demo, and it looks like a robot rather than a slideshow.
python pick_and_place.py --headless -n 100 prints a success rate and a mean placement error.
Success at zero noise is at or very near 100%, and every failure has been chased to a named cause.
The sweep over --noise exists, as five numbers and one plot.
A writeup of about a page: what you built, the numbers, the plot, and a closing paragraph answering “what would it take to make this robust?”

That last paragraph is the assignment. Everything above it is the setup.

The experiment that is the point

Now break it, in the one way that matters.

Add Gaussian noise to the reported block position, and to nothing else. The arm still moves perfectly. The maths is still exact. The only thing that changed is that the number the script was handed no longer matches the world, which is precisely the situation any real camera puts you in.

A dashed circle of one centimetre radius drawn around the block's true position. A reported position that lands inside the circle leads to a successful grasp; a reported position outside it leads to the gripper closing on air. The five states after the grasp run identically in both cases.

Wider than the screen; scroll it sideways.

The failure mechanism has no moving parts. The gripper goes where it was told. The block is somewhere else, off by exactly the perception error. If that offset is smaller than the grasp tolerance, the block is caught, held slightly off-centre, and delivered. If it is larger, the gripper closes on air and the remaining five states run exactly as before, carrying nothing to the bin with perfect precision.

So the success rate is not really a property of your robot. It is the probability that a two-dimensional error vector lands inside a disc of radius 1 cm. For Gaussian error with standard deviation $\sigma$ on each axis, that probability has a closed form:

P(\text{success}) = 1 - \exp\!\left(-\frac{r^{2}}{2\sigma^{2}}\right) \qquad r = 0.01\ \text{m, the grasp tolerance}

Read it as: the chance of missing is $\exp(-r^{2}/2\sigma^{2})$ , which collapses towards zero as the tolerance grows relative to the noise. When $\sigma$ is a fifth of $r$ , the miss probability is under four in a million. When $\sigma$ equals $r$ , it is 61%.

Grasp success rate against perception error sigma. The curve is flat at one hundred per cent while sigma is below about two millimetres, falls to eighty-six per cent at five millimetres, thirty-nine per cent at ten millimetres, and twelve per cent at twenty millimetres. Measured episode counts sit on the closed-form curve.

Wider than the screen; scroll it sideways.

Sweep $\sigma$ over 0, 2, 5, 10 and 20 mm and your five measured points should land on that curve. Mine did: 100%, 100%, 86.1%, 39.2% and 11.5%, over 10,000 episodes at each $\sigma$ , against a prediction of 100%, 100%, 86.5%, 39.4% and 11.8%. If your points sit below the line, the gap is yours: something in your implementation is losing grasps the geometry says you should be winning. That gap is the most informative thing in the project.

Read the shape, not just the endpoints. The curve is flat, then it falls off a cliff. Two millimetres of error costs nothing at all; ten millimetres costs 61 percentage points. That is what a hard tolerance does to a system, and it is why “our perception is accurate to about a centimetre” is not a reassuring sentence.

What the curve is telling you

There is a second number worth pulling out of the sweep. Among the runs that succeed, the mean placement error barely moves: 2.5 mm at $\sigma = 2$ mm, 5.4 mm at $\sigma = 5$ , then it saturates around 6.6 mm and stops responding. The reason is arithmetic. Once $\sigma$ is large, the errors that survive the 1 cm filter are spread almost evenly across the disc, and the mean distance from the centre of an evenly covered disc of radius $r$ is $2r/3$ , which is 6.7 mm. So that number converges to a constant set by the gripper rather than by your perception. The success rate is the only honest metric here.

Which leaves three levers, and only three.

Shrink $\sigma$ . Better perception, better calibration, better lighting. This is what a factory buys.

Widen the tolerance. A bigger gripper aperture, a funnel, a chamfer, a spring that lets the part slide into place. This is mechanical, and it is often cheaper than the first lever.

Close the loop. Keep looking while you move, so the target updates as the hand approaches and the error never gets a chance to be committed to.

Lever three is the rest of this course. Module 2 puts your arm in a physics engine so contact and gravity are real instead of assumed. Module 3 replaces the state machine with a policy learned from demonstrations, and scores that policy against a scripted tracker as the starting pose is moved off the demonstrated one. The perturbation is a different one from yours, but the habit is the same habit: pick the thing the system assumes, vary it, and find where the assumption stops holding.

Check yourself

1. Why does the success curve stay flat and then collapse, instead of degrading smoothly from the start?

Because the grasp is a hard threshold, not a gradient. Success requires the error vector to land inside a disc of radius 1 cm, and while $\sigma$ is small almost every sample lands there, so the rate is pinned at 100%. Once $\sigma$ approaches the radius, the fraction falling outside grows quickly. Formally, the success rate is $1 - \exp(-r^{2}/2\sigma^{2})$ : it is flat while $\sigma \ll r$ because the exponent is hugely negative, and it falls steeply once $\sigma$ and $r$ are comparable. Any system with a fixed tolerance behaves this way, which is why “we are usually within tolerance” is a claim about a cliff edge rather than a slope.

2. You seed every IK call from a fixed constant rather than from the arm’s current joint angles. What do you see, and why?

Occasional violent sweeps between waypoints, most likely on the long carry leg, and possibly the elbow visibly flipping from up to down. A two-link arm reaches most points in two configurations, and damped least squares converges to whichever one the seed’s basin leads to. Seeding from the current pose keeps the solver in the branch the arm is already in. In my reference run the largest joint moved 184 degrees on the carry leg when seeded from the current angles, and 536 degrees when seeded from a constant, for identical start and end poses. Both are “correct” in that the hand arrives; only one is a motion you would run on hardware.

3. At σ = 20 mm the mean placement error among successful runs is about 6.6 mm, barely worse than the 5.4 mm at σ = 5 mm. Why is that number useless for monitoring?

Because it is conditioned on success, and success already filtered out every error larger than 1 cm. Once $\sigma$ is much bigger than the tolerance, the surviving errors are spread nearly evenly over the disc, and the mean distance from the centre of an evenly covered disc of radius $r$ tends to $2r/3$ , or 6.7 mm. The metric saturates at a constant set by the gripper, not by the perception. Meanwhile the success rate has fallen from 86% to 11%. Watching the placement error would have shown you almost nothing while the system fell apart.

4. Your success rate at zero noise is 96%, not 100%. Name three plausible causes and how you would separate them.

First, an IK call that did not converge and returned its best effort: check the convergence flag and the residual on every call rather than trusting the returned angles, and log the iteration count. Second, an elbow flip between waypoints producing a path that leaves the workspace or overshoots: log the per-waypoint joint travel and look for outliers. Third, a frame or sign convention error that only shows up in part of the region, such as a hover offset applied in the wrong direction: plot the failing block positions and see whether they cluster. Clustering is the tell. Failures spread uniformly point at the solver; failures gathered at one edge point at geometry or conditioning.

5. Without touching the arm, name three ways to raise the success rate at σ = 10 mm, and say which lever each one pulls.

Improve the perception so the reported position is closer to the truth, which shrinks $\sigma$ . Fit a wider gripper, a funnel or a compliant mount so a near miss still ends in a grasp, which widens the tolerance $r$ . Or re-read the block position partway through the approach and re-plan the remaining motion, which closes the loop and stops the error being committed to at the start. The first two move the two terms in $1 - \exp(-r^{2}/2\sigma^{2})$ . The third leaves that formula behind entirely, and it is the one the rest of this course is about.

6. Why is the sweep the deliverable rather than the working demo?

Because the demo only tells you the program is correct, and correctness was never in doubt once the maths was right. The sweep tells you the size of the error budget: how far the world may drift from the script’s belief before the script stops working. That is the question Module 3 asks of learned policies too, sweeping the starting pose instead of the perceived block position, and asking it here first is what turns “scripted motion is brittle” from an opinion you were told into a measurement you made.

Do this

Build it. The scaffold is at project/pick_and_place.py, and there is no solution file for this one, deliberately; the whole module has been leading here.

The scaffold gives you the constants, the class skeleton and the command-line interface, and stubs out three things with NotImplementedError: Arm.goto, PickAndPlace.step and run_episode. Fill them in against the five checkpoints above, in order, running the thing at every stage. Import your own libraries from code/ rather than copying code into the project file, so that a bug you fix in ik_2link.py is a bug fixed everywhere.

Two commands drive everything:

python pick_and_place.py
python pick_and_place.py --headless -n 100 --noise 0.005 --seed 1

The first is the animation you show people. The second is the one that produces the writeup. --noise is in metres, so 0.005 is the $\sigma = 5$ mm row of the sweep.

For the sweep, run -n 500 or more at each of $\sigma = 0$ , 0.002, 0.005, 0.010 and 0.020, and plot success rate against $\sigma$ . At -n 100 the sampling noise on a 39% rate is about five percentage points, which is enough to make the curve look bumpy and send you hunting for a bug you do not have. Overlay $1 - \exp(-r^{2}/2\sigma^{2})$ on your measured points. Where they agree, your implementation is sound. Where they disagree, you have found something.

Then write the page. What you built, the numbers, the plot, and an honest answer to what it would take to make this robust. Keep it; Module 3 puts this same scripted approach up against policies that learned the task, and reports the result in the same shape.

What you can now do

You can assemble a complete robot program out of primitives you implemented yourself: a state machine over waypoints, each waypoint resolved by inverse kinematics and reached by interpolated motion, evaluated over hundreds of randomised episodes without a human watching. You can explain why a scripted arm has a hard error budget rather than a soft one, predict that budget in closed form from the grasp tolerance and the perception error, and confirm the prediction by measurement. And you can name the three ways out of it, which is the map for everything that follows.