Where you are. You can describe a robot as a sense-decide-act loop and name its parts. This lesson opens the “sense” box and shows you the literal numbers that fall out of it.
A block of 921,600 integers
Something is sitting on the desk in front of you. Here is what the machine receives.
A block of 921,600 integers, each between 0 and 255, arranged as 480 rows by 640 columns by 3 channels. Row 213, column 447, channel 0 holds the value 178. A new block arrives every 33 milliseconds, forever, at roughly 28 megabytes per second.
That is the entire input. Nothing in it says “mug”. Nothing says how far away the mug is, whether it is glass, or which of the 921,600 numbers belong to it rather than to the desk behind it. Walk out and change nothing, and the next block still will not match this one: values wobble by a few counts because photons arrive at random times, and every few seconds the auto-exposure decides the scene got brighter and shifts all of them at once.
Now name it. That is a camera, the richest sensor on the robot.
Every other sensor works the same way. It hands you an array with a unit, a shape and a rate. It never hands you a fact.
The idea in one paragraph
A sensor returns a measurement, not a state. The measurement is a number with a physical unit, produced at some rate, late by some amount, and wrong by an amount that changes over time. What you want to act on - where the gripper is, where the object is, whether you are gripping hard enough - is in none of those arrays. It has to be reconstructed from them continuously, while they disagree with each other. That reconstruction is state estimation, and it is not an advanced topic to bolt on later. It is the mandatory adapter between the numbers you have and the quantities every controller and every learned policy in this course assumes it can simply read.
Wider than the screen; scroll it sideways.
The joint encoder: an integer, not an angle
An encoder measures one joint’s rotation, and what it returns is a count. Hobby-class servos, including the SO-101’s, typically report position as a 12-bit number: 4096 steps across a full turn. Industrial motors commonly use 17-bit or finer absolute encoders, which is 131,072 counts per revolution.
Converting a count into an angle needs three things the sensor does not supply: a scale in degrees per count, a zero offset saying which count means “straight”, and a sign saying which way counts up. Those live in a calibration file you wrote. Get one wrong and the arm simply moves somewhere else, confidently, with no error raised.
4096 counts per turn is 0.088 degrees each, and anything finer does not exist as far as your code is concerned. For position that is fine. For velocity it is not, because almost no encoder measures velocity; you difference consecutive positions instead. At a 1 kHz loop samples are 1 millisecond apart, so one count of jitter reads as nearly 90 degrees per second of motion that never happened.
The camera: pixels, late
Colour arrives as a uint8 array of shape (height, width, 3), values 0 to 255, typically at 30 Hz. Depth, if you have it, is a separate uint16 array in millimetres where 0 means “no reading here”, not “zero distance”. None of these numbers is proportional to light: a nonlinear tone curve, white balance and auto-exposure sit between the scene and the array, and they keep moving while you watch.
Three physical effects decide what those numbers are worth. Photons arrive as a Poisson process, so shot noise grows as the square root of the signal and dim scenes get unreliable fast. Most CMOS sensors use a rolling shutter, reading out row by row, so the bottom of the frame is captured milliseconds after the top and a wrist camera in motion shears the whole scene. And blur is exposure time times speed, so at 1/60 second a gripper moving 0.5 m/s smears about 8 millimetres; shortening the exposure to fix that lets in less light, which brings the noise back.
Then latency, which people consistently underestimate. Between photons landing and your process holding an array you pay exposure, readout, USB transfer, driver handling and often a JPEG decode. For a consumer webcam that total is commonly tens of milliseconds and can exceed 100 milliseconds. At 0.5 m/s, 80 milliseconds is 4 centimetres. The image describes a world that has already moved.
Wider than the screen; scroll it sideways.
The IMU: six floats and an accumulating lie
An IMU returns six floats: three accelerometer axes in metres per second squared, three gyroscope axes in radians per second, at rates from about 100 Hz to several kilohertz. It is small, cheap, fast, and it works in the dark.
An accelerometer sitting still reads about 9.81 m/s² pointing up, not zero: it measures the force holding it up rather than motion through space, and in free fall it reads zero. Gravity is mixed into every sample, and separating it out means knowing which way is down, which is one of the things you hoped the IMU would tell you.
The error that matters, though, is not the noise but a bias that wanders slowly with temperature and time. An uncalibrated consumer gyro can sit a fraction of a degree per second to several degrees per second off while perfectly motionless. That bias is the whole story, because an IMU is only useful integrated: rate integrated once gives angle, acceleration integrated twice gives position.
Wider than the screen; scroll it sideways.
The force/torque sensor: honest until it warms up
A wrist force/torque sensor returns six floats: three forces in newtons, three torques in newton-metres, at hundreds of hertz to a few kilohertz. Underneath it is a metal flexure with strain gauges bonded to it; the raw signal is a few millivolts from Wheatstone bridges, and a calibration matrix turns those into the six numbers.
It measures everything distal to it, which is more than you asked for. The gripper’s own weight is in there, so the reading at rest is not zero and changes as the wrist rotates; subtracting that gravity term needs the wrist’s orientation from the encoders, so your force reading inherits the encoder’s errors. Accelerating the tool registers as force with nothing touching anything. And the zero drifts as the sensor warms up, which is why you re-zero immediately before contact rather than once at startup.
The cheap substitute is motor current: torque is roughly current times a constant, corrupted by friction, gearing and stiction. That is what a hobby servo gives you. Useful for “did I hit something”, useless for “am I pressing with 2.0 newtons”.
The four, side by side
| Sensor | What it literally returns | Typical rate | Dominant error | How it fails |
|---|---|---|---|---|
| Joint encoder | integer counts, one per joint | 100 Hz - 1 kHz | quantisation, amplified when differenced for velocity | backlash and flex between encoder and joint; a dropped packet returns a stale reading with no error flag |
| Camera | uint8[H,W,3], plus uint16[H,W] mm for depth | 30 Hz | latency of tens of milliseconds | depth holes on shiny, clear and dark surfaces; exposure shifts look like the world changed |
| IMU | 3 accelerations in m/s², 3 rates in rad/s | 100 Hz - several kHz | slowly wandering bias | unbounded drift once integrated; gravity mixed into acceleration |
| Force/torque | 3 forces in N, 3 torques in N·m | 100 Hz - few kHz | thermal zero drift | an overload deforms the flexure and silently shifts calibration; blind until contact |
Why a better sensor does not fix this
Read that last column again. These are not four versions of one problem. The encoder is precise and fast and knows nothing about the world. The IMU is fast and smooth and drifts without bound. The camera is world-referenced and absolute but slow, late, and meaningless until something infers from it. The force sensor is exact and silent until you touch something.
That difference is the resource. Sensor fusion uses one sensor’s strength against another’s weakness: hold a running estimate, push it forward at high rate with the fast signal, and correct it whenever a slow absolute measurement arrives, weighting each by how much you trust it. A Kalman filter is one principled way to choose those weights; the mathematics is bookkeeping on top of that idea.
Two things get built directly on top of this. Module 1 turns counts into geometry: six numbers on a serial bus do not tell you where the gripper is, and getting from joint angles to a pose needs frames, rotations, transforms and forward kinematics. Module 3 attacks the camera end: classical perception builds an explicit pipeline from pixels to object poses and breaks on the eleventh object, while learned policies map pixels to actions directly. The array is not the fact, and both modules are answers to that.
Check yourself
1. Your encoder resolves 0.09 degrees and your loop runs at 1 kHz. Why is differenced velocity useless, and why is averaging not a free fix?
Across 1 millisecond a real joint often moves less than one count, so the differenced velocity is a train of zeros interrupted by spikes worth about 90 degrees per second. That is quantisation noise, not motion. Averaging suppresses it, but a longer window describes an older interval, so you have traded noise for lag, and lag costs stability in a feedback loop. Encoder resolution, filter width and achievable control bandwidth are one connected decision, not three.
2. A wrist camera reports the mug’s position. Pipeline latency is 60 ms and the arm moves at 0.4 m/s. What is wrong with using that number directly?
It describes where the mug was relative to the arm 60 milliseconds ago, and the arm has moved about 2.4 centimetres since. Motion degrades the measurement itself too: rolling shutter shears the geometry because rows were captured at different instants, and the exposure blurs the edges. The fix is not only a faster camera. You propagate the estimate forward between frames with a fast signal such as encoders, and use each frame to correct that estimate rather than replace it.
3. A stationary robot’s accelerometer reads roughly (0, 0, 9.81). Nothing is moving. Explain it, and say what it reads in free fall.
An accelerometer measures the force needed to hold its proof mass in place, not motion through space. At rest the structure pushes up at 9.81 m/s² to oppose gravity, and that is what it reports; in free fall nothing holds the mass, so it reads about zero. The consequence is that gravity dominates the signal, so recovering real acceleration means subtracting a 9.81 m/s² vector whose direction you only know approximately, and orientation error leaks straight into the result.
4. Your force sensor reads 3 N with an empty gripper and nothing in contact. Give two explanations and say how you would separate them.
Either it is reading the weight of everything mounted past it plus a thermal zero offset, or the tool is accelerating and you are measuring its inertia. Hold the arm still: if the reading persists it is gravity plus offset, and if it vanishes it was inertial. Then rotate the wrist in place. A component that changes with orientation is gravity; one fixed in sensor axes is the thermal offset, which re-taring removes. If no taring keeps the zero stable, suspect an earlier overload deformed the flexure.
5. Forward kinematics says the gripper is at m; the camera says 0.34 m. Which is right?
The question is malformed, and noticing that is the point of the lesson. Neither number is a measurement of gripper position. One is joint counts pushed through a kinematic model assuming exact link lengths, zero backlash and a correct calibration offset; the other is pixels pushed through a camera calibration and a detector. Both are inferences with their own errors, and 4 centimetres of disagreement is ordinary. An estimator weights the two by their expected error and produces a third number matching neither; a persistent bias in what is left over is how you discover a wrong link length or camera mounting.
Do this
Ten minutes in a Python REPL, to make the integration trap concrete rather than rhetorical.
import numpy as np
dt, T = 1 / 200, 60.0 # 200 Hz IMU, one minute of standing perfectly still
n = int(T / dt)
bias = np.deg2rad(0.5) # 0.5 deg/s of gyro bias
noise = np.random.normal(0, 0.02, n) # rad/s, zero mean
angle_bias = np.cumsum(np.full(n, bias)) * dt
angle_noise = np.cumsum(noise) * dt
print("bias ->", round(float(np.rad2deg(angle_bias[-1])), 1), "deg")
print("noise ->", round(float(np.rad2deg(angle_noise[-1])), 1), "deg")
accel_bias = 0.01 # m/s^2, about one thousandth of gravity
vel = np.cumsum(np.full(n, accel_bias)) * dt
pos = np.cumsum(vel) * dt
print("position ->", round(float(pos[-1]), 1), "m")The bias line prints 30.0 degrees every run. The noise line differs every run and is typically under two degrees. The position line prints 18.0 metres. Sit with that ratio: the zero-mean noise that dominates the raw signal on a plot contributes a rounding error next to a bias too small to see by eye.
Then change one thing at a time and predict both drift figures before running. Halve bias. Double T. Raise the rate with dt = 1 / 1000. One of the three leaves both figures untouched, and one quadruples the position error rather than doubling it. Working out which is which without running the code is the difference between having the numbers and having the model.
What you can now do
You can say what a joint encoder, a camera, an IMU and a force/torque sensor literally return, in units and shape and rate, and name each one’s dominant error and silent failure mode. You can explain why integrating a biased signal is a trap and why error grows faster for position than for angle. And you can argue from the shape of the errors, rather than from authority, that state estimation is a structural requirement of any robot rather than an optional refinement.