Where you are. You can drive the arm and choose what its actuators mean. This lesson is the other half of the loop: what comes back, in what units, and at what cost.
An arm with no eyes and total recall
Load the SO-101 scene, step it once, and ask it what it can sense.
print(len(data.sensordata)) # 0Zero. Not a small number: an empty array. The best-maintained open model of this arm, thirteen state variables, forty-eight shapes, a camera mount modelled down to its twelve grams, declares no sensors at all.
Now ask an unfair question. Where is the green box on the table?
print(data.body("box").xpos) # [0.5 0. 0.03]Instantly, to the last bit of a float, with nothing pointed at it and no light in the room. Step a hundred times and ask again: [0.5 0. 0.0299388], still free, still exact, now including the fraction of a millimetre it has sunk into the table. The simulator does not perceive the box; it is the box.
That is the shape of the problem. Lesson 0.4 opened with 921,600 integers and no facts. A simulator hands you the opposite by default: every fact, no integers. Neither is what a robot has, and only one of them is a trap.
The idea in one paragraph
In simulation, sensing is declared, not discovered. You add a <sensor> element for each quantity you want measured, MuJoCo computes it every step and writes it into one flat array, and nothing appears in that array unless you asked for it. Cameras work the same way: a <camera> is a pose in the body tree, and a Renderer turns it into pixels only when your code says so. This inverts the usual difficulty. On real hardware you fight to get any signal at all; here the signal is free and the discipline is deciding what to leave out, because the simulator will happily hand your policy the exact position of an object it could never see, and a policy that learns to use that number has learned something no camera can supply.
The sensor block is a request list
Wider than the screen; scroll it sideways.
A <sensor> element names a kind and a target. The compiler assigns each one a slice of data.sensordata in declaration order, and that is the whole mechanism. Notice what most of them point at: a site, the massless collision-free marker from MJCF anatomy. Sites exist to be pointed at, and this is what they are for; a sensor that takes a body measures the body’s own frame, which is rarely where the sensing surface is.
<sensor>
<jointpos name="shoulder_q" joint="shoulder"/>
<framepos name="pad_xyz" objtype="site" objname="pad_site"/>
<touch name="pad_touch" site="pad_site"/>
<rangefinder name="tip_range" site="eye"/>
</sensor>Eleven sensors on the bench in this lesson produce eighteen floats, because some of them are wider than one number: a framepos owns three, a framequat owns four. model.sensor_adr and model.sensor_dim are the only record of which is which. Read the array by hand and you are counting offsets; read it as data.sensor("pad_touch").data and you are not.
Three families, and only two of them are sensors
| Family | Examples | Has a real counterpart |
|---|---|---|
| Proprioceptive | jointpos, jointvel, actuatorfrc | yes, this is what an encoder gives you |
| Contact and ray | touch, force, torque, rangefinder, accelerometer, gyro | yes, each is a part you can buy |
| Frame | framepos, framequat, framelinvel | no |
The third family is the interesting one. framepos reports the world position of any body, site or geom you name. That is a lookup, not a measurement. Point it at the arm’s own fingertip and it is forward kinematics, which a real robot can compute from its encoders. Point it at the cube and it is clairvoyance.
Run the bench and compare the two ways of asking:
data.body('cube').xpos = [0.3 0. 0.02]
sensor('cube_xyz') = [0.3 0. 0.02]Identical, digit for digit. Wrapping a fact in a <sensor> element does not measure it; it moves it into a different array.
Privileged information, and where it belongs
Wider than the screen; scroll it sideways.
Privileged numbers are not forbidden. They are essential, in the right place. Your scripted expert in the pick-and-place project needs to know where the cube is in order to reach for it, and computing that from rendered pixels would be a research project standing between you and a demonstration dataset. Success checks need it: “the cube’s centre is inside the bin volume” is a one-line assertion with framepos and an unsolved perception problem without it.
The line is not “avoid privileged information”. It is privileged numbers may decide what happens; they may not become an observation you record.
What a real sensor looks like when it is honest
Watch a single descent onto the cube on the bench: the pad comes down at nine centimetres per second, touches, and settles. Four numbers over three seconds tell the whole story, and each is honest in a different way.
pad_touch reads exactly 0.000 for one and a half seconds, then 4.499 newtons in a single 50 Hz tick, peaks at 8.05, and settles at 3.87 as the contact solver finds equilibrium. Contact is a step change; there is no gradual approach in that signal, which is why touch is superb for detecting an event and useless for anticipating one.
tip_range starts at -1. Not zero: minus one, MuJoCo’s sentinel for a ray that hit nothing, because at the home pose the wrist points out into empty air. It jumps to half a metre once the ray catches the table, then falls smoothly to a few millimetres as the pad descends. If you had treated -1 as a distance you would have concluded the sensor was reading a metre through the floor.
shoulder_tau, the actuatorfrc sensor, holds around 1.36 N·m while the arm descends and drops to 0.05 once the pad is resting on the cube: the table is now carrying the weight. That is the actuator force from the last lesson, reachable through a sensor rather than through data.actuator_force, and reaching it through a sensor does not make it any more available on hardware.
Cameras are model objects, not code
Wider than the screen; scroll it sideways.
A <camera> sits inside a body, so a camera on the wrist moves with the wrist for free. Its parameters are the ones a lens has: a pose, a field of view, a resolution. Installing MuJoCo covered the mechanics of getting pixels back; what matters here is what you choose to put in the scene.
Two placements cover almost every manipulation setup, and the bench has one of each. A workspace camera is fixed to the world and sees the table, the arm and the object; it gives context and loses the object behind the gripper at exactly the moment the grasp happens. A wrist camera is bolted to the hand and sees the object close up and consistently framed; it has no idea where the table is. Almost every serious imitation-learning setup records both, for exactly that reason. The SO-101 model ships a camera_mount body with a wrist_cam already in it, at 1920 by 1080.
The renderer returns three different arrays, and they are modes rather than separate objects. Colour comes back uint8 with shape (h, w, 3). Turn on depth and the same renderer returns float32 with shape (h, w), in metres from the lens. Turn on segmentation and it returns int32 with shape (h, w, 2), holding a geom id and an object type per pixel, which gives you a free, pixel-perfect mask of the cube for building a success check or a training label.
Rendering is the expensive part
Physics is cheap. Pictures are not.
| Operation | Time | In physics steps |
|---|---|---|
mj_step at a 2 ms timestep | 0.028 ms | 1 |
| render 320 × 240 | 9.0 ms | 318 |
| render 480 × 360 | 10.1 ms | 357 |
| render 640 × 480 | 11.3 ms | 400 |
n = 200 trials · one physics step and one overhead frame, medians on a 2018 four-core laptop with integrated graphics · 2026-08-09
Your machine will produce different numbers; the ratio is the point, and so is its shape. Quadrupling the pixel count from 320 × 240 to 640 × 480 adds only a quarter to the cost, because most of a frame is fixed overhead rather than shading, so rendering smaller buys much less than you would hope. One 480 × 360 frame costs 357 physics steps, which at a 2 ms timestep is three quarters of a second of simulated time. A loop that renders after every step therefore spends over 99% of itself drawing pictures nobody asked for.
The fix is the architecture from Lesson 0.1: a fast layer and a slow one. Step physics at 500 Hz, run the controller at 50 Hz, and render at 10 to 30 Hz because that is what the real camera does anyway. The bench script uses exactly those three rates, and it is not an optimisation; it is a model of the real machine, where the camera genuinely cannot deliver faster.
Check yourself
1. data.sensordata has length 18 and your model declares 11 sensors. Explain, and say how you would find which sensor owns index 7.
Sensors have widths. A jointpos occupies one float, a framepos three, a framequat four, so eleven declarations can easily total eighteen. model.sensor_adr[i] gives the starting index of sensor i and model.sensor_dim[i] its width, so index 7 belongs to whichever sensor’s address-plus-width straddles it. In practice you never do this arithmetic: data.sensor(name).data returns the right slice.
2. You add a framepos sensor on the cube so that “the arm can sense the cube’s position”. What have you actually built?
A lookup with a sensor’s name on it. framepos copies the simulator’s own bookkeeping into data.sensordata; it is identical to data.body("cube").xpos digit for digit and requires no camera, no line of sight and no lighting. The cube can be behind the arm, in the dark, inside a box, and it still reports perfectly. It is privileged information, useful for scripting and scoring, and disqualifying in an observation vector.
3. A rangefinder reads -1. Your controller averages the last ten readings and steers on the result. What happens, and what is the general rule?
The -1 values drag the average negative and the controller steers on a distance that is not merely wrong but impossible. -1 is MuJoCo’s out-of-band value for “the ray hit nothing”, not a measurement. The rule is that a sensor’s sentinel is part of its type: check for it before any arithmetic, exactly as you would check for null rather than letting it coerce to zero.
4. Your training loop renders a frame after every mj_step and runs far slower than you expected. Two things are wrong. Name both.
Cost and realism. A frame costs roughly three hundred physics steps, so rendering every step means well over 99% of the run is drawing pictures. And a camera that produces an image every 2 ms is 500 Hz, which no camera you will ever attach to this robot can do, so the dataset would be teaching the policy to expect observations at a rate that does not exist. Render at the rate the hardware camera runs, typically 10 to 30 Hz.
5. You render depth for a debug image, then later the same script writes a PNG that comes out black. What is the most likely cause?
The renderer is still in depth mode. enable_depth_rendering() is a sticky flag, so every later render() on that object returns float32 metres rather than uint8 colour, and an image writer given metres in the range 0.4 to 1.5 produces something nearly black. Disable the mode in the same block that enabled it, or use a separate renderer for each mode.
6. The SO-101 Menagerie model declares no sensors at all. Is that a gap in the model?
No, it is the right default. A model describes a machine; which quantities you choose to measure is a property of your experiment, not of the arm. Two people using the same file may want completely different observation vectors, and the model has no basis for guessing. The consequence for you is that the observation design is yours, and nobody will warn you when you get it wrong.
Do this
Finish code/sensors_and_cameras.py. Three # TODO(you) markers, about thirty minutes. It runs under plain python; no viewer, no mjpython.
1. Two sensor declarations. The <sensor> block is missing a touch sensor on pad_site and a rangefinder on the eye site. Add them; data.sensordata should grow from 16 floats to 18, and the layout printout should show them at addresses 13 and 14.
2. read(data, name). One line: return np.array(data.sensor(name).data). The np.array matters. .data is a live view onto sensordata, so a value you keep without copying changes under you on the next step, and a log built that way ends up as N copies of the final reading.
3. The depth render. Enable depth mode, render the overhead camera again, and disable it. The script prints the dtype, shape and range so you can check it, and writes sensor_views.png with all four panels.
Then two experiments:
- Add
<jointpos name="elbow_noisy" joint="elbow" noise="0.01"/>and print it beside the cleanelbow_qacross a hundred steps. They are identical. Now write your own:read(data, "elbow_q") + rng.normal(0, 0.01), and notice that you had to decide on a seed, a distribution and a standard deviation, all of which the XML attribute silently declined to do for you. - Change
CAM_HZfrom 10 to 500, so the camera fires on every physics step, and watch the wall clock the script prints. Three seconds of simulation went from0.58 sto15.08 son the machine those timings came from, a 26-fold slowdown, and not one number in the physics changed. Then write down the frame rate of the webcam on the desk in front of you, and setCAM_HZback.
What you can now do
You can declare sensors in MJCF, find any reading by name without counting offsets, and say which of MuJoCo’s sensor families correspond to hardware you could buy. You can render colour, depth and segmentation from a camera you placed yourself, and you know which of the renderer’s switches will bite you later. You can state the cost of a frame in physics steps and pick three rates for a loop rather than one. Most importantly, you can look at an observation vector and answer the question that decides whether a policy will ever leave your laptop: could the real robot compute every number in here?