Where you are. You can build a task scene, randomise it, and run a scripted expert that picks the cube up most of the time. This lesson turns those runs into the file Module 3 trains on.
Play it back
Record one episode. It comes out as 166 rows covering 6.64 seconds. Each row holds six measured joint angles, six commanded joint angles, and one 128 by 128 image. Nothing exotic; you could open it in a spreadsheet if the pictures were not in the way.
Now throw the robot away. Reset the simulator, put the cube back where it started, delete your inverse kinematics, delete the waypoints, delete the state machine. Feed the simulator column two of that table, one row at a time, and step.
The cube goes in the bin. Not approximately: it lands on exactly the same three numbers as the original run, bit for bit, and the joint angles you pass through on the way match the ones in column one exactly.
Now do it with the table your first attempt produced. The cube still goes in the bin. Same row count, same file size, same column names, same outcome. The only difference is that the angles you pass through on replay disagree with the ones in column one by up to 11.7 degrees, because every observation in that file was written one frame too late. Nothing you can look at says so. Only playing it back and comparing does.
The idea in one paragraph
A recorded episode is a contract with a model you have not written yet. The contract says: here is what the robot could see, here is what it did next, and doing that thing from that state really did produce the state on the following row. A dataset format enforces the parts of that contract it can see - names, dtypes, shapes, a fixed frame rate, episode boundaries - and it is completely blind to the parts that matter most, which are whether your actions are aligned with your observations, whether you recorded anything a real robot cannot sense, and whether you saved the one number you need to reproduce the run. Getting the schema right takes an afternoon. Getting the contract right is the difference between fifty demonstrations and fifty megabytes.
Two rates, and one whole number
Your physics runs at whatever model.opt.timestep says; the task scene uses 0.005 seconds, so 200 steps per simulated second. Your dataset should not run at 200 Hz: a policy emitting a command every 5 ms would have to be tiny, and real cameras top out near 30 frames per second.
So a dataset frame covers several physics steps, and the number of steps per frame has to be a whole number.
Wider than the screen; scroll it sideways.
At 25 frames per second, one frame is seconds, which is exactly 8 steps of 0.005. At 30 frames per second it is 6.667 steps, which is not a number of steps at all, and every way of rounding it is wrong differently. Hold each action for 7 steps and every frame runs 1.67 ms long, so the clock drifts without bound. Snap each frame to the nearest step instead and the actions get held for 6 or 7 steps unpredictably, with timestamps sitting up to 2.5 ms - half a timestep - off the grid.
Observe, then command, then step
That last sentence in the term box is the whole alignment contract, and it is three lines of code in an order that is easy to get wrong.
Read the observation first, from a state nothing has acted on yet. Then decide the action and write it to data.ctrl. Then let the physics run for exactly one frame’s worth of steps with that command held. The row you just wrote says “from here, this was done” - and the next row’s observation is what doing it produced.
There is a smaller version of the same idea in the dtype. The schema stores actions as float32; MuJoCo’s data.ctrl is float64. Cast the command to float32 before you write it into ctrl, not on the way into the file, and the number the simulator executed is the number the file holds. Skip that and replay lands 2.4 micrometres away from the recorded run, which is harmless here and is exactly the sort of quiet mismatch that stops being harmless when the thing reading your file is a neural network being asked to predict it.
The schema, named exactly
Names are not yours to choose. They are how a policy trained on your simulated arm ends up readable by a training script written for the real one.
Wider than the screen; scroll it sideways.
Four keys go in, and the writer computes the rest. observation.state is the six measured joint angles as float32. action is the six commanded angles, same dtype and shape. observation.images.<camera> is a uint8 array of height, width and three channels, one key per camera. And task is a plain sentence, repeated on every frame of the episode, because a modern policy is conditioned on language rather than on an episode id.
Each numeric feature also carries a names list, one label per element. For the SO-101 those labels are shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos and gripper.pos, in actuator order. This is the payoff for having used the robotstudio_so101 model back in reading the arm model: its actuator names already match the hardware, so your simulated columns line up with columns recorded from a physical arm without a translation layer in between.
What must never be a feature
Your simulator knows the cube’s exact position. data.body("cube").xpos is right there, it is free, and a policy trained on it will solve this task almost immediately.
It will also be useless: on real hardware that column does not exist, and there is no sensor that returns it. You would have trained a policy on an input you cannot supply, and you would find out last. This is the oracle problem from sensors and cameras, and the dataset is where it does its real damage, because a feature written into a file is a decision that outlives the afternoon you made it in.
The rule is mechanical. A key under observation. must correspond to something the physical robot can measure: joint encoders, cameras, a force sensor if you have one. Everything else goes beside the dataset, not inside it.
The things the format has no column for
There is a second list, and it is the one people forget, because nothing rejects a dataset for lacking it.
| What | Why you will want it | Where it goes |
|---|---|---|
| The random seed | Regenerate the episode at a higher resolution, or with a second camera, without re-scripting anything | Episode metadata |
| The cube’s placed pose | The replay test needs the exact starting state, and arm.cube() after settling is not it | Episode metadata |
| Success, and how it was judged | Filtering demonstrations later; comparing a policy against the expert on the same criterion | Episode metadata |
| Simulator and model versions | A minor MuJoCo release changes contact behaviour, and your old episodes will not reproduce | Dataset metadata |
The seed one is worth dwelling on. Driving the sim from Python established that the same starting state plus the same commands gives bit-identical results. That property turns a saved seed into a compressed episode: keep it, and the pixels are regenerable at any resolution you later wish you had used. Lose it, and the 128 by 128 images you happened to save are all you will ever have.
Where the files go, and why it does not look like a folder per episode
Wider than the screen; scroll it sideways.
The v3.0 layout has four parts. meta/info.json holds the schema, the fps and the path templates. meta/stats.json holds per-feature mean, standard deviation, min and max, which is what normalises your inputs at training time. meta/tasks.parquet maps task strings to integer ids. meta/episodes/ holds one record per episode: its length, its task, and its offsets. Then data/ holds the numeric columns as Parquet shards and videos/ holds one MP4 per camera per shard.
The design principle is that storage is decoupled from the user API. An episode is not a file. It is a row range inside a file that holds many episodes, resolved through the metadata when you ask for it. That is why adding your fifty-first episode does not add a fifty-first file, and it is why the metadata is not optional bookkeeping - it is the index without which the bytes mean nothing.
Record here, write there
Recording needs a physics engine. Writing a LeRobotDataset needs the lerobot package, which needs PyTorch. Keep the boundary between them explicit: a recorder that emits validated frames, and a writer that consumes them.
The writer is short, and it is the documented sequence:
from lerobot.datasets import LeRobotDataset
dataset = LeRobotDataset.create(repo_id="you/so101_pick_sim", fps=25,
features=features(), robot_type="so101_follower")
for episode in episodes:
for frame in episode: # each frame is the four-key dict above
dataset.add_frame(frame)
dataset.save_episode()
dataset.finalize() # not optional
dataset.push_to_hub()Everything hard happened before this block ran.
Check yourself
1. Your recorder samples every 6.67 physics steps to hit 30 fps on a 0.005 s timestep, rounding to 7. What breaks, and when do you find out?
The timestamps drift. Each frame advances the simulator by 0.035 s instead of 0.0333, so by frame 100 your clock is 0.17 s ahead of what frame_index / fps says. Nothing breaks at record time - the arrays are all the right shape - and it surfaces at load time, when the dataset checks timestamps against the nominal grid within tolerance_s, which defaults to a tenth of a millisecond. Fix it at the source: pick an fps whose period is a whole number of timesteps, or change the timestep.
2. Why does the alignment rule say “observe, then command, then step”, and what does the wrong order produce?
The row is a claim that from state the robot did , and that this produced . Reading the observation before anything acts on it is what makes the claim true. If you step first and record afterwards, each row pairs an action with the state it already caused, so the training target leaks the future. The resulting policy is trained to output the action appropriate to a state one frame ahead of the one it is shown, which on hardware looks like a lag or a tuning problem rather than a data bug.
3. You add observation.cube_pose to the schema so the policy has an easier time. What have you actually done?
Trained a policy on an input that does not exist. Nothing on the physical arm measures the cube’s pose directly; that number comes from the simulator’s own state. The policy will learn quickly, score well in sim, and have no way to run anywhere else. Keep the cube’s pose - it is useful for scoring, debugging and analysis - but store it as episode metadata beside the dataset, not as a feature inside it. The test is: name the device that produces this number on real hardware.
4. Replay has two assertions: the outcome matches, and the states match. Why is the second one not redundant?
Because the action column is a valid open-loop trajectory whether or not it lines up with the observations stored beside it. Shift the observation column by one row and the replay still puts the cube in the bin, so the outcome assertion passes on a file that is unusable. The state assertion is the one that fails, by up to 11.7 degrees here. The outcome check catches a wrong start state, a wrong decimation, a dtype mismatch and an action column that stored something other than the executed command; the state check catches phase. Neither proves the demonstrations are good, varied or learnable: a faithful recording of a bad expert is still a bad dataset.
5. Your recording script exits cleanly after the last save_episode() and the directory is the size you expected. Why might it still be unreadable?
Because finalize() never ran. The v3 writer appends to Parquet files incrementally and buffers the metadata, so the footers - the part that says where the row groups are - are written at the end. Without them the files are the right size and are not valid Parquet. The failure mode is nasty because every signal you would normally trust, file count and byte count, looks correct.
6. You have 50 episodes at 128 by 128 and now wish you had recorded at 480 by 640 with a wrist camera as well. What decides whether you have to re-script anything?
Whether you saved the seed and the exact starting state of each episode. The simulator is deterministic, so with those two you can replay the recorded actions and re-render at any resolution, from any camera in the model, without touching the controller. Without them, the pixels you have are the pixels you get. This is the cheapest insurance in the file and it costs two numbers per episode.
Do this
Open module-02-simulation/code/record_dataset.py. It needs your finished so101_pick.py and task_scene.xml from building a task scene next to it, with their own TODOs done: without a cube in the scene there is nothing to record. Four things here are marked TODO(you); do them in this order, because each one makes a command work.
1. The frame, in the right order. Fill in the inner frame() function in run_episode: cast the command to float32, call mj_forward, build and add the frame dict, write data.ctrl, then step exactly steps times. Then:
cd module-02-simulation/code
python record_dataset.py --replayIt records one episode with no recorder attached, replays the actions alone, and prints both outcomes. You are looking for identical: True. If it prints False, compare the two cube positions: a difference of a few micrometres means you cast to float32 in the wrong place, and a difference of centimetres means the ordering is wrong.
2. The schema. Fill in features(). Print it and read it back against the figure above; the names are not negotiable.
3. The validator. Fill in check_frame(). Then break something on purpose: return data.qpos[:6] without the astype(np.float32), or leave the task key out, and confirm the error message tells you which key and what was wrong with it. A validator that fires on episode one is worth an hour; one that fires on episode fifty is worth a day.
4. finalize(). Fill it in, including the refusal to run with an episode still open. Then record and check:
python record_dataset.py -n 3
python record_dataset.py --check pick_demos--check reloads the directory, verifies every column’s dtype, shape and length against info.json, verifies the timestamps sit on the frame grid, and replays the first three episodes to confirm both that the actions reproduce the recorded outcome and that the states they pass through are the recorded observations, exactly. All of it has to pass.
Then earn the second assertion. Roll the observation column forward by one row before saving, rerun --check, and watch the outcome assertion pass while the state assertion fails by about 11.7 degrees. That is the shape of the bug this whole lesson is about, and it is worth seeing once with your own numbers.
Then, once it does, do the thing that is actually the point: time three episodes with --no-images, then three with images, and compare both the seconds and the bytes. On the laptop this lesson was written on, three episodes are 2.7 seconds and 29,724 bytes without pixels, and 12.5 seconds and 24,508,487 bytes with one 128 by 128 camera. Four and a half times the wall clock; eight hundred times the storage. That ratio is why the real format encodes video, and it is the budget you will spend in the milestone.
What you can now do
You can record a scripted episode as a table of frames in the LeRobot v3.0 schema, with the right feature names, dtypes and shapes, at a frame rate that divides your physics rate cleanly. You can state the alignment contract that makes a row a demonstration rather than a log line, and prove your recording honours it by replaying the actions back through the simulator. You can say which columns must never be features and which facts must be saved even though the format has no column for them. And you know where the writer runs, why finalize() is not optional, and what a finished dataset costs.