The data engine

Where you are. You can build a scene, drive a simulated arm and record demonstration episodes. This lesson is about what makes a recording worth training on, which is a different question from whether it recorded.

The week you spent on the wrong half

You have probably had this week. A model that used to work stops working. You try a bigger encoder, three learning rates, a different optimiser. Nothing moves. On Friday, out of ideas, you open the training data.

A third of the rows arrived during one week when the upstream service was retrying, so they are duplicates. Another slice was labelled by whoever was on rota, using a dropdown whose default happened to be the most common class. You drop the duplicates, drop the rota week, retrain the original model unchanged, and it beats everything you tried.

Nothing in that story is specific to machine learning. It is the oldest rule in data engineering. What changes in robotics is who caused it. There is no upstream service and no rota. You collected the data, with your own hands, one attempt at a time, and every defect in it is a decision you made without noticing.

The idea in one paragraph

A demonstration dataset is not a record of what happened; it is the specification of what the robot will be able to do. Everything inside the region your demonstrations covered, the policy has a chance at. Everything outside it, the policy has never seen and has no reason to handle. That makes dataset design an engineering activity with its own review checklist, not a chore you do before the interesting part. And the economics are unusual: you cannot scrape more robot data off the internet, so the unit of investment is not GPU-hours but minutes of a human being’s hands. A working policy for a small arm is built from a few dozen episodes and a few minutes of teleoperation. When the numbers are that small, one careless half hour of recording is a large fraction of everything the robot knows.

What is actually in an episode

Each frame carries two vectors of the same shape. observation.state is what the arm reported about itself: six joint angles, read from the encoders. action is what it was told to do at that instant: six target joint angles, which is exactly the interface the servos accept and nothing more. The camera frames live in separate video files, indexed by the timestamp.

The contract between those columns is the single easiest thing to get wrong. The action stored at frame $t$ is the command that carried the robot from the state at $t$ to the state at $t+1$. Off by one in either direction and you are training a policy to predict what already happened.

Formally, the thing you are building is a set of observation-action pairs drawn from whatever the demonstrator happened to do:

which reads: for each of the $N$ episodes, and each frame in it, one pair of what was seen and what was done. Behaviour cloning, two lessons from now, is nothing more elaborate than fitting a function to that set. Which is why everything interesting about the result is decided here, before any training runs.

Coverage is the currency

Volume is the number that feels like progress. Coverage is the one that predicts success.

Two datasets, both fifty episodes, both twelve thousand frames. In the first, the brick starts in the same place every time. In the second, it starts anywhere on the mat. They cost the same to record and they are not remotely the same dataset: the first one has taught the policy one trajectory very well and told it nothing at all about where the brick can be.

The honest statement of what a dataset taught is the range each joint actually visited. Here is that range, measured on all 11,939 frames of the real dataset:

Two things jump out, and both are the kind of thing you want to know before you spend an evening training. The shoulder pan swept 181 of its roughly 200 available units, so the policy has seen the arm point almost anywhere. The gripper never went past 33, on a scale where two other joints reach exactly 100. Whatever this policy learns about grasping, it has never once seen the fingers open wide.

The other half of coverage is where episodes begin, because that is the distribution the robot will be reset into. On this dataset the shoulder pan starts anywhere across 19 units and the wrist roll across 21, but the elbow starts within 0.63 units every single time. Every demonstration begins from essentially one elbow angle. That is not a defect - it is a decision, made implicitly by whoever set up the rig - but it is a decision the dataset makes on your behalf and does not tell you about unless you look.

Consistency, and the cost of a sloppy hour

Coverage says which situations are in the dataset. Consistency says whether the answers inside it agree.

The same fifty episodes run from 183 to 306 frames, so the longest attempt takes 1.67 times as long as the shortest. Some of that is genuine task variation. Some of it is a human being who was quicker before lunch.

And 14.2 percent of all frames carry no new command at all: the action is bit-identical to the previous frame’s. The median episode has three such frames at the front before anything happens. The worst has 76, which is two and a half seconds of a recording light being on while nobody moved.

Trimming is a judgement call, not a rule. A short settle at the start of an episode is often deliberate, giving the physics or the exposure time to settle. Two and a half seconds is not that. The point of the audit is to make you look, decide, and write down what you decided.

Design the episode before you record it

Almost every dataset problem is a specification problem wearing a data costume. So write the specification first. Six questions, and the answers are the design:

The format, and why it is boring on purpose

The format your recorder writes, and the format the whole open robot-learning ecosystem reads, is LeRobotDataset. It is Parquet files, JSON metadata and MP4 video in a directory. That is the entire idea, and the boringness is the feature: the audit script in this lesson reads a published 50-episode dataset with pandas and huggingface_hub and nothing else. Your data outlives whichever training framework is fashionable this quarter, as long as you keep it in something a dataframe can open.

Why it is called an engine

One last thing, and it is the reason this lesson is first rather than last. Collecting data is not a step you complete. You record, you train, you deploy, you watch where it fails, and you record more demonstrations in exactly that region. Then again.

That loop is the whole shape of modern robot learning, and it is a direct consequence of the problem you will meet in the covariate-shift lesson: a policy generates its own test distribution, so the places it needs data are places you cannot know until it has run. A dataset built once, in one sitting, is a snapshot of your guesses about which situations matter. The engine is what turns those guesses into measurements.

Check yourself

Do this

Audit a real dataset before you ever train on it. code/dataset_audit.py has three TODO(you) functions; code/demos.py is given to you complete and does the reading.

pip install numpy pandas pyarrow huggingface_hub
python dataset_audit.py                      # 50 real SO-101 episodes, from the Hub
python dataset_audit.py --path pick_demos    # your own Module 2 recording

It downloads about 450 kilobytes - the metadata and the numeric columns only, never the 80 megabytes of video - and finishes in a few seconds. When your three functions are right, the report reproduces these measured numbers:

All seven rows were measured on 2026-08-09 over every one of the 11,939 frames, not sampled.

Then do the part that matters:

1. Run it against your own Module 2 recording. Your scripted expert is perfectly consistent and never gets tired, so its numbers should look very different from a human’s. Write down which of the seven rows differ and why. If your dead-air number is high, find out whether that is your settle phase or a bug.
2. Break something on purpose. Truncate one episode’s rows without updating the length in the metadata, and confirm episode_lengths raises rather than returning a plausible-looking number. A cross-check that never fires has not been tested.
3. Add an eighth row to the report. A good one: for each episode, the distance between its starting state and the mean starting state, so you can see at a glance whether any episode started somewhere unusual. You will want this again in the evaluation lesson, where “which conditions did you actually test” becomes the whole argument.

What you can now do

You can read a LeRobot-format dataset with nothing but a dataframe library, and say what is in it: how long the episodes are, how much they disagree, which joint ranges were visited and which never were, how much of the recording is dead air, and how tightly the starting poses cluster. You can state the frame contract that ties an action to the state it produced, and explain why confusing the action column with the state column yields an excellent loss curve and a motionless robot. And you can write the six-line specification for an episode before you record a single one, which is the difference between a dataset that says what you meant and a dataset that merely exists.