Where you are. You have trained policies on data you generated in simulation and measured them with an evaluation harness you wrote yourself. The next lesson opens with a bill of materials and a total of about $230. This lesson is the fork in front of it.
Ninety seconds, and eleven thousand frames
Open a terminal and pull down somebody else’s afternoon.
from huggingface_hub import hf_hub_download
import pandas as pd
path = hf_hub_download("lerobot/svla_so101_pickplace",
"data/chunk-000/file-000.parquet", repo_type="dataset")
frames = pd.read_parquet(path)
print(frames.shape) # (11939, 7)That is 11,939 rows. Fifty episodes of a real arm, on a real bench, in somebody’s real room, picking up a pink lego brick and putting it in a transparent box. Thirty rows per second. Six joint angles measured, six commanded, two camera streams at 480 by 640. A human hand drove every one of those rows.
Nothing was simulated and nothing was synthesised. It cost ninety seconds and no money.
So here is the question this lesson answers: what did the person who spent $230 get that you just did not?
The idea in one paragraph
A physical arm on your bench supplies two things at the same time, and they are separable. It supplies real data, with the noise, the backlash and the hesitating human hand that no model of a robot contains. And it supplies a closed loop, a way to put a trained policy back into the world and find out whether the task actually gets done. Buying an arm buys both in one purchase, which is why the two get discussed as one thing. Without an arm you can still buy each of them, separately: real robot data is published on the Hub by the hundred, and a closed loop is what MuJoCo has been giving you since Module 2. What you cannot buy separately is both at once, and that gap is precisely where most of the difficulty in real robotics lives. This module runs on either path. The milestone at the end has two acceptance criteria, and neither of them is the easy one.
Wider than the screen; scroll it sideways.
What each path actually supplies
| Hardware path | Simulation path | |
|---|---|---|
| Training data | recorded by your hand, 50-60 episodes, one afternoon | downloaded, 50-206 episodes, ninety seconds; or teleoperated in MuJoCo |
| Evaluation | 20 physical trials, roughly 12 seconds and a human reset each | 200 or more scripted trials, seconds each, reset by a function call |
| Cost of a trial | irreversible; a dropped brick stays dropped | free; the seed reproduces it exactly |
| What the number means | the task, on your bench, in your light | the task, in a model whose every constant is a guess |
| Confidence interval at 70% | about 48-85% on 20 trials | about 63-76% on 200 trials |
| Money | $230 to $400, plus weeks of lead time | none |
The interval row is worth reading twice, because it points the wrong way from the usual assumption. The reader with the arm publishes a wide, honest interval because trials are expensive. The reader without one has no excuse for a wide interval at all, and the milestone holds them to it.
Track A: real data, no robot
The Hub hosts LeRobot datasets in the same format Module 3 consumed. They are parquet files plus JSON metadata plus MP4 video, so pandas and pyarrow read them with no robotics library installed at all.
That first dataset is the closest public match to what this module asks you to record: same class of arm, same joint count, same frame rate, two cameras, one task sentence. Train on it and you are training on a real robot, driven by a real hand, with everything that implies.
Public data also arrives unpruned, which is a feature. lerobot/svla_so100_stacking contains an episode one frame long, and three shorter than thirty frames. Nobody removed them for you. Deciding what to drop and writing down the rule you dropped by is the same discipline the hardware reader learns with the discard key, arriving from a different direction.
Track B: a closed loop, no real data
The MuJoCo scene from Module 2 is the other half. It has an SO-101 model, a task, a success predicate that needs no human to judge it, and a reset that costs a function call. It is where your policy gets to fail two hundred times before dinner.
That loop is what lets a simulation reader do the thing this module is actually about: turn the data-quality loop. Record, train, roll out, watch the failures, sort them, aim the next batch of episodes at the biggest bucket, and turn it again. The loop is the content. The bench is the venue.
What you are actually giving up
Not a small thing, and this is where an honest account earns its keep. Four properties of a physical robot are missing from every simulator, and between them they account for most of what makes real robotics hard.
| What is missing | What it does on a real bench | What sim does instead |
|---|---|---|
| Calibration drift | The map from encoder counts to joint angles moves. A servo is nudged, a bracket loosens, and a dataset recorded on Tuesday quietly means something else on Thursday. | The model is exact and stationary. Frame conventions never move under you. |
| Lighting change | The sun crosses the window and the policy stops seeing the object. Pixels are the observation space and pixels are not stable. | Rendering is deterministic. You can randomise light, but only along axes you thought of. |
| Servo heating | A stalled joint dissipates every watt as heat. Torque sags, the arm droops, and the last ten trials of an afternoon are not the same experiment as the first ten. | Actuators never get tired. Trial 200 is identical to trial 1. |
| Demonstration quality | Your hand is the label. It is careful at episode one, grooved by episode four, cutting the corner by episode eight, and it hesitated once because your phone buzzed. | A scripted controller is perfectly consistent, which is a kind of data no human produces. Keyboard teleoperation is inconsistent in ways no human hand is. |
Those four are not a footnote on the simulation path. They are three of the four topics in debugging by symptom, and the fourth is the entire subject of what makes a good demonstration.
How the rest of the module reads
Lessons 1 through 17 each carry a short section headed Without hardware, sitting just before their Check-yourself questions. It names the public dataset that replaces the recording step, the simulated task that replaces the physical one, and the thing to measure instead. The prose above it is unchanged and worth reading on either path, because the mechanism is the content: you should understand what a servo id collision does whether or not six motors are sitting on your desk.
Lessons 1 through 6 are the build, and on the simulation path they are read rather than done. Lessons 7 through 17 are the loop, and on the simulation path they are done. The milestone in lesson 18 states both acceptance criteria in full.
Wider than the screen; scroll it sideways.
Check yourself
1. A real arm supplies two things that a reader without one has to buy separately. Name both, and say which is harder to substitute.
Real data and a closed loop. Real data substitutes well: public LeRobot datasets are recordings of real arms driven by real hands, in the same format your own recording would produce, and downloading one costs ninety seconds. The closed loop substitutes badly, because a simulated loop is closed around a model rather than around the world, so its success rate measures the model. The join is what is genuinely unavailable: you cannot evaluate a policy in a closed loop against real physics without a real robot, and that join is where the reality gap lives.
2. You train a policy on lerobot/svla_so101_pickplace and report an action-prediction error on held-out episodes. Why is that not a success rate?
Because it is measured open loop. Every prediction is scored against a recording the policy could not influence, so a mistake at frame 200 never changes what the policy sees at frame 201. On a robot, mistakes move the arm somewhere the demonstrations never visited and the errors compound; that is covariate shift, and it is the thing offline metrics structurally cannot see. Two policies with the same held-out error can have very different success rates, which is why the simulation milestone asks you to measure that disagreement rather than assume it.
3. Which of the four missing properties makes a dataset recorded on Tuesday mean something different on Thursday, and why does no amount of extra data fix it?
Calibration drift. The dataset stores joint angles, but the angles are only meaningful through the count-to-angle map that was in force when they were recorded. If a servo horn is nudged or the arm is recalibrated under a new id, the same stored number now names a different physical pose, so the old episodes and the new ones disagree about what the task is. Adding episodes recorded under the second calibration makes the contradiction bigger rather than smaller. The fix is upstream: one calibration per arm, backed up, and a replay check before you trust an afternoon.
4. Your simulated policy clears 200 trials at 88%. What may you claim, and what must the sentence contain?
You may claim that a policy trained on your simulated demonstrations completed your simulated task 88% of the time under a success predicate you wrote, and the sentence has to carry the venue, the trial count, the seed, and the predicate. What you may not claim is anything about hardware. A simulator is kinder than the world about contact and drift, and harsher than the world when a contact parameter you never tuned makes a real grasp impossible in the model, so the error is not even signed. The interval is also part of the claim: 88% over 200 trials is roughly 83-92%, which is a genuinely tighter statement than a hardware reader can make on 20 trials, and it is tighter about a smaller thing.
5. Why does the simulation milestone ask for 200 evaluation trials when the hardware milestone asks for 20?
Because the constraint on the hardware reader is wall-clock, and it is real: 20 trials is roughly an hour of resetting a bench by hand, and the resulting interval near 70% runs from about 48% to 85%. In simulation a trial costs milliseconds and a reset costs a function call, so the same wide interval would be a choice rather than a constraint. Equal rigour between the two paths does not mean equal trial counts; it means each path is held to what it can afford. The simulation path can afford a tight interval, so it is required to produce one.
Do this
Twenty minutes, and it decides the next six weeks.
1. Pull a real dataset and print its metadata. Nothing but pandas, pyarrow and huggingface_hub in the environment you already have.
import json
from huggingface_hub import hf_hub_download
info = json.load(open(hf_hub_download(
"lerobot/svla_so101_pickplace", "meta/info.json", repo_type="dataset")))
print(info["codebase_version"], info["robot_type"], info["fps"])
print(info["total_episodes"], "episodes,", info["total_frames"], "frames")
for key, spec in info["features"].items():
print(f" {key:32s} {spec['dtype']:8s} {spec.get('shape')}")Read the feature list against the frame anatomy in lesson 12. Every field the recording lesson describes is in there, produced by somebody else’s hand.
2. Confirm your simulator still runs. Open the Module 2 scene, run one scripted episode, and check the success predicate fires. If it does not, fix that now rather than in six weeks with a training run waiting on it.
3. Choose, and write it down. One line in your lab log: which path, and why. Cost, lead time, what you want the artefact at the end to be. If you order hardware later, this line is the dated note that says what you expected to gain.
4. Write the sentence you intend to publish, with the numbers left blank. "__% over __ trials on [bench], success criterion: ______". Both paths fill in the same sentence. Only one of them gets to leave the venue out, and it is neither.
What you can now do
You can say exactly what a physical robot supplies that a simulator does not, in terms of two separable goods rather than one vague advantage. You can load a real LeRobot dataset with three ordinary Python packages and read its metadata rather than trusting its name. You can name the four properties of real hardware that no simulator reproduces, and point at the lessons in this module that exist because of them. And you can state a robotics result with its venue attached, which is the one habit that makes numbers from the two paths comparable at all.