Where you are. You have a task scene, a scripted expert that succeeds, and a working understanding of why its grasp holds. This lesson is about deliberately making the world vary, and about the surprisingly hard question of whether the varying did anything.
Two randomisers, forty episodes, one number
Take the scene and shake the physics as hard as you dare. Cube mass from 30 g to 1.2 kg, a factor of forty. Fingertip friction across a factor of fourteen. Servo gains across a factor of twenty, which on real hardware would be four different robots. Run twenty episodes.
Twenty successes.
Now put the physics back and scramble the appearance instead: object colours, table colour, the light’s position and colour, the camera nudged. The rendered frames come out different enough that the average pixel moves by 51 values out of 255. Run twenty episodes.
Twenty successes.
Two completely different interventions, forty episodes, and the number you were watching is 100% both times. If success is your instrument, you have just learned nothing at all, twice.
The idea in one paragraph
Domain randomisation means varying the things you know your model gets wrong, so that whatever learns from the data cannot come to depend on them. It is not a difficulty setting and it is not noise for its own sake. Two consequences follow and both are easy to miss. First, the right instrument is the spread of the recorded data, not the success rate: a randomiser that leaves every trajectory identical has done nothing, however violently it shook the parameters. Second, randomisation has a budget, set by your expert’s competence: widen past the point where the scripted policy still succeeds and you are no longer collecting harder demonstrations, you are collecting failures with an expert’s name on them.
Three axes, and what each one actually moves
Run the same episode under three families of randomisation, recording every joint angle at every step, and compare how much the trajectories differ from one another.
Wider than the screen; scroll it sideways.
| what you randomise | success | spread in the joint trace |
|---|---|---|
| appearance: colours, light, camera | unchanged | 0.000 degrees |
| dynamics: mass, friction, servo gains | unchanged | 0.083 degrees |
| object pose: where the cube starts | unchanged at this width | 2.730 degrees |
The zero is exact and it should be. Nothing in a colour or a light position enters the equations of motion, so the physics is bit-for-bit identical and only the pixels differ. That is the whole point of visual randomisation: it changes the observation without touching the answer.
The 2.730 against the 0.083 is the interesting comparison. Moving the cube by a few centimetres changes what the expert does, so the actions themselves vary. Multiplying the servo gains by twenty changes only how well the arm tracks the commands it was already going to send, and the position controller absorbs almost all of it.
The budget is your expert, not the task
Widen the cube’s starting square and watch where the scripted expert gives out.
| half-width of the start square | episodes that ended with the cube in the bin |
|---|---|
| 0 cm | 20/20 |
| 4 cm | 20/20 |
| 8 cm | 20/20 |
| 12 cm | 16/20 |
| 16 cm | 15/20 |
n = 20 trials · scripted pick and place, cube start uniform in a square of half-width w · 2026-08-09
Wider than the screen; scroll it sideways.
Eight centimetres is free. Twelve costs you a fifth of your episodes, and the reason is exactly the crescent from Lesson 2.11: the corners of a 12 cm square reach past where a straight-down grasp pose exists.
The danger is not the failures. It is that the failures are indistinguishable from successes unless something checks. A scripted expert does not know it failed. It executes its waypoints, opens its fingers over the bin, retreats, and the episode ends looking exactly like the nineteen that worked. This is precisely why Lesson 2.11 insisted the verdict be code: the success predicate is the filter standing between a wide randomisation range and a poisoned dataset.
You have two ways to spend the budget. Randomise inside the expert’s competence and keep everything, or randomise wider and discard the failures, accepting that you now pay compute for episodes you throw away. Both are legitimate. Silently keeping the failures is not.
What the evidence actually says
Here the honest answer diverges sharply from the folklore, so it is worth being careful about which claims are measured, which are reported by somebody else, and which are neither.
A 2026 mechanistic follow-up asks why co-training works, and its answer is the most useful thing in this lesson.
Wider than the screen; scroll it sideways.
Transfer is driven mostly by structured representation alignment: the policy’s internal features for simulated and real episodes must end up close together but not identical. That accounts for roughly half the variance in transfer performance in their analysis, with the mixing ratio accounting for about a fifth. Then the sting. When the gap between simulation and reality is both visual and physical, alignment correlates with success at roughly 0.6 to 0.8. When the gap is physics only, the correlation goes negative. Pushing sim and real features together when the only real difference is the dynamics actively hurts.
There is also a large 2026 empirical study that ablates multi-level randomisation, photorealistic rendering, physics-realistic modelling and reinforcement-learning fine-tuning across more than ten thousand real-world trials. I was not able to extract its quantitative rankings from the paper, so I am not quoting any. It is worth reading before you commit a month to any of these strategies, and worth more than my summary of it would be.
What to randomise on this arm
Concretely, in rough order of how well the evidence supports them.
- Camera pose and lighting. The most reliable axis, and the one your real setup will violate first, because nobody re-mounts a camera in exactly the same place twice.
- Object pose, colour and size. Inside the expert’s competence, measured rather than assumed.
- Object mass and friction. Cheap, and remember to scale inertia with mass.
- Servo gains. The SO-101 model’s XML says out loud that its
sts3215gains are not a one-to-one mapping of the gains LeRobot uses on the real servo. That comment is a marked gap in the model, and a marked gap is the honest place to randomise. - Backlash. The same model defines a
backlashjoint class of plus or minus half a degree and then does not attach it to any joint. It is a modelled effect somebody decided not to enable. Enabling and randomising it is one of the few dynamics knobs here with a clear physical justification. - Latency. Not built into the scene, and the one Lesson 2.1 showed can cost you 60 points of success on its own. Delay your commands by a random one to four control steps and you have modelled it.
Review
Success is the wrong instrument
Shake the physics as hard as you dare and run twenty episodes: twenty successes. Put the physics back, scramble the appearance instead, run twenty more: twenty successes. Two completely different interventions, forty episodes, and the number you were watching is one hundred percent both times. If success is your instrument, you have just learned nothing at all, twice. The right instrument is the spread of the recorded data. Randomising colours, lights and camera moved the joint trace by zero degrees exactly, because nothing in a colour enters the equations of motion. Randomising mass, friction and servo gains moved it by eight hundredths of a degree. Moving where the cube starts moved it by two point seven. Ask of every randomiser whether it changes the observations, the actions, or neither.
The budget is your expert, not the task
Randomisation has a budget, and it is set by your scripted expert’s competence rather than by the task. Widen the cube’s starting square and the expert holds at four centimetres and at eight, then drops to sixteen of twenty at twelve, and fifteen of twenty at sixteen. Eight centimetres is free; twelve costs you a fifth of your episodes. The danger is not the failures, it is that they are indistinguishable from successes unless something checks. A scripted expert does not know it failed. It executes its waypoints, opens its fingers over the bin, retreats, and the episode ends looking exactly like the nineteen that worked. That is why the success predicate has to be code. You may randomise inside the expert’s competence and keep everything, or randomise wider and discard the failures. Silently keeping them is not an option.
What the evidence supports, and what it does not
The cheap intuition that more randomisation means more robustness survives neither the measurements nor the literature. The most-quoted current recipe is not randomisation alone but sim-and-real co-training: a mixture of a lot of simulated data and a little real data, with the best mixtures reported between fifty and ninety percent simulation. A mechanistic follow-up finds transfer is driven mostly by structured representation alignment, the policy’s internal features for simulated and real episodes ending up close together but not identical. Then the sting. When the gap is both visual and physical, alignment correlates with success at around zero point six to zero point eight. When the gap is physics only, the correlation goes negative. And do not randomise what you can measure: every parameter randomised instead of measured makes the policy hedge everywhere and commit nowhere.
Check yourself
1. You randomise the cube’s mass across a factor of forty and your success rate does not move. Has the randomisation done anything?
Possibly very little, and the success rate cannot tell you. Record the joint trajectories and measure their spread across episodes: on this arm that randomisation moves them by 0.083 degrees, which is a rounding error next to the 2.730 degrees you get from moving the cube a few centimetres. Success is a one-bit summary of an episode; the data a policy learns from is the whole trajectory, and that is what you should be measuring.
2. Why is a success rate of 15 out of 20 at a wide randomisation range worse than 20 out of 20 at a narrow one, even though the wide range covers more of the world?
Because the five failures do not announce themselves. The scripted expert runs its waypoints, releases and retreats whether or not the cube is in the bin, so unless the success predicate filters them out those five episodes enter your dataset as demonstrations of how to do the task. Wide ranges are fine; wide ranges without a filter are how you teach a policy to drop things.
3. A colleague proposes forcing the sim and real feature distributions as close together as possible. When is that a mistake?
When the gap between them is mainly physical rather than visual. The 2026 mechanistic analysis found alignment correlating with transfer success at roughly 0.6 to 0.8 when the gap is visual and physical, and correlating negatively when the gap is physics only. Features being close is not the goal; close but still distinguishable is. Collapsing them destroys information the policy needed.
4. You set model.body_mass to randomise the object’s weight. What have you silently broken?
The rotational inertia, which lives in model.body_inertia and does not follow. The compiler derived it from the original mass and geometry, so a cube you have made fifteen times heavier still spins as though it weighed 64 grams. It is physically impossible and completely silent. Scale the inertia by the same factor when you scale the mass.
5. Randomising the cube’s yaw across plus or minus 45 degrees leaves the success rate at 8 out of 8. Is the randomisation pointless?
Nearly, for this object, and for a reason worth naming: a cube is symmetric under a quarter turn, so a 45 degree yaw range covers every distinct orientation and the grasp sees essentially the same problem each time. The trajectory spread does move a little, to about 0.29 degrees, because the contact geometry shifts and the arm’s tracking responds. Randomising a symmetry is randomising nothing; swap in an object without that symmetry and the same range becomes meaningful.
6. Given the current evidence, what would you do first with ten real demonstrations and a working simulator?
Co-train rather than transfer: use the real demonstrations alongside a much larger pile of simulated ones, aiming for a mixture in the 50 to 90 percent simulation range rather than an even split. Spend the randomisation budget on the visual gap, where the evidence is strongest and your camera mount is genuinely uncertain, and be sceptical of heavy dynamics randomisation until you have a reason to believe the dynamics are what is wrong.
Do this
code/randomize.py has one TODO(you), in randomise_appearance(). Fill it in and run the file; it takes about five minutes at the default of ten episodes per row. The twenty-episode table above came from python randomize.py --full, which takes about ten. Budget 45 minutes with the follow-ups.
1. Reproduce the three axes. Run the file as shipped and check that the appearance row gives a trajectory spread of exactly zero. If it is not exactly zero, you have randomised something with mass, friction or a gain in it, and finding which one is the exercise.
2. Find your own budget. Extend position_sweep to 20 and 24 cm. The success rate does not fall off a cliff, it sags, and knowing the shape of your own sag is what lets you pick a width on purpose.
3. Filter, then count. At a 12 cm half-width, run 20 episodes and write both numbers down: how many episodes you ran and how many you would keep. That ratio is the real cost of a wide range, and it is the number to quote when someone asks why the dataset took a day.
4. Scale inertia with mass. Add the missing line so body_inertia tracks body_mass, rerun the dynamics sweep, and compare the trajectory spread with and without it. Decide for yourself whether the difference is worth the line.
5. Randomise the thing that is actually wrong. Add a latency randomiser: hold each new ctrl value in a small queue and apply it one to four steps late. Measure the trajectory spread. Compare it with what mass and friction randomisation bought you, and write one sentence about which of the three you would spend your compute on.
What you can now do
You can pick randomisation axes deliberately instead of by reflex, and you can tell whether an axis did anything by measuring the spread of the recorded trajectories rather than watching a success rate that stays at 100% regardless. You can size a range against your expert’s measured competence and say what a wide range costs in discarded episodes. And you can state the current evidence without overselling it: co-training with a small amount of real data is the recipe with the strongest reported support, visual randomisation is the axis with the longest track record, and forcing sim and real closer together is documented to backfire when the gap between them is physical rather than visual.