Where you are. You can derive a policy gradient and you know what PPO’s clip protects. This lesson is the one job the whole apparatus was built for, and the first thing in this course that your laptop genuinely cannot do.
Open the file that defines walking
Somebody hands you the configuration for a quadruped locomotion task and asks you to find the part that describes walking.
You will not find it. What is in the file is a list of terms with weights. One of them pays for matching a commanded velocity. The rest are subtractions: lose points for the body pitching, for torque, for how fast the joints move, for how much the action changed since last step, for feet sliding while in contact. That is the whole task definition, and it is about twenty lines.
There is no mention of a gait. No phase, no stance-to-swing ratio, no diagonal pairs, no swing trajectory. Nobody wrote down that the feet should alternate.
Run it for a few hours on a rented GPU and the robot trots. Not a wobble that gets somewhere - a real gait, with the diagonal pairs synchronised, that transfers to hardware and copes with terrain it never saw.
The gait was not designed. It is simply what scores highest once you have ruled out every cheaper way of moving.
The idea in one paragraph
Locomotion is the case where reinforcement learning is unambiguously the right tool, and the recipe is now standard enough to state in one paragraph. Put thousands of copies of the robot in one simulator on one accelerator. Give each a random velocity command, random ground friction, random payload, random terrain. Reward each for tracking its command and penalise it for everything that makes a policy useless in reality: falling, burning torque, jittering, dragging. Run PPO over the whole population for on the order of a hundred million steps. The output is a small network - a few hundred kilobytes - that maps joint angles, body orientation and the commanded velocity to twelve joint targets at 50 Hz. Nothing about the recipe is subtle. What makes it work is throughput, and what makes it hard is that every weight in that reward is hand-tuned.
Wider than the screen; scroll it sideways.
The reward, term by term
Only the first row is the task. Everything else exists because the optimiser found a way to satisfy the earlier rows that you did not want.
| Term | What it is for | What happens without it |
|---|---|---|
| Velocity tracking | move at the commanded forward, sideways and turning rate | nothing happens; this is the only term that says what the job is |
| Orientation / height | keep the body level and off the floor | the robot goes fast lying on its belly, which tracks velocity perfectly |
| Torque and joint velocity | a small cost per unit of effort | violent, high-frequency motion that no real actuator survives |
| Action rate | penalise change in the command since last step | audible buzzing at the control frequency; servos overheat |
| Foot slip, air time | shape what the feet actually do | scooting, skating, and a gait with no flight phase |
| Termination | end the episode when the body touches the ground | it learns to fall usefully, because falling is fast |
That last point is easy to skim past and it is structural. If the command were not in the observation, the policy would have to guess what speed it was meant to be doing, and the best it could do is the average. Putting it in the input turns “walk” into “walk at whatever this input says”, and it is the same trick that lets one language-conditioned manipulation policy do several tasks.
Why the whole loop lives on the accelerator
A hundred million steps is the number that dictates the architecture. Everything else follows.
Measure your own machine. code/sim_throughput.py builds a twelve-degree-of-freedom quadruped in MuJoCo - four legs, three joints each, floating trunk, feet in contact with the floor - and steps it as fast as one CPU core can, first with random torques and then with a locomotion-sized network choosing the actions.
On my 2018 four-core laptop, idle, I measured 31,000 steps per second for the physics alone and 8,500 with the policy forward pass included. Under load those fell to about 13,000 and 2,800, a factor of three either way, so run it yourself rather than reading mine. Note which number matters: the network costs several times more than the physics, because a batch of one is the worst possible shape for a matrix multiply.
The MuJoCo Playground authors report 417,451 ± 2,955 steps per second for a Unitree Go1 on flat terrain on a single A100 (their Table VII; I have not reproduced it). That is about fifty times what I measured on an idle laptop and a hundred and fifty times what I measured on a busy one, and essentially all of the difference is batching: thousands of robots stepped in one call, with the policy evaluated on all of them at once.
Wider than the screen; scroll it sideways.
What I ran, and what I did not
This section exists because the honest answer is uncomfortable and you need it before you plan an afternoon.
I ran the throughput measurement above. Real MuJoCo, real quadruped, real numbers, on the machine this course was written on. You can reproduce it in thirty seconds.
I did not train a locomotion policy. Not “it was slow” - it is structurally unavailable on this machine, for a reason worth understanding:
- MuJoCo Playground’s speed comes from MJX, which is MuJoCo reimplemented in JAX so the physics runs batched on a GPU.
- JAX’s own installation documentation (read 2026-08-09) states plainly that JAX is not supported on Mac GPUs, and directs Mac users to the CPU build. There is no Apple GPU installation path.
- So on any Mac, Apple silicon or Intel, Playground runs JAX on the CPU. Since the entire method is GPU batching, that removes the point.
Where to actually run it
Three options, in the order I would try them.
The Colab notebooks, free tier. The Playground repository ships notebooks for locomotion, manipulation, the DM Control Suite and vision. This is the lowest-friction way to see a quadruped learn, and it costs nothing. I have not timed the Go1 flat-terrain job on a free T4 and will not guess: the paper’s five minutes is on two RTX 4090s, and a T4 is a different class of device. Time it yourself and write the number down - that measurement is more valuable to you than any figure I could quote.
A rented consumer GPU. On RunPod’s community pricing, checked 2026-08-09, an RTX 4090 was $0.34/hour and an A100 80 GB was $1.19/hour, billed by the second. At those rates the arithmetic in sim_throughput.py prices a full quadruped curriculum in single-digit dollars. Prices in this market move every few months; re-check before you quote them.
A cloud job from the command line. If you are already pushing datasets and models to the Hub, Hugging Face Jobs bills by the minute and integrates with the training CLI. It costs roughly two to four times a raw GPU rental, and you are paying for not managing a box.
Check yourself
1. A locomotion reward has one positive term and six negative ones. Why is that the normal shape rather than a sign of a badly designed objective?
Because the single positive term is the only statement of the task, and every negative term rules out a cheaper way of satisfying it. Velocity tracking alone is satisfied by dragging along the ground, by vibrating, by falling in the right direction. Each penalty closes one of those doors. The asymmetry is not pessimism; it reflects that there are many more degenerate ways to hit a target than good ones.
2. Why is the velocity command part of the observation rather than part of the reward function only?
Because the policy has to be able to condition on it. If the command only appears in the reward, the network never sees which speed it is being graded against, so the best it can do is a compromise gait that scores tolerably across the whole command distribution. Feeding the command in turns one policy into a family of gaits indexed by the input, which is also what makes it drivable with a joystick afterwards.
3. Your throughput measurement shows the physics costing far less than the policy forward pass. Why does that flip completely on a GPU?
Because the laptop measurement runs a batch of one, which is the worst case for a matrix multiply: you pay the full overhead of a network call to compute twelve numbers. On the GPU the same network is evaluated for thousands of robots in a single batched call, so the per-robot cost of the network collapses while the physics work stays proportional to the number of robots. Batching helps the network far more than it helps the solver.
4. Somebody claims their new algorithm trains a quadruped “twice as fast” as the baseline. What do you need before that means anything?
Whether “fast” means environment steps or wall-clock, and on what hardware. Halving the step count is an algorithmic claim; halving the wall-clock might just be a bigger GPU or better batching. The two come apart badly here - PPO is sample-inefficient and throughput-friendly, and a more sample-efficient method that batches worse can be slower in wall-clock while being better on the axis being claimed. Ask for both numbers, the hardware, and the seed count.
5. Why can Apple silicon not simply use its GPU for MuJoCo Playground, the way it can for training a small imitation policy?
Because the two use different frameworks. PyTorch has a Metal backend, so an imitation policy can train on an Apple GPU. Playground’s speed comes from MJX, which is built on JAX, and JAX has no supported Apple GPU backend at all - its own installation docs send Mac users to the CPU build. It is a missing backend, not a performance shortfall, so no amount of better hardware on that machine changes it.
6. The gait was never specified. What does that imply about debugging a locomotion policy that walks badly?
That the bug is almost never in the gait and almost always in the reward or the randomisation. There is no gait code to inspect. If it skates, look at the foot-slip and air-time terms. If it buzzes, look at the action-rate term. If it stands still, the effort penalties are probably outweighing the tracking reward. Debugging moves from reading code to reading the objective, which is a genuinely different skill.
Do this
1. Measure your own machine. Fill in wall_clock_hours and the arithmetic block in code/sim_throughput.py and run it. Thirty seconds. Write down three numbers: steps per second with the policy in the loop, the hours a hundred-million-step run would take here, and the same for twenty runs, which is what tuning a reward actually costs. Compare them against the rented-GPU line the script prints.
2. Then go somewhere it can finish. Open the MuJoCo Playground locomotion notebook in Colab on the free tier and train the Go1 flat-terrain task. Record: wall-clock to a stable gait, steps per second reported, and the GPU you were assigned. Compare your wall-clock with the paper’s five minutes on two RTX 4090s and write one sentence explaining the gap. If the free tier will not carry it, rent an hour of a consumer GPU; at the prices above this experiment costs less than a coffee.
3. Break the reward on purpose. In the notebook’s reward configuration, take the torque or action-rate penalty and multiply its weight by fifty. Re-run for a short budget and watch the resulting behaviour. You are looking for the policy discovering that standing still is cheaper than walking. Then set the same weight to zero and watch it buzz. Those two runs will teach you more about reward design than any amount of reading, and they are why lesson 3.17 can talk about transfer at all: a policy that jitters in simulation is not a policy you can put on hardware.
What you can now do
You can read a locomotion reward and say what each term is defending against, and you can predict the specific degenerate behaviour that appears when one is missing or badly weighted. You can explain why the simulator, not the network, is the reason the training loop lives on an accelerator, and you have measured the gap on your own machine rather than taking it on faith. You know exactly why this particular job cannot run locally on a Mac, that the reason is a missing JAX backend rather than a slow one, and where to run it instead for a few dollars.