ACT, part by part

ACT is a transformer that turns one observation into the next hundred actions, trained as a conditional autoencoder so it can absorb the variation between demonstrations.

Where you are. You know chunking is the idea that made imitation learning work on cheap arms. This lesson is the specific architecture that carried it, read from the implementation rather than the abstract.

A comment in a config file

Open LeRobot’s configuration_act.py and scroll to the decoder. There is a comment sitting above one number:

Although the original ACT implementation has 7 for n_decoder_layers, there is a bug in the code that means only the first layer is used. Here we match the original implementation by setting this to 1.

The paper describes a seven-layer decoder. The published results were produced by a one-layer decoder, because of a bug (tonyzhaozh/act#25). And the most-used reimplementation deliberately reproduces the bug, because matching the numbers matters more than matching the prose.

Nothing here is a scandal. It is the single most useful thing you can learn about reading machine-learning papers: the reference implementation is the artefact that carries the result, and the paper is a description of it that may be wrong. Read this lesson with the config file open.

That one number also decides how big the model is, which is why you will see ACT quoted at two different sizes and both are right. A decoder layer at this width costs about 5.4 million parameters, so a seven-layer decoder puts the model at roughly 80 million, which is the figure LeRobot’s own documentation quotes. The one-layer decoder that --policy.type=act actually builds puts it at about 52 million. The difference is six decoder layers and nothing else, so when you meet a training-time or memory number for ACT, the first thing to establish is which of the two it was measured on. The wall-clock figure in Lesson 4.13 is measured on the 52-million one, which is the one you will train.

The idea in one paragraph

ACT is a transformer that takes one observation - camera frames plus current joint angles - and emits the next hundred joint commands in a single forward pass. Around 80 million parameters as the paper configures it, about 52 million in the version you will actually train, trainable from scratch on about fifty demonstrations. Two design decisions do the work. The first is the chunk itself, which you met in the last lesson. The second is that it is trained as a conditional variational autoencoder: during training a second network reads the actions the human actually performed and compresses them into a small latent vector, which the policy is allowed to see. That gives the model somewhere to put the variation between two demonstrations of the same task that no camera could have predicted. At inference the latent is set to zero and the extra network is thrown away.

The pipeline, in tokens

ACT as a token pipeline: camera images through a ResNet-18 into many tokens, the joint state into one token, the latent into one token, all mixed by a four-layer transformer encoder, then read by a one-layer decoder holding one hundred learned queries

Wider than the screen; scroll it sideways.

Six pieces. Each one earns its place.

One observation, not a history. LeRobot’s ACT sets n_obs_steps = 1 and raises a ValueError if you ask for more. Only the current frame goes in. That looks like an omission until you notice that the chunk already spans time on the output side; ACT’s answer to “the task is not Markov” is to commit to a stretch of future rather than to look at a stretch of past.

A ResNet-18 per camera. ImageNet-pretrained, and the feature grid it produces becomes a token per spatial cell rather than a single pooled vector. Keeping the grid is what lets attention ask “where in this image is the thing I am reaching for” instead of receiving a summary that has already thrown the answer away.

Joint angles as one more token. No special treatment. A linear layer maps the current joint vector into the same width as everything else and it joins the sequence.

A four-layer transformer encoder mixes all of those into a memory. This is the part that is genuinely just a transformer.

A one-layer decoder holding a hundred learned queries. This is the piece worth slowing down on.

A linear head turns each decoder output into one action vector. For a bimanual setup that is 14 numbers; for your SO-101 it is 6.

The part that is not obvious: the latent

Record the same pick twice. Same object, same starting pose, same person. The two action sequences will not match. The human paused for a beat on one take, approached a shade faster on the other, gripped a few millimetres higher. From one photograph of the scene, both futures are correct and nothing in the image distinguishes them.

A network trained to output one answer per observation is being asked to reconcile that, and the loss it is trained under has an opinion: it converges toward the average of the demonstrated actions. Sometimes the average is fine. Sometimes it is a motion nobody performed. Lesson 9 is entirely about when it is not fine, so here take only the modest version of the claim: there is variation in the demonstrations that the observation does not explain, and the model needs somewhere to put it.

The two paths through ACT: during training the demonstrated chunk is read by a CVAE encoder that produces a distribution over the latent, which is sampled and handed to the policy; at inference the chunk does not exist, so the latent is fixed at zero and the same policy runs unchanged

Wider than the screen; scroll it sideways.

That somewhere is a 32-number latent vector $z$ . During training a separate four-layer transformer reads the joint state and the demonstrated action chunk, and outputs a mean and a spread. Sample $z$ from that, hand it to the policy as one more token, and the policy’s job gets easier: it no longer has to explain the demonstrator’s hesitation from the pixels, because the hesitation is in $z$ .

The loss has two terms:

\mathcal{L} = \underbrace{\big\lVert \hat{a}_{1:k} - a_{1:k} \big\rVert}_{\text{reconstruct the chunk}} \;+\; \beta \cdot \underbrace{D_{\mathrm{KL}}\!\left(q(z \mid s, a_{1:k}) \,\big\Vert\, \mathcal{N}(0, I)\right)}_{\text{keep } z \text{ near the prior}}

The first term says: predict the actions that were actually demonstrated. The second says: whatever you encode into $z$ , keep the distribution of $z$ close to a plain unit Gaussian centred on zero. LeRobot sets $\beta$ to 10, which is a heavy thumb on the scale.

At inference there is no demonstrated chunk to read, so the encoder cannot run and $z$ is set to the prior’s mean: a vector of zeros. Every one of those 32 numbers is zero on every real robot step ACT has ever taken. The whole apparatus exists to shape training, and then it is deleted. Run act_lite.py at its own defaults and it prints the split: 3,192,640 of its 7,430,214 parameters sit in the CVAE encoder and never see a robot.

Reading the config as the spec

These are LeRobot’s defaults at tag v0.6.1, read on 2026-08-09. Pin your version; this table has a shelf life of weeks.

Setting	Default	What it decides
`n_obs_steps`	1	one frame in; more raises `ValueError`
`chunk_size`	100	actions predicted per forward pass
`n_action_steps`	100	actions executed before re-querying
`temporal_ensemble_coeff`	`None`	ensembling off; the paper’s value is 0.01
`vision_backbone`	`resnet18`	ImageNet-pretrained, one per camera
`dim_model`, `n_heads`	512, 8	transformer width
`n_encoder_layers`	4	observation encoder depth
`n_decoder_layers`	1	see the top of this lesson
`use_vae`, `latent_dim`	`True`, 32	the latent above
`n_vae_encoder_layers`	4	the encoder that is thrown away
`kl_weight`	10.0	$\beta$ in the loss
`optimizer_lr`, `lr_backbone`	1e-5, 1e-5	AdamW, and no learning-rate schedule

Two of those rows disagree with the paper, and both disagreements are worth understanding.

Why the paper’s own trick is off by default

Predicting a hundred actions and executing all hundred is the blind window from Lesson 6. The obvious fix is to predict a fresh chunk every step and, for any given timestep, average all the predictions that cover it.

Four chunks predicted one step apart, drawn as overlapping bars, with a vertical line at one timestep crossing all four, feeding a weighted average

Wider than the screen; scroll it sideways.

That is temporal ensembling, and it is a named contribution of the ACT paper. Older predictions get less weight; the freshest gets most. It smooths the joins where one chunk hands over to the next, and it keeps the policy looking at the world every single step.

LeRobot ships it disabled. Look at what enabling it requires: n_action_steps must drop to 1, because you cannot form an ensemble for step $t$ unless you queried at step $t$ . Executing one action per query, on a model whose whole design assumes you execute a hundred, means

\text{forward passes per episode} \;:\; \frac{T}{100} \;\longrightarrow\; T

a hundred times more inference. On a workstation with a GPU that is a nuisance. On the embedded board actually bolted to a robot it is often the difference between running at control rate and not running at all.

What reproduction actually looks like

The other place the paper and the implementation part company is the decoder from the opening. Here is what makes that story useful rather than cynical.

LeRobot’s own reproduction numbers, published on the model cards: ACT on the simulated cube-transfer task scores 83.0% over 500 evaluation episodes. An equivalent model trained with the original ACT repository and evaluated in the same harness scores 68%. Same algorithm, same task, same evaluator, fifteen points apart.

There is one more of these worth knowing about. LeRobot v0.6.0 fixed a loss-normalisation bug affecting padded actions in ACT and Diffusion Policy (PR #3442), which means numbers measured before and after that release are not strictly comparable. Lesson 8 shows you the padding it refers to.

Is ACT still what you should use in 2026?

Split the question.

Starting out, with one cheap arm and fifty demonstrations: yes, and LeRobot says so in its own documentation - ACT is “the first model we recommend when you’re starting out”, for fast training, low compute and strong performance. That recommendation matches what a laptop and a desk-sized budget can actually do.

As a live baseline: yes. It still appears in August 2026 papers, including one running a quantised ACT on a bimanual SO-101 driven by a Jetson Orin Nano (arXiv:2608.03938).

As the frontier: no. The frontier is action-chunking flow-matching vision-language-action models, which Module 5 covers. But notice what happened to ACT’s actual idea: chunking won so completely that it is now inside those models. What is dated about ACT is the CVAE transformer it was wrapped in, not the design decision it was built to demonstrate.

Check yourself

1. At inference, what is

z

, and what is the CVAE encoder doing?

$z$ is a vector of 32 zeros - the mean of the prior the KL term pushed the posterior toward. The CVAE encoder is doing nothing, because it needs the demonstrated action chunk as input and there is no demonstration at inference. It exists only during training and its parameters are dead weight afterwards. This also means the deployed policy is deterministic: same observation, same chunk, every time.

2. Someone sets kl_weight to 0 and reports a much lower training loss. What have they built?

A policy that will not work on a robot. With no KL penalty, nothing stops the encoder from writing the demonstrated actions straight into $z$ and the decoder from copying them back out, so training loss falls toward zero by memorisation through a channel that does not exist at deployment. At inference $z$ is zero, that channel carries nothing, and the policy has learned very little about mapping observations to actions. Low training loss with a rising KL term is the signature.

3. Why does the decoder use learned queries instead of generating the chunk one action at a time?

Speed and stability. All hundred slots are computed in one parallel forward pass, so inference cost does not scale with chunk length, and there is no autoregressive loop in which an early mistake conditions every later action. The price is that the actions in a chunk are only weakly coordinated - each is computed from the shared observation memory, but slot 99 never sees slot 98’s answer.

4. Temporal ensembling is the paper’s own contribution and the library disables it. Give the reason, in cost terms.

It requires re-querying every step. n_action_steps must be 1 to form an ensemble at every timestep, so a policy that ran $T/100$ forward passes per episode now runs $T$ - a hundredfold increase in inference. That is affordable in an offline benchmark and frequently not affordable on the hardware attached to the robot. The default trades the paper’s smoothness for deployability.

5. The same algorithm scores 83% in one implementation and 68% in another. What does that tell you to do before comparing two methods?

Compare implementations, not descriptions. Run both methods in one harness, with one dataloader, one normalisation scheme and one evaluation protocol, and pin the versions - a fifteen-point implementation gap is wider than most claimed improvements, so an unpinned comparison measures engineering rather than ideas. It also tells you which artefact to trust when the paper and the code disagree: the code produced the number.

6. Why does ACT only look at one frame when the task clearly has history in it?

Because it puts the temporal extent on the output instead of the input. A hundred-step chunk is a commitment to a stretch of future, which covers much of what a stretch of past would have told you, and it is far cheaper than pushing several frames of several cameras through a ResNet on every query. The variation that a single frame genuinely cannot explain - a demonstrator’s pauses and speed differences - is what the latent absorbs during training.

Do this

About twenty-five minutes.

1. Build it. code/act_lite.py declares every layer and leaves three holes: encode_latent, decode, and the forward that decides between them. Fill them in, plus act_loss. Then:

python act_lite.py

The report should show the training pass consuming a chunk and returning one, the inference pass consuming only a state, and two consecutive inference calls returning identical tensors. If they differ, you are still sampling $z$ at inference.

2. Watch the latent collapse. The same file has a second mode that trains one small model twice, once with the KL term switched off:

python act_lite.py --collapse

On this laptop it prints a training loss of 0.36 with kl_weight = 0 against 0.46 with kl_weight = 10, and then the same two models scored on the deployed path where $z$ is zero: 0.72 and 0.32. The run with the better training loss is more than twice as bad on the robot, and its KL term sits at 103 instead of 0.02. That is question 2, reproduced in under a minute. Your numbers will differ; the ordering should not.

3. Read the real thing. Open LeRobot’s configuration_act.py and modeling_act.py for the version you actually installed, and find three things: where temporal_ensemble_coeff is checked, what happens if you set n_obs_steps above 1, and where $z$ is replaced by zeros on the inference path. Write the three line numbers down. The next lesson trains this, and when something misbehaves you will want to already know where to look.

What you can now do

You can name every component of ACT - the frozen-in-time single observation, the per-camera ResNet, the token-mixing encoder, the learned decoder queries, and the latent that exists only during training - and say what breaks if you remove each one. You can read the shipped config as the specification, explain why the paper’s own temporal ensembling is off by default in cost terms, and articulate why a fifteen-point gap between two implementations of one algorithm is the most important number in this lesson.