Where you are. You can diagram the major vision-language-action models, fine-tune one on data you collected yourself, and read a results table without being fooled by it. This lesson covers the other half of the frontier: models that predict what happens next rather than what to do next.
Push a book off the table without touching it
Close your eyes. Picture a hardback lying half over the edge of your desk. Push it with one finger. It tips, pivots on the edge, drops, and lands flat with a slap.
Now notice what you did not do. You did not decide what colour the carpet was. You did not render the light falling across the page edges, or the exact shadow under the desk, or the pattern on the cover. You predicted the consequence and skipped the picture almost entirely.
Do it once more with a full mug instead. You flinched, and you flinched before any picture finished forming.
That gap - between the small set of things you bothered to predict and the enormous set you did not - is the whole design question in this lesson.
The idea in one paragraph
A world model is a network trained to answer one question: given what I can see now, and the action I am about to take, what will I see next? Feed it enough video and it absorbs a rough statistical physics. Things fall. Pushed objects slide. Liquid spills when the cup tips past some angle. That capability buys three separate things, and keeping them separate is most of the value of this lesson, because in August 2026 they are at three completely different stages of maturity. You can use a world model to generate training data you never collected. You can use it to plan, by imagining a few candidate actions and scoring the futures they lead to. Or you can use it as an encoder, a perception backbone whose representations already know something about how objects behave. There is a tempting fourth use - replace your physics simulator, train policies inside a dreamed world, deploy to hardware - and that one does not work yet, for reasons that are structural rather than a matter of waiting.
What your simulator gives you that a learned model does not
You already own a simulator. MuJoCo, from the simulation module, is a hand-written physics engine, and it is worth listing exactly what “hand-written” bought you, because a world model gives you almost none of it.
Wider than the screen; scroll it sideways.
| Physics engine | Learned world model | |
|---|---|---|
| How it was made | equations written by a person | gradient descent on video |
| State | you can read every position and velocity exactly | there is no state, only an observation |
| Reset | deterministic, to the same initial condition, forever | none; you can prompt it, not reset it |
| Reward | computable from state, so reinforcement learning works | not computable; nothing is ground truth |
| Contact | approximate but consistent and debuggable | plausible-looking and unreliable |
| What it covers | exactly what you modelled, and nothing else | anything it has seen video of |
That last row is the trade in one line. The engine is narrow and trustworthy; the model is broad and untrustworthy. Neither is a strictly better version of the other, which is why in 2026 they are used together rather than one replacing the other.
Predicting pixels is the expensive way to be right
Go back to the book. Your prediction was cheap because it was coarse. A model that predicts the next frame has no such luxury: it must commit to every pixel, including the wood grain, the shadows and the specular highlight on the mug, none of which affect any decision you will ever make.
The alternative is to predict in latent space: run the current frame through an encoder to get an embedding, predict the next embedding directly, and never generate an image at all. You lose the ability to show a human the imagined future. You keep everything that matters for deciding.
Wider than the screen; scroll it sideways.
V-JEPA 2 (Meta, 11 June 2025) is the clean example. 1.2 billion parameters, pretrained on over a million hours of internet video, predicting in latent space rather than pixels. The action-conditioned variant, V-JEPA 2-AC, was then trained on only 62 hours of unlabelled robot video from the DROID dataset - a tiny amount by the standards of anything else in this module.
Planning by imagining
Here is what V-JEPA 2-AC does with that, and it is not what most people assume. It is not a policy. It is a planner.
That loop is model-predictive control, and you have met its shape before: it is a feedback controller whose “decide” step is a short search over futures. What is new is that the model doing the imagining was learned from video rather than written down.
Deployed this way, zero-shot on Franka arms in two different labs, it reached 65 to 80% success on pick-and-place with new objects in environments it had never seen. That is a genuine, reproduced-across-labs result and it is the strongest evidence in this lesson that world models do something real.
Read the limitations just as carefully. The goal has to be supplied as an image, so anything longer than a single reach needs a human to hand it a sequence of visual subgoals. Meta states the model operates at a single time scale, which is the same limitation in different words: it cannot decide for itself that “make coffee” decomposes into six steps. And planning is not free - secondary write-ups quote roughly 16 seconds per planning step, against something like four minutes for approaches that generate pixels. I could not verify either of those two timings against the paper itself, so treat the ratio as the message and the absolute numbers as hearsay. Either way, this is nowhere near the 30 to 50 Hz your control loop needs, which is why it plans and something else executes.
The three slots in a stack you already own
Draw your own pipeline: teleoperated demonstrations, a dataset, a fine-tuned policy, a control loop. A world model can enter it in exactly three places, and they have wildly different maturity.
Wider than the screen; scroll it sideways.
| Slot | What it does | State in Aug 2026 |
|---|---|---|
| 1. Data generator | invents plausible extra trajectories to widen a small real dataset | the mainstream production use, and openly licensed |
| 2. Planner | scores imagined futures against a supplied goal image | works, narrowly; needs a goal image and a slow loop |
| 3. Encoder | supplies the pretrained vision tower a policy is built on | already shipping inside models you have used |
Slot three is the one people miss, because it does not look like a world model when it arrives. Cosmos-Reason2-2B is the vision-language backbone inside GR00T N1.7, the model you drew in comparative anatomy. A model from the Cosmos world-model line is doing perception and reasoning duty inside a VLA, not simulating anything.
Cosmos and Genie, with dates attached
Two families are worth knowing by name, as of August 2026.
NVIDIA Cosmos 3 is described as a two-tower mixture-of-transformers that unifies physical reasoning, world generation and action generation. Reported sizes are a 16B “Nano” and a 65B “Super”, plus image-to-video and text-to-image configurations. Two things would make it practically relevant rather than a demo reel. First, the licence: the Linux Foundation’s OpenMDW-1.1, permitting commercial use and derivatives with a “Built on NVIDIA Cosmos” attribution, with code, checkpoints, curated synthetic datasets and an evaluation benchmark all released. Second, it can emit numerical robot data - joint angles, gripper positions, trajectory points - not only video. A model that outputs numbers fits in a pipeline. A model that outputs video fits in a keynote.
DeepMind’s Genie line is the interactive branch: Genie 1 (February 2024) generated playable 2D worlds, Genie 2 (December 2024) 3D ones, and Genie 3 (August 2025) added higher resolution and longer memory, so a generated world stays consistent when you look away and look back. A consumer-facing “Project Genie” prototype followed in 2026; I did not verify its rollout dates against a primary source, so check them before repeating any. In November 2025 DeepMind showed SIMA 2 improving at tasks inside Genie-generated worlds with no new human demonstrations, which is the most striking result in this whole area and also the most misread. SIMA 2 is a game agent. Nothing in that loop touched a physical robot.
Promise versus practice, stated plainly
Genuinely useful today:
- Synthetic data generation. Cosmos-class models widening a small real dataset. Open licence, real pipelines.
- Latent planning against a goal image. V-JEPA 2-AC, transferred zero-shot across two labs. Narrow, but real and independently exercised.
- A world model as a perception backbone. Cosmos-Reason2 inside GR00T N1.7.
Still promise:
- Training a policy in a generated world and deploying it to hardware. No published result of this on a physical arm that I can point to as of August 2026. SIMA 2 inside Genie 3 is a game agent.
- Replacing MuJoCo or Isaac. No reliable contact, no ground-truth state, no computable reward, no deterministic reset. This is not a quality gap that scale closes; it is a category difference.
- Long-horizon planning. V-JEPA 2 is explicitly single-time-scale and needs human-supplied subgoals. Hierarchical latent world models are an active research area with no consolidated result.
Check yourself
1. Why can you not run the reinforcement-learning loop from the simulation module inside a generated world?
Because the loop needs three things a generated frame cannot provide. A reward, which has to be computed from ground-truth state, and a generated image has no state behind it to query. A deterministic reset, so that two runs start from identical conditions, which prompting a generative model does not give you. And contact you can trust, since the whole point of the exercise is usually the moment of contact. What you get back is a picture that depicts an outcome, with nothing underneath it that knows whether the outcome occurred.
2. What is the argument for predicting in latent space rather than pixels, in one sentence, and what do you give up?
Almost all the information in a frame - texture, lighting, shadow detail - has no effect on any decision the robot will make, so generating it spends capacity and compute on a rendering step no downstream consumer reads. Predicting the embedding directly skips it. What you give up is the ability to show the imagined future to a human, which matters for debugging and for generating training video, and is exactly why the pixel-generating branch still exists.
3. V-JEPA 2-AC reached 65 to 80% pick-and-place on unseen objects zero-shot. Why is it still not a policy you could put on your arm?
Because it plans rather than acts, and it plans slowly against a goal you have to supply as an image. Each step is a short search over imagined futures, quoted at seconds rather than milliseconds, so it cannot hold a control loop by itself. Anything longer than one reach needs a human-supplied sequence of visual subgoals, because the model is single-time-scale and will not decompose a task for you. It is the deliberative half of a fast-slow split with nothing fast attached.
4. Cosmos-Reason2-2B sits inside GR00T N1.7. Which of the three slots is that, and why is it the least glamorous one?
The encoder slot: a pretrained vision-language backbone whose representations already encode something about how objects behave, with an ordinary action head bolted on top. It is unglamorous because nothing is being simulated and nothing is being dreamed - the world model is doing perception. It is also, as of August 2026, the world-model use with the most deployed mileage, which is a useful correction to the framing the research papers lead with.
5. A vendor claims their generative world model produced synthetic data that improved a policy. What is the one experiment that would convince you?
Hold out real out-of-distribution trials, and compare a policy trained on real data alone against one trained on real plus generated data, evaluated on those held-out trials rather than on the training distribution. Synthetic data that only lifts in-distribution numbers has most likely taught the policy the generator’s artefacts, which is a way of overfitting with extra steps. The claim is about generalisation, so the measurement has to be about generalisation, with the trial count stated.
6. Why does this lesson insist on separating “generate data”, “plan” and “encode” when all three use the same underlying model?
Because they have different maturity, different failure modes and different costs, and a vendor announcement will usually blur them into a single capability claim. Data generation is in production and openly licensed. Planning works narrowly, needs a goal image, and runs orders of magnitude too slowly for a control loop. Encoding already ships inside models you have used and is barely discussed. If you accept the blurred version, a demonstration of one becomes evidence for all three, which is precisely the move that makes the field hard to read.
Do this
About forty minutes, no GPU.
1. Hunt three licences. Open the model card or repository for the current Cosmos release, for V-JEPA 2, and for one VLA you have actually fine-tuned. For each, write down two lines: the licence covering the code, and the licence covering the weights. They differ more often than they agree. I could not confirm V-JEPA 2’s weights licence at the time of writing, so treat that one as genuinely unknown until you have read it yourself, and record the date you checked.
2. Design the synthetic-data experiment. Half a page, no code. You have your own dataset from the hardware module. Suppose you could generate 200 extra plausible episodes for one task. Write down: what you would hold out, what the control condition is, how many trials per cell, and the result that would make you discard the synthetic data. If your design cannot produce that last outcome, it is a demonstration and not an experiment.
3. Put the re-check in your calendar. Ninety days out, a twenty-minute block, with the four questions from the shelf-life note above pasted into the invitation. The field moves faster than your memory of it does, and a recurring calendar entry is a more reliable trigger than a feeling of staleness.
What you can now do
You can define a world model precisely, and separate it from a policy and from a physics engine. You can explain why predicting embeddings beats predicting pixels, and what that costs. You can name the three slots a world model occupies in a stack like yours, say which of them is production-ready in August 2026 and which is not, and give the structural reason a generated world cannot yet replace the simulator you already run. And you know what to re-check in ninety days, which is the only durable form this knowledge takes.