Give the World Model a Truth Channel

Video world models have an evaluation problem: you can watch their rollouts, but you can’t check them. Latent world models like V-JEPA have it worse. They predict in an embedding space nobody can even watch. The field mostly responds by scoring what’s scoreable: frame quality, human preference, or a proxy reconstruction of the scene. I built one of those proxies and it works, but a proxy is still a proxy.

So I tried the opposite move. Instead of better proxies, pick a world where exact ground truth is free. 2D games expose their entire internal state every step if you ask. Crafter hands you the full world map, including everything off-screen, plus inventory and position alongside every frame. MiniGrid encodes the whole board, door states and all, while the agent sees only a partial view. Train and probe world models there, and every claim about “what the model knows” becomes a measurement against literal truth. No hand-written verifiers anywhere; verification is a state diff.

Two days on a MacBook, one frozen V-JEPA 2 checkpoint, about zero dollars of cloud compute. Here’s what fell out. Code and repro for everything is on GitHub.

The truth channel caught a real bug within minutes of existing

My first determinism test failed strangely: two runs with identical seeds and identical actions produced identical pixels but different world states. Two swapped cells, a cow versus grass. The cause turned out to be upstream, in Crafter itself: creature balancing iterates a Python set of objects and indexes into it with its RNG, so memory-address ordering decides which cow despawns, and the divergence leaks into terrain because cows graze grass into path. Pixel-based supervision could never have seen this, because the pixels matched. A position-sorted patch restored byte-exact reproducibility.

That’s the thesis in miniature. If you can’t check state, you don’t know your substrate is broken.

What frozen V-JEPA 2 knows about 2D worlds

Probe the frozen encoder against exact state, with probes split by rollout seed and shuffled-target floors under every number. At 1,440 clips: attentive probes beat mean-pooling on every target, V-JEPA wins where semantics beat pixels (HUD vitals, R² 0.55), and a dumb pixel baseline wins where the answer is literally drawn on screen (resource icons, 0.66). Nobody, no frozen per-clip encoder, decodes off-screen state. Memory requires dynamics.

The result that surprised me came later, when I needed a renderer: a small decoder reconstructs frames from frozen, randomly projected V-JEPA 2 tokens at 0.5-of-255 mean error. Near-lossless. The frozen tokens are essentially a complete encoding of the board. All the linear probing in the world had only ever recovered a fraction of what was sitting in there.

JEPA energy is a real failure detector, with a measured blind spot

Train a small action-conditioned GRU over per-slice V-JEPA latents on clean play only. Seven seconds of training. Its prediction residual becomes an anomaly detector, and because the substrate has a truth channel, I could score it against real anomalies rather than vibes: I teleported the player mid-rollout via engine surgery, rendered as real frames, with exact labels for when the corruption happened.

Teleports: AUC 0.977. The naive baseline, frame-to-frame latent distance, gets 0.565, which is chance. Normal play also changes every pixel; the model knows which changes the actions explain.

Then the honest part. A +5 wood inventory spike, a few HUD pixels, scores 0.57 ± 0.01. Global pooling dilutes small anomalies 256 to 1. Scoring residuals per spatial region and taking the max lifts it to 0.80 ± 0.04, and the HUD region self-identifies as the anomaly location in every seed (32–55 of 120 cases). A runtime verifier that reports where the violation happened, trained on nothing but clean data, with per-failure-class ROC curves. That per-class breakdown is exactly what I’d want published before trusting any energy-based monitor in production.

One scar worth sharing: my first regional implementation let the model learn the projection its own energy was measured in. It promptly shrank that space to make prediction easy and detection died, 0.977 to 0.645. That’s the JEPA representation-collapse failure mode reproduced in a 25-second training run. A frozen random projection fixed it. Anti-collapse isn’t a training nicety; it’s load-bearing for any energy-based monitor.

Latent versus generative, decided by exact state

The argument about whether world models should predict pixels or embeddings usually runs on intuition. With a truth channel it runs on numbers. I trained matched-budget pairs from scratch on identical data: a pixel model (conv encoder, action-conditioned GRU, deconv decoder, next-frame MSE) and a JEPA-style model (same encoder and core, next-embedding prediction against an EMA target, variance regularized).

The generative model’s internals are state-blind. Every probe lands within noise of its shuffled floor. In both environments. At 10 and 40 epochs, where the curves diverge: one objective accumulates state with training, the other never does. With linear probes and MLP probes, so it isn’t an accessibility artifact. The JEPA-style model meanwhile decodes the full MiniGrid board, including the off-view door and key state, at R² 0.46 ± 0.06 from partial observations (shuffled floor ≈ −0.3), and its state signal measurably decays under free-run rollout, which is what compounding error looks like when you can finally see it.

The mechanism reads almost obvious in hindsight. Next-frame MSE on a mostly-static world rewards copying the frame. Latent prediction can’t copy; it has to track what changes.

Scope, honestly: tiny models, grid worlds, one budget. The claim is exactly that wide. Whether the ranking holds at a billion parameters on natural video is precisely the kind of question that needs this instrument at larger scale.

Then I made it playable, and the bugs got interesting

A world model with a truth channel is also the ideal harness for a Genie-style toy: WASD in the browser, every frame hallucinated by the model, exported to ONNX so it runs client-side. You can play it here; the model downloads once and every frame after that is computed on your machine. Building it produced a compressed tour of a decade of world-model failure modes, each caught by an automated ship-gate before reaching the page.

The first two attempts shipped a frozen dream. A bottleneck conv-GRU cannot express Crafter’s view-scrolling, so MSE’s optimum is the mean frame; the fix was switching to a static-board world where dynamics are local deltas, plus wiring the action directly into the decoder. Then key pickups didn’t render, because successful pickups were both rare in random play and small enough that my own event detector was silently discarding them, a small-footprint blindness I had literally just finished measuring in the verifier work and then reproduced in my own tooling.

The deepest one was door logic. Players noticed the locked door opened without the key. I rebuilt the data around the game’s actual logical cases, generated targeted negatives (a policy that walks to the door and knocks without the key), and added a two-sided gate: keyless toggle must not open the door, keyed toggle must. Nine gated training iterations later, here’s what the diagnostics established. The model learns the rule perfectly; teacher-forced on real frames, keyless toggles predict nothing and keyed toggles open the door. The hidden state is irrelevant; swapping it changes nothing. Pasting the real key pixels into a drifted dream frame fully restores correct behaviour. The entire failure is that the model’s own renders dim the key sprite, one step, the moment the agent gets near it, because in training data proximity almost always preceded pickup. The model learned a superstition, then acted logically on its own hallucination.

Data fixes, loss reweighting, occlusion augmentation, deployment-matched rollout training, and a skip decoder each moved the violation down, from 54 to 39 on the gate’s scale. Scaling the model 1.5x moved it to 18. Scaling it 2x moved it to zero. Exactly zero: the keyless toggle now changes the door region by 0.00 while the keyed toggle opens it at 52, and the tenth candidate shipped. The trend was monotone with capacity from the moment the logic existed, which is its own small finding: the rule was learned early and cheaply, and everything after that was buying enough rendering fidelity for the model’s imagination to stop violating its own knowledge. The nine failing candidates were all blocked from shipping by the gates, which is the part I’d defend hardest: the instrument that made the diagnosis possible is the same one that kept the regressions off the page.

What I think this adds up to

Small worlds with exact state turn world-model claims from arguments into measurements. In two days that bought: a reproducibility bug found in a popular benchmark, a calibrated per-failure-class verifier on frozen JEPA latents, a decided micro-scale version of the latent-versus-generative question, evidence that frozen V-JEPA 2 tokens are a near-lossless world encoding, and a precise mechanistic story for why a playable model that knows a rule still breaks it.

The instrument is the contribution. Everything else is what it measured first.

Update (Aug 2026): the headline detector and probe numbers are now means ± std over 5 model-training seeds, after a multi-seed replication. Every qualitative claim held in every seed; three originally published single-seed values (0.584, 0.827, R² 0.51) sat at the favorable end of the spread and read slightly lower as means. Per-seed artifacts live in the repo under reports/multiseed/.