How the Dream Works
The playable demo is a ~33M-parameter neural network pretending to be a game engine. This page explains the machine: the inference loop your keystrokes drive, where the training data came from, why the loss function has six terms, and the gates that decided what was allowed to ship. Longer story in the essay; everything here is reproducible from the repo.
The loop
One keypress runs one step of this, entirely in your browser (ONNX Runtime Web, WebAssembly):
frame (64×64×3) ─▶ conv encoder ─▶ embedding ─┐
│ ├─▶ GRU ─▶ hidden state ─┐
action (0..6) ─▶ embedding ───────────────────┘ (memory) │
│ ▼
└──────────── spatial skip ────────▶ delta decoder
│
frame + delta ─▶ clamp ─▶ snap to palette ─▶ NEXT FRAME ───────────────┘
▲ │
└────────────── feeds back ────────────────┘ Three design choices matter. The decoder predicts a residual delta over the current frame rather than a whole image, because on a mostly-static board the honest prediction is "almost nothing changed," and a model forced to repaint everything collapses to painting the average. The action embedding is wired directly into the decoder, so "what you pressed" has a short path to pixels. And the output snaps each pixel to the game's discrete color palette, so tiny per-step errors can't compound into smear.
Where the data comes from
The environment is MiniGrid DoorKey, and the useful property is that it has a truth channel: the engine will tell you its exact state every step — the full board, what the agent carries, what's in front of it. That turns data generation and evaluation into engineering instead of guesswork. Every transition in the corpus is labeled by a classifier over exact state into one of ten logical cases:
forward_move forward_blocked rotate noop pickup_key pickup_noop face_key_no_pickup toggle_locked_no_key toggle_locked_with_key toggle_unlocked_door
Four scripted policies generate the rollouts, and each exists because random play starves a case the model needed:
| policy | what it does | the case it exists for |
|---|---|---|
| explorer | biased random movement | general dynamics |
| seeker | fetches the key, opens the door, heads to goal | pickups and keyed door-opens, which random play almost never produces |
| knocker | walks to the locked door without the key and toggles it | the negative: keyless toggles that must do nothing |
| tease | approaches the key, lingers facing it, walks away | anti-superstition: being near the key must not make it vanish |
Clips containing rare cases are oversampled by case (keyless door-knocks 8×, key-lingering 6×, and so on), so a few hundred logical events get the gradient share of thousands.
A loss assembled from failures
The training objective is next-frame prediction plus five corrections, and every correction is a patch over a specific observed failure:
| term | the failure it fixes |
|---|---|
| input noise | self-fed frames are imperfect; without noise the model has never seen its own output distribution (the GameNGen trick) |
| change-weighted MSE | motion is a tiny fraction of pixels; unweighted MSE learns "copy the frame" |
| no-change boost | the mirror problem: a correct keyless toggle predicts nothing, which earns no gradient under change-weighting — so the door learned to open unconditionally |
| key-region weight | the key sprite is 64 pixels the whole plot depends on; the model dimmed it whenever the agent came near, then treated its own dimming as "key taken" |
| key occlusion | hide the key in inputs sometimes so the recurrent state must remember the pickup rather than re-reading it off the screen |
| deterministic tail rollouts | the last 5 steps of every training clip always feed the model its own predictions — one-step noise never exposes deep self-fed drift, and deployment is nothing but deep self-fed drift |
Even with all of that, the door logic only fully resolved with capacity. The keyless-toggle violation went 54 → 39 through data, loss, and architecture fixes, 18 at 1.5× width, and exactly 0.00 at 2× width, while the keyed door kept opening at 52. The rule was learned early and cheaply; everything after was buying enough rendering fidelity for the model's imagination to stop contradicting its own knowledge.
The gates
Nothing ships by eyeball. Every candidate model runs a gauntlet scored against the engine's exact state, and nine of ten candidates in the door campaign were refused by it:
| gate | requirement | shipped model |
|---|---|---|
| liveness | self-fed rollouts keep changing (correct wall-blocking excepted) | 24–27/255 on movement |
| action sensitivity | different keys produce different frames | 23.6/255 |
| pickup renders | at real pickup events, beat the copy-the-frame baseline | 56× better |
| door opens (keyed) | replayed key→door script opens the door in dream mode | 52/255 |
| door stays shut (keyless) | replayed keyless knock changes the door region by <8/255 | 0.00 |
From checkpoint to your browser
pytorch checkpoint ─▶ single-step ONNX graph (frame, action, h) → (frame', h')
─▶ fp16 conversion (132MB → 66MB, max 1/255 drift, logic identical)
─▶ GitHub release asset (versioned with the code)
─▶ Cloudflare Pages Function proxies it same-origin, edge-cached
─▶ ONNX Runtime Web executes each step on your machine The fp16 model was re-verified against the same door gates before shipping: keyless 0.00, keyed 52.15, identical to fp32. Nothing about the model runs server-side; after the one download, the dream is yours.