World Models Melt

An open world model will hand you a thirty-second rollout that looks great. Every frame, on its own, is a plausible photograph. Then you play it back and something is wrong in a way that’s hard to name at first. Pan the camera across a room and pan back, and the furniture has quietly rearranged itself. A doorway that was on the left is a window now. The model was never holding a room in its head. It was generating the next plausible frame, and a stack of plausible next frames does not add up to a consistent place.

I’ve started calling this scene melt, and the thing that bothers me is that our standard metrics don’t see it at all.

Why the usual numbers miss it

The metrics people reach for on generated video, FVD and the VBench family, are measuring the wrong axis for this. FVD compares the distribution of generated clips to real ones. It tells you the frames look like video, not that the video describes a stable world. Per-frame quality scores have the same blind spot by construction, because every individual frame in a melting rollout can be sharp, well lit, and photoreal. The failure isn’t in any one frame. It’s in the relationship between frame 12 and frame 240, and that relationship is three-dimensional, so a stack of 2D frame scores walks right past it.

You can catch it by eye if you stare at enough loops, but staring is not a metric. I wanted a number.

Reconstruct the scene and watch it fail

The idea is simple to state. If a rollout is geometrically consistent, you should be able to reconstruct a coherent 3D scene from it. If it melts, the reconstruction should fall apart in ways you can measure. So run feed-forward 3D reconstruction over the video and read the reconstruction’s own behaviour as the signal.

The reconstruction part is what recently got cheap. The DUSt3R and VGGT lineage of feed-forward models will take ordinary video and return per-frame depth, camera poses, and a point cloud, with no per-scene optimisation and no camera calibration. The one I used runs locally on Apple Silicon: fifteen to twenty-five minutes for a thirty-to-sixty second clip on an idle laptop. Slow, but fine for an eval you run offline. You are not training anything. You are pointing a model that already understands 3D at a video and watching what it does.

Four things fall out of that, and each catches a different way for geometry to be wrong.

Reconstruction confidence over time. These models report how sure they are, frame by frame. On coherent footage that confidence is steady; when the underlying geometry stops making sense, it sags, and the shape of the sag across a rollout is itself a signal.

Revisit drift, which is the one I care about most and the actual scene-melt detector. When the camera returns to a spot it has already seen, does the newly reconstructed geometry line up with what was there the first time? If the scene stayed put, it aligns. If the room rearranged itself while you weren’t looking, the revisited geometry lands somewhere else, and the size of that gap is the melt.

Depth temporal consistency. Reproject the depth from one frame into the next along the recovered camera motion. In a stable scene the surfaces agree; in a melting one they slide.

Pose plausibility. The recovered camera trajectory should look like something a real camera could do. Teleports and impossible accelerations are a tell that the model isn’t moving through a place, it’s dreaming frames one at a time.

Revisit drift is the piece I haven’t seen framed quite this way. Loop closure is old news in SLAM, but scoring a world model against its own past that way, asking flatly whether the world stayed the same, wasn’t in the prior art I found, WorldScore and PDI-Bench included.

Making sure I was measuring the model, not my tools

A metric you can’t calibrate is a vibe with a decimal point. The trap here is specific and worth naming. The reconstruction model can fail on its own, and if it fails on synthetic-looking footage the same way a melting world model would, I’d be measuring the recon model’s confusion and calling it the world model’s. So the calibration set is built to pull those two apart.

It has three groups. Real captured video, which should score high because the geometry is real. Generated rollouts from open world models, which are the thing under test. And deliberately corrupted video as a negative control, which should sit at the floor. If real doesn’t beat corrupted by a wide margin, the metric is broken and nothing else it says matters.

It held up. Real footage came in at 66.4, generated at 47.8, and corrupted at the bottom at 19.7. That 18.6-point separation between real and generated is the whole point: the score tells a genuine scene from a generated one with no human in the loop. And revisit drift did exactly what it was built to do on a case you can also just watch. One generated rollout drifted 0.893 on revisit across 51 loop closures and scored a zero on that axis, while a geometrically sound reference clip scored 97.2. The number agreed with my eyes, which is the minimum bar for trusting it on the clips where my eyes can’t keep up.

What’s still hard

I don’t want to oversell it. Monocular reconstruction has no absolute scale, so everything is recovered up to a similarity transform; you compare shapes and trajectories after alignment, not raw distances. Stylised footage is genuinely out of distribution for models trained on real video, so a blocky voxel world can confuse the reconstruction for reasons that have nothing to do with the world model, which is precisely why the corrupted control earns its place. And at fifteen-plus minutes a clip on a laptop it is nowhere near cheap enough to sit inside a training loop, though that’s a hardware problem rather than a method one: a discrete GPU with batched, streamed inference should cut it by an order of magnitude.

Why I bothered

World-model video is improving fast, and the demos are now good enough that “looks right” is doing a lot of load-bearing work in how people judge them. Looks right is a frame property. The thing that will actually matter, as these models get pointed at games, simulators, and anything an agent has to act inside, is whether they hold a coherent world while you aren’t looking at it. That’s a geometry property, and for now the generation is running well ahead of the measurement. Reconstruction is a cheap way to close some of that distance. Point a model that genuinely understands 3D at one that’s only pretending to, and the pretending shows up as drift.