Why Evals Are The Moat

Everyone building on top of frontier models runs into the same thing eventually. The model gets better, the prompt scaffolding you were quietly proud of stops mattering, and the product you thought you had turns out to be a UI over someone else’s weights.

So the question I keep coming back to is which part of this doesn’t get eaten by the next checkpoint. For us the answer has been measurement. Not the model, not the app, but the ability to say with evidence whether a system is actually good at the specific job you care about.

A benchmark is a number, an eval is a machine

People use the words interchangeably and they shouldn’t. A benchmark is a score you cite. An eval is the thing that produced it: a way to generate tasks, run a system against them, and grade the output so the grade tracks something real. If you can regenerate the tasks, turn the difficulty up, and re-score against a new model the afternoon it drops, you have an asset that keeps paying out. If all you kept was last month’s leaderboard screenshot, you have a citation.

Why it holds up

When the next model lands, your eval is what tells you whether it’s better for your use case, which is exactly the question the model provider can’t answer for you, because they don’t know what your use case is. The taste that goes into it, what “correct” even means in your domain, is slow to build and annoying to copy. And every run leaves you a pile of graded trajectories, which are error analysis today and, once you clean them up, training data tomorrow.

The catch

Evals are unglamorous. Nobody has ever been wowed by a scoring rubric in a demo. The teams that end up owning the layer above the model are going to be the ones who were willing to treat the boring measurement work as the actual product and the app as the thing wrapped around it. We’re betting the company on that being true.