What benchmarks actually measure (and what they don't)

You will see public benchmarks like the Artificial Analysis text-to-image leaderboard and various video benchmarks ranking these models against each other. It is worth understanding what these benchmarks measure and what they miss.

Most public benchmarks are based on human preference, people are shown two outputs from different models and asked which one they prefer for the same promptPromptThe text description you provide to a model to specify what you want it to generate., and the winning rates are aggregated into ELOEloA rating derived from head-to-head human preference votes, borrowed from chess, used to rank models on leaderboards.-style ratings. This measures something real but it has well-known biases. People tend to prefer images that look more polished, more colorful, more conventionally attractive, which means models tuned to produce 'pretty' outputs do well even if they are bad at following specific instructions or rendering accurate details. Models with strong opinions (Midjourney, Krea-1) sometimes get penalized because the AI judges or human raters do not appreciate their aesthetic choices.

More specific benchmarks, text rendering accuracy, prompt following, anatomical correctness, physical plausibility for video, give you more useful information for specific tasks but are less commonly published. The honest truth is that the best way to evaluate models is to test them on your own use cases with your own prompts and judge the results yourself. Benchmarks are useful as a sanity check and as a starting point, but they should not be the final word.

The deeper problem is that the thing most benchmarks measure, average preference across a broad prompt set, is not the thing most production work needs, which is reliability on a narrow one. A model that wins the arenaArenaA public leaderboard that ranks models from large numbers of blind pairwise human preference votes, which tends to reward average appeal over fitness for a specific job. by being pleasing on landscapes and portraits can still fail every time you ask for a hand holding a specific object, legible text on a package, or the same character twice. Aggregate scores hide these tail failures, and the tail is exactly where professional work lives. Two other distortions are worth naming. Benchmarks reward the median output, but a skilled operator cares about the best output out of ten, which is a different quantity entirely. And blind-preference tests strip away the controllability, editing, and workflow integration that decide whether a model is usable at all, so a model can top the leaderboard and still be the wrong tool for a real job. Read the numbers as a rough filter, not a ranking, and trust a weekend of your own prompts over any leaderboard.