What is a generative visual model, really?

Let us start with the simplest possible question. When you type a sentence into Midjourney or Sora and an image or video appears that matches your description, what is actually happening?

The literal answer is: a very large neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples., meaning a mathematical function with billions of adjustable numbers inside it, called parameters, has been trained on hundreds of millions of images from the internet, and has learned the statistical patterns that connect words to pictures. When you give it a new sentence, it uses those learned patterns to construct a fresh image that has never existed before, one pixel value at a time, in a way that is consistent with both your sentence and everything it has ever seen.

But that answer is too compressed to mean anything yet. Let us unpack it with an analogy.

The sculptor analogy

Imagine a sculptor who has spent her entire career studying the human form. She has dissected anatomy textbooks, sketched live models for thousands of hours, examined Greek statues, watched ballet dancers move, studied how light falls on skin. Her hands and eyes have absorbed an enormous amount of implicit knowledge about what a human body looks like, how it is proportioned, how it moves, how it is shaped by light and shadow.

Now you give her a rough block of marble and say: 'carve me a young woman in mid-stride, looking back over her shoulder.' She does not need a reference photo. She does not need to plan every chisel strike in advance. She picks up her tools and starts removing material. At first she works in broad strokes, establishing the volume, the pose, the rough silhouette. Then she refines: shoulders, hips, the tilt of the head, the curve of the back. Then finer still: the muscles in the calf, the texture of the hair, the fall of the fabric. At every step, her hands are guided by the vast library of implicit knowledge she has accumulated. The statue emerges from the block not because she planned it, but because she has internalized what 'a young woman in mid-stride' should look like, and she is removing everything that does not match that internal picture.

This is essentially what a generative image model does. It starts with a block of pure visual noise, a grid of random colored pixels, like the static on an old television, and it removes material (well, transforms material) until what is left matches your description. The 'implicit knowledge' is the billions of parameters inside the neural network, learned from training on a colossal dataset of images and their captions. The 'description' is your text promptPromptThe text description you provide to a model to specify what you want it to generate., which gets fed into the network as guidance. The 'chisel strikes' are individual passes through the network, each one making the image a little bit cleaner, a little bit more coherent, a little bit closer to what you asked for.

This metaphor is not just a teaching device. It is, almost literally, what is going on. The technical name for this approach is a diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image., and the chisel-strike steps are called denoisingDenoisingThe core loop of diffusion. Starting from pure noise, the model repeatedly predicts and subtracts a little noise until a coherent image emerges. steps. Hold onto this picture, we will refine it as we go, but the core intuition will not change.

Three families of generators, and why one of them won

Diffusion is not the only way to build a generative visual model. Historically, there have been three main approaches, and it is worth understanding all three so you can see why diffusion ended up dominating.

The first family is GANs, which stands for Generative Adversarial Networks. This was the dominant approach from roughly 2014 to 2021. A Generative Adversarial Network (GANGAN (Generative Adversarial Network)An older family of generative models where two networks, a generator and a discriminator, train against each other. The generator tries to produce realistic images while the discriminator tries to tell real from fake. GANs made sharp images but were unstable to train and largely gave way to diffusion.) works by pitting two neural networks against each other: a generator that tries to create realistic fake images, and a discriminator that tries to tell real images from fakes. They train together in a kind of arms race, every time the discriminator gets better at spotting fakes, the generator has to get better at fooling it. After enough rounds, the generator can produce images that are indistinguishable from real ones, at least at small sizes. GANs gave us the original 'this person does not exist' websites, deepfakes, and a lot of academic art. They have a major problem, though: they are notoriously hard to train. The arms race is delicate, and it frequently collapses into a state called 'mode collapseMode collapseA failure, classically in GANs, where the generator produces only a few kinds of output regardless of input, losing diversity. Distinct from model collapse.' where the generator just produces the same few images over and over. They also struggled to scale, making a GAN bigger did not reliably make it better, which meant they hit a quality ceiling that nobody could break through.

The second family is autoregressiveAutoregressiveGenerating output one piece at a time, each conditioned on what came before, the way language models write text. Some image and video models work this way instead of by diffusion. models, which generate images the way large language models generate text, one piece at a time, left to right, top to bottom, with each new piece conditioned on everything that came before. OpenAI's original DALL-E from January 2021 worked this way. So does Google's Parti. The advantage is that these models inherit all the scaling tricks the LLM world has figured out over the last decade. The disadvantage is that they are slow at inferenceInferenceRunning a trained model to produce output. The fast, cheap part of the model lifecycle, in contrast to training. time (you have to generate every patch sequentially) and they have a hard time with global structure, because the model is always reasoning locally about what comes next instead of holistically about the whole image.

The third family is diffusion models, the sculptor approach we just described. They were first proposed in a 2015 Stanford paper that borrowed its core idea from physics, the way a drop of ink spreads through water, and were then largely ignored for five years. That story, and why diffusion eventually won, comes later in the history chapters.

The reason diffusion eventually won is partly empirical and partly architectural. Empirically, it produces higher quality and more diverse images than GANs, and it does so without GAN-style training instabilities. Architecturally, it scales gracefully, when you make a diffusion model bigger and train it on more data, it reliably gets better, in a way that mirrors how large language models scale. By 2022, every major research lab had quietly converged on diffusion as the path forward. By 2024, it was so dominant that the only remaining question was how to make it faster and how to extend it from images to video.