How diffusion really works, in plain English

We will start with a single image. Imagine a 512x512 photograph of a golden retriever sitting on grass. We want to teach a neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples. to generate images like this from scratch. Here is the setup.

First, we define a forward processForward processIn diffusion, the fixed procedure that gradually destroys an image by adding a little noise at each step until nothing but static remains. It involves no learning. that gradually destroys the image by adding noise to it. We are going to do this in many small steps, let us say a thousand. At step zero, the image is pristine. At step one, we add a tiny bit of random noise to every pixel. The image looks almost identical to the original, but slightly grainier. At step two, we add a tiny bit more noise. By step a hundred, the image is visibly grainy but still clearly a golden retriever. By step five hundred, the dog is barely visible through the static. By step a thousand, the image is indistinguishable from pure random noise. The dog is gone.

This forward process is mathematically precise. At every step, we know exactly how much noise we added, and we can write down the relationship between the image at step t and the image at step t minus one. This relationship is just: image at step t equals image at step t minus one plus a small amount of noise. There is no learning involved here. The forward process is a fixed, deterministic procedure.

Now here is the magic. If we knew the inverse of the forward process, that is, if we had a function that could take a noisy image and reliably tell us what it looked like one step earlier, before the most recent noise was added, then we could start with pure random noise and run the inverse a thousand times to get back to a clean image. The image we got back would not be the original golden retriever, because we threw away the original information. But it would be some clean image consistent with the noise we started from. Specifically, it would be a sample from the distribution of all possible clean images that the model knows about.

So the entire problem of generative modeling reduces to: can we train a neural network to learn the inverse of the forward process?

Yes, we can, and the way we do it is conceptually beautiful. We take our training dataset, billions of images, and for each image, we randomly pick a step number t between one and a thousand. We run the forward processForward processIn diffusion, the fixed procedure that gradually destroys an image by adding a little noise at each step until nothing but static remains. It involves no learning. up to step t, producing a noisy version of the image. We then ask our neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples.: 'given this noisy image, and the fact that we are at step t, predict the noise that was added.' We compare the network's prediction to the actual noise we added (which we know, because we just added it), compute the error, and use that error to nudge the network's parameters in a direction that makes its prediction slightly more accurate next time. Repeat this billions of times across the training dataset. The network slowly learns to predict noise at every level of corruption.

Once the network is trained, generation works like this. We start with a tensor of pure random noise, the same shape as our images, but completely meaningless. We tell the network: 'this is at step one thousand, predict the noise.' The network gives us its best guess. We subtract that predicted noise from our image, which gives us a slightly less noisy version. We tell the network: 'this is at step nine hundred ninety-nine, predict the noise.' We subtract again. And again. And again. After a thousand steps, we have a clean image. The image is whatever the network's accumulated training has biased it to produce, usually something that resembles the training data, because that is what the network has learned the structure of.

In practice, modern models do not actually use a thousand steps. They use somewhere between twenty and fifty, with a clever sampling procedure that takes bigger jumps. But the principle is the same.

Here is what is wild about this. The neural network never directly tries to generate an image. All it ever does is predict noise. It looks at a noisy thing and says 'here is the noise that is in it.' But by doing that prediction repeatedly, starting from pure randomness, you get an image.

So far we have described unconditional generation, the model produces some random image from the distribution it learned. But you wanted a golden retriever, not whatever the model felt like producing. How does your text promptPromptThe text description you provide to a model to specify what you want it to generate. influence the result?

The mechanism is called conditioningConditioningAny extra input that steers generation beyond the prompt, such as a depth map, pose, edge map, or reference image., and it is built into the noise prediction step. Instead of just asking the network 'predict the noise in this image at step t,' we ask 'predict the noise in this image at step t, given that the image is supposed to depict a golden retriever sitting on grass.' The text of your prompt gets passed through a separate neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples. called a text encoderText encoderA neural network that converts text prompts into numerical embeddings that the diffusion model can use. Modern models use combinations including CLIP and T5., which converts the words into a long list of numbers (a vector) that represents the meaning of the prompt in a form the diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image. can use. That vector gets fed into the diffusion model as additional input, and the model learns during training to use it to bias its noise predictions.

The result is that the same starting noise will produce different images depending on what prompt you condition on. Same noise plus 'golden retriever on grass' gives you a dog. Same noise plus 'a medieval castle at sunset' gives you a castle. The starting noise determines the random seed of the generation, it controls the specific layout, pose, and details, but the prompt steers the generation toward a particular region of the image distribution.

There is a clever trick used to make conditioning stronger, called classifier-free guidance. The idea is that during training, you sometimes show the model the prompt and sometimes show it nothing (an empty prompt). It learns to do both conditional and unconditional prediction. At inferenceInferenceRunning a trained model to produce output. The fast, cheap part of the model lifecycle, in contrast to training. time, you compute both predictions and take a weighted combination, pushing the result strongly in the direction of the prompt and away from the unconditional baseline. The strength of this push is the 'guidance scaleGuidance scale (Classifier-Free Guidance (CFG) scale)A parameter that controls how strongly the model adheres to the prompt versus producing diverse outputs. Higher values mean stricter prompt following. Typical range 5-10.' parameterParameterOne individual weight. Model size is quoted in parameters, in the millions or billions. More parameters can mean more capability and more cost. that you see in tools like Stable Diffusion. Higher guidance means the model sticks more rigidly to the prompt; lower guidance gives more diversity but less prompt adherencePrompt adherenceHow faithfully a model follows the specifics of your prompt, meaning the objects, counts, colors, spatial relationships, and text you asked for. High adherence means you get what you described, not just something in the right neighborhood..

Here is the deep reason diffusion ended up beating GANs, expressed without math. A GANGAN (Generative Adversarial Network)An older family of generative models where two networks, a generator and a discriminator, train against each other. The generator tries to produce realistic images while the discriminator tries to tell real from fake. GANs made sharp images but were unstable to train and largely gave way to diffusion. tries to learn the entire distribution of real images in one shot, it has to map a single random vector all the way to a finished image in one forward pass. That is an enormous, ill-conditioned learning problem, which is why GANs are so finicky to train. A diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image., by contrast, breaks the problem into many small steps. Each step only has to do something easy: 'remove a little bit of noise.' Removing a little noise is a much simpler problem than generating an image from scratch. The cumulative effect of many easy steps is the same as one impossible step, but the training is far more stable and the results are far better.

Another way to put it: diffusion models trade compute for quality. They are slower than GANs at inferenceInferenceRunning a trained model to produce output. The fast, cheap part of the model lifecycle, in contrast to training. time, a GAN can generate an image in a single forward pass, while a diffusion model needs twenty to fifty, but they produce dramatically better results. As compute has gotten cheaper, that trade has looked better and better. A lot of the engineering work since 2022 has been about reducing the number of steps a diffusion model needs while keeping the quality, which is a topic we will return to when we discuss distillationDistillationA technique where a smaller, faster 'student' model is trained to mimic the outputs of a larger 'teacher' model in fewer sampling steps. Used to produce fast variants of slow models (FLUX schnell, SDXL Turbo, etc.)..