The deep history, part one: from 2015 to Stable Diffusion

In March 2015, four researchers at Stanford published the paper that would, eventually, give us all of this. The lead author was Jascha Sohl-Dickstein, a physicist by training who had moved into machine learning. His co-authors were Eric Weiss, Niru Maheswaranathan, and the well-known theoretical neuroscientist Surya Ganguli.

Their idea was strange and beautiful. They observed that physics gives us a precise mathematical description of how an ordered system slowly turns into a disordered one, how, for instance, a drop of cream stirred into coffee starts as a structured pattern and gradually becomes uniform brown. This forward processForward processIn diffusion, the fixed procedure that gradually destroys an image by adding a little noise at each step until nothing but static remains. It involves no learning. is easy to write down: it is just gradual addition of random Gaussian noise (that is, random numbers drawn from the bell curve). What if, they asked, we could train a neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples. to learn the reverse of that process? To take a noisy image and predict, step by step, what it must have looked like before the noise was added? If you could do that, you could start with pure random noise, the maximally disordered state, and run the reverse processReverse processIn diffusion, the learned procedure that starts from pure noise and repeatedly removes a little predicted noise until a clean image emerges. Generation is the reverse process run to completion. to produce an ordered, structured image.

They got the math to work. They trained the model. It produced recognizable but blurry digits. The paper was accepted at the International Conference on Machine Learning. And then it sat there. For five years. Almost nobody built on it. The deep learning community in 2015 was obsessed with GANs, which were producing flashier results. Diffusion was a curiosity from a physics-flavored corner of the field that did not seem to scale.

Sohl-Dickstein later joined Google Brain and continued doing influential work in machine learning theory. The diffusion paper now has over seven thousand citations. He has said in interviews that he had no idea, at the time, what he had stumbled onto.

The five-year hibernation ended in 2019 when a Stanford PhD student named Yang Song, working with his advisor Stefano Ermon, published a paper called 'Generative Modeling by Estimating Gradients of the Data Distribution.' Song had independently arrived at a closely related idea from a different angle, instead of thinking about diffusion from physics, he thought about it as estimating something called the 'score' of the data distribution, which is a mathematical object that points in the direction of higher probability. The two approaches turned out to be different views of the same underlying math. Song's framing made it possible to scale the technique in ways that the original 2015 formulation could not.

Then in June 2020, three researchers at UC Berkeley, Jonathan Ho, Ajay Jain, and Pieter Abbeel, published a paper called 'DenoisingDenoisingThe core loop of diffusion. Starting from pure noise, the model repeatedly predicts and subtracts a little noise until a coherent image emerges. Diffusion Probabilistic Models,' or Denoising Diffusion Probabilistic Models (DDPMDDPM (Denoising Diffusion Probabilistic Model)The 2020 method that made diffusion practical for image generation. It defines a forward process that adds noise and trains a network to reverse it step by step. Most modern image models descend from this idea.). This is the paper that really started everything. Ho and his collaborators took Sohl-Dickstein's framework, simplified the math, fixed some training instabilities, and showed for the first time that diffusion models could match the image quality of the best GANs of the era. They produced photorealistic 256x256 images of bedrooms, faces, churches. The paper landed in the deep learning community like a bomb. Suddenly the curiosity from 2015 was not a curiosity anymore. It was the new state of the art.

DDPM was a revelation, but it was also extraordinarily expensive. The model worked directly on pixels, and the denoising process required hundreds or thousands of steps, each of which was a full pass through a large neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples.. Generating a single 256x256 image could take minutes. Training a model that could handle 1024x1024 images would have been prohibitive on any reasonable hardware. The next breakthrough had to be about making diffusion efficient enough to actually use.

That breakthrough came from a small computer vision research group at Ludwig Maximilian University of Munich, called CompVis, headed by Professor Björn Ommer. The key figures were two PhD students, Robin Rombach and Andreas Blattmann, working with a third researcher, Dominik Lorenz. They were collaborating with Patrick Esser, a researcher at the New York startup Runway. (Yes, that Runway, the same company we will discuss later as a major video model labModel labA company that trains and ships the frontier generation models themselves, such as Black Forest Labs, Runway, or Google.. We are at the point in the story where many of the people involved are about to become very important.)

In December 2021, this group published a paper called 'High-ResolutionResolutionThe pixel dimensions of an image or video, for example 1024x1024. Higher resolution shows more detail but costs more memory and time. Image Synthesis with Latent DiffusionLatent diffusionDiffusion that operates in a compressed 'latent' space produced by a VAE, rather than directly in pixel space. Dramatically reduces compute cost. Used by essentially every modern image and video model. Models.' The insight was deceptively simple, and it is the single most important architectural trick in all of modern generative imagery, so we are going to spend some time on it.

The problem with running diffusion directly on pixels is that pixels are an extraordinarily inefficient representation of an image. A 1024x1024 RGB image is about three million numbers, but the actual visual content, what makes the image meaningful, what distinguishes it from other images, is much lower dimensional than that. Most of those three million numbers are redundant. Adjacent pixels are highly correlated. Smooth regions of color are described, in some sense, by very little information. If you could find a way to compress the image into a much smaller representation that preserved the meaningful content, you could run the entire diffusion process in that compressed space, do all the expensive math on a fraction of the data, and only convert back to pixels at the very end.

The Munich team realized they could do this with a separate neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples. called a variational autoencoder, or Variational Autoencoder (VAEVAE (Variational Autoencoder)A neural network that learns to compress images into a small numerical representation (a latent) and decompress them back. Used in latent diffusion models to make the generation process tractable.). A VAE is a model that learns to compress images into a small numerical representation called a latent, typically about 64 times smaller than the original, and then decompress that latent back into a faithful reconstruction of the image. You train the VAE first, on lots of images, until it can reliably round-trip images through the compression with minimal loss of quality. Then you freeze the VAE and use it as a kind of universal translator between the pixel world and the latent world.

Once you have a good VAE, you can run your diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image. entirely in latent spaceLatent spaceThe compressed numeric representation an image is turned into before diffusion. Working here instead of on raw pixels is what makes modern image models fast enough to run.. The forward processForward processIn diffusion, the fixed procedure that gradually destroys an image by adding a little noise at each step until nothing but static remains. It involves no learning. adds noise to latents, not to pixels. The reverse processReverse processIn diffusion, the learned procedure that starts from pure noise and repeatedly removes a little predicted noise until a clean image emerges. Generation is the reverse process run to completion. denoises latents, not pixels. The neural network never has to look at a full-resolution image. Only when the diffusion is complete and you have a clean latent do you push it through the VAE decoder one final time to get a real image out. The savings in compute are enormous, you are doing diffusion on something roughly the size of a 128x128 image, even though your output is 1024x1024.

This is what 'latent diffusion' means, and it is the architecture underlying every single image and video model in this document. When you read that a model 'works in latent space' or 'uses a VAE,' this is what they are talking about. The Munich paper got the trick working at high quality, and the resulting architecture became known as a Latent Diffusion Model, or Latent Diffusion Model (LDM).

In early 2022, the Munich team's work caught the attention of a British entrepreneur named Emad Mostaque, who had founded a small AI company called Stability AI. Stability had money, Mostaque had personal wealth and had raised more from venture capital, and crucially, it had access to compute. Mostaque struck a deal: Stability would provide the GPUs needed to train a much larger version of the latent diffusionLatent diffusionDiffusion that operates in a compressed 'latent' space produced by a VAE, rather than directly in pixel space. Dramatically reduces compute cost. Used by essentially every modern image and video model. model on a much larger dataset, and the resulting model would be released to the public as open source.

The dataset was LAIONLAIONA German nonprofit that assembled the LAION-5B dataset of 5 billion image-text pairs used to train Stable Diffusion.-5B, a collection of nearly six billion image-caption pairs scraped from the public internet by a German nonprofit called LAION. The compute was a cluster of 256 NVIDIA A100 GPUs, the most powerful graphics processors available at the time. The training run cost an estimated six hundred thousand dollars. The result was a 860-million-parameterParameterOne individual weight. Model size is quoted in parameters, in the millions or billions. More parameters can mean more capability and more cost. latent diffusion model that the team called Stable Diffusion.

On August 22, 2022, Stable Diffusion was released to the public. Anyone with a reasonably modern graphics card could download the weightsWeightsThe learned numbers inside a model that encode everything it knows. 'Open weights' means these are downloadable., the actual numerical values of the model's parameters, and run the model on their own computer. There was no waitlist, no Application Programming Interface (APIAPI (Application Programming Interface)The programmatic endpoint you call to run a hosted model from your own code instead of a web interface.) key, no ten-dollar-per-image fee, no corporate gatekeeper deciding what you could and could not generate. The model was just there, on the internet, free.

The impact was immediate and seismic. Within two months, Stable Diffusion had over ten million users globally. The GitHub repository hit thirty thousand stars in a few weeks, one of the fastest climbs in the platform's history. An entire ecosystem of tools, custom-trained variants, web interfaces, fine-tunes, and creative communities sprang up around it almost overnight. For the first time, the technology of generative imagery had escaped from inside corporate research labs and into the hands of anyone who wanted it.

This is the moment that, more than any other, made AI image generation a cultural phenomenon. DALL-E 2, OpenAI's much-hyped image generator, had been announced earlier in 2022, but it was only available through a closed beta and remained behind an invitation system for months. Midjourney existed, but it ran on Discord and was not technically open. Stable Diffusion was the first model that anybody could actually have. The downstream consequences are still playing out, copyright lawsuits, the entire concept-art and stock-photography industries getting upended, a generation of independent artists discovering they could iterate on visual ideas at a speed that was previously inconceivable.

The five people who built Stable Diffusion, Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer, are about to become a recurring cast in this story. Most of them later left Stability AI in 2024 and founded a new company called Black Forest Labs in Freiburg, Germany. Black Forest Labs is the company behind FLUX, which we will spend a lot of time on later. Patrick Esser had already been at Runway, which independently became one of the biggest names in video generation. The Munich diaspora is, no exaggeration, one of the most consequential research lineages in the history of the field.