Stable Diffusion in 2022 was a U-Net (a U-shaped convolutional network) based model. The U-Net is a particular kind of neural network architecture, originally designed in 2015 for medical image segmentation, that became the standard backbone for diffusion models in the DDPM era. We do not need to get deep into how U-Nets work mechanically, but the high-level idea is this: a U-Net takes an image, progressively compresses it through a series of convolutional layers down to a small bottleneck representation, then expands it back out through a mirror series of layers, with shortcut connections between the compression and expansion sides. The shape of the data flowing through the network looks like the letter U, hence the name.
U-Nets are good at processing images because they are built around convolutions, which are mathematical operations that look at small local patches of the image and combine them in learned ways. This is well-suited to image data, which has a lot of local structure, adjacent pixels are usually related, edges are local features, textures are local patterns. Convolutional networks dominated computer vision from roughly 2012 through 2020. They were the natural choice for the inside of a diffusion model.
But convolutions have a weakness, and that weakness becomes a serious problem at high resolution and even more so for video. Convolutions only see local context. For information to flow from one side of a 1024x1024 image to the other, it has to pass through many layers of the network, getting compressed and re-expanded each time. The U-Net's bottleneck is meant to address this, at the bottom of the U, the spatial resolution is small enough that everything is effectively in 'view' of everything else, but the bottleneck also discards information. There is a fundamental tension between local detail and global coherence in convolutional architectures, and as people pushed diffusion models to higher resolutions and more complex prompts, that tension started to bite.
Meanwhile, in a completely different corner of machine learning, the transformer architecture was eating the world.
A brief detour into transformers
Transformers are a neural network architecture that was introduced in a 2017 Google paper called 'Attention Is All You Need.' They were originally designed for language translation. The central idea is something called the attention mechanism, which lets every position in a sequence directly interact with every other position, regardless of distance. In a sentence like 'the cat sat on the mat,' a transformer can let the word 'cat' attend directly to the word 'mat,' without having to pass information through 'sat' and 'on' first. This is in contrast to previous architectures (called RNNs, or recurrent neural networks) where information had to be carried forward step by step.
Attention turned out to be the ingredient that allowed neural networks to scale gracefully. Where convolutional and recurrent architectures hit quality ceilings as you made them bigger, transformers kept getting better. This is the discovery that gave us GPT-2 in 2019, GPT-3 in 2020, and eventually ChatGPT in 2022. By 2022, every state-of-the-art language model on Earth was a transformer. The architecture had also crossed over into vision, where 'Vision Transformers' (ViTs) had begun to challenge convolutional networks on image classification tasks. The pattern was clear: if you wanted scale, you wanted attention.
So it was natural to ask: could you replace the U-Net inside a diffusion model with a transformer? Would it work? Would it scale the way transformers do for language?
December 2022, William Peebles, Saining Xie, and the paper that nobody wanted
The answer came from a paper by William Peebles, then a PhD student at UC Berkeley, and Saining Xie, a professor at New York University who had previously worked on Vision Transformers at Meta. Their paper, posted to arXiv in December 2022, was titled 'Scalable Diffusion Models with Transformers.' The architecture they proposed was called the Diffusion Transformer, or Diffusion Transformer (DiT).
The DiT recipe is straightforward in retrospect. You take the latent that comes out of the VAE, you split it into small patches (just like a Vision Transformer splits an image into patches), you treat each patch as a token, and you feed all the tokens into a transformer. The transformer's attention mechanism lets every patch attend to every other patch, which solves the long-range coherence problem that U-Nets struggled with. You add some clever conditioning machinery to inject the timestep and the text prompt into the transformer, and you train the whole thing on a denoising objective just like a normal diffusion model.
The Peebles and Xie paper showed that this worked, and crucially, that it scaled. The bigger they made the DiT, the better it got, and the scaling curve looked like the curves for language models, clean, predictable, and apparently endless. They demonstrated state-of-the-art results on the ImageNet image generation benchmark with a model called DiT-XL/2.
Then they submitted the paper to CVPR 2023, the most prestigious computer vision conference in the world. It was rejected. The reviewers said it 'lacked novelty.' In their defense, the paper was, in some sense, just combining two existing ideas, diffusion and transformers. There was no clever new mathematical trick. There was no new dataset. The argument of the paper was essentially: 'this works and it scales, here are the curves, the end.' But machine learning conferences in that era prized novelty over scaling evidence, and the paper was passed over.
Peebles and Xie resubmitted to ICCV, the next major computer vision conference, where it was accepted. Then, in February 2024, OpenAI released Sora, the text-to-video model that astonished the world with its sixty-second photorealistic video clips. The accompanying technical report had a section that said, in essence: Sora is a Diffusion Transformer. The architecture comes from the Peebles and Xie paper. The lead author of that paper, William Peebles, had joined OpenAI shortly after publishing it, and was now leading the Sora team.
Yann LeCun, Meta's chief AI scientist, posted on social media: 'The Diffusion Transformer paper, by my former-FAIR-and-current-NYU colleague Saining Xie and former-Berkeley-student-and-current-OpenAI engineer William Peebles, was rejected from CVPR 2023 for lack of novelty, accepted at ICCV 2023, and apparently forms the basis for Sora.' It became one of the most-cited examples in machine learning history of conference review processes failing to recognize a foundational contribution. By 2024, every frontier image and video model was a Diffusion Transformer or a close variant. The U-Net era was over.
There is a moral here that is relevant to anyone trying to read the field. The biggest architectural innovations in machine learning are often not flashy. They are often combinations of existing ideas that turn out to scale unusually well. The thing to watch for is not 'this is mathematically elegant' but 'this gets predictably better with more compute and more data.' DiT was the latter, and that is why it won.