The landmark papers, and why they mattered

If you wanted to understand the current state of the field through reading papers, you would start with the seven listed here. Each one contained an idea that became foundational. Most of their authors are now at frontier labs building production models. Most of the papers are surprisingly readable for an outsider, modern machine learning papers are written in a particular style that mixes precise math with informal English prose, and the high-level argument can usually be followed even without working through the equations.

1. Sohl-Dickstein et al. (2015), the paper nobody read

Title: 'Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.' Authors: Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli (Stanford). This is the paper that introduced the concept of diffusion models for generative image synthesis. The mathematical machinery was borrowed from physics, specifically from nonequilibrium thermodynamics, which describes how ordered systems become disordered. The key insight was that you can run that process in reverse, with a learned neural networkNeural networkA mathematical function built from layers of simple operations, with millions or billions of adjustable numbers inside it, that learns a task by having those numbers tuned on examples., to turn noise into ordered images. The paper produced tiny, blurry results. It sat largely uncited for four years. Today it has over seven thousand citations. Sohl-Dickstein has said in interviews that he had no idea, at the time, what he had stumbled onto.

2. Ho, Jain, Abbeel (2020), diffusion finally works

Title: 'DenoisingDenoisingThe core loop of diffusion. Starting from pure noise, the model repeatedly predicts and subtracts a little noise until a coherent image emerges. Diffusion Probabilistic Models.' Authors: Jonathan Ho, Ajay Jain, Pieter Abbeel (UC Berkeley). This is the paper that turned Sohl-Dickstein's idea into something that could actually compete with GANs. It simplified the math, fixed training instabilities, and demonstrated photorealistic generation on CIFAR-10 and CelebA at a quality level that nobody had seen before from a non-Generative Adversarial Network (GANGAN (Generative Adversarial Network)An older family of generative models where two networks, a generator and a discriminator, train against each other. The generator tries to produce realistic images while the discriminator tries to tell real from fake. GANs made sharp images but were unstable to train and largely gave way to diffusion.) method. It is the foundation of every modern image and video generation model. The Berkeley lab where this work was done, Pieter Abbeel's group, has become one of the most important nodes in the entire generative AI research network. Ho went on to co-found Ideogram in 2022.

3. Rombach et al. (2021), latent diffusionLatent diffusionDiffusion that operates in a compressed 'latent' space produced by a VAE, rather than directly in pixel space. Dramatically reduces compute cost. Used by essentially every modern image and video model. makes it economical

Title: 'High-ResolutionResolutionThe pixel dimensions of an image or video, for example 1024x1024. Higher resolution shows more detail but costs more memory and time. Image Synthesis with Latent Diffusion Models.' Authors: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer (LMU Munich and Runway). This is the paper that made high-resolution diffusion practical by introducing the trick of running the entire diffusion process in a compressed latent spaceLatent spaceThe compressed numeric representation an image is turned into before diffusion. Working here instead of on raw pixels is what makes modern image models fast enough to run. rather than directly in pixels. Without this trick, generating a 1024x1024 image with diffusion would still cost minutes on the best hardware, instead of seconds. The architecture became the foundation of Stable Diffusion the following year, and through Stable Diffusion it became the foundation of essentially every subsequent image and video model. Four of the five authors later founded Black Forest Labs.

4. Radford et al. (2021), Contrastive Language-Image Pretraining (CLIPCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward.) and the dawn of text conditioningConditioningAny extra input that steers generation beyond the prompt, such as a depth map, pose, edge map, or reference image.

Title: 'Learning Transferable Visual Models From Natural Language Supervision.' Authors: Alec Radford and twelve colleagues at OpenAI. This is the paper that introduced CLIP, Contrastive Language-Image Pretraining, the technique that taught a neural network to map images and text into a shared semantic space. It is the reason text-to-image generation works at all. Before CLIP, getting a neural network to understand 'a photo of a golden retriever on grass' required either a hand-built ontology or a tiny labeled dataset. CLIP learned the mapping from 400 million scraped image-caption pairs and produced a model that could understand essentially any visual concept described in natural language. CLIP was the text encoderText encoderA neural network that converts text prompts into numerical embeddings that the diffusion model can use. Modern models use combinations including CLIP and T5. in the original Stable Diffusion. It is still used in modern models alongside larger language model encoders like Text-to-Text Transfer TransformerTransformerThe neural network architecture built on attention that underpins modern language and, increasingly, image and video models. It scales well and handles long-range relationships better than convolutional networks. (T5).

5. Peebles and Xie (2022), the transformer takes over

Title: 'Scalable Diffusion Models with Transformers.' Authors: William Peebles (UC Berkeley) and Saining Xie (NYU). This is the paper that introduced the Diffusion Transformer architecture, demonstrating that you could replace the U-NetU-NetA convolutional neural network architecture with a characteristic U shape (compress, then expand) used as the backbone of early diffusion models. Largely replaced by Diffusion Transformers in 2023-2024. (a U-shaped convolutional network) backbone of a diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image. with a Transformer and get a model that scales much more cleanly. It was rejected from CVPR 2023 for 'lack of novelty,' accepted at ICCV later that year, and within fifteen months had become the architectural basis for Sora, Stable Diffusion 3, FLUX, and essentially every frontier model that followed. Peebles joined OpenAI shortly after publishing the paper and led the Sora team. The story of this paper, rejected, then vindicated, has become the canonical example of conference review processes failing to recognize foundational work.

6. Esser et al. (2024), MultimodalMultimodalHandling more than one kind of data, for example text and images together, in a single model. Diffusion Transformer (MM-DiTMM-DiT (Multimodal Diffusion Transformer)A variant of DiT that processes text and image (or audio) tokens in separate parallel streams joined by attention. Used by Stable Diffusion 3, FLUX, and most newer models. Improves prompt following significantly.) and the multimodal turn

Title: 'Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.' Authors: Patrick Esser and colleagues at Stability AI. This is the Stable Diffusion 3 paper, and it introduced the dual-stream MM-DiT architecture (parallel transformer paths for text and image, joined by attention) along with the use of rectified flow as the training objective. These two ideas, joint multimodal processing and straight-line trajectory training, define the current generation of image and video models. FLUX inherits both directly. Almost every model released in 2025 and 2026 is some variation on this template. The paper is also notable for being one of the last major contributions from the Stability AI research team before most of them left to found Black Forest Labs.

7. Brooks et al. (2024), Sora as world simulator

Title: 'Video generation models as world simulators.' Authors: Tim Brooks, Bill Peebles, and the Sora team at OpenAI. This is the technical report released alongside Sora's preview in February 2024. It is not a traditional research paper, there is no full method description, no reproducible code, no comparison benchmarks. What it is, instead, is a manifesto. The argument: scaling video generation models produces something that is qualitatively different from a video generator. It produces a system that has, by accident, learned an implicit model of physics, object permanence, scene composition, and the temporal logic of the real world. The paper claims (and demonstrates with cherry-picked examples) that Sora can simulate counterfactual scenarios, model interactions between objects, and exhibit emergent properties consistent with having absorbed a world modelWorld modelA model that learns an internal simulation of how a scene evolves, aiming to keep physics, objects, and space consistent over time. A frontier for long, coherent video.. Whether this claim is true at the level the report suggests is one of the open debates in current AI research, but the paper has shaped how everyone thinks about what video models are for.

If you only read one of these papers, read Sohl-Dickstein 2015. It is the most surprising of the seven. It contains an idea that was correct but unrecognized for half a decade, and reading it after the fact gives you a sense of what foundational research actually looks like, quiet, slightly weird, drawing on ideas from outside the immediate field, with results that look modest enough to be safely ignored.