Refinements on the basic recipe

The original diffusion math, from the DDPMDDPM (Denoising Diffusion Probabilistic Model)The 2020 method that made diffusion practical for image generation. It defines a forward process that adds noise and trains a network to reverse it step by step. Most modern image models descend from this idea. paper, treats the forward processForward processIn diffusion, the fixed procedure that gradually destroys an image by adding a little noise at each step until nothing but static remains. It involves no learning. as a stochastic random walk, at every step you take a small random kick in some direction, accumulating to a curved, jittery path between the clean image and pure noise. The reverse processReverse processIn diffusion, the learned procedure that starts from pure noise and repeatedly removes a little predicted noise until a clean image emerges. Generation is the reverse process run to completion. has to learn to undo that curved path. The curves make the problem harder than it needs to be. They mean the model has to learn a complicated, twisting trajectory through the latent spaceLatent spaceThe compressed numeric representation an image is turned into before diffusion. Working here instead of on raw pixels is what makes modern image models fast enough to run..

In 2022, a researcher at UT Austin named Xingchao Liu and his collaborators published a paper called 'Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.' The idea was: instead of training the model to follow these curved paths, what if you trained it to follow straight lines from noise to data? Straight lines are easier to learn and easier to integrate at inferenceInferenceRunning a trained model to produce output. The fast, cheap part of the model lifecycle, in contrast to training. time. The math is closely related to diffusion but reformulated in a way that the field calls 'flow matching', the model is learning a vector field that smoothly transports points from a noise distribution to a data distribution.

The practical payoff is fewer sampling steps for the same quality. Where a traditional diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image. might need fifty steps, a rectified flow model can often get there in twenty. Stable Diffusion 3 and Stable Diffusion 3.5 use rectified flow. So does FLUX. So do most of the newer video models. When you read that a model is 'flow matched' or uses 'rectified flow,' what they mean is: this model trains the network to push noise toward images along straighter trajectories than classical diffusion, and as a result it samples faster. From the user's perspective, you do not see anything different; from the engineering perspective, this is one of the biggest reasons the 2024 generation of models felt like a step change.

The original Stable Diffusion used a text encoderText encoderA neural network that converts text prompts into numerical embeddings that the diffusion model can use. Modern models use combinations including CLIP and T5. called Contrastive Language-Image Pretraining (CLIPCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward.), developed by OpenAI in 2021. CLIP was a major breakthrough at the time, it learned to map images and text into a shared embeddingEmbeddingA list of numbers that represents the meaning of a word, image, or concept in a form the model can compute with. Similar things sit close together in this numeric space. space by training on hundreds of millions of image-caption pairs from the internet. CLIP is genuinely good at understanding the rough semantic content of a phrase. It knows what 'dog' means, it knows what 'cat' means, it knows what 'beach at sunset' looks like.

But CLIP has a serious limitation: it is not really a language model. It does not understand grammar or syntax very well. It tends to treat prompts as a bag of concepts rather than as structured sentences. If you write 'a red ball on top of a blue cube,' early Stable Diffusion would frequently give you a blue ball on top of a red cube, because CLIP did not strongly enforce which color went with which object. Compositional prompts, anything involving multiple objects with specific relationships, were a known weakness.

The fix turned out to be: use a real language model as your text encoder. Specifically, use T5, a large encoder model developed by Google, that has billions of parameters and a much deeper understanding of language structure. T5 is the kind of model that knows the difference between 'the cat chased the dog' and 'the dog chased the cat.' It understands prepositions, nesting, modifiers. When you swap CLIP for T5, your model's promptPromptThe text description you provide to a model to specify what you want it to generate. following improves dramatically.

Modern models do not use just one text encoder; they use several in combination. Stable Diffusion 3 uses three: two CLIP variants (for visual concept alignment, which they are still good at) and T5-XXL (for syntactic and structural understanding). FLUX uses a similar combination. The text from your prompt gets fed through all of them, the resulting embeddings are concatenated, and the diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image. is conditioned on the combined representation. The reason your prompts now work the way you expect, getting the colors right, the spatial relationships right, the counts of objects right, is largely because of T5 and its peers replacing pure CLIP.

In 2024, Stability AI published the Stable Diffusion 3 paper, written by Patrick Esser and the team that had stayed at the company. The paper introduced an architectural variant called MM-DiTMM-DiT (Multimodal Diffusion Transformer)A variant of DiT that processes text and image (or audio) tokens in separate parallel streams joined by attention. Used by Stable Diffusion 3, FLUX, and most newer models. Improves prompt following significantly., for MultimodalMultimodalHandling more than one kind of data, for example text and images together, in a single model. Diffusion TransformerTransformerThe neural network architecture built on attention that underpins modern language and, increasingly, image and video models. It scales well and handles long-range relationships better than convolutional networks.. This is the architecture that FLUX, SD3, and a long list of newer models all use, and it is worth understanding because it represents a meaningful departure from the original DiTDiT (Diffusion Transformer)A diffusion model architecture that uses a Transformer instead of a U-Net. Introduced by Peebles and Xie in 2022. Scales much better than U-Nets and is the basis for almost all current frontier models. recipe.

In a vanilla DiT, the text encoderText encoderA neural network that converts text prompts into numerical embeddings that the diffusion model can use. Modern models use combinations including CLIP and T5. produces an embeddingEmbeddingA list of numbers that represents the meaning of a word, image, or concept in a form the model can compute with. Similar things sit close together in this numeric space., and that embedding gets injected into the transformer through cross-attentionCross-attentionThe step where the image being generated attends to the text embedding. It is the mechanism by which the prompt actually steers the picture. layers, the image patches attend to the text tokens and pull in semantic information, but the text and image streams are not really equals. The text is auxiliary information that conditions the image generation. The image is the main thing being processed.

In MM-DiT, the architecture is genuinely multimodal. There are two parallel streams running through the transformer: one for image patches, one for text tokens. They have separate weightsWeightsThe learned numbers inside a model that encode everything it knows. 'Open weights' means these are downloadable., the image stream is processed by image-specific parameters, the text stream by text-specific parameters, but they exchange information through joint attention layers at every step. The transformer is reasoning about words and pixels at the same time, at every level of abstraction. The text representation gets refined by what is happening in the image just as the image representation gets refined by what is in the text.

The practical effect is dramatically improved alignment between promptPromptThe text description you provide to a model to specify what you want it to generate. and output, especially for complex prompts that involve text rendering, spatial layout, or compositional structure. MM-DiT is a big part of why FLUX can generate images with legible text in them, something that earlier diffusion models were notoriously bad at. It is also a big part of why models like Nano Banana Pro can take fourteen reference images plus a text prompt and produce a coherent composite, the architecture is built to handle multimodal inputs as first-class citizens, not as bolted-on conditioningConditioningAny extra input that steers generation beyond the prompt, such as a depth map, pose, edge map, or reference image..