By 2024, the basic recipe, flow-matched MM-DiT in latent space conditioned on T5 embeddings, was established. The remaining engineering work was about pushing the speed-quality frontier in both directions. Faster models for interactive use, larger models for higher quality, and various tricks to get more out of the same compute. We will cover the most important techniques because they explain a lot of the version differences between models.
Distillation: making slow models fast
A standard diffusion model needs maybe twenty to fifty sampling steps to produce a good image. Each step is a full forward pass through the network. If you have a model like FLUX with twelve billion parameters, those steps add up, generating a single image can take five to ten seconds on a high-end Graphics Processing Unit (GPU). That is fine for a finished render, but it is too slow for interactive applications where you want to type and see the image update in real time.
The solution is distillation, a technique borrowed from the language model world. You train a smaller, faster 'student' model to mimic the output of a larger 'teacher' model, but in fewer steps. The student does not have to learn how to do diffusion from scratch; it just has to learn to reproduce what the teacher does. With clever training techniques, you can compress fifty steps of the teacher into four steps of the student, or even one step, with relatively modest quality loss.
The most famous example is FLUX.1 [schnell], the fast version of FLUX, which is a four-step distillation of FLUX.1 [pro]. 'Schnell' means fast in German, which is appropriate for a Munich-Freiburg lab. There is also FLUX.1 [dev], which is an eight-to-twenty step distillation that trades a little speed for higher quality. Stable Diffusion 3.5 Large Turbo is a similar distillation, generating in four steps. SDXL Turbo and SDXL Lightning are earlier examples of the same technique.
The thing to know about distilled models is that they have some real limitations. They generally cannot use classifier-free guidance the way their teachers can, the guidance is baked into the weights during distillation. They tend to produce slightly less diverse outputs, because the student has learned a particular path through the space rather than the full distribution the teacher knows. But for most uses, the speed gain is worth it. When you use Krea's real-time canvas and see images updating as you type, you are using distilled models.
Mixture of Experts: scaling capacity without scaling compute
The other major technique borrowed from language models is Mixture of Experts, or MoE. The idea is that instead of a single dense network where every parameter is active for every input, you have many smaller 'expert' networks and a routing mechanism that selects which experts to activate for any given input. The total parameter count can be very large, but the compute per inference step is only a fraction of the total, because most of the parameters are inactive for any given query.
MoE has been used in language models for years, the rumored architecture of GPT-4 includes MoE, but it took until 2025 for somebody to apply it to diffusion. Alibaba's Wan 2.2 was the first major video model to use MoE, and they did it in a clever way that is specific to diffusion. Instead of routing by input token (the way language models do), they route by noise level. One expert handles the early, high-noise steps where the model is laying out coarse structure. A different expert handles the late, low-noise steps where the model is refining details. The two experts have different parameters because the two phases of denoising are fundamentally different problems, the early phase is about composition and layout, the late phase is about texture and detail.
Wan 2.2-A14B has twenty-seven billion total parameters but only fourteen billion active per step, giving it the capacity of a much larger model at the cost of a normal one. The open-source release of Wan 2.2 was a significant moment because it demonstrated that the MoE-for-diffusion approach worked, and it set a template that other video model labs are now following.
Custom hardware and the cost of doing this at all
Underlying everything is the question of compute. Modern generative models, both image and video, require staggering amounts of GPU time to train. To give you concrete numbers: the original Stable Diffusion in 2022 took about 150,000 A100 GPU-hours to train, costing roughly $600,000 at then-current cloud prices. SDXL, the bigger version released in 2023, was several times more expensive. Stable Diffusion 3 cost upwards of $5 million to train. FLUX is rumored to have cost in the tens of millions.
Video models are much worse. Open-source efforts like Open-Sora 2.0 reported training costs of about $200,000, which they presented as a triumph of efficiency. Mainstream commercial video models are believed to cost tens of millions to over a hundred million dollars per training run. Sora's training cost has never been disclosed, but estimates from people who have looked at the compute requirements put it well into the hundreds of millions of dollars range. Veo 3, with its joint audio-video architecture and 4K output, is presumably comparable.
This is the central economic reality of the field. Training a frontier model is a capital-intensive enterprise that requires not just money but also access to compute clusters that very few organizations possess. The labs that have these resources, Google DeepMind, OpenAI, Meta, Alibaba, Tencent, ByteDance, Kuaishou, MiniMax, Stability AI, Black Forest Labs, Runway, are a small group, and the gap between them and everyone else is widening. Open-source communities have done remarkable things on smaller budgets, but they cannot match the scale of the largest commercial models. We will revisit the economics in more detail in the next chapter.