Stability AI (Stable Diffusion family)

We have already covered Stability AI extensively in the foundations, they were the company that funded and released the original Stable Diffusion in August 2022, and they employed the Munich researchers who built it. The historical importance of Stability AI is enormous. Without their willingness to release the model weightsWeightsThe learned numbers inside a model that encode everything it knows. 'Open weights' means these are downloadable. publicly, the entire open-source AI image ecosystem might not exist.

But Stability AI as a company has had a turbulent recent history that is worth understanding because it explains the current state of the Stable Diffusion product line. Founder Emad Mostaque was a controversial figure, charismatic, ambitious, prone to making large public claims that were difficult to back up. The company raised over $100 million and grew to nearly 200 employees. But by early 2024, internal issues were mounting: the company's commercial revenue was much smaller than the funding implied, several key research staff had left or were considering leaving, and there were disputes over who owned the Intellectual Property (IP) for Stable Diffusion (the original Munich researchers had reasonable claims to credit and ownership). In March 2024, Mostaque resigned as CEO. Shortly after, Robin Rombach and his colleagues left to found Black Forest Labs.

The company nearly collapsed in mid-2024 but was rescued by new investors and a leadership reshuffle. Stability AI today is a smaller, more focused organization that continues to release Stable Diffusion models, but it is no longer the dominant force it was in 2022 and 2023. The creative center of the open-source image generation world has moved to Black Forest Labs.

Stable Diffusion 1.4 and 1.5 (2022)

The original public releases. 860 million parameters, U-NetU-NetA convolutional neural network architecture with a characteristic U shape (compress, then expand) used as the backbone of early diffusion models. Largely replaced by Diffusion Transformers in 2023-2024. architecture, CLIPCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward. text encoding, trained on a subset of LAIONLAIONA German nonprofit that assembled the LAION-5B dataset of 5 billion image-text pairs used to train Stable Diffusion.-5B. SD 1.5 became the dominant base model for the open-source community for nearly a year and a half. An enormous fraction of the LoRAs, fine-tunes, and custom checkpoints from 2022 to 2023 are based on SD 1.5. Even today, in 2026, there are still significant communities running SD 1.5 derivatives because the model is fast, runs on modest hardware, and has well-understood characteristics.

Stable Diffusion 2.0 and 2.1 (late 2022)

Stability's attempt to upgrade the original. Used a different text encoderText encoderA neural network that converts text prompts into numerical embeddings that the diffusion model can use. Modern models use combinations including CLIP and T5. (a CLIP variant called OpenCLIP rather than the original CLIP) and made some architectural changes. The result was a model that scored similar quality on benchmarks but felt different to use, and the community largely rejected it, most people stayed with SD 1.5 or moved to SDXL when it came out. SD 2.x is the only Stable Diffusion release that could fairly be called a misstep.

Stable Diffusion XL (SDXL, July 2023)

A major upgrade. 3.5 billion parameters, two text encoders (regular CLIP plus a larger CLIP variant), native 1024x1024 generation, much better quality across the board. SDXL became the new standard base model for the open-source community and held that position for over a year. The vast majority of fine-tunes and LoRAs released between mid-2023 and mid-2024 are SDXL based.

Stable Diffusion 3 (SD3, June 2024)

This was the architectural breakthrough, the first Stable Diffusion to use the MM-DiTMM-DiT (Multimodal Diffusion Transformer)A variant of DiT that processes text and image (or audio) tokens in separate parallel streams joined by attention. Used by Stable Diffusion 3, FLUX, and most newer models. Improves prompt following significantly. architecture and the rectified flow training objective. 8 billion parameters in the largest variant. It introduced the dual-stream transformerTransformerThe neural network architecture built on attention that underpins modern language and, increasingly, image and video models. It scales well and handles long-range relationships better than convolutional networks. with separate parallel paths for text and image tokens that we discussed in the foundations. The release was somewhat controversial, the initial 'Medium' version shipped with quality issues that the community criticized loudly, but the architectural innovations were sound, and SD3 served as the template for FLUX (which was developed by the same people who designed SD3 before they left to start Black Forest Labs).

Stable Diffusion 3.5 (October 2024)

The current Stable Diffusion flagship. Comes in three variants: 3.5 Large (8 billion parameters, highest quality), 3.5 Large Turbo (a 4-step distilled version of Large), and 3.5 Medium (2.5 billion parameters, designed to run on consumer hardware). Architecturally similar to SD3 but with significant training improvements. The Medium variant introduces an enhanced architecture called MMDiT-X with multi-resolutionResolutionThe pixel dimensions of an image or video, for example 1024x1024. Higher resolution shows more detail but costs more memory and time. training, allowing the model to generate at variable resolutions more reliably. The community license allows free commercial use up to $1 million in annual revenue, with paid licenses for larger organizations.

Stable Diffusion's current strategic position is interesting. The models are still good, competitive with FLUX and the closed alternatives, but the open-source community has largely moved its center of gravity to FLUX-based workflows. SD 3.5 is used by people who want a permissively licensed open-weightsWeightsThe learned numbers inside a model that encode everything it knows. 'Open weights' means these are downloadable. model or who have existing pipelines built on Stable Diffusion infrastructure. For new projects, FLUX is more often the default choice.

The single event that best explains how Stability lost its lead is the Stable Diffusion 3 Medium release on June 12, 2024. The model was fine, but it shipped with a restrictive community license that alienated the exact open-source community that had built the brand. Free commercial use was gated behind revenue, funding, and monthly-active-user thresholds and a monthly image cap, and the terms were read as claiming ongoing control over anything fine-tuned on SD3, even models trained on SD3-generated images, with a clause implying you had to delete your models if the agreement ended. CivitAICivitaiA community website for sharing Stable Diffusion and FLUX models, LoRAs, and embeddings., the community's largest resource hub, temporarily banned all SD3 content to avoid the legal exposure. A company's own core fanbase rejecting its flagship release is a rare and damning signal.

Stable Diffusion spans two clear architectures. Era one, from SD 1.5 through SDXL, is classic latent diffusionLatent diffusionDiffusion that operates in a compressed 'latent' space produced by a VAE, rather than directly in pixel space. Dramatically reduces compute cost. Used by essentially every modern image and video model.: a VAEVAE (Variational Autoencoder)A neural network that learns to compress images into a small numerical representation (a latent) and decompress them back. Used in latent diffusion models to make the generation process tractable. compresses an image into a latent spaceLatent spaceThe compressed numeric representation an image is turned into before diffusion. Working here instead of on raw pixels is what makes modern image models fast enough to run., a U-NetU-NetA convolutional neural network architecture with a characteristic U shape (compress, then expand) used as the backbone of early diffusion models. Largely replaced by Diffusion Transformers in 2023-2024. denoises it there, and CLIPCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward.-style text encoders condition the result. This is the design in the original latent-diffusion paper, and SDXL, at roughly 3.5 billion parameters with two text encoders and a base-plus-refinerRefinerA second model or pass that takes a near-finished image and sharpens its details, used in some two-stage pipelines. pipeline, was the last widely beloved fully-open Stability image base, still heavily fine-tuned today. Era two, from SD3 onward, replaced the U-Net with the MultimodalMultimodalHandling more than one kind of data, for example text and images together, in a single model. Diffusion TransformerTransformerThe neural network architecture built on attention that underpins modern language and, increasingly, image and video models. It scales well and handles long-range relationships better than convolutional networks. and swapped standard diffusion training for rectified flow, described in the arXiv paper on scaling rectified flow transformers.

The MMDiT idea is genuinely important: it keeps separate weight streams for text and image tokens but joins them in the attention operation, which measurably improved typography and prompt adherencePrompt adherenceHow faithfully a model follows the specifics of your prompt, meaning the objects, counts, colors, spatial relationships, and text you asked for. High adherence means you get what you described, not just something in the right neighborhood., and it influenced the whole field, including FLUX, whose founders co-authored that SD3 research. The honest caveat is that the shipped SD3 Medium weightsWeightsThe learned numbers inside a model that encode everything it knows. 'Open weights' means these are downloadable. did not fully live up to their launch-day benchmarkBenchmarkA standardized test used to compare models on a task, such as prompt adherence or image quality. claims, most visibly mangling human anatomy on some prompts, a reminder that a strong architecture and a strong release are not the same thing.

Stability's strengths are its legacy and its breadth. It created the entire open image-generation ecosystem, and that installed base still runs largely on SDXL and SD 1.5, with their deep libraries of fine-tunes, LoRAs, and ControlNets; its models are efficient enough to run on consumer GPUs; the MMDiT and rectified-flow research it published is real and field-shaping; and it spans more modalities than most, across image, video, audio, and 3D experiments. The 2024 reset also brought serious industry backing and distribution through WPP, Amazon Bedrock, and Nvidia.

The weaknesses are the reasons it is now a survivor rather than the leader. FLUX and other newer models overtook it on open-weight image quality and took the community with them, the SD3 launch and its license were a self-inflicted wound, the core researchers and much senior leadership left in 2023 and 2024, and the company went through a documented near-bankruptcy before its rescue. It also carries ongoing copyright exposure, including the Getty Images litigation over training data, and a strategic ambiguity as it spreads across many modalities and shifts toward enterprise licensing that can read as retreating from the open community that made it.

For practical use, the open weightsOpen weightsModels whose trained parameters are publicly downloadable, allowing anyone to run them locally or fine-tune them. Contrasts with closed/API-only models. for SDXL, SD 1.5, and the SD3.5 family (Large, Large Turbo, Medium) live on Hugging FaceHugging FaceThe dominant repository for open-source AI models, including all the open-weight image and video models discussed in this document. under Stability's community license, and the honest advice is that SDXL and SD 1.5 remain the most practical fully-open bases for heavy local fine-tuningFine-tuningContinuing to train an existing model on new data to specialize it for a style, subject, or domain, without training from scratch. because the tooling and checkpointCheckpointA saved set of model weights. The .safetensors or .ckpt files you download to run a model. libraries are deepest there. For hosted access there is the Stability Developer Platform, plus Amazon Bedrock and Nvidia NIM. Read the license per model, because the thresholds on revenue, monthly active users, and volume decide whether you need a paid enterprise license, and budget for that early if your usage is commercial and large.