The two knobs that matter: rank and alpha

The rankRank (LoRA rank)The inner dimension of a LoRA's two small matrices, which sets how much capacity it has to learn. Higher rank captures more complex behavior but risks overfitting and produces larger files. is the inner dimension of the two matrices A and B, the number we called r in the previous section. It controls how many independent directions of change the LoRALoRA (Low-Rank Adaptation)A fine-tuning technique that adapts a base model to a specific style or concept using a small additional file. The standard way to customize open-source models. can express. A rank-4 LoRA can change the model in 4 independent ways per layer. A rank-128 LoRA can change it in 128 independent ways per layer. Higher rank means more capacity, which means the LoRA can learn more complex behaviors, but it also means more parameters to train, more risk of overfittingOverfittingWhen a model memorizes its training data instead of learning general patterns, so it does well on what it has seen and poorly on anything new. A common risk when training LoRAs. on a small dataset, and a larger file at the end. The trade-off is real and not symmetric, going from rank 4 to rank 16 buys a lot of capacity at modest cost, while going from rank 64 to rank 128 buys very little additional capacity at significant cost.

The practical wisdom that has emerged from the community over the past three years is that rank should be matched to the complexity of what you are teaching the model. For a single character or a single specific style, rank 8 to 16 is usually enough. For a complex artistic style with many internal variations, rank 32 to 64 is appropriate. For a domain adaptation that needs to teach the model many new concepts at once (for example, an entire fictional universe with multiple characters, locations, and props), rank 64 to 128 starts to make sense. For Stable Diffusion XL, a typical training run uses rank 32 to 64. For FLUX, which is a more capable base model that needs less help, ranks of 16 to 32 are common. For Stable Diffusion 1.5 in the legacy ecosystem, ranks down to rank 4 or 8 still work for simple subjects.

The alphaAlpha (LoRA alpha)A scaling factor that controls how strongly a LoRA's learned change is applied, effectively its volume knob, tuned separately from rank. is a scaling factor that multiplies the contribution of the LoRALoRA (Low-Rank Adaptation)A fine-tuning technique that adapts a base model to a specific style or concept using a small additional file. The standard way to customize open-source models. at inferenceInferenceRunning a trained model to produce output. The fast, cheap part of the model lifecycle, in contrast to training. time. The math, in slightly simplified form, is that the effective weight update applied to the model is (alpha / rankRank (LoRA rank)The inner dimension of a LoRA's two small matrices, which sets how much capacity it has to learn. Higher rank captures more complex behavior but risks overfitting and produces larger files.) times the product of A and B. So alpha controls the magnitude of the LoRA's influence, while rank controls its dimensionality. The reason alpha is divided by rank is to make the magnitude of the update roughly independent of the rank you chose, so that you can change the rank without having to retune everything else simultaneously.

In language model fine-tuningFine-tuningContinuing to train an existing model on new data to specialize it for a style, subject, or domain, without training from scratch., the conventional wisdom is to set alpha to about 2× the rank, alpha 32 for rank 16, alpha 64 for rank 32. This makes the LoRA's effective contribution moderately strong. In diffusion modelDiffusion modelA generative model that learns to produce images by training a neural network to reverse a noise-adding process. Generates by starting from random noise and iteratively denoising to produce a clean image. fine-tuning, the conventions are different, and the field has not converged on a single rule. Many SDXL training guides recommend alpha equal to half the rank (alpha 16 for rank 32) which produces a gentler LoRA that blends with the base model more subtly. Other guides recommend alpha equal to rank (alpha 32 for rank 32) which produces a stronger effect. FLUX LoRAs tend to be trained with alpha approximately equal to rank, which works because FLUX is a stronger base model that can absorb more aggressive fine-tuning without falling apart.

The practical effect of alpha at inference time is that you can essentially control how strongly any LoRA is applied. In ComfyUIComfyUIThe dominant node-based interface for running diffusion models locally. Started by 'comfyanonymous' in January 2023, now stewarded by Comfy Org. and most other tools, when you load a LoRA you specify a 'strength' or 'weight' parameterParameterOne individual weight. Model size is quoted in parameters, in the millions or billions. More parameters can mean more capability and more cost., typically between 0 and 1, sometimes higher. This multiplies the alpha at inference time. A strength of 0 means the LoRA is effectively off (the original model is used unchanged). A strength of 1 means the LoRA is applied as trained. Strengths of 0.5 to 0.8 are common in practice, they let the LoRA influence the output without overwhelming the base model's capabilities. Strengths above 1 force the LoRA to be applied more strongly than during training, which often produces interesting but sometimes broken results.

If you set the rankRank (LoRA rank)The inner dimension of a LoRA's two small matrices, which sets how much capacity it has to learn. Higher rank captures more complex behavior but risks overfitting and produces larger files. too low, the LoRALoRA (Low-Rank Adaptation)A fine-tuning technique that adapts a base model to a specific style or concept using a small additional file. The standard way to customize open-source models. cannot express the thing you are trying to teach it. The output will look like the base model with a faint suggestion of your training data, the right colors, maybe, but not the right features. The training loss will plateau at a higher value than it should. There is no fix except to retrain at a higher rank.

If you set the rank too high, the LoRA will overfit. It will memorize the specific images in your training set rather than learning the underlying concept. The output will look exactly like one of your training images, regardless of what promptPromptThe text description you provide to a model to specify what you want it to generate. you give it. You can mitigate this with more training data, but the right move is usually to lower the rank.

If you set the alphaAlpha (LoRA alpha)A scaling factor that controls how strongly a LoRA's learned change is applied, effectively its volume knob, tuned separately from rank. too low (or the inferenceInferenceRunning a trained model to produce output. The fast, cheap part of the model lifecycle, in contrast to training. strength too low), the LoRA's effect will be too subtle to see. Your character will have the right hair color but not the right face. Your style will be present in the lighting but not the brushwork. The fix is to increase alpha during training or strength during inference.

If you set the alpha too high (or the inference strength too high), the LoRA will overpower the base model's general capabilities. The output will look like your training data but the prompts will stop working. Asking for 'a person sitting on a park bench at sunset' will produce your character standing in a void, because the LoRA is so strongly applied that it crowds out everything the base model knows about parks, benches, and sunsets. The fix is to lower alpha or strength.