LoRA: Efficient Fine-Tuning Explained.

The Core Idea

Full fine-tuning updates every parameter in a model. For a 7B parameter model, that means storing 7 billion gradient values in memory during training -- requiring 28GB+ of VRAM just for the gradients, plus the model weights themselves. LoRA (Low-Rank Adaptation) exploits a key insight from the original paper by Hu et al. (2021): **the weight updates during fine-tuning have low intrinsic dimensionality.** In plain language, the changes you need to make to a model's weights to adapt it to your task can be represented by a much smaller matrix. Instead of updating the full weight matrix W (dimensions d x d), LoRA decomposes the update into two smaller matrices: ``` Full update: W' = W + deltaW (d x d parameters) LoRA update: W' = W + B * A (d x r + r x d parameters) Where: W = original frozen weight matrix (e.g., 4096 x 4096) B = low-rank down-projection (4096 x r, e.g., 4096 x 16) A = low-rank up-projection (r x 4096, e.g., 16 x 4096) r = rank (typically 8-64) Parameter savings: Full: 4096 x 4096 = 16,777,216 parameters LoRA (r=16): 4096 x 16 + 16 x 4096 = 131,072 parameters Reduction: 99.2% fewer trainable parameters ``` The original model weights are frozen. Only A and B are trained. This means: - Training requires a fraction of the VRAM - The original model is untouched (no catastrophic forgetting risk) - LoRA adapters are tiny files (10-100MB vs 14GB+ for full model) - Multiple adapters can be swapped at inference time