LoRA, QLoRA, and FSDP: Efficient Fine-Tuning for Busy People
LoRA, QLoRA, and FSDP: Efficient Fine-Tuning for Busy People
When compute is tight, the question is not "can I fine-tune?"
The question is: which method buys the most learning per GB?
Use parameter-efficient fine-tuning when full fine-tuning is too expensive.
LoRA
LoRA freezes the base model and trains small adapter matrices.
| Feature | Description | Benefit |
|---|---|---|
| Base Model Weights | Frozen (untrained) | Preserves pre-trained knowledge |
| LoRA Adapters | Small, trainable weight matrices (Low-Rank) added to Transformer layers | Task-specific learning, minimal parameters |
| Training | Only LoRA adapters updated | Memory & compute efficient |
Why it saves memory:
LoRA replaces one big update with two smaller matrices.
| Matrix | Shape | Parameters |
|---|---|---|
| Original Matrix | [4096, 4096] | 16,777,216 |
| LoRA Matrix A | [4096, r] | 4096 * r |
| LoRA Matrix B | [r, 4096] | r * 4096 |
| LoRA Total (r=8) | 65,536 (0.4% of Original) |
-
Formula: LoRA Parameters ≈
2 * (r * Original Matrix Dimension) -
Higher rank
r: more capacity, more memory. -
Lower rank
r: less capacity, less memory.
Use LoRA when you want cheap adaptation without updating the whole model.
QLoRA
QLoRA quantizes the base model, then adds LoRA adapters.
Quantization stores weights in smaller data types.
Example: a 32-bit weight becomes a 4-bit or 8-bit weight.
You lose precision, but save memory.
| Feature | Description | Benefit |
|---|---|---|
| Base Model Weights | Quantize First (e.g., 4-bit) | Drastically reduced base model memory footprint |
| LoRA Adapters | Add on to quantized base model (same as LoRA) | Task-specific learning, minimal parameters |
Use QLoRA when GPU memory is the hard limit.
FSDP
FSDP splits model weights, gradients, and optimizer state across multiple GPUs.
The workflow:
- Break model state into shards.
- Put different shards on different GPUs.
- Move shards when computation needs them.
| Feature | Description | Benefit |
|---|---|---|
| Model Distribution | Weights, gradients, optimizer states divided into "shards" | Train models too large for 1 GPU memory |
| Multi-GPU Usage | Distribute shards across GPUs | Scalable training, faster with more GPUs |
| Communication Management | FSDP handles inter-GPU communication for parameter access during computation | Keep track of shards |
Use FSDP when one GPU cannot hold the training state.
Transformer Layers
Transformers mainly use attention and feedforward layers.
| Layer Component | Function | Benefit |
|---|---|---|
| 1. Multi-Head Attention | Understands relationships and context between words in a sentence | "Paying Attention" to important parts of input (words) |
| 2. Feedforward Network (FFN) | Processes and "thinks" about the context-aware information (Non-linear) | "Processing & Thinking" based on attention |
A feedforward layer usually has:
- Expansion: increase dimensionality.
- Activation: add non-linearity.
- Projection: reduce dimensionality back.
Bottom line:
Use LoRA for cheap adaptation.
Use QLoRA when memory is tight.
Use FSDP when the model is too large for one GPU.