LoRA (Low-Rank Adaptation) is a technique for fine-tuning that trains a small number of additional parameters instead of updating the entire model's weights — dramatically reducing the compute, memory, and storage cost of fine-tuning.
The Core Problem LoRA Solves
Full fine-tuning:
Update ALL of a model's weights (potentially billions of
parameters) → requires substantial compute/memory, and
produces an entirely new full-size copy of the model per
fine-tuning job.
LoRA:
Freeze the original model's weights entirely. Add small,
separate trainable "adapter" matrices alongside the original
layers. Only these small adapters are trained.
→ Far less compute/memory needed, and the resulting adapter
is tiny compared to the full model.The Intuition, Without Heavy Math
LoRA is based on the observation that the actual change needed to adapt a model to a new task can often be represented by a much smaller, lower-dimensional set of numbers than the model's full weight matrices — rather than modifying the huge original weight matrices directly, LoRA learns a small "delta" that gets combined with the frozen original weights at inference time.
Practical Benefits
| Benefit | Why It Matters |
|---|---|
| Much lower training cost | Fewer parameters to update means less compute and memory required |
| Tiny adapter file size | An adapter might be a small fraction of the full model's size — easy to store and distribute |
| Swap adapters without re-loading the whole model | Multiple LoRA adapters (for different tasks/customers) can share one frozen base model |
| Lower risk of "catastrophic forgetting" | Since the original weights are frozen, the base model's general capabilities are less likely to degrade |
Practical Use Case
A company wanting to fine-tune the same base model differently for several different customers/use cases can train a separate small LoRA adapter per use case, rather than maintaining several full-size fine-tuned model copies — a meaningful storage and operational cost saving at scale.
Common Mistakes
- Assuming LoRA always matches full fine-tuning's quality for every task — for some tasks requiring very substantial behavior change, full fine-tuning can still outperform LoRA; the right choice depends on the specific task and should be evaluated
- Not understanding that LoRA still requires the original frozen base model at inference time — the adapter alone isn't a complete, standalone model
Interview Relevance
"What problem does LoRA solve compared to full fine-tuning?" — dramatically reduced compute/memory/storage cost by training a small set of additional parameters instead of updating the entire model's weights.
Practice Question
Explain why a company serving 50 different customers with slightly different fine-tuned behavior would prefer LoRA adapters over 50 separate fully fine-tuned model copies.
