CH 04 · LoRA, Fine-Tuning Giant Models on a Budget · 04/07
Read as one pageHow LoRA works
Instead of editing the original weights, LoRA does this:
- Freeze the original model completely. Not a single original parameter changes.
- Add a small pair of skinny matrices (the low-rank adapter) alongside the layers you want to adapt.
- Train only those small matrices on your new data. They learn the adjustment.
flowchart TD
Frozen[Original giant model<br/>FROZEN, never changes] --> Combine
Adapter[Tiny LoRA adapter<br/>the only thing that trains] --> Combine
Combine[Add the adapter's adjustment<br/>on top of the frozen model] --> Output[Model specialized<br/>for your task]
Because you are training only the skinny matrices, the number of trainable parameters can drop by a factor of thousands. Memory needs plummet, training is fast, and the resulting adapter file is small, often only a few megabytes instead of many gigabytes.
Field check
See it worked. Then try it.
Distinguish the frozen base weights from the small trainable update.
Worked example
For a 100 × 100 matrix, choose rank 4.
A 100 × 4 matrix and a 4 × 100 matrix have 800 parameters together, compared with 10,000 in the dense matrix. Their product has the original shape.
Finished this idea?
Your learning progress stays in this browser.