Skip to slide
28 / 74

CH 04 · LoRA, Fine-Tuning Giant Models on a Budget · 04/07

Read as one page

How LoRA works

Instead of editing the original weights, LoRA does this:

  1. Freeze the original model completely. Not a single original parameter changes.
  2. Add a small pair of skinny matrices (the low-rank adapter) alongside the layers you want to adapt.
  3. Train only those small matrices on your new data. They learn the adjustment.
flowchart TD
    Frozen[Original giant model<br/>FROZEN, never changes] --> Combine
    Adapter[Tiny LoRA adapter<br/>the only thing that trains] --> Combine
    Combine[Add the adapter's adjustment<br/>on top of the frozen model] --> Output[Model specialized<br/>for your task]

Because you are training only the skinny matrices, the number of trainable parameters can drop by a factor of thousands. Memory needs plummet, training is fast, and the resulting adapter file is small, often only a few megabytes instead of many gigabytes.

Low-rank adaptation
Low-rank adaptation

Field check

See it worked. Then try it.

Distinguish the frozen base weights from the small trainable update.

Worked example

For a 100 × 100 matrix, choose rank 4.

A 100 × 4 matrix and a 4 × 100 matrix have 800 parameters together, compared with 10,000 in the dense matrix. Their product has the original shape.

Your turnFor a 200 × 200 matrix and rank 2, how many parameters are in the two LoRA matrices together?

Finished this idea?

Your learning progress stays in this browser.