Skip to content

Chapter 04 · 4 min read · Updated

LoRA, Fine-Tuning Giant Models on a Budget

LoRA freezes a pretrained model and trains only a small pair of low-rank matrices, cutting fine-tuning cost while letting one base model carry many adapters.

Prefer one idea per screen? Read it as 7 slides →

Paper: LoRA: Low-Rank Adaptation of Large Language Models (2021)

By now you can build a model (Chapter 1), scale it (Chapter 2), and align it (Chapter 3). But there is a practical problem. Suppose you have a giant model and you want to teach it a new specialty: your company's writing style, legal documents, medical Q&A. The obvious approach is fine-tuning, which means continuing to train the model on your new data. The trouble is that fine-tuning a model with billions of parameters is brutally expensive. LoRA is the trick that makes it cheap and practical.

04.1

Why normal fine-tuning hurts

When you fine-tune a model the usual way, you adjust all of its parameters. For a model with billions of them, this causes three pains:

  1. Memory. Training needs several times more memory than just running the model, because it has to track how to adjust every single parameter. This can require many expensive GPUs.
  2. Storage. Every fine-tuned copy is a full-size model. If you want ten specialized versions, you store ten enormous files.
  3. Cost and time. Updating billions of dials is slow and burns a lot of money.

For most people and companies, this is simply out of reach. LoRA changes that.

04.2

The key insight: big changes can hide in small matrices

Here is the idea that makes LoRA work. When you fine-tune a model for a new task, the change you make to its weights turns out to be surprisingly simple. Even though the weights are a huge grid of numbers, the adjustment needed to specialize them can be captured by something much smaller. In technical terms, the update has a low rank.

Let us unpack rank with a picture. A model's weights live in big grids of numbers called matrices. A large matrix might be 1000 by 1000, which is one million numbers. But some large matrices can be reconstructed by multiplying two skinny ones together. A 1000 by 1000 matrix can be approximated by a 1000 by 8 matrix times an 8 by 1000 matrix. Count the numbers: that is 8000 plus 8000, which is 16000 numbers instead of one million. Almost the same information, a tiny fraction of the storage.

flowchart LR
    Big["One big update grid<br/>1000 x 1000 = 1,000,000 numbers"]
    Small["Two skinny grids<br/>1000 x 8 and 8 x 1000<br/>= 16,000 numbers"]
    Big -. can be approximated by .-> Small

LoRA bets that the adjustment a model needs in order to learn a new task fits comfortably into those two skinny grids.

04.3

How LoRA works

Instead of editing the original weights, LoRA does this:

  1. Freeze the original model completely. Not a single original parameter changes.
  2. Add a small pair of skinny matrices (the low-rank adapter) alongside the layers you want to adapt.
  3. Train only those small matrices on your new data. They learn the adjustment.
flowchart TD
    Frozen[Original giant model<br/>FROZEN, never changes] --> Combine
    Adapter[Tiny LoRA adapter<br/>the only thing that trains] --> Combine
    Combine[Add the adapter's adjustment<br/>on top of the frozen model] --> Output[Model specialized<br/>for your task]

Because you are training only the skinny matrices, the number of trainable parameters can drop by a factor of thousands. Memory needs plummet, training is fast, and the resulting adapter file is small, often only a few megabytes instead of many gigabytes.

Low-rank adaptation
Low-rank adaptation

Field check

See it worked. Then try it.

Distinguish the frozen base weights from the small trainable update.

Worked example

For a 100 × 100 matrix, choose rank 4.

A 100 × 4 matrix and a 4 × 100 matrix have 800 parameters together, compared with 10,000 in the dense matrix. Their product has the original shape.

Your turnFor a 200 × 200 matrix and rank 2, how many parameters are in the two LoRA matrices together?

04.4

Why this is such a big deal in practice

One base model, many tiny adapters

Since the original model stays frozen and untouched, you can train a separate small adapter for each task and keep them all. Want the model to switch from legal writing to casual chat? Swap the adapter, not the whole model.

flowchart LR
    Base[One frozen base model] --> A[+ Legal adapter]
    Base --> B[+ Medical adapter]
    Base --> C[+ Customer-support adapter]
    A --> O1[Legal assistant]
    B --> O2[Medical assistant]
    C --> O3[Support assistant]

This is wonderfully efficient. Instead of storing ten full models, you store one base model plus ten tiny adapters.

No slowdown when running

A natural worry: does adding an adapter make the model slower to use? No. After training, the adapter's adjustment can be merged back into the original weights, producing a model that runs at exactly the original speed. You get the customization for free at inference time.

A simple analogy

Think of the giant pre-trained model as a printed textbook that is expensive to reprint. Normal fine-tuning rewrites the entire book for each new course. LoRA instead clips a thin set of sticky notes onto the relevant pages. The book underneath never changes, the sticky notes are cheap to make, and you can keep different sets of notes for different courses and swap them in seconds.

04.5

Why it spread everywhere

LoRA democratized fine-tuning. Suddenly a hobbyist with a single GPU, not just a big lab, could specialize a large model. It is now one of the most widely used techniques in the field, and it underpins the huge ecosystem of community-made model variants. Whenever you hear about someone "fine-tuning" an open model at home, they are almost always using LoRA or a close relative of it.

04.6

The one-sentence takeaway

LoRA fine-tunes a giant model cheaply by freezing the original and training only a tiny pair of low-rank matrices that capture the small adjustment a new task requires, which slashes memory and storage, lets one base model wear many swappable adapters, and adds no slowdown when the model runs.

Next: Chapter 5, Mixtral and Mixture of Experts, where models get a bigger brain without getting slower.