Skip to slide
25 / 74

CH 04 · LoRA, Fine-Tuning Giant Models on a Budget · 01/07

Read as one page

LoRA, Fine-Tuning Giant Models on a Budget

LoRA freezes a pretrained model and trains only a small pair of low-rank matrices, cutting fine-tuning cost while letting one base model carry many adapters.

7 slides · 4 min read

Contour map for LoRA, Fine-Tuning Giant Models on a Budget

In this chapter

  1. 01Why normal fine-tuning hurts
  2. 02The key insight: big changes can hide in small matrices
  3. 03How LoRA works
  4. 04Why this is such a big deal in practice
  5. 05Why it spread everywhere
  6. 06The one-sentence takeaway

Paper: LoRA: Low-Rank Adaptation of Large Language Models (2021)

By now you can build a model (Chapter 1), scale it (Chapter 2), and align it (Chapter 3). But there is a practical problem. Suppose you have a giant model and you want to teach it a new specialty: your company's writing style, legal documents, medical Q&A. The obvious approach is fine-tuning, which means continuing to train the model on your new data. The trouble is that fine-tuning a model with billions of parameters is brutally expensive. LoRA is the trick that makes it cheap and practical.

Finished this idea?

Your learning progress stays in this browser.