Skip to content
Agent YAP

What is it?

LoRA

LoRA stands for Low-Rank Adaptation (Chapter 4). It is a cheap way to fine-tune a giant model. Instead of adjusting all of the model's parameters, LoRA freezes the original model and trains only a tiny pair of skinny low-rank matrices that capture the adjustment a new task needs.

This cuts memory and storage dramatically, lets one frozen base model carry many small swappable adapters for different tasks, and adds no slowdown when the model runs, because the adapter can be merged back in. LoRA is what made fine-tuning accessible to people without giant compute budgets.

From the Foundational Modelling glossary

Where LoRA is explained

  1. LoRA, Fine-Tuning Giant Models on a BudgetFoundational Modelling
  2. Start Here: Foundational Modelling, Explained SimplyFoundational Modelling · The big picture
  3. Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · The one-sentence takeaway

Every term, A to Z