What is it?
LoRA
LoRA stands for Low-Rank Adaptation (Chapter 4). It is a cheap way to fine-tune a giant model. Instead of adjusting all of the model's parameters, LoRA freezes the original model and trains only a tiny pair of skinny low-rank matrices that capture the adjustment a new task needs.
This cuts memory and storage dramatically, lets one frozen base model carry many small swappable adapters for different tasks, and adds no slowdown when the model runs, because the adapter can be merged back in. LoRA is what made fine-tuning accessible to people without giant compute budgets.
From the Foundational Modelling glossary