LoRA, Fine-Tuning Giant Models on a Budget

Paper: LoRA: Low-Rank Adaptation of Large Language Models (2021)

By now you can build a model (Chapter 1), scale it (Chapter 2), and align it (Chapter 3). But there is a practical problem. Suppose you have a giant model and you want to teach it a new specialty: your company's writing style, legal documents, medical Q&A. The obvious approach is fine-tuning, which means continuing to train the model on your new data. The trouble is that fine-tuning a model with billions of parameters is brutally expensive. LoRA is the trick that makes it cheap and practical.