Research Papers
Foundational Modelling
Self-attention, scaling laws, alignment, efficient fine-tuning and mixture-of-experts: the architectural and training results every modern model is built on, each explained from scratch rather than summarised.
8 chapters74 slides43 minUpdated
Your route through the subject
0/74 slides completed · 0/8 chapters · 0 slides read
Start with: Start Here: Foundational Modelling, Explained Simply
The chapter map
Follow the route or choose a chapter.- 010/5 completeStart Here: Foundational Modelling, Explained Simply~2 minReadCode
- 020/7 completeThe Transformer, the Engine Inside Every Modern Model~5 minReadPlateCode
- 030/6 completeScaling Laws and Chinchilla, How Big Should a Model Be?~4 minReadCode
- 040/6 completeAlignment, Turning a Text Predictor Into a Helpful Assistant~5 minReadCode
- 050/7 completeLoRA, Fine-Tuning Giant Models on a Budget~4 minReadField checkPlateCode
- 060/7 completeMixtral and Mixture of Experts, More Brain, Same Speed~3 minReadPlateCode
- 070/8 completeJudging Models, How Do We Measure Quality?~3 minReadCode
- 080/28 completeGlossary: Foundational Modelling~18 minRead
Looking for the reading order? Follow this subject’s paper path to see what each paper builds on.
Chapters
Chapter 0: Start Here: Foundational Modelling, Explained Simply
Six papers map the journey from a model's architecture through scale, alignment, efficiency, and evaluation, in the order that builds understanding.
2 min5 slides: Start Here: Foundational Modelling, Explained Simply0/5 complete
Chapter 1: The Transformer, the Engine Inside Every Modern Model
Attention lets every word in a sentence look directly at every other word and decide what matters, the design behind GPT, Claude, and every modern model.
5 min7 slides: The Transformer, the Engine Inside Every Modern Model0/7 complete
Chapter 2: Scaling Laws and Chinchilla, How Big Should a Model Be?
Scaling laws show model quality improves predictably with more parameters, data, and compute, and Chinchilla found trained models were starved of data.
4 min6 slides: Scaling Laws and Chinchilla, How Big Should a Model Be?0/6 complete
Chapter 3: Alignment, Turning a Text Predictor Into a Helpful Assistant
InstructGPT aligns a raw model through supervised fine-tuning, a reward model, and reinforcement learning, while DPO reaches the same goal in one step.
5 min6 slides: Alignment, Turning a Text Predictor Into a Helpful Assistant0/6 complete
Chapter 4: LoRA, Fine-Tuning Giant Models on a Budget
LoRA freezes a pretrained model and trains only a small pair of low-rank matrices, cutting fine-tuning cost while letting one base model carry many adapters.
4 min7 slides: LoRA, Fine-Tuning Giant Models on a Budget0/7 complete
Chapter 5: Mixtral and Mixture of Experts, More Brain, Same Speed
Mixtral sends each word to two of eight expert subnetworks, so it holds the knowledge of a large model while running at the cost of a much smaller one.
3 min7 slides: Mixtral and Mixture of Experts, More Brain, Same Speed0/7 complete
Chapter 6: Judging Models, How Do We Measure Quality?
Judging open-ended chat quality is hard with no single right answer, and a strong model can substitute for a human judge about eighty percent of the time.
3 min8 slides: Judging Models, How Do We Measure Quality?0/8 complete
Chapter 7: Glossary: Foundational Modelling
Neural networks, tokens, embeddings, self-attention, pretraining, RLHF, DPO, LoRA, and Mixture of Experts are defined here for anyone new to language models.
18 min28 slides: Glossary: Foundational Modelling0/28 complete