Skip to content
Foundational Modelling chapter route0001020304050607

Research Papers

Foundational Modelling

Self-attention, scaling laws, alignment, efficient fine-tuning and mixture-of-experts: the architectural and training results every modern model is built on, each explained from scratch rather than summarised.

8 chapters74 slides43 minUpdated

Your route through the subject

0/74 slides completed · 0/8 chapters · 0 slides read

Start with: Start Here: Foundational Modelling, Explained Simply

Begin reading

The chapter map

Follow the route or choose a chapter.
  1. 010/5 completeStart Here: Foundational Modelling, Explained Simply~2 minReadCode
  2. 020/7 completeThe Transformer, the Engine Inside Every Modern Model~5 minReadPlateCode
  3. 030/6 completeScaling Laws and Chinchilla, How Big Should a Model Be?~4 minReadCode
  4. 040/6 completeAlignment, Turning a Text Predictor Into a Helpful Assistant~5 minReadCode
  5. 050/7 completeLoRA, Fine-Tuning Giant Models on a Budget~4 minReadField checkPlateCode
  6. 060/7 completeMixtral and Mixture of Experts, More Brain, Same Speed~3 minReadPlateCode
  7. 070/8 completeJudging Models, How Do We Measure Quality?~3 minReadCode
  8. 080/28 completeGlossary: Foundational Modelling~18 minRead

Looking for the reading order? Follow this subject’s paper path to see what each paper builds on.

Chapters

  1. Chapter 0: Start Here: Foundational Modelling, Explained Simply

    Six papers map the journey from a model's architecture through scale, alignment, efficiency, and evaluation, in the order that builds understanding.

    2 min5 slides: Start Here: Foundational Modelling, Explained Simply0/5 complete

  2. Chapter 1: The Transformer, the Engine Inside Every Modern Model

    Attention lets every word in a sentence look directly at every other word and decide what matters, the design behind GPT, Claude, and every modern model.

    5 min7 slides: The Transformer, the Engine Inside Every Modern Model0/7 complete

  3. Chapter 2: Scaling Laws and Chinchilla, How Big Should a Model Be?

    Scaling laws show model quality improves predictably with more parameters, data, and compute, and Chinchilla found trained models were starved of data.

    4 min6 slides: Scaling Laws and Chinchilla, How Big Should a Model Be?0/6 complete

  4. Chapter 3: Alignment, Turning a Text Predictor Into a Helpful Assistant

    InstructGPT aligns a raw model through supervised fine-tuning, a reward model, and reinforcement learning, while DPO reaches the same goal in one step.

    5 min6 slides: Alignment, Turning a Text Predictor Into a Helpful Assistant0/6 complete

  5. Chapter 4: LoRA, Fine-Tuning Giant Models on a Budget

    LoRA freezes a pretrained model and trains only a small pair of low-rank matrices, cutting fine-tuning cost while letting one base model carry many adapters.

    4 min7 slides: LoRA, Fine-Tuning Giant Models on a Budget0/7 complete

  6. Chapter 5: Mixtral and Mixture of Experts, More Brain, Same Speed

    Mixtral sends each word to two of eight expert subnetworks, so it holds the knowledge of a large model while running at the cost of a much smaller one.

    3 min7 slides: Mixtral and Mixture of Experts, More Brain, Same Speed0/7 complete

  7. Chapter 6: Judging Models, How Do We Measure Quality?

    Judging open-ended chat quality is hard with no single right answer, and a strong model can substitute for a human judge about eighty percent of the time.

    3 min8 slides: Judging Models, How Do We Measure Quality?0/8 complete

  8. Chapter 7: Glossary: Foundational Modelling

    Neural networks, tokens, embeddings, self-attention, pretraining, RLHF, DPO, LoRA, and Mixture of Experts are defined here for anyone new to language models.

    18 min28 slides: Glossary: Foundational Modelling0/28 complete