CH 07 · Glossary: Foundational Modelling · 01/28
Read as one pageGlossary: Foundational Modelling
Neural networks, tokens, embeddings, self-attention, pretraining, RLHF, DPO, LoRA, and Mixture of Experts are defined here for anyone new to language models.
28 slides · 18 min read
In this chapter
- 01Neural network
- 02Parameters (weights)
- 03Layers and depth
- 04Vector
- 05Token
- 06Tokenization
- 07Embedding
- 08Self-attention
- 09Multi-head attention
- 10Feed-forward network
- 11Positional encoding
- 12Softmax
- 13Encoder and decoder
- 14Transformer
- 15Context window
- 16Training the model
- 17Scale and cost
- 18Supervised fine-tuning (SFT)
- 19Reinforcement learning
- 20RLHF
- 21Reward model
- 22PPO
- 23KL divergence
- 24Preference data
- 25DPO
- 26Efficiency tricks
- 27Measuring quality
This is your shared reference for folder 01. Every term that the chapters link to is explained here from scratch, in plain language. Unlike a normal glossary that gives one-line definitions, this one takes the time to make each idea actually click. You can read it straight through as a primer, or jump in whenever a chapter sends you here.
Terms are grouped by theme so related ideas sit together.
- The basic building blocks: Neural network, Parameters (weights), Layers and depth, Vector, Token, Tokenization, Embedding
- Inside the Transformer: Self-attention, Multi-head attention, Feed-forward network, Positional encoding, Softmax, Encoder and decoder, Transformer, Context window
- Training the model: Pre-training, Fine-tuning, Loss function, Gradient descent and backpropagation, Overfitting, Hyperparameter, Perplexity
- Scale and cost: Compute and FLOPs, Scaling law, Emergent ability, In-context learning, Inference
- Alignment: Supervised fine-tuning (SFT), Reinforcement learning, RLHF, Reward model, PPO, KL divergence, Preference data, DPO
- Efficiency tricks: Matrix rank (low-rank), LoRA, Mixture of Experts (MoE), Router (gating network), Sparse and dense models
- Measuring quality: Benchmark, Elo rating
Finished this idea?
Your learning progress stays in this browser.