What is it?
Parameters (weights)
Parameters, also called weights, are the adjustable numbers inside a neural network. They are the heart of what a model "knows." Think of a giant mixing board with billions of sliders. Each slider controls how strongly one piece of information influences another. Training is the process of setting all those sliders to good positions.
When you hear that a model "has 70 billion parameters," it means it has 70 billion of these tunable numbers. More parameters means more capacity to store patterns and knowledge, but also more memory and more cost to run.
From the Foundational Modelling glossary
Where Parameters is explained
- Scaling Laws and Chinchilla, How Big Should a Model Be?Foundational Modelling · Three knobs you can turn
- Mixtral and Mixture of Experts, More Brain, Same SpeedFoundational Modelling · The tension we are trying to escape
- Building the PromptBuilding Coding Agents and Harnesses · Real code: the tool schema is rewritten per provider
- LoRA, Fine-Tuning Giant Models on a BudgetFoundational Modelling
- Alignment, Turning a Text Predictor Into a Helpful AssistantFoundational Modelling · Paper 1: InstructGPT and the RLHF recipe