What is it?
Transformer
The Transformer is the neural network architecture introduced in Attention Is All You Need (Chapter 1), built around self-attention. It processes all words in parallel and lets any word attend to any other, which made it fast to train and good at long-range understanding.
It is the foundation of essentially every modern large language model. When people say GPT, Claude, Llama, or Gemini, they are referring to very large Transformers trained on huge amounts of text.
From the Foundational Modelling glossary
Where Transformer is explained
- The Transformer, the Engine Inside Every Modern ModelFoundational Modelling · The one big idea: attention
- Deep Research Agents and How They Are TrainedBuilding Coding Agents and Harnesses · Concept explanation
- Start Here: Foundational Modelling, Explained SimplyFoundational Modelling · Chapter map
- Mixtral and Mixture of Experts, More Brain, Same SpeedFoundational Modelling · How Mixtral is built
- Scaling Laws and Chinchilla, How Big Should a Model Be?Foundational Modelling