Skip to content
Agent YAP

What is it?

Feed-forward network

Inside each Transformer layer, after attention has gathered context from other words, the feed-forward network processes each word on its own. If attention is "gather information from your neighbors," the feed-forward step is "now sit and think about what you gathered."

It is a small, standard neural network applied separately to each position. It gives the model room to transform and refine the information attention collected. In a Mixture of Experts model (Chapter 5), it is this feed-forward part that gets split into multiple experts.

From the Foundational Modelling glossary

Where Feed-forward network is explained

  1. The Transformer, the Engine Inside Every Modern ModelFoundational Modelling · The rest of the machine
  2. Mixtral and Mixture of Experts, More Brain, Same SpeedFoundational Modelling · How Mixtral is built

Every term, A to Z