Feed-forward network

Inside each Transformer layer, after attention has gathered context from other words, the feed-forward network processes each word on its own. If attention is "gather information from your neighbors," the feed-forward step is "now sit and think about what you gathered."

It is a small, standard neural network applied separately to each position. It gives the model room to transform and refine the information attention collected. In a Mixture of Experts model (Chapter 5), it is this feed-forward part that gets split into multiple experts.