The problem they were trying to solve

Language is about relationships between words that can be far apart. Take this sentence:

The trophy did not fit in the suitcase because it was too big.

What does "it" refer to? The trophy. Now change one word:

The trophy did not fit in the suitcase because it was too small.

Now "it" refers to the suitcase. To understand the sentence, the model has to look back across several words and figure out which earlier word each word is connected to.

Before 2017, the popular tool for this was the recurrent neural network, which read a sentence one word at a time, left to right, carrying a running memory. This had two big problems:

1. It was slow, because word number 100 could not be processed until words 1 through 99 were done. No skipping ahead, no doing things in parallel.
2. It had a bad memory. By the time it reached the end of a long paragraph, it had mostly forgotten the beginning.

The Transformer solves both at once.