In June 2017, eight Google researchers published a paper with a confident title: Attention Is All You Need. It introduced the transformer, an architecture for sequence tasks such as translation. Three years later, almost every state-of-the-art language model is a transformer: BERT (2018), GPT-2 (2019), T5 (2019) and GPT-3 (2020). Last month, the Vision Transformer paper showed the same architecture competing with convolutional networks on image recognition.
This article explains how transformers work, without the heavy maths.
What came before
Before transformers, the standard approach to text was the recurrent neural network (RNN), and its improved variants LSTM and GRU. An RNN reads a sentence one word at a time, updating a hidden state as it goes.
That design has two problems:
- It is sequential. Word 50 cannot be processed until words 1 to 49 have been. This makes training slow and hard to parallelise on modern hardware.
- Long-range dependencies are hard. Information from early in a long passage has to survive many update steps to influence later words, and it tends to fade.
Attention mechanisms were already being added to RNNs to help with the second problem. The transformer's insight was to drop recurrence entirely and use attention for everything.
Self-attention in one paragraph
For each token in a sequence, self-attention asks: which other tokens are relevant to understanding this one, and how much? In "The animal didn't cross the street because it was too tired", attention lets the representation of "it" draw heavily on "animal".
Mechanically, each token's vector is projected into three vectors: a query, a key and a value. The attention weight between two tokens is the similarity of one's query to the other's key. Each token's new representation is a weighted sum of all the values.
import numpy as np
def scaled_dot_product_attention(Q, K, V):
"""Q, K, V: arrays of shape (sequence_length, d_k)."""
d_k = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d_k) # similarity of every query to every key
weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
weights /= weights.sum(axis=-1, keepdims=True) # softmax: each row sums to 1
return weights @ V # weighted mix of valuesBecause every token attends to every other token in one step, distance no longer matters, and the whole computation is a few matrix multiplications that GPUs and TPUs are very good at.
The rest of the block
A transformer layer ("block") combines several pieces:
flowchart TB In[Token embeddings + positional encoding] --> MHA[Multi-head self-attention] MHA --> AN1[Add & layer norm] In --> AN1 AN1 --> FF[Feed-forward network] FF --> AN2[Add & layer norm] AN1 --> AN2 AN2 --> Next[Next block ×N]
- Multi-head attention. Several attention operations run in parallel, each with its own projections, so different heads can focus on different relationships (syntax, coreference and so on).
- Feed-forward network. A small neural network applied to each token independently, adding capacity.
- Residual connections and layer normalisation keep training stable when many blocks are stacked.
- Positional encoding. Attention itself ignores word order, so information about each token's position is added to its embedding.
Stack this block 12, 24 or (in GPT-3's case) 96 times, and you have a transformer.
Three families
The original transformer had an encoder (to read the input) and a decoder (to generate the output). Later models often use just one half:
| Family | Examples | How it attends | Good at |
|---|---|---|---|
| Encoder-only | BERT, RoBERTa | Every token sees the whole input | Classification, search, extraction |
| Decoder-only | GPT-2, GPT-3 | Each token sees only earlier tokens | Text generation, few-shot prompting |
| Encoder-decoder | Original transformer, T5, BART | Encoder reads, decoder generates | Translation, summarisation |
BERT is trained by masking words and predicting them from both sides, which makes it excellent at understanding text. GPT models are trained to predict the next word, which makes them natural generators.
Why transformers won
- Parallelism: training on huge datasets became feasible.
- Scalability: performance keeps improving as models, data and compute grow, as GPT-3 dramatically demonstrated.
- Transfer learning: pre-train once on vast text, then fine-tune or prompt for many tasks.
- Generality: the same architecture works across languages, tasks and now modalities.
Limitations
- Quadratic cost. Every token attends to every other token, so cost grows with the square of the sequence length. Long documents are expensive, and much current research ("efficient transformers") tries to reduce this.
- Data and compute hunger. The largest models need resources few organisations have.
- Interpretability. Attention weights are sometimes treated as explanations, but what a model has actually learned remains hard to inspect.
The bottom line
Transformers are the engine of the current wave of progress in natural language processing, and increasingly beyond it. Understanding self-attention and the encoder/decoder distinction goes a long way towards understanding why models like BERT and GPT-3 behave as they do.
Our agentic AI security team works with transformer-based models every day and helps organisations choose the right model family for their problem.