AI & LLMs

Transformers Explained: The Architecture Behind Modern LLMs

BERT, GPT-3, T5 and now vision models all rest on the same 2017 architecture: the transformer. This explainer walks through how transformers work, from self-attention to positional encoding, why they replaced recurrent networks, and the encoder, decoder and encoder-decoder variants behind today's language models.

By EthicalHat Research · · 3 min read
Infographic: How a transformer processes text

In June 2017, eight Google researchers published a paper with a confident title: Attention Is All You Need. It introduced the transformer, an architecture for sequence tasks such as translation. Three years later, almost every state-of-the-art language model is a transformer: BERT (2018), GPT-2 (2019), T5 (2019) and GPT-3 (2020). Last month, the Vision Transformer paper showed the same architecture competing with convolutional networks on image recognition.

This article explains how transformers work, without the heavy maths.

What came before

Before transformers, the standard approach to text was the recurrent neural network (RNN), and its improved variants LSTM and GRU. An RNN reads a sentence one word at a time, updating a hidden state as it goes.

That design has two problems:

  1. It is sequential. Word 50 cannot be processed until words 1 to 49 have been. This makes training slow and hard to parallelise on modern hardware.
  2. Long-range dependencies are hard. Information from early in a long passage has to survive many update steps to influence later words, and it tends to fade.

Attention mechanisms were already being added to RNNs to help with the second problem. The transformer's insight was to drop recurrence entirely and use attention for everything.

Self-attention in one paragraph

For each token in a sequence, self-attention asks: which other tokens are relevant to understanding this one, and how much? In "The animal didn't cross the street because it was too tired", attention lets the representation of "it" draw heavily on "animal".

Mechanically, each token's vector is projected into three vectors: a query, a key and a value. The attention weight between two tokens is the similarity of one's query to the other's key. Each token's new representation is a weighted sum of all the values.

python
import numpy as np

def scaled_dot_product_attention(Q, K, V):
    """Q, K, V: arrays of shape (sequence_length, d_k)."""
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)                     # similarity of every query to every key
    weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights /= weights.sum(axis=-1, keepdims=True)      # softmax: each row sums to 1
    return weights @ V                                  # weighted mix of values

Because every token attends to every other token in one step, distance no longer matters, and the whole computation is a few matrix multiplications that GPUs and TPUs are very good at.

The rest of the block

A transformer layer ("block") combines several pieces:

flowchart TB
  In[Token embeddings + positional encoding] --> MHA[Multi-head self-attention]
  MHA --> AN1[Add & layer norm]
  In --> AN1
  AN1 --> FF[Feed-forward network]
  FF --> AN2[Add & layer norm]
  AN1 --> AN2
  AN2 --> Next[Next block ×N]
  • Multi-head attention. Several attention operations run in parallel, each with its own projections, so different heads can focus on different relationships (syntax, coreference and so on).
  • Feed-forward network. A small neural network applied to each token independently, adding capacity.
  • Residual connections and layer normalisation keep training stable when many blocks are stacked.
  • Positional encoding. Attention itself ignores word order, so information about each token's position is added to its embedding.

Stack this block 12, 24 or (in GPT-3's case) 96 times, and you have a transformer.

Three families

The original transformer had an encoder (to read the input) and a decoder (to generate the output). Later models often use just one half:

FamilyExamplesHow it attendsGood at
Encoder-onlyBERT, RoBERTaEvery token sees the whole inputClassification, search, extraction
Decoder-onlyGPT-2, GPT-3Each token sees only earlier tokensText generation, few-shot prompting
Encoder-decoderOriginal transformer, T5, BARTEncoder reads, decoder generatesTranslation, summarisation

BERT is trained by masking words and predicting them from both sides, which makes it excellent at understanding text. GPT models are trained to predict the next word, which makes them natural generators.

Why transformers won

  • Parallelism: training on huge datasets became feasible.
  • Scalability: performance keeps improving as models, data and compute grow, as GPT-3 dramatically demonstrated.
  • Transfer learning: pre-train once on vast text, then fine-tune or prompt for many tasks.
  • Generality: the same architecture works across languages, tasks and now modalities.

Limitations

  • Quadratic cost. Every token attends to every other token, so cost grows with the square of the sequence length. Long documents are expensive, and much current research ("efficient transformers") tries to reduce this.
  • Data and compute hunger. The largest models need resources few organisations have.
  • Interpretability. Attention weights are sometimes treated as explanations, but what a model has actually learned remains hard to inspect.

The bottom line

Transformers are the engine of the current wave of progress in natural language processing, and increasingly beyond it. Understanding self-attention and the encoder/decoder distinction goes a long way towards understanding why models like BERT and GPT-3 behave as they do.

Our agentic AI security team works with transformer-based models every day and helps organisations choose the right model family for their problem.

EthicalHat Research

The EthicalHat research team: penetration testers, SOC analysts and engineers writing about security, software engineering and AI.

Want an expert opinion on your own systems?

Email info@ethicalhat.com or call +1-844-838-4422. Our security engineers are happy to talk it through.