The Transformer architecture, explained

Every major LLM — GPT, Claude, Gemini, Llama — is built on the same architecture introduced in 2017: the Transformer. Its core idea sounds almost too simple to explain a revolution: instead of processing a sentence one word at a time in order, let every word look directly at every other word and decide, for itself, which ones matter.

The problem it replaced

Before transformers, sequence models were recurrent (RNNs, LSTMs): they read a sentence left to right, one token at a time, carrying a hidden state forward as a summary of everything seen so far. That has two problems. First, it's sequential — token 50 can't be processed until token 49 is done, so training can't be parallelized across a sequence. Second, that hidden state is a bottleneck: by the time the model reaches token 50, the detail from token 2 has been compressed and re-compressed dozens of times, and often it's effectively lost.

Self-attention fixes both problems at once: every token connects directly to every other token in a single step, and because there's no step-by-step recurrence, every token can be processed in parallel.

Self-attention: queries, keys, and values

For every token, the model produces three vectors by multiplying its embedding against three learned weight matrices:

• Query (Q) — "what am I looking for?"
• Key (K) — "what do I contain?"
• Value (V) — "what do I actually pass along if someone attends to me?"

A token's query is compared against every other token's key (a dot product) to produce a relevance score, those scores are turned into weights with a softmax, and the output is a weighted blend of every token's value — mostly made up of whichever tokens scored highest.

def attention(Q, K, V, d_k):
    scores = (Q @ K.T) / sqrt(d_k)      # how much each token attends to every other
    weights = softmax(scores, axis=-1)  # normalize into a probability distribution
    return weights @ V                  # weighted blend of value vectors

In the sentence "the animal didn't cross the street because it was too tired," this is exactly the mechanism that lets the model figure out "it" refers to the animal, not the street — the query for "it" simply matches the key for "animal" more strongly.

Multi-head attention

One attention pattern can only capture one kind of relationship at a time. Multi-head attention runs several of these Q/K/V projections in parallel — typically 8 to 96 "heads" depending on model size — so one head can learn to track grammatical structure (subject ↔ verb), another can track coreference ("it" ↔ "animal"), and another can track something with no clean human name at all. Their outputs are concatenated and projected back down to the model's working dimension.

Positional encoding

There's a catch: attention has no built-in sense of order. Swap two words in the input and, mechanically, attention scores are unaffected — every token still just compares itself to every other token. So before anything else happens, a positional encoding is added to each token's embedding, injecting information about where it sits in the sequence.

The full block: encoder and decoder

A transformer stacks these pieces into repeatable blocks. The original architecture had two stacks — an encoder that builds a rich representation of the input, and a decoder that generates the output one token at a time, attending both to what it has generated so far and to the encoder's output. Modern LLMs like GPT are decoder-only: there's no separate encoder pass — the model just keeps predicting the next token, attending back over everything generated so far.

Two details make deep stacks of these blocks trainable at all:

• Residual connections — each sub-layer's output is added back to its input ("Add & Norm"), giving gradients a direct path back through dozens of layers instead of vanishing.
• The feed-forward network — after attention mixes information across tokens, a small two-layer MLP is applied to each token independently, giving the model room to transform what attention just gathered.

Where it goes from here

Everything downstream — a model's vocabulary of tokens, the embeddings it computes, the retrieval systems built on top of it — assumes this architecture underneath. Understanding self-attention is the foundation for the next two pieces of the puzzle: how meaning gets turned into vectors in the first place (embeddings), and how those vectors get used to ground a model in real data (RAG).