Attention: Self-Attention vs. Cross-Attention

Every explanation of transformers eventually arrives at the same three letters: Q, K, V. Query, Key, Value. It's a genuinely simple idea dressed up in unfamiliar notation, and once it clicks, the difference between "self-attention" and "cross-attention" stops looking like two mechanisms and starts looking like one mechanism used two different ways — the only thing that changes is where the inputs come from.

The shared mechanism

Strip away the terminology and attention is just a weighted average. You have a set of value vectors, and you want to combine them into one output vector, but not with equal weight — some values matter more than others for the thing you're currently trying to compute. To decide the weights, you compare a query (what you're looking for right now) against a set of keys (a label attached to each value, describing what that value has to offer). The comparison is a dot product — a similarity score — which gets scaled down and pushed through a softmax so the weights are non-negative and sum to one. That's the entire mechanism. Everything else is bookkeeping.

What makes self-attention and cross-attention feel like different tools isn't the math — it's a single design choice: does the query come from the same sequence as the keys and values, or from somewhere else?

Self-attention: looking within one sequence

In self-attention, Q, K, and V are all linear projections of the same sequence. Every token produces its own query ("given what I mean, what am I looking for elsewhere in this sentence?"), and every token also produces a key and a value ("here's what I represent, and here's the content I'll contribute if someone finds me relevant"). Each token then attends over every token in the sequence, including itself, and comes away with an updated representation that blends in whatever context turned out to matter.

This is exactly what lets a transformer resolve a sentence like "The trophy didn't fit in the suitcase because it was too big." The token "it" starts out meaning almost nothing on its own — a pronoun is just a placeholder. But its query vector, compared against the keys for "trophy" and "suitcase," ends up assigning most of the attention weight to "trophy," because that's the token whose key best matches what "it" is looking for in this context. The output for "it" is now a vector that has effectively absorbed "trophy"'s content — the ambiguity is resolved entirely inside one sequence, using nothing but the sequence's own tokens as Q, K, and V.

Cross-attention: looking from one sequence into another

Cross-attention keeps the identical formula but breaks the single source: the query comes from one sequence, while the keys and values come from a different sequence entirely. The canonical example is the original encoder-decoder Transformer built for machine translation. The encoder reads the full source sentence and produces a set of key/value pairs — one per source token, computed once and reused throughout decoding. The decoder, generating the target sentence one token at a time, produces a query at each step that effectively asks: "given everything I've generated so far, which parts of the source sentence are relevant right now?"

Translating "the black cat" into French, when the decoder is about to emit "noir" (black), its query at that step attends most strongly to the key for "black" in the encoded source sentence — not to "cat" or "the". The value it pulls back is the encoder's representation of "black," carried across into a completely different sequence and vocabulary. Self-attention could never do this on its own, because self-attention never leaves the sequence it started in — cross-attention is specifically the mechanism for letting one sequence borrow information computed over another.

The same code, two different wirings

The snippet below is deliberately the same attention() function called twice. The only thing that differs between the two calls is which tensors get fed in as Q, K, and V.

def attention(Q, K, V):
    # Q: queries   -> "what am I looking for?"
    # K: keys      -> "what do I have to offer?"
    # V: values    -> "the actual content to return"
    d_k = K.shape[-1]

    scores = Q @ K.T / sqrt(d_k)   # similarity of each query to each key
    weights = softmax(scores, axis=-1)
    return weights @ V             # weighted average of the values

# Self-attention: Q, K, V all come from the SAME sequence
X = encode_tokens(sentence)
self_out = attention(Q=X @ Wq, K=X @ Wk, V=X @ Wv)

# Cross-attention: Q comes from one sequence, K/V from a DIFFERENT one
D = decoder_hidden_states          # target sequence so far
E = encoder_output                 # source sequence
cross_out = attention(Q=D @ Wq, K=E @ Wk, V=E @ Wv)

In the self-attention call, Q, K, and V are all projections of the same tensor X — one sequence, three roles. In the cross-attention call, Q is a projection of the decoder's own hidden states D, but K and V are projections of the encoder's output E — a second, independently produced sequence. Nothing about softmax(QK^T / sqrt(d_k)) V changes; only the source of three tensors does.

Masking, and why decoders can't peek ahead

One detail matters for self-attention specifically: when a decoder attends over its own sequence, it must not be allowed to look at tokens it hasn't generated yet, or training would be trivially cheating — the model could just copy the answer. This is enforced by masking: before the softmax, scores for "future" positions are set to negative infinity, so they collapse to zero weight. Cross-attention doesn't need this, because the encoder's entire source sequence is already fully known and available — there's no future to hide.

Why modern LLMs mostly skip cross-attention

Most of today's large language models — GPT-style, Claude-style decoder-only architectures — don't use an encoder-decoder split at all. There's a single stack of layers, and every layer uses masked self-attention over one continuously growing sequence: the prompt plus whatever the model has generated so far. There's no second sequence to cross-attend into, because there's only ever one sequence in play. When you paste a document into a chat prompt, the model isn't cross-attending to it the way a translation decoder cross-attends to an encoder — it's self-attending, with the document simply concatenated into the same context window as everything else.

That's a deliberate simplification, not a limitation waiting to be fixed. It scales cleanly, it needs only one kind of attention layer, and it turns out to be enough: the same query/key/value machinery that lets a token find its antecedent within a sentence is equally capable of letting a token find the relevant instruction three thousand tokens earlier in the same context. Cross-attention still shows up where two genuinely distinct sequences need to talk to each other — multimodal models attending from text tokens into image patches are a common modern example — but for a single running context, self-attention alone does the job.