Positional Encoding: Why Sine and Cosine?

"The cat sat on the mat" and "the mat sat on the cat" contain exactly the same five tokens. To a recurrent network, the difference is obvious — it reads one word at a time, in order, and its hidden state naturally accumulates "what came before what." To a transformer's self-attention layer, which looks at all tokens simultaneously and computes similarity scores between every pair of them, the two sentences are indistinguishable. Attention is fundamentally a function over an unordered set of vectors — swap two tokens' positions and the set of pairwise dot products is identical.

That was the price of dropping recurrence for parallelism: the transformer gained speed and long-range gradient flow, but lost the one thing recurrence gave away for free — an inherent sense of sequence order. Positional encoding is the fix the original "Attention Is All You Need" paper proposed, and the sine/cosine scheme it introduced is still one of the cleanest pieces of engineering in the whole architecture.

Injecting position into the embedding

The idea itself is simple: for every position in the sequence, compute a fixed vector of the same dimensionality as the token embeddings, and add it to that token's embedding before the first attention layer ever sees it. The word "bank" at position 3 and the word "bank" at position 30 start out as the identical embedding vector, but after this addition they become two different vectors — close in meaning, but distinguishable by where they sit in the sequence. Every downstream layer inherits that distinction for free, without needing any recurrence or convolution to propagate it.

The hard part is designing that vector. It needs to give every position in a sequence — potentially thousands of tokens long — its own distinct signature, it needs to behave consistently whether the sequence is 10 tokens or 10,000, and ideally it should let the model reason about relative distance ("this token is three positions away from that one"), not just absolute position index. A raw integer position value satisfies none of these well on its own.

The formula

For a model dimension d_model and position pos, the encoding at dimension index i is:

• PE(pos, 2i) = sin(pos / 10000^(2i / d_model)) — even dimensions
• PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model)) — odd dimensions

Each pair of dimensions oscillates at its own frequency, controlled by the exponent 2i / d_model. Early dimensions (small i) divide the position by a number close to 1, so they complete a full sine/cosine cycle almost every token — a fast, high-frequency wiggle. Late dimensions divide by numbers approaching 10,000, so they barely move across a short sequence — a slow, low-frequency drift. Stack all of those frequencies together across the embedding and every position ends up with a unique combination of values, the same way a specific moment in time gets a unique combination of hour, minute, and second hands.

Working it out for a toy d_model = 8 makes the pattern concrete — four sine/cosine pairs, each pair oscillating slower than the last:

import math

def positional_encoding(pos: int, d_model: int) -> list[float]:
    pe = [0.0] * d_model
    for i in range(0, d_model, 2):
        denom = 10000 ** (i / d_model)
        pe[i] = math.sin(pos / denom)
        pe[i + 1] = math.cos(pos / denom)
    return pe

d_model = 8
for pos in range(4):
    values = [round(v, 4) for v in positional_encoding(pos, d_model)]
    print(f"pos={pos}: {values}")

# pos=0: [0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0]
# pos=1: [0.8415, 0.5403, 0.0998, 0.995, 0.01, 1.0, 0.001, 1.0]
# pos=2: [0.9093, -0.4161, 0.1987, 0.9801, 0.02, 0.9998, 0.002, 1.0]
# pos=3: [0.1411, -0.99, 0.2955, 0.9553, 0.03, 0.9996, 0.003, 1.0]

Notice position 0 is always [0, 1, 0, 1, 0, 1, 0, 1] — sin(0) is always 0, cos(0) is always 1, regardless of frequency. From there, the fastest-moving dimensions (0 and 1) swing dramatically between positions 1, 2, and 3, while the slowest-moving dimensions (6 and 7) barely change at all across the same three steps. That mix of fast and slow signals is what makes each position's full 8-dimensional vector unique.

Why sine and cosine, specifically

It would have been far simpler to just add the raw position index — 0, 1, 2, 3, and so on — directly to the embedding. That approach fails for a few concrete reasons the sinusoidal scheme was built to avoid.

First, boundedness. Sine and cosine only ever output values between -1 and 1, no matter how large the position gets. A raw position index for token 10,000 in a long document would dwarf a typical embedding value, distorting everything it's added to. The sinusoidal encoding stays the same scale whether the sequence has 10 tokens or 10,000.

Second, a unique fingerprint per position. Because each dimension pair oscillates at a different frequency, the combination of values across all dimensions is (for practical sequence lengths) distinct for every position — and it varies smoothly, so nearby positions get similar-looking encodings rather than arbitrary, unrelated ones.

Third, and most subtly, relative position becomes a linear function. Trigonometric angle-addition identities mean that sin(pos + k) and cos(pos + k) can each be written as a linear combination of sin(pos), cos(pos), sin(k), and cos(k). Concretely, for any fixed offset k, there exists a fixed linear transformation — independent of pos — that maps PE(pos) to PE(pos + k).

Not the only answer

Sinusoidal positional encoding was the original transformer's solution, but it isn't the only one, and later architectures made different trade-offs on the same underlying problem. Some models replace it with learned positional embeddings — a plain trainable lookup table indexed by position, letting the model fit whatever positional pattern the data actually rewards instead of a fixed mathematical one. Others inject position directly into the attention computation itself rather than into the input embedding, through relative positional encodings or schemes like rotary positional embeddings (RoPE), which rotate query and key vectors by an angle proportional to position. Each approach is chasing the same goal — giving an inherently order-blind mechanism some notion of sequence — and the fact that so many different solutions have proven workable is a good sign the original sine/cosine trick was never the only way to do it, just an unusually elegant first one.