Autoregressive vs. Non-Autoregressive Generation

Ask an LLM a question and watch the tokens stream back one after another, and you're watching autoregressive generation in action. The name sounds intimidating, but the idea is simple: the model regresses on its own prior outputs — each new token is produced by conditioning on every token that came before it, including the ones the model itself just generated a moment ago. That feedback loop, feeding the output back in as part of the input, is the entire mechanism behind "GPT-style" decoding.

One token, then the next, then the next

Concretely, autoregressive decoding is a loop. The model looks at the full sequence so far — prompt plus whatever it has generated in this response — and produces a probability distribution over the next token. It samples (or picks the most likely) token, appends it to the sequence, and repeats. The token generated at step 5 was chosen with full knowledge of tokens 1 through 4, including token 4, which the model itself wrote one step earlier.

def generate(prompt_tokens: list[int], max_new_tokens: int) -> list[int]:
    tokens = prompt_tokens[:]

    for _ in range(max_new_tokens):
        # Every previous token — prompt AND everything generated
        # so far — is fed back in as context for the next step.
        logits = model(tokens)
        next_token = sample(logits[-1])
        tokens.append(next_token)

        if next_token == EOS:
            break

    return tokens

That dependency on "everything generated so far" is exactly what the word autoregressive is naming, and it's the reason a language model can stay coherent across a long paragraph: every new word is chosen with the entire preceding sentence already fixed in place.

Why this became the dominant approach

Autoregressive decoding isn't just a convenient engineering choice — it matches how language actually unfolds. The most natural next word in a sentence genuinely depends on every word before it: "the trophy didn't fit in the suitcase because it was too small" only resolves what "it" refers to once you've read the whole clause. A model that predicts strictly left-to-right, conditioned on everything so far, is modeling that dependency directly rather than working around it.

It also makes training remarkably simple. Rather than requiring the model to generate a token, wait, and then learn from whether that generated token was any good, training uses a shortcut called teacher forcing: at every position, the model is shown the real, correct previous tokens from the training text — not its own (possibly wrong) predictions — and is simply asked to predict the next one. That turns generation, which is sequential and slow, into a single forward pass over an entire training sequence with every position's loss computed in parallel. Training is parallel even though inference isn't.

The cost hiding in that loop

The teacher-forcing trick that makes training fast doesn't carry over to inference. At generation time there is no ground-truth "next token" to peek at — the model has to actually produce token N before it can even begin computing token N+1, because token N+1's prediction depends on token N being part of the input. There is no way to skip ahead.

The parallel alternative

Non-autoregressive generation takes a different approach: predict some or all of the output tokens at once, in parallel, without any one token waiting on another. Instead of 500 sequential steps, you might do it in a small, fixed number of steps regardless of output length — a huge win for latency, since the expensive part of inference is no longer proportional to how much text you're generating.

Historically, that speed came at a real quality cost. When positions are predicted independently, none of them can see what the others are choosing. Two positions that both need to reference the same earlier noun might independently pick inconsistent phrasing; two adjacent positions might both hedge toward the same "safe" common word and produce an oddly repetitive phrase; a fact stated in one clause might not match a fact restated in another, because nothing forced the second prediction to check its work against the first.

Why the gap exists, not just that it does

The underlying reason is that language carries strong sequential dependencies: subject-verb agreement, pronouns and other referring expressions that resolve against something said earlier, and factual consistency that has to hold across an entire sentence or paragraph. Autoregressive decoding enforces that consistency automatically, almost as a side effect — by the time the model predicts a later token, every earlier token is already fixed and visible, so agreement and consistency are trivially available to condition on. A model predicting positions independently and in parallel has no equivalent mechanism. It can approximate consistency by conditioning on shared context, but it can't observe theother predictions it is making at the same instant, which is precisely what autoregressive decoding gives for free.

Why the search continues anyway

Given that trade-off, it might seem like autoregressive decoding should simply win by default — and for general-purpose language generation, it largely has. But the latency cost of pure autoregressive decoding is significant enough at production LLM scale that non-autoregressive and semi-autoregressive approaches — generating chunks of tokens together, then refining them, or loosening strict left-to-right ordering in more limited ways — remain an active area of work. None of this makes token-by-token generation obsolete; it's better understood as chipping away at one specific, well-understood cost of an approach that otherwise maps very naturally onto how language actually works.