Tokenization & Normalization: Preparing Text for a Model

A neural network only understands numbers — specifically, indices into a fixed-size table of embeddings. Raw text, on the other hand, is an open-ended stream of characters with no built-in boundaries: no model can have an embedding for "every possible string," because there are infinitely many of them. Before any attention layer, any matrix multiply, any gradient ever touches a sentence, it has to be converted into a finite sequence of discrete units drawn from a fixed vocabulary. That conversion step is tokenization, and the cleanup that happens just before it is normalization. Neither is glamorous, but both decide, silently, how well everything built on top of them will actually work.

The naive approach: one token per word

The most intuitive scheme is to split text on whitespace and punctuation and give every distinct word its own slot in the vocabulary. It's easy to reason about, and it matches how humans think about language. It also breaks down almost immediately at any real scale.

Natural language has an enormous — effectively unbounded — set of distinct word forms once you count inflections ("run," "runs,""running,""ran"), compounds, typos, product names, and rare technical terms. A word-level vocabulary either has to grow huge to cover all of them, which bloats the embedding table and the final output layer (both scale directly with vocabulary size), or it has to cap itself at, say, the 50,000 most common words and map everything else to a single <UNK> token. The second option is what most systems were forced into, and it's quietly destructive: a misspelled product name, a rare medical term, and a word in a language the vocabulary wasn't built for all collapse into the exact same "unknown" signal, and the model loses whatever information distinguished them.

The opposite extreme: one token per character

Swing all the way to the other end and you get character-level tokenization: a vocabulary of maybe a few hundred symbols — letters, digits, punctuation, a handful of Unicode categories — which never needs an unknown token, because any string can be spelled out character by character. The out-of-vocabulary problem disappears entirely.

The cost shows up somewhere else. A sentence that was 10 tokens long at the word level might be 60 tokens long at the character level, and since a transformer's attention cost grows with the square of sequence length, that's a real, expensive tax on every forward and backward pass. Worse, the model now has to learn everything about word structure — that "t," "h," "e" usually combine into a meaningful unit, that "-ing" is a common suffix — completely from scratch, spending model capacity on a problem a smarter tokenizer could have solved for free.

Subword tokenization: splitting the difference

The approach almost every modern language model actually uses sits between those two extremes. Algorithms like Byte-Pair Encoding (BPE), WordPiece, and Unigram all start from individual characters (or bytes) and build the vocabulary upward: scan a large training corpus, find the pair of adjacent units that occurs most frequently, merge it into a single new unit, and repeat — thousands of times — until the vocabulary reaches a target size, typically somewhere in the tens of thousands.

The effect is that common words end up represented as a single token (because their component letters were merged together so often during training that they became one unit), while rare or never-before-seen words gracefully fall back to a handful of smaller, still-familiar pieces instead of a single opaque <UNK>. Nothing is ever truly out of vocabulary — in the worst case, an unfamiliar word just decomposes all the way down to individual characters the vocabulary does contain.

A small worked example makes the merging process concrete. Imagine a vocabulary that has already learned a handful of frequent pairs from training, and watch what happens to a word it has never seen as a whole:

# Illustrative BPE-style merge, not a real tokenizer

word = "tokenization"
tokens = list(word) + ["</w>"]
print(tokens)
# ['t', 'o', 'k', 'e', 'n', 'i', 'z', 'a', 't', 'i', 'o', 'n', '</w>']

# Suppose training on a large corpus already learned these
# frequent adjacent pairs, most-merged first:
learned_merges = ["t o", "i z", "a t", "i o", "iz e", "en t"]

def merge_pair(tokens: list[str], pair: str) -> list[str]:
    a, b = pair.split(" ")
    merged = []
    i = 0
    while i < len(tokens):
        if i < len(tokens) - 1 and tokens[i] == a and tokens[i + 1] == b:
            merged.append(a + b)
            i += 2
        else:
            merged.append(tokens[i])
            i += 1
    return merged

for pair in learned_merges:
    tokens = merge_pair(tokens, pair)

print(tokens)
# ['t', 'o', 'k', 'en', 'iz', 'at', 'io', 'n', '</w>']
# A common word would collapse much further, often to one token;
# an unseen word like this falls back to a handful of familiar pieces.

Notice the process never asks "have I seen this exact word before?" — it only ever asks "have I seen this exact pair before?" That's what lets a tokenizer trained on general text handle a brand-new product name or a technical term it never encountered as a whole word, by falling back to the familiar sub-pieces it was built from.

Normalization comes first

Tokenization assumes it's receiving reasonably clean, consistent text — and getting it there is a separate step, normalization, that runs before any splitting happens. It typically covers: lowercasing (so "Model" and "model" aren't treated as unrelated strings, when the task doesn't need case sensitivity), Unicode normalization (collapsing visually identical but differently-encoded characters — an accented "é" built from one code point versus an "e" plus a combining accent mark — into one canonical form), and consistent handling of whitespace and punctuation.

What makes this step easy to get wrong isn't any single rule — it's consistency between training and inference. If a model was trained on lowercased, whitespace-collapsed text, and the production service that calls it at inference time skips that step, the model sees inputs that don't resemble anything in its training distribution. Nothing crashes. There's no exception, no error log, no obvious signal at all — the model just quietly produces worse embeddings and worse predictions, and the mismatch is often only found much later, if it's found at all.

Why the choice of tokenizer keeps mattering

Vocabulary size is a direct trade-off dial: a larger vocabulary means shorter sequences (more gets said in fewer tokens) but a bigger embedding table and output layer; a smaller vocabulary means a leaner model but longer sequences for the same text. Every model family picks a point on that trade-off, and it becomes a fixed property of the model that can't be changed without retraining.

The choice also has consequences that aren't evenly distributed across languages, because a subword vocabulary is built from a training corpus that is rarely balanced across the world's languages and scripts.