The Journey from ANNs to Transformers

It's easy to look at a transformer and see it as a single clever idea. It isn't — it's the last step in a decades-long argument about how a neural network should represent sequence. Every architecture in this story was a direct response to a specific failure of the one before it. Understanding that chain of failures is the fastest way to understand why transformers look the way they do.

The starting point: plain feedforward networks

A basic artificial neural network — a multilayer perceptron (MLP) — is a stack of layers, each one a weighted sum of its inputs followed by a nonlinearity. Stack enough of these layers and, in principle, you get a universal function approximator: given enough width and depth, an MLP can approximate any continuous function mapping inputs to outputs.

But that power comes with a quiet assumption baked in: every input is a fixed-size, flat vector, and every dimension of that vector is treated independently of the others' position. Feed an MLP an image, a sentence, or a stock price history, and it has no built-in notion that dimension 5 came "before" dimension 6, or that the input might be shorter or longer next time. It just sees numbers in slots.

Why that breaks for sequential data

Language is exactly the case this assumption fails on. A sentence isn't a fixed-size bag of words — it's a variable-length sequence where order carries most of the meaning ("dog bites man" and "man bites dog" use the same words). A plain feedforward network has two problems here: it has no natural way to accept inputs of different lengths, and even if you pad every sentence to some fixed size, it has no memory — no mechanism that says "what I do with word 10 should depend on what word 3 was." Every input dimension is processed as if it were independent of every other.

What was needed wasn't a bigger MLP. It was a different notion of computation — one where the network could carry information forward as it moved through a sequence.

Recurrent neural networks: giving the network a memory

Recurrent neural networks (RNNs) solve this with a simple but powerful trick: reuse the same weights at every position in the sequence, and carry a hidden state forward from one step to the next. At each time step, the network combines the current input with the hidden state it inherited from the previous step, producing a new hidden state that summarizes everything seen so far.

h = zeros(hidden_size)
for token in sequence:
    # same weights, applied again and again — this is why it's "recurrent"
    h = tanh(W_x @ embed(token) + W_h @ h + b)
    # h is the network's entire memory of everything seen so far

This finally gives a network something like memory, and it handles variable-length input naturally — you just run the loop for as many steps as there are tokens. For a while, this was the default architecture for anything sequential: language modeling, translation, speech.

But that same loop is also the source of a deep problem. Training an RNN means backpropagating the error signal backward through time — through every step of the sequence, using the same weight matrix over and over. If that matrix tends to shrink values slightly, the gradient shrinks geometrically at every step it passes through, and by the time it reaches token 1 of a 100-token sequence it has vanished to effectively zero. If the matrix tends to grow values, the gradient explodes instead. Either way, the network struggles to learn dependencies that span more than a handful of steps — exactly the "it" in a sentence that refers back to a noun three clauses earlier.

LSTMs and GRUs: gating what gets remembered

Long Short-Term Memory networks (LSTMs), and later the simpler Gated Recurrent Units (GRUs), tackled the vanishing-gradient problem head-on by adding explicit gates — small learned sub-networks that decide, at every step, what to keep and what to throw away. An LSTM cell maintains a separate memory cell alongside its hidden state, and uses a forget gate to decide what old information to discard, an input gate to decide what new information to write in, and an output gate to decide what to expose to the rest of the network. Because the memory cell is updated additively rather than by repeated multiplication, gradients have a much more direct path back through time, and the network can, in principle, preserve a signal across hundreds of steps instead of a handful.

This was a genuine breakthrough — LSTMs powered most of the serious advances in machine translation and speech recognition through the 2010s. But two limitations remained, and they turned out to matter more than they seemed to at the time. First, gating helps a great deal but doesn't fully solve long-range dependency learning; very long sequences still degrade. Second, and more consequentially, an LSTM is still fundamentally sequential: step N's computation depends on step N-1's hidden state, full stop. You cannot compute step 50 before step 49 finishes. That makes training slow and hard to parallelize, no matter how much GPU compute you throw at it — the hardware sits partially idle waiting for one step to feed the next.

Attention: stop compressing everything into one vector

The next idea attacked a different assumption: that all of a sequence's relevant history has to be squeezed into one fixed-size hidden state. Attention, introduced around 2014-2015 in the context of neural machine translation, let a decoder look back directly at every encoder hidden state from the input sequence — not just the last one — and learn, for each output it was generating, a weighted combination of which input positions were relevant right now. Crucially, this was still bolted onto an RNN: the encoder and decoder were still recurrent, still sequential, but attention gave the decoder a way to route around the single-hidden-state bottleneck instead of relying on it for everything.

This immediately improved translation quality, especially on long sentences, because the model no longer had to compress an entire source sentence into one vector before it could start generating a translation. But it also raised an obvious question: if attention is doing the heavy lifting of connecting relevant positions across the sequence, how much is the recurrence even still contributing?

The Transformer: keep the attention, drop the recurrence

The 2017 "Attention Is All You Need" paper answered that question by removing recurrence entirely. A transformer has no hidden state carried step by step — every token attends directly to every other token in a single operation, computed for the whole sequence at once. That single change solves the RNN/LSTM family's deepest structural problem: because there's no step N depending on step N-1, every position in a sequence can be processed in parallel on a GPU during training.

Removing recurrence has a side effect, though: a pure attention mechanism has no inherent sense of order — swapping two tokens doesn't change how attention scores are computed between them. RNNs got ordering for free, as a byproduct of processing tokens one at a time; transformers have to add it back explicitly, via positional encodings injected into each token's embedding before attention ever runs.

Where this leaves the rest of the story

This progression — MLP, RNN, LSTM/GRU, attention-augmented RNN, transformer — is really one continuous argument about how to give a network access to the past without paying for it with either untrainable gradients or untrainable training time. Each fix solved the previous architecture's most urgent limitation while introducing a smaller one that the next architecture would go on to solve. From here, the natural next questions are how positional encoding actually represents order inside a vector, how self-attention differs from the cross-attention used between an encoder and a decoder, and what it actually takes to train a model this size in practice — each easily worth a deep dive of its own.