Activation Functions: Why Neural Networks Need Them

It's easy to treat activation functions as a minor detail — a small nonlinear stamp applied after every layer, worth a line in a model definition and little more thought. But that stamp is doing something load-bearing. Remove it, and a twenty-layer network stops being a twenty-layer network in any meaningful sense.

Why nonlinearity is the whole point

A single layer of a neural network computes something like y = Wx + b — a linear transformation. Stack a second linear layer on top and you get y = W2(W1x + b1) + b2, which multiplies out to y = (W2W1)x + (W2b1 + b2). That's still just y = W'x + b' for some combined weight matrix and bias — a single linear layer wearing a costume. No matter how many linear layers you stack, the composition of linear functions is always another linear function. A hundred-layer network built entirely from linear layers has exactly the same expressive power as a network with one.

That's the entire reason activation functions exist. By inserting a nonlinear function between layers, each additional layer stops being redundant algebra and starts being a new transformation of the input space — bending, folding, and warping it in ways a straight line never could. Depth without nonlinearity buys you nothing beyond a single hyperplane decision boundary; depth with nonlinearity is what lets a network approximate arbitrarily complex functions, which is the whole premise of deep learning.

Sigmoid: the historical default, and its problem

The sigmoid function squashes any real-valued input into the range (0, 1), which made it an obvious early choice — its output looks like a probability, and its smooth, differentiable S-curve was easy to reason about mathematically. For a long time it was the default activation in the layers of a network, not just the output.

The trouble shows up during training. Sigmoid flattens out sharply for large positive or large negative inputs — the curve is nearly horizontal at both extremes. Since backpropagation multiplies gradients together layer by layer, and the gradient of a flat curve is close to zero, a neuron pushed into either extreme contributes almost nothing to the gradient flowing backward. Stack several sigmoid layers and those near-zero gradients multiply together, shrinking exponentially as they travel back toward the earlier layers. This is the vanishing gradient problem, and it made deep sigmoid networks painfully slow — or effectively impossible — to train.

Tanh: a zero-centered cousin, same underlying flaw

Tanh has the same S-shaped silhouette as sigmoid but stretches the output range to (-1, 1) and is zero-centered. That centering is a genuine, if modest, improvement: sigmoid's outputs are always positive, which biases the gradients flowing into the next layer in a consistent direction and can slow convergence. Tanh's symmetry around zero avoids that bias, and it tended to train a bit better in practice as a result.

But tanh flattens out at its extremes for exactly the same reason sigmoid does — the curve saturates. Deep tanh networks still run into vanishing gradients once activations drift toward -1 or 1, so it never fully solved the problem sigmoid had; it just pushed the pain point back a little.

ReLU: cheap, effective, and its own new failure mode

The Rectified Linear Unit reframes the problem entirely: ReLU(x) = max(0, x). For positive inputs it's the identity function — output equals input, gradient equals exactly 1, no saturation, no shrinking. For negative inputs it's just zero. That's the whole function, and it's almost free to compute compared to an exponential like sigmoid or tanh.

Because the gradient never shrinks for positive activations no matter how many layers a signal passes through, ReLU let networks go far deeper than sigmoid or tanh ever practically allowed. That single property is a big part of why ReLU became the default activation across most of deep learning's growth through the 2010s — it wasn't a clever trick, it was a removal of the thing that was breaking everything else.

ReLU trades one failure mode for another, though. If a neuron's weights end up pushing its input negative for every example in the training set, its output is zero, and — critically — its gradient is also exactly zero for any negative input. No gradient means no weight update, which means that neuron can never recover on its own. It has simply died. This is the dying ReLU problem, and in a large enough network a meaningful fraction of neurons can end up permanently inactive, quietly wasting capacity.

Leaky ReLU and GELU: patching the dead-neuron problem

Leaky ReLU is the most direct fix: instead of clamping negative inputs to exactly zero, it lets a small fraction through — something like 0.01x instead of 0. That tiny nonzero slope means the gradient is never exactly zero, so a neuron that drifts negative still gets a (small) signal to correct course instead of dying permanently. It costs almost nothing extra to compute and directly targets the failure mode ReLU introduced.

GELU (Gaussian Error Linear Unit) takes a different approach: rather than a sharp corner at zero, it weights each input by a smooth, probabilistic curve, so the transition from "mostly off" to "mostly on" is gradual rather than a hard cutoff. That smoothness gives it slightly better gradient behavior near zero than ReLU's hard hinge, and it has become the standard activation inside the feed-forward blocks of modern transformer architectures — not because it's dramatically more powerful in theory, but because it consistently trains a little better in practice at the scale those models operate at.

The practical takeaway

Activation function choice isn't an arbitrary hyperparameter to sweep blindly — it's a decision with a real mechanical reason behind it. Sigmoid and tanh saturate and starve deep networks of gradient; ReLU and its descendants don't, at least not in the same way, which is exactly why the ReLU family and GELU family dominate hidden-layer activations in modern architectures while sigmoid survives mainly at output layers that genuinely need a probability, and tanh shows up mostly in older recurrent designs. The field converged on this default for a concrete, well-understood reason, not by convention alone.