The ReLU function explained

A neural network is a stack of linear maps, and a stack of linear maps is still one linear map: W_2(W_1 x) = (W_2 W_1)x. Depth buys nothing until something bends between the layers. The bend that made deep networks trainable is the simplest one you could write down.

Definition #

The rectified linear unit, or ReLU, keeps positive numbers and turns everything else into zero:

The Derivative #

At x = 0 the slope from the left is 0 and the slope from the right is 1, so no derivative exists there. This does not stop training: libraries pick a value, usually 0, and carry on.

Why It Replaced the Sigmoid #

Backpropagation is the chain rule, so the gradient that reaches the first layer holds one factor of the activation's slope for every layer. In a chain of n layers with activation f, where a_k is the input to the activation in layer k:

The sigmoid \sigma(x) = 1 / (1 + e^{-x}) has the slope \sigma'(x) = \sigma(x)(1 - \sigma(x)), which is never above \frac{1}{4}. Across ten layers, those factors multiply to at most

and that is the best case, with every unit sitting exactly at x = 0. At x = 5 the slope is already 0.0066. The function \tanh does better at the centre, where its slope is 1, but it flattens in the same way.

A ReLU that is switched on has a slope of exactly 1, so it passes the gradient through unchanged, however deep the network is. The function is old: Fukushima used it in 1969,^[1] and around 2010 it was brought into deep networks.^[2][3] The AlexNet paper made the case in 2012: a four-layer convolutional network with ReLU reached 25% training error on CIFAR-10 about six times sooner than the same network with \tanh.^[4]

One Kink at a Time #

If every piece is a straight line, how does a network learn a curve? By adding kinks. Three ReLUs already make a bump:

Each term switches on at its own point and changes the slope there, which goes from 0 to 1, then to -1, then back to 0.

The coefficient of each ReLU is the change in slope at its kink. So any broken line through the points t_0 < t_1 < \dots < t_n, with slope m_i on the piece that ends at t_i, can be written on [t_0, t_n] as

That sum is exactly a network with one hidden layer. With enough kinks it can follow any continuous function on an interval as closely as you like. Six units already trace a sine wave with a maximum error of 0.134, and twelve bring that down to 0.033: double the kinks and the error falls to a quarter.

Why Depth Matters #

A new unit adds one kink. A new layer multiplies them, because each layer folds the output of the one before. Take this tent on [0, 1], made of two units:

Feed it into itself, and every layer doubles the number of teeth.

Ten such layers use 20 units and produce 512 teeth with 1,023 kinks. One hidden layer would need at least 1,023 units to draw the same shape.^[5] This does not mean that every deep network reaches that count, only that depth can buy complexity far more cheaply than width can.

The Dying ReLU Problem #

The flat side has a price. Take a unit \mathrm{ReLU}(wx + b) with inputs in [-1, 1], w = 1 and b = -10. Then wx + b \le -9 for every input, so the output is always 0 and so is the gradient. Neither w nor b will ever change again, and the unit is dead.

A single large update can push a unit there, which is why a learning rate that is too high can quietly remove part of a network. To check for it, measure the share of units that output zero for a whole batch.

Variants #

Leaky ReLU gives the left side a small slope, so that no unit is ever fully off. The usual choice is \alpha = 0.01.^[6]

GELU replaces the kink with a smooth curve, where \Phi is the standard normal distribution function. It dips to about -0.17 near x = -0.75.^[7]

In language models the choice has moved over time. The original Transformer used plain ReLU in its feed-forward layers, \max(0, xW_1 + b_1)W_2 + b_2,^[8] and BERT moved to GELU.^[9]

Initialization #

For an input that is symmetric around zero, ReLU discards half of the signal: \mathbb{E}[\mathrm{ReLU}(z)^2] = \frac{1}{2}\mathbb{E}[z^2]. Without a correction, the variance would halve at every layer, so ReLU networks draw their weights with twice the usual variance, where n is the number of inputs to the layer.^[10]

In Code #

import numpy as np

def relu(x):
    return np.maximum(0, x)

def bump(x):
    return relu(x) - 2 * relu(x - 1) + relu(x - 2)

print(bump(np.array([-1.0, 0.5, 1.0, 1.5, 3.0])))
# [0.  0.5 1.  0.5 0. ]

Common Mistakes #

1. Putting a ReLU on the output of a regression whose target can be negative. It cannot produce a negative number.
2. Using an initialization designed for \tanh, which lets the signal shrink layer by layer.
3. Blaming the data when a network stops improving, without checking how many units have died.
4. Worrying about the kink at zero. An input that lands exactly on it is rare, and the choice of slope there rarely matters.