GELU (Gaussian Error Linear Unit) is a smooth activation function that weights its input by how likely that input is under a standard normal distribution. It's the default activation inside every major Transformer-based model — BERT, GPT, and essentially every modern large language model.
Formula
\(\Phi(z)\) is the cumulative distribution function of the standard normal distribution — the probability that a standard normal random variable is less than \(z\) (see Probability Distributions). Intuitively: GELU multiplies the input by "the probability this input would be kept if we randomly decided whether to zero it out, weighted by how large it is" — a smooth, probabilistic version of ReLU's hard \(z\ge0\) gate.
A Practical Approximation
Computing \(\Phi(z)\) exactly is expensive, so implementations commonly use a fast approximation:
Graph
Similar overall shape to ReLU, but smooth everywhere, with a small negative dip near zero instead of a hard corner.
GELU vs ReLU — The Key Differences
| Property | ReLU | GELU |
|---|---|---|
| Smoothness | Sharp corner at \(z=0\) | Smooth (infinitely differentiable) everywhere |
| Negative inputs | Always exactly 0 | Small negative values allowed near \(z=0\), approaching 0 for very negative \(z\) |
| Monotonicity | Monotonic (never decreases) | Slightly non-monotonic — dips very slightly before rising |
| Interpretation | A hard, deterministic gate | A smooth, probabilistic gate |
Code
import torch.nn as nn
import torch
layer = nn.GELU() # PyTorch's default is the exact (erf-based) formulation
x = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0])
print(layer(x))
# Approximate version, matching the tanh-based approximation formula above
layer_approx = nn.GELU(approximate='tanh')
print(layer_approx(x))Where It's Used Today
GELU is the standard activation inside the feed-forward blocks of Transformer architectures (see the Transformers category) — used in BERT, GPT-style models, and most modern large language models. Its smoothness is believed to help with the optimization dynamics of very large, very deep Transformer stacks, though it's more computationally expensive than plain ReLU.
Common Mistakes
- Assuming GELU is simply "ReLU with better branding" — the probabilistic interpretation and smooth, slightly non-monotonic shape are genuinely different mathematical objects, not a cosmetic variation.
- Not distinguishing between the exact and tanh-approximated GELU implementations when comparing published model architectures — some models specify one variant explicitly, and results can differ subtly.
Interview Relevance
Q: "Why do Transformer architectures like BERT and GPT typically use GELU instead of ReLU?" GELU is smooth everywhere (no sharp corner at zero, unlike ReLU) and weights inputs probabilistically rather than with a hard cutoff, which empirically tends to help optimization in very deep, large-scale Transformer stacks. It has become something of a de facto standard in Transformer feed-forward blocks, though the specific advantage over ReLU is more empirical than proven.
Practice Question
Using the tanh-based approximation formula, would GELU(z) be closer to 0 or closer to z for a large positive z? What does that tell you about GELU's behavior compared to ReLU for large positive inputs?
