Gaussian Error Linear Units (GELUs)

The GELU is an activation function introduced in the paper Gaussian Error Linear Units. If you're unfamiliar with activation functions or want a refresher, you can check out this blog where I explain them in detail.

GELU gained broad attention after Google adopted it in BERT, and later when OpenAI used it in GPT-2. Throughout the early Transformer era, models such as the GPT series, BERT variants, and Google’s Vision Transformer (ViT) relied heavily on GELU because its smooth, probabilistic formulation often led to better training stability and performance compared to ReLU.

However, while GELU was the dominant choice in those earlier architectures, modern large language models have largely moved on to more effective gated activations like SwiGLU (used in LLaMA and PaLM) and SiLU/Swish. I will write blogs on them as well.

Building the Foundations: Key Probability Concepts

What is a Random Variable?

When people hear “random variable” they often imagine something complicated but the idea is incredibly simple. A random variable is just a number we don’t know yet, but one that is generated by some process or an outcome generated by a random experiment.

• If the outcomes are countable (like 1 to 6 on dice) → Discrete Random Variable
• If the outcomes can be any real number in a range (like height, weight, temperature) → Continuous Random Variable

GELU uses a continuous random variable (the standard normal one) so keep that in mind.

Probability Distribution (A description of how likely different values are)

A probability distribution is basically the full description of the behavior of a random variable. It tells you:

• what values the random variable could take, and
• how likely each of those values are.

For discrete variables (like dice rolls) this really is a literal list value → probability. But for continuous variables like height, time, temperature, neural activations the story changes. You can’t just “list” infinitely many real numbers. Instead, we use a function to describe the likelihood landscape.

That function is the PDF.

Probability Density Function (PDF)

When a random variable is continuous, you don’t assign probabilities to individual values, because that probability is always zero (there are infinitely many possibilities).

The PDF is the tool that answers this. A PDF is a curve over the x-axis whose height at each x tells you how dense or concentrated the probability is around that value. It doesn’t give you the probability of that exact value because a single point has zero width and therefore zero area. But it tells you the intensity of probability around that area.

If the PDF is tall around x = 0 and flat near x = 5, it means the random variable strongly prefers values near 0 and rarely lands near 5.

The important part is this:

• To get actual probability, you look at area under the PDF.
• The area between two points, say 1 and 2, gives the probability that the random variable falls in that interval.

The total area under the whole PDF is 1.

Standard Gaussian Distribution (the most iconic PDF of all)

Now that we know what a PDF is, the standard Gaussian (or standard normal) distribution becomes easy to understand. It is simply a PDF with a very specific shape: the bell curve (as you can see in the figure above).

It is centered at zero, symmetric around zero, and its “spread” is controlled by the variance. For the standard Gaussian, we fix:

• mean = 0
• variance = 1

So the formula becomes:

• the exponential controls how fast the tails fall off,
• the constant out front makes sure the total area = 1,
• and the fact that it’s symmetric means values equally distant from 0 are equally likely.

This shape appears everywhere in nature and machine learning noise, weights , initializations, error behaviors because of deep mathematical laws like the Central Limit Theorem.

GELU uses this distribution because its activation idea is tied to the probability that a Gaussian random variable is less than a certain point.

To express that probability, we need something else: the cumulative view.

Cumulative Distribution Function (CDF) — probability accumulated up to x

This is the point I wanted to reach when I started explaining the basics, because the CDF shows up directly in the GELU formula.

While the PDF tells us how dense the probability is at a point, the CDF tells us how much probability has been accumulated up to that point. Whatever area you’ve swept out is the probability that the random variable is less than or equal to x.

And we can simply obtain CDF of a PDF by integrating it. Don't worry I will do full integration of PDF of Standard normal distribution below.

1. if \(x=0 \rightarrow CDF(0) = 0.5\). Because half the curve is left of zero

2. if \(x=2 \rightarrow CDF(2) \approx 0.977\). Because 97.7% of the bell curve is to the left of 2

3. if \(x=-2 \rightarrow CDF(-2) \approx 0.022\).

So we get the following intuition:

• Large positive x CDF close to 1
• Large negative x CDF close to 0
• x around 0 CDF near 0.5

The GELU Formula: Bringing It All Together

Now that we have the basics, let's introduce GELU. The GELU activation function for an input \(x\) is defined as:

This has a cool intuitive interpretation: For large positive \(x\) , \(Φ(x)≈1\) so GELU passes \(x\) through almost unchanged (like the identity function). For large negative \(x\) , \(Φ(x)≈0\), so GELU suppresses the signal (like ReLU). But unlike ReLU, it's smooth and allows a tiny bit of gradient flow for negatives, inspired by stochastic regularization techniques like dropout.

The key is that transition from suppressing to passing is gradual and smooth(You can observe this in the graph below) because \(Φ(x)\) is smooth. There is no hard cutoff like RELU's where everything (-)ve becomes exactly 0 and that smoothness gives nicer gradients for optimization.

As promised, let’s finally see how the CDF actually comes from integrating the PDF of the standard normal distribution. This is where things get interesting, because this integral behaves very differently from the simple integrals you’ve seen before in high school. The integral of the Gaussian does not simplify into any combination of elementary functions. Instead, mathematicians define a special function the error function, written as erf specifically to represent this integral. Using this function, the Gaussian CDF can be written in a clean, exact form. Let's see the integration.

The PDF of the standard normal is:

To get the CDF, we need to integrate from −∞ to some value x:

let \(u = \frac{x}{\sqrt{2}}\)

so \(x = \sqrt{2}u\) and \(dx = \sqrt{2} du\)

limits \(x = -\infty \rightarrow u = -\infty\)

\(x = a \rightarrow u = \frac{a}{\sqrt{2}}\)

Substitution

Split at 0 and use symmetry

Evaluating the First Part

This evaluates to \(\frac{1}{2}\). Click on the drop-down button to see exactly how it evaluates to \(\frac{1}{2}\).

Evaluating the Second Part

Let's rearrange the erf definition to solve for the integral part, which is what we need:

First, pull the constant \(\frac{1}{\sqrt{\pi}}\) out of our second integral:

$$\frac{1}{\sqrt{\pi}} \left[ \frac{\sqrt{\pi}}{2} erf\left(\frac{a}{\sqrt{2}}\right) \right]$$

The \(\sqrt{\pi}\) on the outside cancels with the \(\sqrt{\pi}\) in the numerator: $$\frac{1}{\cancel{\sqrt{\pi}}} \left[ \frac{\cancel{\sqrt{\pi}}}{2} erf\left(\frac{a}{\sqrt{2}}\right) \right] = \frac{1}{2} erf\left(\frac{a}{\sqrt{2}}\right)$$ And that's the result for the second integral.

Now, we just add them back together:

$$\Phi(a) = (\text{Part 1}) + (\text{Part 2})$$

$$\Phi(a) = \frac{1}{2} + \frac{1}{2} erf\left(\frac{a}{\sqrt{2}}\right)$$

Factoring out the \(\frac{1}{2}\) gives the final, clean expression :

$$\Phi(a) = \frac{1}{2} \left[ 1 + erf\left(\frac{a}{\sqrt{2}}\right) \right]$$

This formula is the closed-form solution for the Standard Normal Cumulative Distribution Function (CDF), \(\Phi(a)\), expressed using the error function, erf(z).

Why erf Is Expensive (and Why Deep Learning Avoids It)?

Here’s the catch: even though libraries like PyTorch expose a direct torch.erf function, computing \(erf(x)\) numerically is expensive. The reason is simple evaluating the error function requires complicated numerical approximations: piecewise polynomials, rational approximations, and branching logic for different ranges of x.

Computing erf(x) is nothing like computing \(e^x\) or \(tanh(x)\), which have fast, fused hardware instructions on modern GPUs. There is no native GPU instruction for erf. So PyTorch (and every other framework) must fall back to a software implementation, which internally uses:

• multiple polynomial approximations
• multiple exponentials
• conditional branches depending on x
• table-based or rational approximations

GPUs hate branching and conditionals because they force entire warps of threads to wait while others execute different branches. So even though torch.erf works and is available, the GPU is still running a slow, divergence-heavy software routine underneath.

When you use GELU, this matters a lot. A transformer computes GELU billions of times during training. Using the exact erf-based GELU means paying this cost every single time, which substantially slows down training.

This is why modern deep learning frameworks avoid evaluating the exact erf version of GELU repeatedly, and instead rely on a fast approximation (like the tanh-based one) that behaves almost identically but runs dramatically faster on hardware.

Let's Look at them.

The Practical Fix: A Fast Tanh-Based Approximation of GELU

Researchers noticed something elegant:

• the normal CDF \(Φ(x)\)
• and the tanh function

have very similar S-shaped curves.

So instead of computing:

we use a fast, smooth approximation of Φ(x) that runs extremely well on GPUs.

The tanh-based approximation is:

This formula may look random at first, but every constant inside it was chosen carefully by curve fitting so that:

• it matches the Gaussian CDF almost perfectly
• it is numerically stable
• it is dramatically faster than calling erf

And because GPUs have optimized vectorized implementations of tanh, this version is not just faster it’s much faster on every modern deep learning accelerator.

Wrapping Up

We’ve now traced the entire story of GELU from the first principles of random variables and probability, all the way through the structure of the Gaussian distribution, the origin of the CDF, the appearance of the error function, and finally how all of that condenses into a single activation function used across deep learning.

What makes GELU beautiful is that it isn’t just another heuristic nonlinearity. It’s a function with deep probabilistic meaning, smooth gradients, and practical engineering trade-offs that shaped the Transformer era. The exact version comes straight from the mathematics of the Gaussian, while the tanh approximation arises from clever numerical insight designed to make training fast on GPUs.

If you’ve followed everything up to here, you now understand not just what GELU is, but why it exists, how it’s derived, and why modern deep learning uses the approximate version in practice. And that means you understand it far more deeply than most people who just “use it because the paper said so.”

If You’d Like to Reach Out

If you have any questions, doubts, or want to discuss anything from this blog, feel free to comment or reach out to me on my socials. And if you notice any mistake or think something could be improved, please let me know I’m always learning and refining as I go.