Sigmoid Function

The sigmoid function squashes any real number into the range (0, 1) — the exact mathematical property that lets logistic regression's linear combination of features become a valid probability.

Formula

\(z\) can be any real number, positive or negative, arbitrarily large. \(e\) is Euler's number (\(\approx 2.718\)). As \(z\) grows very large and positive, \(e^{-z} \to 0\), so \(\sigma(z) \to 1\). As \(z\) grows very large and negative, \(e^{-z} \to \infty\), so \(\sigma(z) \to 0\). At \(z=0\): \(\sigma(0) = 1/(1+1) = 0.5\) exactly.

Graph — The S-Curve

Numerical Example

Using \(z = 0.8x - 4\) (from a fitted model), for \(x=6\): \(z = 0.8(6)-4 = 0.8\).

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for x in [3, 5, 6, 8]:
    z = 0.8 * x - 4
    print(x, round(sigmoid(z), 3))
# 3   0.168
# 5   0.500
# 6   0.690
# 8   0.917

Notice \(x=5\) gives exactly 0.5 — this is the decision boundary point, where \(z=0\).

The Derivative — Why It's Elegant, and Why It Matters

The sigmoid's derivative can be written entirely in terms of the sigmoid's own output — no need to recompute \(z\) or \(e^{-z}\) separately. This elegance is exactly why sigmoid was historically popular for gradient-based training: computing gradients during backpropagation is cheap once you already have \(\sigma(z)\) from the forward pass.

Practical Use Cases

• The final layer activation for binary classification, in both logistic regression and neural networks
• Converting any unbounded score into an interpretable, bounded probability-like value

Limitations

• For very large or very small \(z\), the derivative approaches 0 (the curve flattens) — known as "vanishing gradients," a real problem in deep neural networks that use sigmoid in hidden layers (modern networks generally prefer ReLU-family activations for hidden layers, keeping sigmoid mainly for the output layer of binary classifiers)

Common Mistakes

• Confusing the sigmoid's output with a guaranteed frequency — a well-calibrated 0.7 output should be right about 70% of the time on average across many predictions, not a guarantee for any single one.
• Forgetting the sigmoid saturates (flattens) for large \(|z|\) — extremely confident (near 0 or 1) predictions barely change their gradient, which can slow further learning on those examples.

Interview Relevance

Q: "Why is the sigmoid derivative written as σ(z)(1-σ(z)) useful in practice?" It lets you reuse the already-computed sigmoid output during backpropagation without recomputing the exponential from scratch — a small but real computational efficiency that mattered a lot in early neural network implementations.

Practice Question

Compute \(\sigma(z)\) by hand for \(z=2\) and \(z=-2\), and verify the two results are symmetric around 0.5 (i.e. \(\sigma(2) + \sigma(-2) = 1\)).

Related ML Notes

• Logistic Regression
• Logistic Regression Intuition
• Logistic Regression Cost Function