Sigmoid vs Tanh

This comparison note puts sigmoid and tanh side by side directly — useful both for interviews and for the concrete decision of which to use inside an LSTM/GRU gate, where both still appear regularly.

Side-by-Side Comparison

Why Both Still Appear Inside LSTM/GRU Cells

Both survive specifically because their bounded ranges are semantically meaningful there, not because they avoid vanishing gradients (they don't, within a single cell's internal computation) — sigmoid's (0,1) range is exactly right for a "gate" controlling how much information to keep or discard, and tanh's (-1,1) range is exactly right for a candidate value that can push a cell's memory in either direction. This is covered in full in the LSTM & GRU category.

Code — Plotting the Difference

import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))
def tanh(z): return np.tanh(z)

z = np.linspace(-5, 5, 5)
print("sigmoid:", sigmoid(z))
print("tanh:   ", tanh(z))
# sigmoid stays entirely positive; tanh is symmetric around 0

Common Mistakes

• Assuming tanh solves vanishing gradients because it's zero-centered — it saturates just as severely as sigmoid for large \(|z|\); zero-centering only addresses a separate, secondary optimization issue.
• Using sigmoid for a hidden layer "because it's simpler to reason about" in a deep feedforward network — for hidden layers, ReLU (or a variant) is virtually always the better modern default for both of these reasons.

Interview Relevance

Q: "If both sigmoid and tanh suffer from vanishing gradients, why does an LSTM still use both internally?" Their bounded ranges carry specific semantic meaning inside a gated cell: sigmoid's (0,1) output is used directly as a "how much to let through" gate value, and tanh's (-1,1) output is used as a signed candidate update. The vanishing-gradient concern is mitigated in LSTMs by a separate mechanism — the cell state's largely additive update path — not by avoiding these activations.

Practice Question

For \(z=3\), compute both \(\sigma(3)\) and \(\tanh(3)\). Which one is closer to its respective saturation bound?

Related DL Notes

• Sigmoid Function
• Tanh Function
• Forget Gate