Sigmoid vs Softmax

This closing comparison note resolves a question nearly every beginner hits when building their first classifier: sigmoid or softmax for the output layer? The answer depends entirely on whether classes are mutually exclusive.

Side-by-Side Comparison

Worked Example — Why the Choice Matters

An image-tagging model that can output any combination of "outdoor," "daytime," "has-people" (these can all be simultaneously true — not mutually exclusive) should use independent sigmoid outputs, one per tag: \(P(\text{outdoor})=0.9\), \(P(\text{daytime})=0.8\), \(P(\text{has-people})=0.3\) — these don't need to sum to anything in particular.

A digit classifier that outputs exactly one of {0,1,...,9} (mutually exclusive — an image is only ever one digit) should use softmax: \(P(0)=0.02, P(1)=0.85, \ldots\) — forced to sum to 1, since exactly one answer is correct.

The Special Case: Binary Classification

For exactly 2 mutually exclusive classes, a single sigmoid output (representing \(P(\text{class}=1)\), with \(P(\text{class}=0)=1-P(\text{class}=1)\) implied) is mathematically equivalent to a 2-output softmax — using softmax with 2 outputs for binary classification is technically valid but redundant; a single sigmoid neuron is the simpler, standard choice.

Code

import torch
import torch.nn as nn

# Multi-label: independent sigmoid per tag
multi_label_logits = torch.tensor([2.0, 1.5, -0.5])   # outdoor, daytime, has-people
print(torch.sigmoid(multi_label_logits))
# each interpreted independently -- no constraint that they sum to 1

# Multi-class, mutually exclusive: softmax across all classes
multi_class_logits = torch.tensor([0.1, 3.2, 0.5, -1.0])   # e.g. 4 possible digit classes
print(torch.softmax(multi_class_logits, dim=0))
# sums to exactly 1.0 -- exactly one class is "the" answer

Common Mistakes

• Using softmax for a multi-label problem — forcing probabilities to sum to 1 when multiple labels can genuinely co-occur artificially suppresses valid simultaneous predictions (predicting one tag strongly necessarily lowers every other tag's probability, which is wrong when tags are independent).
• Using independent sigmoids for a genuinely mutually-exclusive multi-class problem — this doesn't enforce that exactly one class "wins," and the resulting scores don't form a coherent probability distribution over the classes.

Interview Relevance

Q: "You're building a model to detect which of several diseases (assume a patient can have more than one simultaneously) are present in a medical scan. Sigmoid or softmax for the output layer?" Sigmoid, applied independently to each disease's output neuron — since diseases can co-occur (not mutually exclusive), softmax's forced-sum-to-1 constraint would incorrectly suppress the ability to predict multiple diseases confidently at once.

Key Takeaways — Activation Functions

• Step is historical only; it has zero gradient almost everywhere and cannot be used with backpropagation.
• Sigmoid and tanh saturate for large \(|z|\), causing vanishing gradients in deep networks — they're now mostly reserved for output layers (sigmoid) and gated cells (both).
• ReLU is the modern hidden-layer default: cheap, non-saturating for positive inputs, but can "die." Leaky ReLU, PReLU and ELU each address dying neurons differently — fixed slope, learned slope, and smooth exponential floor respectively.
• GELU and Swish are smooth, non-monotonic ReLU alternatives that dominate in Transformers specifically.
• Softmax is for mutually-exclusive multi-class outputs; sigmoid (applied per-class) is for binary or multi-label outputs.

Next: Loss Functions covers every major loss formula — building directly on the MLE derivations from Probability & Statistics for DL and the activation choices from this category — with worked numerical examples for each.

Practice Question

A weather model predicts, for a given day, the probability it will be sunny, rainy, cloudy or snowy — assume exactly one of these four labels is correct per day. Which activation function belongs on the output layer, and what loss function pairs with it?

Related DL Notes

• Sigmoid Function
• Softmax Function
• Binary Cross-Entropy