Logistic regression uses log-loss (binary cross-entropy) as its cost function — not MSE — specifically because MSE combined with the sigmoid's nonlinearity produces a non-convex surface that gradient descent can get stuck in.
Formula
\(y_i\) is the true label (0 or 1), \(\hat{y}_i = \sigma(z_i)\) is the predicted probability. Only one of the two terms is ever "active" for a given example — if \(y_i=1\), the \((1-y_i)\log(1-\hat{y}_i)\) term becomes zero, leaving just \(-\log(\hat{y}_i)\); if \(y_i=0\), it's the reverse.
Why This Specific Formula — Reading It as a Penalty
| True Label | Active Term | Behavior |
|---|---|---|
| y = 1 | \(-\log(\hat{y})\) | Near 0 when \(\hat{y}\) is close to 1 (correct, confident) — grows toward infinity as \(\hat{y} \to 0\) (confidently wrong) |
| y = 0 | \(-\log(1-\hat{y})\) | Near 0 when \(\hat{y}\) is close to 0 (correct, confident) — grows toward infinity as \(\hat{y} \to 1\) (confidently wrong) |
Numerical Example
True label \(y=1\). Compare two predictions:
The badly wrong prediction is penalized over 20 times more heavily than the good one — log-loss punishes confident wrongness far more severely than a simple "was it right or wrong" metric would.
from sklearn.metrics import log_loss
import numpy as np
y_true = [1]
print(log_loss(y_true, [0.9], labels=[0,1])) # ~0.105
print(log_loss(y_true, [0.1], labels=[0,1])) # ~2.303Why This Cost Function Stays Convex
Unlike MSE-with-sigmoid (which is non-convex), the log-loss formula, combined specifically with the sigmoid function, produces a cost surface that is provably convex — a single global minimum, same guarantee linear regression's MSE has. This is exactly why log-loss (not MSE) is the standard choice for training logistic regression via gradient descent.
The Gradient — A Surprisingly Elegant Result
This has exactly the same form as linear regression's gradient (average residual, weighted by features) — even though the cost function and the model's output are completely different. This elegant coincidence is a direct mathematical consequence of pairing the sigmoid function with log-loss specifically, and it's why the two algorithms' training loops look almost identical in code.
Practical Use Cases
- Training every binary classifier that outputs a probability — logistic regression, and the output layer of binary-classification neural networks
- Evaluating a classifier's calibration quality — a model with low log-loss isn't just getting labels right, it's confidently right
Common Mistakes
- Using MSE to train a logistic regression model instead of log-loss — technically possible, but abandons the convexity guarantee and typically trains worse.
- Feeding a probability of exactly 0 or 1 into the log-loss formula — \(\log(0)\) is undefined; scikit-learn and most implementations clip probabilities to a tiny epsilon away from the extremes to avoid this.
Interview Relevance
Q: "Why does log-loss penalize confident wrong predictions so much more than MSE would?" Because of the logarithm — as a predicted probability for the true class approaches 0, \(-\log(\hat{y})\) grows without bound, while MSE's squared-error penalty stays finite and comparatively small; log-loss is specifically designed to punish confident wrongness severely, which matters a lot for probability-calibrated applications.
Practice Question
For true label \(y=0\), compute the log-loss for predictions \(\hat{y}=0.05\) and \(\hat{y}=0.95\), and explain the large difference.
