Logistic Regression Intuition

This note answers the "why," not the "what": why linear regression fails at classification, why the sigmoid function specifically fixes it, and what the log-odds interpretation actually means.

Why Not Just Use Linear Regression for Classification?

Linear regression's output is unbounded — it can be any real number, positive or negative, of any size. A "probability" of -0.3 or 1.4 is nonsensical. Linear regression also assumes a constant-rate linear relationship, but the actual relationship between a feature and a class probability is naturally S-shaped: probability changes slowly near the extremes (very unlikely stays very unlikely) and fastest near the middle (where the outcome is genuinely uncertain) — exactly the shape the sigmoid function has, and a straight line doesn't.

The Log-Odds (Logit) Interpretation

Logistic regression is secretly still linear — just not in probability space. Rearranging the sigmoid equation:

The left side, \(\ln(\hat{y}/(1-\hat{y}))\), is the log-odds (or "logit") of the positive class. This equation says: logistic regression is a perfectly ordinary linear regression — but on the log-odds of the outcome, not the probability itself. This is exactly why the algorithm is called "logistic regression," not "logistic classification."

What a Coefficient Actually Means, in Log-Odds Terms

In linear regression, a coefficient \(b_1\) means "a one-unit increase in \(x\) changes \(\hat{y}\) by \(b_1\)." In logistic regression, a coefficient \(b_1\) means "a one-unit increase in \(x\) changes the log-odds of the outcome by \(b_1\)" — equivalently, it multiplies the odds by \(e^{b_1}\).

import numpy as np

b1 = 0.8   # a fitted coefficient, in log-odds units
odds_multiplier = np.exp(b1)
print(odds_multiplier)   # 2.23 -- each unit increase in x roughly DOUBLES the odds of the positive class

Why the Cost Function Also Had to Change

Linear regression's MSE cost function, if applied directly to sigmoid outputs, produces a non-convex surface — full of local minima that gradient descent can get stuck in. Logistic regression instead uses log-loss, chosen specifically because it stays convex (bowl-shaped) even with the sigmoid nonlinearity in the mix.

Common Mistakes

• Believing "logistic regression" means it's somehow a regression algorithm used for continuous targets — the name refers to the log-odds linearity, not the type of prediction task.
• Interpreting a coefficient directly as a probability change rather than a log-odds (or odds-multiplier) change — a very common source of misstatement.

Interview Relevance

Q: "In what sense is logistic regression still 'linear'?" It's linear in the log-odds of the outcome — \(\ln(\hat{y}/(1-\hat{y})) = w^Tx+b\) is a perfectly ordinary linear equation; the sigmoid function is just the transformation that converts those log-odds back into a bounded [0,1] probability.

Practice Question

A logistic regression coefficient for "years_of_experience" is \(b_1 = 0.4\). Compute \(e^{0.4}\) and explain what it means about how the odds of the outcome change per additional year.

Related ML Notes

• Sigmoid Function
• Logistic Regression Cost Function
• Logistic Regression vs Linear Regression