In the context of the bias-variance tradeoff, bias measures how far a model's average prediction (across many hypothetical training sets) is from the true value — systematic error caused by a model that's fundamentally too simple to capture the real pattern.
Formula
\(f(x)\) is the true underlying function generating the data. \(\hat{f}(x)\) is the model's prediction, and \(E[\hat{f}(x)]\) is its expected prediction, averaged over many different training sets drawn from the same distribution. Bias is how far off that average is from the truth — it captures a systematic error, not random variation.
Intuition — Why "High Bias" Means "Too Simple"
A linear model fit to a genuinely curved relationship will, no matter how much data it's trained on or how many times you retrain it on different samples, consistently miss the curve in the same systematic way — that's high bias. More data doesn't fix high bias, because the model's fundamental assumption (linearity) is wrong; only a more flexible model (or better features) can reduce it.
Model Bias vs Statistical Bias — The Same Underlying Idea
This is conceptually the same idea as an unbiased estimator in statistics (like why sample variance divides by \(n-1\)) — a systematic, predictable deviation from the truth, as opposed to random noise around it.
High Bias vs Low Bias — Practical Examples
| High Bias (underfits) | Low Bias (flexible) |
|---|---|
| Linear/Logistic Regression on non-linear data | Deep Decision Tree |
| A decision stump (single split) | Random Forest, Gradient Boosting |
| Heavily regularized model | Unregularized, high-capacity model |
Practical Use Cases
- Diagnosing underfitting — a large training-set error is a direct symptom of high bias
- Reasoning about which fix (more capacity vs more regularization) actually addresses the specific problem observed
Common Mistakes
- Confusing model bias (systematic error from oversimplification) with fairness-related bias (systematically unfair outcomes for a demographic group) — these are entirely different, unrelated concepts that happen to share a name.
- Assuming bias can be fixed with more training data — it usually can't; bias is about the model's fundamental assumptions, not the amount of data available.
Interview Relevance
Q: "Why doesn't collecting more training data fix a high-bias model?" Bias comes from the model's structural assumptions being wrong (e.g. fitting a straight line to curved data) — more examples of the same wrong-shaped fit don't change that the model class itself can't represent the true relationship; only a more flexible model or better features address bias directly.
Practice Question
Explain why a linear regression model fit to a clearly quadratic relationship will show high bias regardless of how much training data it's given.
