Overfitting happens when a model learns the training data's noise and quirks in addition to its genuine patterns — producing excellent training performance that fails to generalize to new data.
The Clearest Signal — The Train/Validation Gap
Training error keeps falling while validation error starts rising — the growing gap between the two curves is the textbook signature of overfitting.
Diagnosing It in Code
from sklearn.metrics import accuracy_score
train_acc = accuracy_score(y_train, model.predict(X_train))
val_acc = accuracy_score(y_val, model.predict(X_val))
print(f"Train: {train_acc:.3f}, Validation: {val_acc:.3f}, Gap: {train_acc - val_acc:.3f}")
# A large, positive gap (e.g. 0.99 vs 0.72) is the clearest overfitting signalCommon Causes
| Cause | Why It Overfits |
|---|---|
| Model too complex for the amount of data | Enough capacity to memorize noise instead of learning the general pattern |
| Too many features relative to samples | More opportunity to fit noise in any single feature |
| Training for too long (iterative models) | Later iterations increasingly fit training-set-specific noise |
| Noisy or mislabeled training data | A flexible model can memorize the noise as if it were signal |
The Fixes
- Regularization — directly penalize model complexity
- Pruning or depth limits, for tree-based models
- Early stopping, for iterative training
- More training data, if obtainable
- Feature selection, to reduce unnecessary complexity
- Switching to an ensemble like Random Forest, which averages away individual overfitting
Practical Use Cases
Every model-building project needs to actively check for overfitting — it's not an edge case, it's the default failure mode of any sufficiently flexible model trained without appropriate safeguards.
Common Mistakes
- Only checking training accuracy and never comparing it against validation performance.
- Assuming a complex model is automatically better — extra flexibility only helps if it's matched with enough data and appropriate regularization.
Interview Relevance
Q: "How do you detect overfitting during model development?" Compare training performance against validation/cross-validation performance — a small gap is healthy and expected; a large, growing gap (especially alongside near-perfect training scores) is the clearest evidence the model has started memorizing training-set-specific noise instead of general patterns.
Practice Question
A model achieves 98% training accuracy and 71% test accuracy. Name three concrete changes you'd try, in order of what you'd attempt first.
