Decision trees are among the most overfitting-prone algorithms in classical ML — their flexibility to keep splitting until every leaf is pure is exactly what makes them so easy to explain, and exactly what makes them memorize noise if left unchecked.
Why Trees Overfit So Readily
A tree with no depth limit can always achieve 100% training accuracy — simply keep splitting each impure node further until every leaf contains samples of only one class. But splits made to isolate a single noisy training example don't represent any real, generalizable pattern; they represent that one example's specific noise. This is a fundamentally different failure mode than underfitting — the tree isn't too simple, it's needlessly, brittlely complex.
Training vs Validation Accuracy as Depth Grows
Training accuracy keeps climbing toward 100% as depth increases; validation accuracy peaks, then declines as the tree starts fitting training-set noise instead of general patterns.
Diagnosing Overfitting in Code
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
for depth in [2, 3, 5, 8, 12, None]:
model = DecisionTreeClassifier(max_depth=depth, random_state=42)
model.fit(X_train, y_train)
train_acc = accuracy_score(y_train, model.predict(X_train))
test_acc = accuracy_score(y_test, model.predict(X_test))
print(f"depth={depth}: train={train_acc:.3f}, test={test_acc:.3f}")
# Watch for the gap between train and test accuracy widening as depth increases --
# a large, growing gap is the clearest signal of overfittingThe Fixes, In Order of Where They Fit
| Fix | How It Helps |
|---|---|
| Pruning (pre or post) | Directly limits tree complexity |
Increase min_samples_leaf | Prevents leaves from isolating single noisy examples |
| More training data | Makes it harder for the tree to fit noise that only appears in a small sample |
| Switch to Random Forest | Averages many overfitting-prone trees together, canceling out individual trees' noise-fitting |
That last row is genuinely important context: Random Forest doesn't fix any single tree's tendency to overfit — it exploits it deliberately, training many separately overfit trees on different random subsets and averaging away their individual noise, which is a fundamentally different (and often more effective) strategy than pruning a single tree carefully.
Common Mistakes
- Judging a tree's quality from training accuracy alone — an unpruned tree can trivially reach ~100% training accuracy while performing poorly on new data.
- Assuming a single, carefully-pruned tree will always outperform an ensemble — in practice, Random Forest or gradient boosting usually beats even a well-tuned single tree.
- Pruning so aggressively the tree starts underfitting instead — always validate with cross-validation, not by eye.
Interview Relevance
Q: "How would you tell if a decision tree is overfitting?" Compare training accuracy to validation/test accuracy — a large, growing gap as depth increases (near-perfect training accuracy alongside much worse validation accuracy) is the clearest signal; fix with pruning, minimum leaf size constraints, more data, or an ensemble method.
Practice Question
A decision tree achieves 99% training accuracy and 68% test accuracy. Name two specific hyperparameter changes you'd try first, and explain your reasoning for each.
