A decision tree predicts by asking a sequence of yes/no questions about the features — each question splits the data further, until it reaches a final decision. It's the algorithm that most closely resembles how a human would actually reason through a classification problem, step by step.
Tree Structure and Terminology
Root node → internal (decision) nodes → leaf nodes. Every path from root to leaf is one chain of yes/no questions ending in a prediction.
| Term | Meaning |
|---|---|
| Root node | The first, top-level split — considers the entire dataset |
| Internal (decision) node | A further split on some subset of the data |
| Leaf node | A final prediction — no further splitting |
| Depth | The number of splits from root to the deepest leaf |
| Branch | A path connecting one node to a child node |
How a Tree Decides Where to Split
At every node, the algorithm considers every possible feature and threshold, and picks whichever split makes the resulting two groups most "pure" — most dominated by a single class. Purity is measured with Gini impurity or entropy, and the improvement from a split is quantified as information gain. This process repeats recursively on each resulting subset until a stopping condition is met.
Classification vs Regression Trees
| Classification Tree | Regression Tree | |
|---|---|---|
| Leaf predicts | A class label (majority class in that leaf) | A number (average target value in that leaf) |
| Split criterion | Gini impurity or entropy | Variance / MSE reduction |
Minimal Working Example
from sklearn.tree import DecisionTreeClassifier
import numpy as np
X_train = np.array([[1,0],[2,0],[2,1],[3,1],[4,1],[5,2],[6,2],[7,3]])
y_train = np.array([0,0,0,0,1,1,1,1])
model = DecisionTreeClassifier(max_depth=2, random_state=42)
model.fit(X_train, y_train)
print(model.predict([[4, 2]])) # predicted classPractical Use Cases
- Credit approval, medical diagnosis — anywhere a human-readable decision path matters
- As the building block of Random Forest and Gradient Boosting, the two most widely used tabular-data algorithms in practice
Advantages
- Highly interpretable — the exact reasoning path for any prediction can be read off directly
- No feature scaling required — splits are threshold-based per feature, not distance-based
- Naturally handles non-linear relationships and feature interactions
Limitations
- Prone to overfitting if grown too deep, memorizing noise instead of general patterns
- Unstable — small changes in training data can produce a very different tree structure
- Individually, usually less accurate than ensemble methods built from many trees
Common Mistakes
- Growing a tree to full depth without any stopping criteria or pruning — this almost always overfits.
- Assuming a single decision tree will match the accuracy of Random Forest — ensembles of trees are used in practice precisely because a single tree is comparatively unstable.
Interview Relevance
Q: "Why don't decision trees need feature scaling?" Because splits are based on a single feature crossing a threshold at a time (e.g. "income > 50000"), never on distance or a weighted sum across features — the relative scale between different features never affects which split is chosen.
Practice Question
Sketch, in words, what the first split of a decision tree predicting loan default might look like, and why that particular feature/threshold might be chosen first.
