A complete walkthrough of how a classification tree picks its very first split — computing impurity before and after a candidate split, and choosing whichever split reduces impurity the most.
The Dataset
10 historical examples of whether a customer played tennis, based on the day's outlook:
| Outlook = Sunny? | Played Tennis | Count |
|---|---|---|
| Yes (Sunny) | No | 3 |
| Yes (Sunny) | Yes | 1 |
| No (Not Sunny) | Yes | 5 |
| No (Not Sunny) | No | 1 |
Root node: 10 examples total, 6 "Yes", 4 "No".
Step 1 — Impurity Before the Split
Step 2 — Impurity After Splitting on "Sunny?"
| Group | Counts | Gini | Entropy |
|---|---|---|---|
| Sunny (4 examples) | 1 Yes, 3 No | \(1-(0.25^2+0.75^2)=0.375\) | \(\approx 0.811\) |
| Not Sunny (6 examples) | 5 Yes, 1 No | \(1-(0.833^2+0.167^2)\approx 0.278\) | \(\approx 0.650\) |
Step 3 — Weighted Impurity and Information Gain
See Gini Impurity, Entropy and Information Gain for the formulas in full depth. In practice, the tree would repeat this exact calculation for every candidate feature and threshold, then pick whichever split produces the highest information gain (or Gini reduction).
Python Implementation
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt
# outlook_sunny (1/0), humidity_high (1/0) -> played (1/0)
X_train = [[1,1],[1,1],[1,0],[1,0],[0,1],[0,0],[0,0],[0,0],[0,0],[0,1]]
y_train = [0,0,0,1,1,1,1,1,0,1]
model = DecisionTreeClassifier(criterion="entropy", max_depth=3, random_state=42)
model.fit(X_train, y_train)
plt.figure(figsize=(10,6))
plot_tree(model, feature_names=["outlook_sunny","humidity_high"], class_names=["No","Yes"], filled=True)
plt.show()
print(model.predict([[1, 0]])) # sunny, low humidity -> predictioncriterion="entropy" tells scikit-learn to use information gain (based on entropy) for split selection; criterion="gini" (the default) uses Gini impurity instead. In practice, the two criteria usually produce very similar trees.
How the Tree Grows Beyond the Root
After the first split, the algorithm repeats the exact same process independently on each resulting subset — the Sunny group and the Not-Sunny group each get their own best-split search, considering all remaining features again. This recursive splitting continues until a stopping condition (max depth, minimum samples per leaf, or zero remaining impurity) is reached — see Decision Tree Pruning for how these stopping rules are chosen deliberately.
Practical Use Cases
- Any binary or multi-class classification problem where an interpretable decision path adds real value
Common Mistakes
- Assuming Gini and entropy always pick the same split — they usually agree, but not always exactly, since they weight impurity slightly differently.
- Growing the tree without limiting depth, letting it split all the way down to single-example leaves — a direct path to overfitting.
Interview Relevance
Q: "How does a decision tree decide which feature to split on first?" It evaluates every candidate feature and threshold, computes the impurity reduction (information gain) each would produce, and selects whichever split reduces impurity the most — exactly the calculation worked through above.
Practice Question
Using the impurity formulas, verify that Gini(Not Sunny) with 5 Yes / 1 No comes out to approximately 0.278.
