Information Gain

Information gain measures exactly how much a candidate split reduces uncertainty (entropy) — it's the metric a decision tree actually compares across every candidate feature and threshold to decide where to split next.

Formula

\(\text{Entropy(parent)}\) is the impurity before splitting. The sum is the weighted average entropy of the resulting child nodes, weighted by how many samples \(n_k\) landed in each child out of the total \(n\). Information gain is simply the drop in entropy the split achieved.

Full Worked Calculation

Continuing the running example from Decision Tree Classification: root has 10 samples (6 Yes, 4 No), split on "Sunny?" into Sunny (4 samples: 1 Yes, 3 No) and Not-Sunny (6 samples: 5 Yes, 1 No).

def entropy(labels):
    import numpy as np
    from collections import Counter
    counts = Counter(labels)
    n = len(labels)
    probs = [c/n for c in counts.values()]
    return -sum(p * np.log2(p) for p in probs)

def information_gain(parent, children):
    n = len(parent)
    weighted_child_entropy = sum((len(c)/n) * entropy(c) for c in children)
    return entropy(parent) - weighted_child_entropy

parent = [1,1,1,1,1,1,0,0,0,0]
sunny = [1,0,0,0]
not_sunny = [1,1,1,1,1,0]

print(information_gain(parent, [sunny, not_sunny]))   # 0.2565

How the Tree Uses This — Comparing Multiple Candidates

In a real dataset, the tree computes information gain for every possible feature and threshold, then picks whichever produces the highest IG:

Here, "Outlook = Sunny?" wins and becomes the root split, because it reduces uncertainty the most out of every option considered.

Practical Use Cases

• The core decision-making mechanism inside every entropy-based decision tree split
• Feature selection — features that never produce meaningful information gain across any split are effectively uninformative for this target

Common Mistakes

• Forgetting to weight child entropy by group size — an impure but tiny child group shouldn't count as heavily as a large one.
• Assuming a feature with high information gain at the root will also be useful deeper in the tree — relevance can change once the data has already been split by other features.

Interview Relevance

Q: "How does a decision tree choose which feature to split on at each node?" It computes information gain (or Gini reduction) for every candidate feature and threshold at that node, and selects whichever split yields the highest value — repeating this process independently at every subsequent node.

Practice Question

A split produces two children with entropy 0.4 and 0.9, weighted 70% and 30% of the parent's samples respectively. If the parent's entropy was 0.85, compute the information gain.

Related ML Notes

• Entropy
• Gini Impurity
• Decision Tree in Python