Entropy

Entropy, borrowed directly from information theory, measures the amount of "surprise" or uncertainty in a node's class distribution — a pure node has zero entropy (no uncertainty at all), and entropy peaks when classes are perfectly mixed.

Formula

\(p_i\) is the proportion of class \(i\), and the logarithm is base 2 — entropy is measured in bits. For binary classification, entropy ranges from 0 (pure) to a maximum of 1 (perfectly 50/50 split).

The Information-Theoretic Intuition

Entropy answers: "how many yes/no questions, on average, would you need to correctly guess a randomly drawn sample's class?" A pure node (all one class) needs zero questions — you already know the answer. A perfectly 50/50 node needs exactly 1 bit of information (one well-chosen yes/no question) to resolve, on average — which is exactly why maximum binary entropy equals 1.

Graphical Comparison — Entropy vs Gini

Numerical Example

Same root node: 6 Yes, 4 No out of 10.

import numpy as np

def entropy(labels):
    from collections import Counter
    counts = Counter(labels)
    n = len(labels)
    probs = [count/n for count in counts.values()]
    return -sum(p * np.log2(p) for p in probs)

root = [1,1,1,1,1,1,0,0,0,0]
print(entropy(root))   # 0.9710
print(entropy([1,0,0,0]))          # Sunny group: ~0.8113
print(entropy([1,1,1,1,1,0]))       # Not-Sunny group: ~0.6500

These are the exact same values used to compute Information Gain in Decision Tree Classification (IG ≈ 0.257).

Practical Use Cases

• The alternative split criterion in scikit-learn's decision trees (criterion="entropy")
• The foundation of information gain, which directly uses entropy before and after a split

Common Mistakes

• Using base-10 or natural log instead of log base 2 — the choice of log base changes entropy's numeric scale (its unit), though it doesn't change which split gets chosen as best.
• Assuming entropy and Gini will always rank candidate splits identically — they usually agree closely, but not with mathematical guarantee.

Interview Relevance

Q: "Why is entropy measured in bits, and what does that actually mean?" It comes from information theory — 1 bit represents the amount of information needed to resolve one binary (yes/no) uncertainty; a node with entropy 0.971 has nearly as much uncertainty as a perfectly balanced 50/50 split (entropy exactly 1.0).

Practice Question

Compute the entropy of a node with 8 samples: 2 of class A, 6 of class B, and compare it to the Gini impurity you'd compute for the same node.

Related ML Notes

• Gini Impurity
• Information Gain
• Probability Distributions