Random Forest's built-in feature importance ranks each feature by how much it reduced impurity across every split, in every tree — a genuinely useful, nearly-free byproduct of training that doesn't require any extra computation step.
How It's Computed — Mean Decrease in Impurity (MDI)
For every split that uses feature \(f\), in every tree, record how much that split reduced impurity (Gini or entropy), weighted by how many samples passed through that split. Sum these contributions for feature \(f\) across all splits and all trees, then average across the \(m\) trees. Features that are chosen for high-impact splits often, across many trees, end up with high importance.
Python Implementation
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
import pandas as pd
import matplotlib.pyplot as plt
data = load_breast_cancer()
model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(data.data, data.target)
importances = pd.Series(model.feature_importances_, index=data.feature_names).sort_values(ascending=False)
print(importances.head(10))
importances.head(10).plot(kind="barh")
plt.gca().invert_yaxis()
plt.xlabel("Importance (mean decrease in impurity)")
plt.show()Expected output: importance values that all sum to 1.0 across every feature — each one represents that feature's relative share of the total impurity reduction achieved by the whole forest.
A Known Bias — Why MDI Can Mislead
Mean Decrease in Impurity has a real, well-documented bias: it tends to favor features with many unique values (high-cardinality numeric features, or categorical features with many categories) — simply because such features offer more possible split points to choose from, mechanically increasing their chance of appearing in a high-impact split, independent of their true predictive value.
import numpy as np
# Demonstrating the bias: add a completely random, high-cardinality feature
np.random.seed(0)
X_with_noise = np.column_stack([data.data, np.random.rand(len(data.data), 1) * 1000])
model_noise = RandomForestClassifier(n_estimators=200, random_state=42)
model_noise.fit(X_with_noise, data.target)
print(model_noise.feature_importances_[-1]) # the pure-noise feature often still gets NON-trivial importanceThis is exactly why permutation importance is often recommended alongside (or instead of) MDI — it measures importance by directly shuffling a feature and observing the resulting drop in actual model performance, which isn't biased toward high-cardinality features the same way.
Practical Use Cases
- A fast first-pass feature ranking during feature selection
- Sanity-checking a model — an unexpectedly dominant feature can be an early signal of data leakage
- Communicating, at a high level, which factors a model relies on most
Common Mistakes
- Treating feature importance as proof of causation — a feature can be "important" to the model's predictions without being a genuine cause of the target.
- Comparing MDI importance across features with very different cardinalities without accounting for the bias described above.
- Not checking permutation importance as a cross-check when MDI produces a surprising ranking.
Interview Relevance
Q: "Why can Random Forest's built-in feature_importances_ be misleading for high-cardinality features?" Mean Decrease in Impurity is computed from how often and how impactfully a feature is chosen for splits — a feature with many unique values simply has more candidate split points available, mechanically inflating its apparent importance even if it's not genuinely more predictive; permutation importance avoids this specific bias.
Practice Question
Your Random Forest ranks a customer ID-like column as the second most important feature. What should you suspect, and what would you check next?
