Feature importance ranks how much each feature actually contributed to a trained model's predictions — turning "the model works" into "the model works, and here's what it's actually paying attention to."
Why This Matters Beyond Curiosity
Feature importance isn't just for explaining a model after the fact — it's a practical tool during feature engineering itself: it tells you which engineered features were worth the effort, which raw features might be safe to drop, and whether a model is relying on a feature that shouldn't logically be predictive (a strong warning sign for data leakage).
Three Ways to Measure It
| Method | How It Works | Best For |
|---|---|---|
| Model coefficients | The learned weight on each feature (larger magnitude = more influence, on scaled features) | Linear/logistic regression |
| Built-in tree importance | How much each feature reduces impurity across all splits in the model | Decision Tree, Random Forest, XGBoost — see Random Forest Feature Importance |
| Permutation importance | Shuffle one feature's values and measure how much performance drops | Any model — see Permutation Importance |
A Quick Example — Coefficients vs Tree Importance
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
import pandas as pd
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names
log_model = LogisticRegression(max_iter=5000).fit(X, y)
importance_lr = pd.Series(log_model.coef_[0], index=feature_names).abs().sort_values(ascending=False)
print(importance_lr.head())
rf_model = RandomForestClassifier(random_state=42).fit(X, y)
importance_rf = pd.Series(rf_model.feature_importances_, index=feature_names).sort_values(ascending=False)
print(importance_rf.head())Important caveat: these two rankings frequently disagree, because they measure fundamentally different things — a linear coefficient reflects a feature's linear contribution, while tree importance reflects how useful a feature was for splitting, including any non-linear or interaction effects. Neither is "more correct" in general; they answer slightly different questions.
Why Raw Coefficient Magnitude Alone Can Mislead
A linear model's coefficient reflects the effect of a one-unit change in that feature — but "one unit" means very different things for "age in years" versus "income in rupees." Comparing raw coefficients across unscaled features is meaningless; always standardize features first if you intend to compare their coefficients as a measure of importance.
Practical Use Cases
- Deciding which engineered features actually earned their place in the final model
- Explaining model decisions to stakeholders in an interpretable, ranked way
- Sanity-checking a suspiciously high-performing model for leakage — an unexpectedly dominant feature is a red flag worth investigating
Limitations
- Correlated features can "split" their true importance between them, making each look individually less important than the underlying signal they jointly represent
- Tree-based importance can be biased toward high-cardinality numerical features, which offer more possible split points
Common Mistakes
- Comparing unscaled linear model coefficients directly as if their magnitudes were already comparable.
- Treating feature importance as proof of causation — a feature can be highly "important" to a model's predictions without being a genuine cause of the outcome.
- Only checking one importance method — coefficients and tree-based importance can tell meaningfully different stories about the same model.
Interview Relevance
Q: "Your Random Forest and Logistic Regression models rank feature importance very differently on the same data — why might that happen?" They measure different things — coefficients capture linear, additive contribution, while tree importance captures split-based usefulness, including non-linear effects and interactions the linear model can't represent at all.
Practice Question
A feature ranks as the single most important feature in your fraud model, but it's an internal database ID that shouldn't logically predict fraud. What should you suspect, and what would you check next?
