SHAP

SHAP (SHapley Additive exPlanations) assigns each feature a precise contribution to a specific prediction, grounded in Shapley values — a concept from cooperative game theory with real mathematical guarantees, not just a heuristic.

The Game Theory Behind It

Imagine features as "players" cooperating to produce a prediction (the "payout"). A Shapley value fairly divides that payout among the players by averaging each feature's marginal contribution across every possible order in which features could be added to the prediction — accounting for the fact that a feature's contribution can depend on which other features are already present.

Formula

\(\phi_i\) is feature \(i\)'s Shapley value. \(F\) is the full set of features, \(S\) is a subset not containing feature \(i\), and \(v(S)\) is the model's prediction using only the features in \(S\). This sums feature \(i\)'s marginal contribution across every possible subset it could be added to, weighted appropriately.

Worked Example — Two Features

A model's average output with no features (baseline) is 50. Using feature A alone: 60. Using feature B alone: 55. Using both: 72.

Each Shapley value averages the feature's contribution across the two possible "orders" it could be added (first, or second, after the other). Check the efficiency property: \(\phi_A + \phi_B = 13.5+8.5 = 22 = 72-50\) — the Shapley values exactly and completely account for the full difference between the baseline and final prediction, with nothing left over.

Python Implementation

import shap
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=200, random_state=42).fit(X_train, y_train)

explainer = shap.TreeExplainer(model)   # optimized for tree-based models
shap_values = explainer.shap_values(X_test)

# Local: explain ONE specific prediction
shap.plots.waterfall(shap_values[0])

# Global: aggregate across all predictions for overall feature importance
shap.summary_plot(shap_values, X_test)

TreeExplainer is a specialized, efficient version for tree-based models; KernelExplainer works on any model but is significantly slower, since it approximates the Shapley calculation via sampling rather than exploiting tree structure directly.

Why SHAP Is Both Global and Local

A SHAP value exists for every feature, for every individual prediction — that's inherently a local explanation. But averaging the absolute SHAP values across many predictions produces a global feature importance ranking — this dual nature is exactly why SHAP appears in both roles in Global vs Local Explanations.

Practical Use Cases

• Explaining an individual high-stakes prediction (loan denial, medical risk score) with mathematically grounded, additive feature contributions
• Both global feature ranking AND per-prediction explanation from a single, consistent framework

Common Mistakes

• Using the slow, model-agnostic KernelExplainer when a faster, model-specific explainer (like TreeExplainer) is available for the model type in use.
• Interpreting SHAP values as proof of real-world causation rather than the model's learned association.

Interview Relevance

Q: "What does the 'efficiency property' of Shapley values guarantee?" The sum of every feature's Shapley value exactly equals the difference between the model's prediction for a specific instance and its average baseline prediction — no contribution is left unaccounted for and none is double-counted, a genuine mathematical guarantee, not just an approximate heuristic.

Practice Question

For a 2-feature model with baseline=100, feature A alone=130, feature B alone=110, both=150, compute the Shapley value for each feature and verify they sum to 50.

Related ML Notes

• LIME
• Global vs Local Explanations
• Permutation Importance