Stacking

Stacking trains a "meta-model" to learn the best way to combine several different base models' predictions — rather than using a fixed rule like averaging or majority vote, it learns the combination directly from data.

The Architecture

Why Stacking Can Beat a Fixed Voting Rule

A voting classifier combines models with a fixed rule (e.g. simple average) — every base model gets equal say, everywhere. Stacking's meta-model can learn a more nuanced combination: "trust the SVM more when the logistic regression and random forest disagree," or "downweight the tree-based model specifically for this feature region" — patterns a fixed rule structurally can't express.

Python Implementation

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC

base_models = [
    ("logreg", LogisticRegression(max_iter=1000)),
    ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
    ("svm", SVC(probability=True)),
]

meta_model = LogisticRegression()   # often kept simple, to avoid overfitting on top of base predictions

stacked_model = StackingClassifier(estimators=base_models, final_estimator=meta_model, cv=5)
stacked_model.fit(X_train, y_train)
print(stacked_model.score(X_test, y_test))

The cv=5 parameter is important: scikit-learn trains base models using cross-validation internally, so the meta-model learns from out-of-fold base predictions — preventing the meta-model from learning to over-trust a base model just because it saw and memorized the same training data.

Practical Use Cases

• Machine learning competitions, where squeezing out extra accuracy by combining diverse model types is a common winning strategy
• Combining genuinely different model families (linear, tree-based, distance-based) that make different kinds of errors

Advantages

• Can outperform any single base model or a fixed-rule ensemble, when base models are diverse
• Learns the combination rule from data instead of relying on a hand-picked one

Limitations

• More complex to build, tune and maintain than bagging or a simple voting ensemble
• Slower — every prediction requires running all base models plus the meta-model
• Higher overfitting risk if not using proper cross-validation to generate the meta-model's training data

Common Mistakes

• Training the meta-model on base models' predictions from the same data those base models were trained on — this leaks information and inflates the meta-model's apparent performance; proper out-of-fold predictions (as cv=5 handles) are essential.
• Using highly correlated base models (e.g. three near-identical tree-based models) — stacking adds the most value when base models genuinely disagree in different situations.

Interview Relevance

Q: "Why is cross-validation used when generating training data for a stacking meta-model?" To avoid leakage — if a base model's predictions on its own training data are used to train the meta-model, the meta-model learns to over-trust predictions the base model has essentially memorized rather than genuinely generalized, inflating stacked performance in a way that won't hold on new data.

Practice Question

You have three base models that all make very similar predictions on most inputs. Would stacking likely provide much benefit over simply averaging them? Explain.

Related ML Notes

• Voting Classifier
• Ensemble Learning
• Cross-Validation