Ensemble learning combines multiple models into one stronger predictor — the same "many imperfect opinions averaged together beat any single opinion" principle behind Random Forest, generalized to many more combination strategies and base model types.
The Four Main Families
| Family | Core Idea | Full Note |
|---|---|---|
| Bagging | Train many models in parallel on random data subsets, average/vote | Random Forest is the classic example |
| Boosting | Train models sequentially, each correcting the previous ones' mistakes | AdaBoost, Gradient Boosting, XGBoost |
| Stacking | Train a meta-model to learn how to best combine several different base models | — |
| Voting | Simple, fixed combination rule (majority vote or averaged probabilities) across different algorithms | — |
Bagging vs Boosting — The Fundamental Split
Bagging's models never see each other's output — they're purely independent, combined only at the end. Boosting's models are built one after another, each specifically targeting what the previous ones got wrong.
Why This Distinction Actually Matters
| Bagging | Boosting | |
|---|---|---|
| Primarily reduces | Variance | Bias |
| Base learners | Usually deep, low-bias, high-variance trees | Usually shallow, high-bias, low-variance "weak learners" |
| Parallelizable? | Yes, fully | No — inherently sequential |
| Overfitting risk | Lower | Higher, if trained for too many rounds |
Practical Use Cases
- Tabular data competitions and production systems, where XGBoost-family boosting is often the top-performing approach
- Combining fundamentally different model types (a tree, a linear model, an SVM) via stacking or voting when they make different kinds of mistakes
Common Mistakes
- Assuming any ensemble automatically beats a single well-tuned model — ensembles add real value when base models are reasonably accurate and make different kinds of errors; ensembling several near-identical, highly correlated models adds little.
- Confusing bagging and boosting's failure modes — bagging is fairly overfitting-resistant by design, while boosting can overfit if run for too many rounds without regularization.
Interview Relevance
Q: "Why does boosting typically reduce bias while bagging reduces variance?" Boosting's base learners are deliberately weak (high bias) and trained sequentially to correct remaining errors — repeatedly targeting bias directly; bagging's base learners are typically already low-bias but high-variance, and averaging many independently-trained ones cancels out variance without touching bias much.
Practice Question
You have three different classifiers (a logistic regression, a decision tree, and an SVM) that each perform reasonably well but make different mistakes. Which ensemble family would you reach for first, and why?
