Boosting builds an ensemble one model at a time, sequentially — each new model deliberately focuses on the mistakes the ensemble-so-far is still making, rather than being trained independently like bagging.
The General Boosting Pattern
| Step | What Happens |
|---|---|
| 1 | Train a simple "weak learner" (often a shallow tree) on the data |
| 2 | Identify what it got wrong — either by reweighting misclassified points (AdaBoost) or by fitting the next model to the residual errors (Gradient Boosting) |
| 3 | Train the next weak learner specifically targeting those mistakes |
| 4 | Combine all weak learners into a final, weighted prediction |
| 5 | Repeat for a set number of rounds, or until performance stops improving |
The Two Main Boosting Strategies
| Strategy | How It Corrects Mistakes | Full Note |
|---|---|---|
| Adaptive (AdaBoost) | Increases the weight of misclassified points, so the next learner focuses on them harder | AdaBoost |
| Gradient-based | Fits the next model directly to the current residual errors (the negative gradient of the loss) | Gradient Boosting |
Why "Weak" Learners, Specifically
Boosting deliberately uses simple, high-bias models (often decision "stumps" — trees with just one split) as its building blocks. A single weak learner barely does better than random guessing on its own — but combining many of them, each correcting the last's specific mistakes, produces a strong, accurate combined predictor. This is the opposite intuition from bagging, which works best with already-strong (low-bias, high-variance) base learners.
Minimal Working Example
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
# A "weak learner" -- a decision stump, max_depth=1
weak_learner = DecisionTreeClassifier(max_depth=1)
model = AdaBoostClassifier(estimator=weak_learner, n_estimators=50, random_state=42)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))Practical Use Cases
- Tabular data problems where achieving the highest possible accuracy matters more than training speed or simplicity
- Situations where different subsets of the data are systematically harder to predict, benefiting from adaptive, targeted correction
Advantages
- Often achieves higher accuracy than bagging on many tabular problems
- Directly reduces bias, addressing systematic underfitting in a way bagging alone can't
Limitations
- Inherently sequential — can't parallelize across rounds the way bagging can
- More prone to overfitting if trained for too many rounds without regularization
- More sensitive to noisy data and outliers — a mislabeled point can get progressively more weight/focus round after round
Common Mistakes
- Training for far more rounds than needed without early stopping or validation monitoring, risking overfitting to training-set noise.
- Using an already-strong, low-bias base learner (like a deep, unpruned tree) for boosting — this often overfits quickly, since boosting's benefit comes from repeatedly correcting a genuinely weak learner's bias.
Interview Relevance
Q: "Why does boosting use weak learners instead of strong ones?" A strong learner already fits the data well, leaving little systematic bias for subsequent rounds to correct — and combining many already-strong, similar models adds little (that's bagging's territory). Boosting's power comes specifically from sequentially chaining together many high-bias, low-variance weak learners, each fixing a distinct piece of the remaining error.
Practice Question
Explain why boosting is inherently harder to parallelize across multiple CPU cores than bagging.
