Gradient Boosting builds its sequence of models differently from AdaBoost — instead of reweighting misclassified points, each new model is trained to directly predict the residual errors of the ensemble built so far.
The Core Idea — Fitting Residuals
| Step | What Happens |
|---|---|
| 1 | Start with a simple initial prediction — often just the mean of the target (for regression) |
| 2 | Compute the residuals: \(r_i = y_i - \hat{y}_i\) (actual minus current prediction) |
| 3 | Train a new weak learner (usually a shallow tree) to predict these residuals |
| 4 | Update predictions: \(\hat{y} \leftarrow \hat{y} + \eta \cdot \text{new tree's prediction}\), where \(\eta\) is the learning rate |
| 5 | Recompute residuals using the updated predictions, repeat from step 3 |
"Gradient" refers to the fact that, for squared-error loss, the residual \(y_i-\hat{y}_i\) is exactly the negative gradient of the loss function with respect to the prediction — so "fit the next tree to the residuals" is mathematically "fit the next tree to the negative gradient," which generalizes cleanly to other loss functions beyond squared error (like log-loss for classification).
Worked Example — Two Rounds
4 target values: \(y=[10,20,30,40]\). Initial prediction (mean): \(\hat{y}_0 = 25\) for every point.
| Point | y | Initial ŷ | Residual |
|---|---|---|---|
| 1 | 10 | 25 | -15 |
| 2 | 20 | 25 | -5 |
| 3 | 30 | 25 | 5 |
| 4 | 40 | 25 | 15 |
Suppose a shallow tree fit to these residuals predicts \(-10\) for points 1-2 and \(+10\) for points 3-4 (a simple one-split tree). With learning rate \(\eta=0.5\):
New residuals shrink: point 1's residual goes from \(-15\) to \(10-20=-10\) — smaller in magnitude, meaning the ensemble is getting closer to the true values with each round.
from sklearn.ensemble import GradientBoostingRegressor
import numpy as np
X = np.array([[1],[2],[3],[4]])
y = np.array([10, 20, 30, 40])
model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=1, random_state=42)
model.fit(X, y)
print(model.predict(X)) # converges close to the true y values over many small, additive stepsWhy the Learning Rate Matters So Much
A smaller \(\eta\) means each tree contributes only a small correction — slower convergence, but typically better generalization, since no single tree gets to dominate the final prediction. A larger \(\eta\) converges faster but risks overfitting, similar in spirit to the learning rate tradeoff in plain gradient descent. In practice, \(\eta\) and \(n\_estimators\) are tuned together — a smaller learning rate typically needs more rounds to compensate.
Practical Use Cases
- Tabular regression and classification where top-tier predictive accuracy is the priority
- The foundational algorithm behind XGBoost, LightGBM and CatBoost — all three are optimized, extended implementations of this same core idea
Common Mistakes
- Setting a high learning rate with many estimators — often overfits severely; lower learning rate with more estimators is the more common, more robust combination.
- Not using early stopping (monitoring validation loss and halting once it stops improving) — plain gradient boosting will happily keep fitting training-set noise indefinitely if allowed to.
Interview Relevance
Q: "How does Gradient Boosting differ from AdaBoost in how it corrects errors?" AdaBoost reweights training points, forcing subsequent weak learners to focus more on previously misclassified examples; Gradient Boosting instead has each new model directly predict the current residual errors (the negative gradient of the loss), additively refining the ensemble's predictions round by round.
Practice Question
Using the worked example, compute the residual for point 4 after the first boosting round (learning rate 0.5, tree predicting +10 for points 3-4).
