Bagging (Bootstrap Aggregating) trains many copies of a model on random bootstrap samples of the data, then averages or votes across their predictions — the general technique Random Forest applies specifically to decision trees, but usable with any base model.
The Variance-Reduction Formula
For \(n\) models each with prediction variance \(\sigma^2\), correlated with each other by \(\rho\): if the models were completely independent (\(\rho=0\)), variance of their average would shrink all the way to \(\sigma^2/n\) — the more models, the better. But real bagged models are somewhat correlated (they're trained on overlapping bootstrap samples of the same data), so variance only shrinks toward the floor \(\rho\sigma^2\) — it can never go below that, no matter how many models you add.
This formula directly explains why Random Forest adds a second randomization (random feature subsets per split) on top of bagging alone — it further reduces \(\rho\) (the correlation between trees), pushing that variance floor lower than bagging alone could achieve.
Bagging Applied to Any Base Model
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)
# Bagging works with ANY base estimator, not just trees
bagged_trees = BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=42)
bagged_trees.fit(X_train, y_train)
print("Bagged trees:", bagged_trees.score(X_test, y_test))
bagged_logreg = BaggingClassifier(LogisticRegression(max_iter=1000), n_estimators=100, random_state=42)
bagged_logreg.fit(X_train, y_train)
print("Bagged logistic regression:", bagged_logreg.score(X_test, y_test))Why Bagging Helps High-Variance Models Much More Than Low-Variance Ones
Decision trees (especially deep, unpruned ones) are naturally high-variance — small data changes produce very different trees, exactly the situation bagging's variance formula addresses well. Logistic regression is already fairly low-variance (stable coefficients across similar datasets), so bagging typically helps it far less — there's simply less variance left to reduce. This is precisely why bagging became famous specifically through Random Forest (bagged trees), not bagged linear models.
Practical Use Cases
- Stabilizing any inherently high-variance, unstable base model
- The foundation underneath Random Forest specifically
Common Mistakes
- Applying bagging to an already low-variance, stable model and expecting a large improvement — there's little to reduce.
- Forgetting that bagging alone (without also randomizing features per split) still leaves base learners more correlated than Random Forest's fuller randomization achieves.
Interview Relevance
Q: "Why does bagging help decision trees so much more than it helps logistic regression?" Decision trees are naturally high-variance — unstable to small changes in training data — which is exactly what averaging many bagged versions cancels out; logistic regression's coefficients are already comparatively stable (low variance), so there's much less variance for bagging to reduce.
Practice Question
If two bagged models have correlation \(\rho=0.3\) and individual variance \(\sigma^2=4\), what's the theoretical floor their averaged variance can never go below, no matter how many models are added?
