A Random Forest trains many decision trees on randomized versions of the data and features, then combines their predictions — turning a collection of individually noisy, overfitting-prone trees into a single, far more stable and accurate model.
The Core Idea — "Wisdom of Crowds," Made Precise
A single decision tree overfits easily, as covered in Decision Tree Overfitting — it can memorize noise in its specific training set. Random Forest's insight isn't to prevent any individual tree from overfitting; it's to train many trees, each seeing a different random slice of the data and features, so each tree's noise is different and uncorrelated. Averaging their predictions cancels out most of that individual noise, while the genuine signal — present in every tree, since it's real — survives the averaging intact.
The Two Sources of Randomness
| Randomization | What It Does |
|---|---|
| Bagging (Bootstrap Aggregating) | Each tree trains on a random sample of rows, drawn with replacement from the original data |
| Random feature subsets | At each split, only a random subset of features is considered — not all of them |
Both forms of randomness push the individual trees to be different from each other — see How Random Forest Works for the full mechanics.
Ensemble Voting, Visually
5 differently-trained trees vote 3-to-2 for class 1 — each tree's individual mistakes are different, so they don't all agree, and the majority tends to be right more often than any single tree.
Minimal Working Example
from sklearn.ensemble import RandomForestClassifier
import numpy as np
X_train = np.array([[1,0],[2,0],[2,1],[3,1],[4,1],[5,2],[6,2],[7,3]])
y_train = np.array([0,0,0,0,1,1,1,1])
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
print(model.predict([[4, 2]])) # majority vote across 100 trees
print(model.predict_proba([[4, 2]])) # fraction of trees voting each classPractical Use Cases
- Tabular business data — credit risk, churn, fraud — where Random Forest is a strong, low-maintenance default
- Situations needing a reasonable balance of accuracy and interpretability (via feature importance), without the full complexity of gradient boosting
Advantages
- Far more resistant to overfitting than a single decision tree, with little manual tuning required
- Handles non-linear relationships and feature interactions natively
- No feature scaling required, same as individual trees
- Provides a useful, built-in feature importance ranking
Limitations
- Loses the direct interpretability of a single decision tree — you can't trace one clean reasoning path anymore
- Slower to train and predict than a single tree, and uses more memory (many trees stored)
- Usually outperformed by gradient boosting methods (XGBoost, LightGBM) on many tabular benchmarks, though it remains a strong, lower-effort baseline
Common Mistakes
- Assuming more trees (
n_estimators) always meaningfully improves accuracy — returns diminish quickly past a few hundred trees for most datasets, while training time keeps growing. - Expecting to explain an individual prediction as easily as with a single decision tree — see Explainable AI for the tools built specifically for this gap.
Interview Relevance
Q: "Why does averaging many overfit trees produce a better model than any single tree?" Each tree overfits to different noise, because each sees a different random sample of rows and considers different random feature subsets at each split — averaging cancels out noise that varies randomly across trees, while genuine signal (present consistently across trees) survives the averaging.
Practice Question
Explain in your own words why training 100 identical decision trees on the exact same data (no randomization at all) would provide no benefit over a single tree.
