Random Forest Regression

For regression, Random Forest predicts by averaging every tree's individual prediction — the same ensemble idea as classification, with a mean replacing a vote.

Formula

\(T_i(x)\) is regression tree \(i\)'s predicted value (the mean of its leaf, as covered in Decision Tree Regression), and \(m\) is the number of trees.

Worked Example

House prices from a small forest of 5 trees, each trained on a different bootstrap sample, predicting for the same 1300 sq ft house:

Notice this averages out the individual trees' disagreement (ranging from 45 to 55) into a single, more stable estimate of 50 — the same variance-reduction principle as averaging repeated samples in statistics.

Python Implementation

from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score
import numpy as np

X_train = np.array([[800],[1000],[1200],[1500],[1800],[2000]])
y_train = np.array([35, 42, 48, 60, 72, 80])

model = RandomForestRegressor(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

print(model.predict([[1300]]))   # averaged prediction across all 200 trees

# Inspecting individual trees' predictions
individual_preds = [tree.predict([[1300]])[0] for tree in model.estimators_]
print(np.mean(individual_preds), np.std(individual_preds))   # the mean IS the forest's prediction

Why Random Forest Regression Smooths Out the "Staircase"

A single regression tree produces a blocky step function (see Decision Tree Regression). Averaging many trees, each with slightly different split thresholds (due to different bootstrap samples), smooths those sharp steps into a much more gradual, continuous-looking prediction curve — even though every individual contributing tree is still a step function underneath.

Practical Use Cases

• House price, sales, and demand forecasting on tabular data with non-linear relationships
• Any regression problem where a single tree's instability is a concern but full interpretability isn't required

Common Mistakes

• Expecting Random Forest regression to extrapolate beyond the training data's target range — like individual trees, it still only averages observed leaf values, so it can't predict outside what it's seen.
• Not checking model.estimators_'s individual predictions when debugging an unexpected forest output — inspecting the spread across trees can reveal whether the forest is confidently agreeing or actually quite uncertain.

Interview Relevance

Q: "Can a Random Forest regressor predict a value never seen in the training targets, like extrapolating beyond the max observed price?" No — since every tree's leaf prediction is a mean of observed training targets, and the forest averages those means, the output is mathematically bounded within the training data's target range, same limitation as a single regression tree.

Practice Question

A forest of 4 trees predicts \([60, 65, 58, 100]\) for the same input. Compute the forest's final prediction, and comment on whether the spread of individual tree predictions is concerning here.

Related ML Notes

• Random Forest Classification
• Decision Tree Regression
• Mean Absolute Error