Decision Tree Regression

For regression, a decision tree splits to minimize variance within each resulting group instead of classification impurity — and each leaf predicts the simple average of the target values that landed in it.

Formula — Variance Reduction as the Split Criterion

This is exactly variance, the same formula covered in Statistics — a split is chosen to minimize the weighted average variance of the two resulting child nodes, the direct regression analogue of minimizing Gini impurity for classification.

Worked Example

House sizes and prices: \((800,35), (1000,42), (1200,48), (1500,60), (1800,72), (2000,80)\) — same dataset as KNN Regression's worked example.

Root node: all 6 prices \([35,42,48,60,72,80]\), mean \(=56.17\), variance \(\approx 259.5\).

Candidate split: "size < 1350?"

A large variance reduction (211.3) signals this is a genuinely useful split — the two resulting groups are far more internally consistent than the original mixed group.

Predicting From a Leaf

Once a new query point lands in a leaf during prediction, the tree returns the mean of the training targets that landed in that same leaf during training — for the "size ≥ 1350" leaf above, a new house of size 1600 would be predicted at \(70.67\) (the leaf's mean), regardless of whether it's 1400 or 1900 sq ft, as long as it falls in that same leaf.

from sklearn.tree import DecisionTreeRegressor
import numpy as np

X_train = np.array([[800],[1000],[1200],[1500],[1800],[2000]])
y_train = np.array([35, 42, 48, 60, 72, 80])

model = DecisionTreeRegressor(max_depth=2, random_state=42)
model.fit(X_train, y_train)

print(model.predict([[1300]]))   # predicts the mean price of whichever leaf 1300 sqft falls into
print(model.predict([[1900]]))    # a different leaf, likely predicting the higher-price group's mean

The Step-Function Nature of Tree Predictions

Unlike linear regression's smooth line, a regression tree's predictions form a step function — constant within each leaf's region, jumping abruptly at each split threshold. This means regression trees, unlike KNN regression's local averaging, produce identical predictions for every point within the same leaf, no matter how close or far each is from the threshold.

Practical Use Cases

• Regression problems with strong non-linear, threshold-like effects (e.g. a feature that matters a lot above a certain value but not below it)
• As the building block of Random Forest Regression and gradient boosting regressors

Common Mistakes

• Expecting smooth, continuous predictions the way linear regression produces — regression trees are inherently step functions.
• Growing a regression tree too deep, letting leaves shrink to very few samples — this overfits to noise in exactly the same way classification trees do.

Interview Relevance

Q: "Why do decision tree regression predictions look like a staircase instead of a smooth curve?" Because every point falling within the same leaf receives the identical prediction (that leaf's mean target value) — predictions can only change at a split threshold, producing constant, flat regions punctuated by jumps.

Practice Question

For the split above ("size < 1350?"), what would the tree predict for a 900 sq ft house and a 1700 sq ft house?

Related ML Notes

• Decision Tree Classification
• Variance
• Random Forest Regression