For regression, KNN predicts a continuous value by averaging the target values of the k nearest neighbors — the same "find similar points" idea as classification, with averaging replacing voting.
Formula
\(y_{(1)}, \dots, y_{(k)}\) are the target values of the \(k\) nearest neighbors. This is a simple, unweighted average by default — every one of the k neighbors contributes equally.
Worked Numerical Example
House sizes (sq ft) and prices (lakh): \((800, 35), (1000, 42), (1200, 48), (1500, 60), (1800, 72), (2000, 80)\). Query: predict the price for a 1300 sq ft house, using \(k=3\).
| Size | |1300 − size| | Price |
|---|---|---|
| 1200 | 100 | 48 |
| 1500 | 200 | 60 |
| 1000 | 300 | 42 |
| 800 | 500 | 35 |
| 1800 | 500 | 72 |
| 2000 | 700 | 80 |
The 3 nearest neighbors by size are 1200, 1500 and 1000 sq ft — averaging their prices gives a prediction of ₹50 lakh for the 1300 sq ft house.
from sklearn.neighbors import KNeighborsRegressor
import numpy as np
X_train = np.array([[800],[1000],[1200],[1500],[1800],[2000]])
y_train = np.array([35, 42, 48, 60, 72, 80])
model = KNeighborsRegressor(n_neighbors=3)
model.fit(X_train, y_train)
print(model.predict([[1300]])) # [50.] -- matches the hand calculation exactlyDistance-Weighted Regression
model_weighted = KNeighborsRegressor(n_neighbors=3, weights="distance")
model_weighted.fit(X_train, y_train)
print(model_weighted.predict([[1300]]))
# Closer neighbors (1200, distance 100) pull the prediction more than farther ones (1000, distance 300)With weights="distance", the 1200 sq ft neighbor (distance 100, very close) has more influence than the 1000 sq ft neighbor (distance 300, farther) — usually a more accurate approach than treating all k neighbors identically.
Practical Use Cases
- Price estimation based on similar past sales (real estate, used cars)
- Any regression problem where "similar inputs should have similar outputs" is a reasonable, defensible assumption
Advantages and Limitations, Specific to Regression
- Naturally captures local, non-linear relationships without needing explicit polynomial features
- Predictions are always bounded within the range of observed target values in the neighborhood — KNN regression can never extrapolate beyond the training data's value range, unlike linear regression
Common Mistakes
- Expecting KNN regression to extrapolate sensibly beyond the range of training data — since it only averages nearby observed values, it can't predict a value outside what it's already seen nearby.
- Using unweighted averaging on data where the nearest neighbor is dramatically closer than the k-th nearest — weighted averaging is usually the better default.
Interview Relevance
Q: "Can KNN regression predict a value higher than any value seen in training?" No — since predictions are always an average (weighted or not) of observed neighbor values, the output is mathematically bounded within the range of the training targets, unlike linear regression which can extrapolate arbitrarily far.
Practice Question
Using the house price table above, predict the price for a 1600 sq ft house with k=3, showing your distance calculations.
