KNN Distance

KNN's entire prediction hinges on one question: how do you measure "closeness" between two points? The distance metric you choose directly determines which neighbors get consulted — and therefore what the model predicts.

Euclidean Distance — The Default

This is the ordinary straight-line distance — the Pythagorean theorem, extended to \(n\) dimensions. For 2D points \(x=(2,3)\) and \(y=(6,6)\):

Geometric Intuition

Manhattan Distance — The Alternative

Instead of a straight diagonal line, Manhattan distance sums the absolute differences along each axis — like navigating city blocks instead of flying directly. For the same points: \(d = |6-2|+|6-3| = 4+3 = 7\).

Minkowski Distance — The General Formula

This is a family that includes both special cases: \(p=2\) gives Euclidean distance, \(p=1\) gives Manhattan distance. scikit-learn's KNN implementation defaults to Minkowski with \(p=2\) — i.e., ordinary Euclidean distance.

Python Implementation

import numpy as np
from scipy.spatial import distance

a = np.array([2, 3])
b = np.array([6, 6])

print(distance.euclidean(a, b))    # 5.0
print(distance.cityblock(a, b))     # 7.0 -- Manhattan/cityblock distance
print(distance.minkowski(a, b, p=2))  # 5.0 -- same as Euclidean when p=2
print(distance.minkowski(a, b, p=1))  # 7.0 -- same as Manhattan when p=1

Why the Choice of Metric Matters

Euclidean distance is sensitive to the overall magnitude of differences across all dimensions simultaneously; Manhattan distance treats each dimension's contribution independently and can be more robust to outliers in a single dimension, since it doesn't square (and thereby amplify) large individual differences. For high-dimensional data, both distances tend to become less discriminating — see the curse of dimensionality.

Why Scaling Is Non-Negotiable for Any Distance Metric

# WITHOUT scaling: income (thousands) completely dominates age (tens)
person_a = [25, 40000]   # age, income
person_b = [55, 41000]

import numpy as np
d = np.sqrt((25-55)**2 + (40000-41000)**2)
print(d)   # ~1000.4 -- the 30-year age gap barely registers next to the 1000-unit income gap

This is precisely why feature scaling is described as non-negotiable for KNN — without it, whichever feature happens to have the largest numeric range silently dominates every distance calculation, regardless of its actual real-world importance.

Common Mistakes

• Using Euclidean distance on unscaled features, letting one large-range feature dominate.
• Applying Euclidean/Manhattan distance to categorical data without proper encoding first — raw category codes have no meaningful numeric distance between them.

Interview Relevance

Q: "Why does KNN require feature scaling, but a decision tree doesn't?" KNN's predictions come directly from distance calculations across all features simultaneously, so an unscaled feature with a large numeric range dominates the distance; a decision tree splits on one feature's threshold at a time, so relative scale between different features never affects its split decisions.

Practice Question

Compute both the Euclidean and Manhattan distance between points \((0,0)\) and \((3,4)\) by hand.

Related ML Notes

• Vectors
• Feature Scaling
• How KNN Works