Dimensionality reduction compresses a dataset's many features into fewer, while preserving as much of the meaningful structure as possible — useful for visualization, faster training, noise reduction, and fighting the curse of dimensionality.
Why Fewer Dimensions Can Mean More Signal
Real datasets often have redundant or correlated features — as covered in Vector Space, two nearly-parallel feature vectors (like temperature in Celsius and Fahrenheit) add almost no new information despite counting as two separate columns. Dimensionality reduction finds the handful of directions that actually carry the data's real structure, discarding the rest.
The Two Main Families
| Family | How It Works | Full Notes |
|---|---|---|
| Linear | Finds new features as linear combinations of the originals | PCA |
| Non-linear (manifold learning) | Preserves local neighborhood structure, even when it's curved/non-linear | t-SNE, UMAP |
Minimal Working Example
from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
data = load_digits() # 64 features (8x8 pixel images)
print(data.data.shape) # (1797, 64)
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(data.data)
print(X_reduced.shape) # (1797, 2) -- compressed from 64 to 2 dimensions
print(pca.explained_variance_ratio_.sum()) # how much of the original information survivedPractical Use Cases
- Visualizing high-dimensional data in 2D/3D during EDA
- Speeding up training for algorithms sensitive to feature count
- Reducing noise by discarding low-variance directions
- Preprocessing before distance-based algorithms (like KNN) on high-dimensional data
Common Mistakes
- Applying dimensionality reduction before splitting into train/test — like any fitted preprocessing step, it should be fit on training data only.
- Assuming dimensionality reduction always improves downstream model accuracy — sometimes it does, but it's also a genuine loss of information, and tree-based models in particular often don't need it at all.
- Confusing dimensionality reduction (creating new, combined features) with feature selection (keeping a subset of the original features) — see PCA vs Feature Selection.
Interview Relevance
Q: "Why would you reduce dimensionality before applying KNN to a 500-feature dataset?" To fight the curse of dimensionality — in very high-dimensional space, distances between points become less discriminating, so KNN's core "nearest neighbor" concept breaks down; reducing to the handful of dimensions that actually carry signal restores meaningful distance comparisons.
Practice Question
You have a dataset with 200 highly correlated numeric features. Would you reach for dimensionality reduction or feature selection first, and why might you eventually want both?
