PCA (Principal Component Analysis)

PCA (Principal Component Analysis) finds the directions along which data varies the most, and re-expresses the data in terms of those directions — the most widely used dimensionality reduction technique, built directly on eigenvalues and eigenvectors.

The Core Idea

Instead of the original (often arbitrary) feature axes, PCA finds new axes — principal components — aligned with the directions of maximum variance in the data. The first principal component (PC1) captures the most variance possible in a single direction; PC2 captures the most remaining variance, subject to being perpendicular to PC1; and so on.

Formula

\(\Sigma\) is the data's covariance matrix. The principal components are exactly its eigenvectors \(v\), and each corresponding eigenvalue \(\lambda\) tells you how much variance that component captures — this is precisely the eigenvalue equation from Math for ML, applied specifically to a covariance matrix.

Geometric Intuition

Worked Numerical Example

4 mean-centered points: \((2,1),(1,2),(-1,-2),(-2,-1)\).

Solving \(\det(\Sigma-\lambda I)=0\): \((3.333-\lambda)^2 - 2.667^2 = 0 \Rightarrow \lambda = 3.333 \pm 2.667\), giving \(\lambda_1 = 6.0\) and \(\lambda_2 = 0.667\).

The eigenvector for \(\lambda_1=6.0\) works out to the direction \([1,1]\) (normalized: \([0.707, 0.707]\)) — matching the diagram, since the data clearly spreads along the \(y=x\) diagonal.

import numpy as np
from sklearn.decomposition import PCA

X = np.array([[2,1],[1,2],[-1,-2],[-2,-1]])

pca = PCA(n_components=2)
pca.fit(X)

print(pca.explained_variance_)         # [6.0, 0.667] -- matches the hand-solved eigenvalues
print(pca.explained_variance_ratio_)    # [0.9, 0.1]
print(pca.components_)                   # [[0.707, 0.707], [0.707, -0.707]] -- the eigenvectors

See PCA Step by Step for this exact computation done fully by hand, using the same underlying algorithm scikit-learn runs internally.

Why Keeping Only the Top Components Works

Since PC1 alone captures 90% of the variance here, projecting the data onto just PC1 (reducing from 2D to 1D) loses only about 10% of the total spread — often an acceptable tradeoff for the benefits of a lower-dimensional representation.

Practical Use Cases

• Compressing highly correlated numeric features before modeling
• Visualizing high-dimensional data in 2D by keeping just the top 2 components
• Noise reduction — later, lower-variance components often correspond more to noise than genuine signal

Advantages

• Mathematically well-understood, deterministic, and fast to compute
• Components are guaranteed orthogonal (uncorrelated with each other)
• Explained variance ratio gives a direct, quantifiable sense of how much information was kept

Limitations

• Only captures linear relationships — see t-SNE for non-linear structure
• Principal components are linear combinations of original features, making them harder to interpret directly than the raw features
• Sensitive to feature scale — always standardize before PCA

Common Mistakes

• Running PCA on unscaled features — a feature with a naturally larger numeric range will dominate the variance calculation regardless of its true importance.
• Trying to interpret a principal component as if it were a single original feature — it's a weighted mix of all of them.

Interview Relevance

Q: "Why must you scale features before applying PCA?" PCA finds directions of maximum variance — a feature measured in a naturally larger range (like income vs age) will have artificially larger variance purely due to units, dominating the principal components regardless of its true informational importance, unless every feature is put on a comparable scale first.

Practice Question

If a dataset's first two principal components explain 60% and 25% of variance respectively, how much information is lost by keeping only those two components?

Related ML Notes

• Eigenvalues
• PCA Step by Step
• PCA in Python