PCA Step by Step

The complete PCA algorithm, broken into explicit, hand-computable steps — from raw data to the final reduced representation, so nothing about "how PCA actually works" is left as a black box.

The Algorithm, As Explicit Steps

Full Worked Example

Starting data (already mean-centered, so step 1's mean-subtraction is already done): \((2,1),(1,2),(-1,-2),(-2,-1)\).

Step 2 — Covariance matrix (sample covariance, dividing by \(n-1=3\)):

Step 3 — Eigenvalues and eigenvectors (solved in PCA): \(\lambda_1=6.0\) with eigenvector \(v_1=[0.707, 0.707]\); \(\lambda_2=0.667\) with eigenvector \(v_2=[0.707,-0.707]\).

Step 4 — Sort: \(\lambda_1 > \lambda_2\), so \(v_1\) is PC1, \(v_2\) is PC2 — already in order here.

Step 5 — Project: to reduce to 1D, project each point onto \(v_1\) using the dot product:

import numpy as np

X = np.array([[2,1],[1,2],[-1,-2],[-2,-1]])

# Step 2: covariance matrix
cov_matrix = np.cov(X, rowvar=False)
print(cov_matrix)

# Step 3: eigen-decomposition
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print(eigenvalues, eigenvectors)

# Step 4: sort descending
order = np.argsort(eigenvalues)[::-1]
eigenvalues, eigenvectors = eigenvalues[order], eigenvectors[:, order]

# Step 5: project onto the top component (PC1)
pc1 = eigenvectors[:, 0]
projected_1d = X @ pc1
print(projected_1d)   # e.g. [2.121, 2.121, -2.121, -2.121] -- matches the hand calculation

Why Every Step Exists

• Standardization ensures no feature dominates purely due to its scale (see PCA's common mistake section)
• Covariance matrix summarizes how every pair of features varies together — the raw material PCA analyzes
• Eigen-decomposition finds the directions (eigenvectors) and magnitudes (eigenvalues) of variance embedded in that covariance structure
• Sorting ensures you keep the most informative directions first when reducing dimensions
• Projection is the actual dimensionality reduction step — re-expressing each point in terms of the new, fewer axes

Practical Use Cases

• Understanding exactly what PCA().fit_transform() computes internally, useful for debugging unexpected results
• Implementing PCA from scratch as an interview or learning exercise

Common Mistakes

• Skipping standardization before computing the covariance matrix — this single omission is the most common PCA implementation bug.
• Forgetting to sort eigenvectors by eigenvalue before selecting the "top" components — numpy.linalg.eig doesn't guarantee any particular order.

Interview Relevance

Q: "Implement PCA from scratch, reducing a dataset to 1 dimension." The five-step code block above is exactly this answer — standardize, compute covariance, eigen-decompose, sort, project — and being able to explain why each step exists (not just recite it) is what separates a strong answer from a memorized one.

Practice Question

Using the projection formula, compute the 1D projected value for the point \((-1,-2)\) onto \(v_1=[0.707, 0.707]\).

Related ML Notes

• PCA
• Covariance
• PCA in Python