An eigenvalue tells you how much a special direction (its matching eigenvector) gets stretched or shrunk by a matrix transformation — without changing direction. This pair is the mathematical engine behind PCA.
Formula
\(A\) is a square matrix (the transformation), \(\vec{v}\) is an eigenvector (a direction that doesn't rotate under this transformation), and \(\lambda\) (lambda) is the eigenvalue — the scalar amount \(\vec{v}\) is stretched or shrunk by.
Solving for Eigenvalues — The Characteristic Equation
For \(A = \begin{bmatrix}2&1\\1&2\end{bmatrix}\):
import numpy as np
A = np.array([[2, 1], [1, 2]])
eigenvalues, eigenvectors = np.linalg.eig(A)
print(eigenvalues) # [3. 1.] (or [1. 3.], order isn't guaranteed)
print(eigenvectors) # columns are the matching eigenvectorsGeometric Intuition
An eigenvector keeps pointing the same way after the transformation — only its length changes, by a factor of λ.
Why This Matters for ML
PCA finds the eigenvectors of a dataset's covariance matrix — these directions are the axes along which the data varies the most (and least), ranked by their eigenvalues. See PCA for the full algorithm built on this idea.
Common Mistakes
- Assuming every matrix has real-valued eigenvalues — some (like rotation matrices) have complex eigenvalues, though covariance matrices used in PCA are guaranteed to have real, non-negative eigenvalues.
- Confusing the eigenvalue (a single number, the scale factor) with the eigenvector (a direction/vector).
Interview Relevance
Q: "What do the eigenvalues of a covariance matrix represent in PCA?" Each eigenvalue represents the amount of variance in the data along its corresponding eigenvector's direction — sorting by eigenvalue tells you which directions capture the most information.
Practice Question
For \(A = \begin{bmatrix}4&0\\0&1\end{bmatrix}\), identify the eigenvalues by inspection (hint: diagonal matrices make this direct) without solving the characteristic equation.
