An eigenvector is a direction that a matrix transformation doesn't rotate — it only stretches or shrinks it, by a factor equal to the matching eigenvalue. Finding these special directions is exactly what PCA uses to find a dataset's "natural axes."
Solving for an Eigenvector, Given Its Eigenvalue
Once you know \(\lambda\), find \(\vec{v}\) by solving \((A - \lambda I)\vec{v} = 0\). Continuing the example from Eigenvalues, for \(A = \begin{bmatrix}2&1\\1&2\end{bmatrix}\), \(\lambda = 3\):
For \(\lambda = 1\): solving \((A-I)\vec{v}=0\) gives \(v_1 = -v_2\), so \(\vec{v} = \begin{bmatrix}1\\-1\end{bmatrix}\).
import numpy as np
A = np.array([[2, 1], [1, 2]])
eigenvalues, eigenvectors = np.linalg.eig(A)
for val, vec in zip(eigenvalues, eigenvectors.T):
print(f"eigenvalue: {val:.1f}, eigenvector direction: {vec}")
# eigenvalue: 3.0, eigenvector direction: [0.707 0.707] -- same direction as [1, 1], just normalized
# eigenvalue: 1.0, eigenvector direction: [-0.707 0.707] -- same direction as [1, -1]NumPy returns eigenvectors normalized to length 1 — the direction is what matters, not the specific length, since any scalar multiple of an eigenvector is also a valid eigenvector for the same eigenvalue.
Why Eigenvectors Are Orthogonal for Covariance Matrices
A special, useful property: for a symmetric matrix (which a covariance matrix always is), the eigenvectors are guaranteed to be perpendicular (orthogonal) to each other. This is exactly why PCA's principal components don't overlap in the information they capture — each one explains a distinct, independent direction of variance.
Practical Use Cases
- PCA — the principal components ARE the eigenvectors of the covariance matrix, ranked by their eigenvalues
- Understanding the "natural directions" a linear transformation stretches data along, useful for debugging why a linear model behaves a certain way
Common Mistakes
- Expecting a unique eigenvector for a given eigenvalue — any non-zero scalar multiple of an eigenvector is equally valid; only the direction is fixed, not the magnitude or sign.
- Assuming eigenvectors are always orthogonal — true for symmetric matrices (like covariance matrices) but not guaranteed for matrices in general.
Interview Relevance
Q: "Why are PCA's principal components always orthogonal to each other?" Because they're the eigenvectors of a covariance matrix, which is always symmetric — and eigenvectors of a symmetric matrix are guaranteed to be mutually orthogonal.
Practice Question
For the diagonal matrix \(A = \begin{bmatrix}5&0\\0&2\end{bmatrix}\), identify the two eigenvectors by inspection.
