A vector space is the full set of vectors reachable by adding and scaling a chosen set of "basis" vectors. In ML, the number of features in your dataset defines the dimension of the space every data point lives in.
Basis and Dimension
A basis is the smallest set of vectors that can combine (via addition and scalar multiplication) to reach every point in the space. The number of vectors in a basis is the space's dimension.
Any 2D vector \([a,b]\) can be built from these two basis vectors — which is why 2D space needs exactly 2 basis vectors, not more, not fewer.
Why This Matters for ML: Feature Space
A dataset with 20 features doesn't just have "20 columns" — each data point is a vector in a 20-dimensional vector space. This reframing is what makes concepts like distance, similarity and dimensionality reduction precise: KNN measures distance in this space, PCA finds a lower-dimensional subspace that still captures most of the data's structure.
Span — What a Set of Vectors Can and Can't Reach
import numpy as np
# Two vectors that are NOT parallel span all of 2D space (any point is reachable)
v1 = np.array([1, 0])
v2 = np.array([0, 1])
# Two PARALLEL vectors only span a 1D line — you can never reach off that line
v3 = np.array([2, 4])
v4 = np.array([1, 2]) # v4 is just 0.5 * v3 -- same direction, no new informationThis directly explains why highly correlated features add little new information — two nearly-parallel feature vectors span almost the same subspace, so one of them is close to redundant.
Common Mistakes
- Treating "more features" as automatically "more information" — if new features are linear combinations of existing ones, they add dimension count without adding real span/information.
- Confusing the number of features (columns) with the true dimensionality of the meaningful signal in the data — this is exactly the gap dimensionality reduction exploits.
Interview Relevance
Q: "What does it mean for a dataset to have 'intrinsic dimensionality' lower than its feature count?" It means the real information in the data can be captured by fewer directions (a smaller basis) than the number of raw features — highly correlated or redundant features inflate the feature count without expanding the space the data actually occupies.
Practice Question
Two features, "temperature in Celsius" and "temperature in Fahrenheit," are included in a dataset. Explain, using the idea of span, why these two columns add almost no new dimensionality to the feature space.
