Variance

Variance measures how spread out a dataset is — specifically, the average of each value's squared distance from the mean. Two datasets can share the exact same mean and look completely different once you account for variance.

Why "Squared Distance," Not Just "Distance"

You might expect spread to be measured as the average distance from the mean — but plain distances (some positive, some negative) always sum to exactly zero around the mean by definition, canceling out and hiding any real spread. Squaring each deviation before averaging fixes this: every term becomes positive, so larger deviations (in either direction) always increase the total. This is also why variance is expressed in squared units (e.g. "dollars²"), which is exactly why standard deviation — variance's square root — exists, to bring the units back to something interpretable.

Formula — Population Variance vs Sample Variance

\(x_i\) is each value, \(\mu\)/\(\bar{x}\) is the population/sample mean, \(N\)/\(n\) is the population/sample size. Notice the sample formula divides by \(n-1\), not \(n\) — this is Bessel's correction, and it isn't arbitrary.

Why Sample Variance Divides by n − 1, Not n

A sample's own mean \(\bar{x}\) is computed from that same sample — which means the sample's deviations are, on average, slightly smaller than the true deviations from the unknown population mean \(\mu\) would be (the sample mean is, by construction, the value that minimizes the sum of squared deviations for that specific sample). Dividing by \(n-1\) instead of \(n\) corrects for this built-in underestimate, making \(s^2\) an unbiased estimator of the true population variance \(\sigma^2\). With small samples this correction matters a lot; as \(n\) grows large, the difference between dividing by \(n\) and \(n-1\) becomes negligible.

Graphical Intuition

Step-by-Step Numerical Example

Dataset: \(2, 4, 4, 4, 5, 5, 7, 9\) (a classic textbook example, chosen so every step comes out clean).

Notice the sample variance (4.571) is slightly larger than the population variance (4) — exactly the correction Bessel's adjustment is designed to apply.

Python Implementation

# From scratch
def variance_from_scratch(data, sample=True):
    n = len(data)
    mean = sum(data) / n
    squared_deviations = [(x - mean) ** 2 for x in data]
    denominator = (n - 1) if sample else n
    return sum(squared_deviations) / denominator

data = [2, 4, 4, 4, 5, 5, 7, 9]
print(variance_from_scratch(data, sample=False))   # 4.0  (population)
print(variance_from_scratch(data, sample=True))     # 4.571428571...  (sample)

# NumPy / Pandas
import numpy as np
import pandas as pd

print(np.var(data))            # 4.0   -- NumPy defaults to POPULATION variance (ddof=0)
print(np.var(data, ddof=1))     # 4.571428571...  -- pass ddof=1 for SAMPLE variance
print(pd.Series(data).var())     # 4.571428571...  -- Pandas defaults to SAMPLE variance

This default mismatch between NumPy and Pandas is a genuinely common source of bugs — always check which one a function is using, especially when comparing results computed with both libraries on the same data.

Practical Use Cases

• Quantifying feature spread during EDA — a near-zero-variance feature carries almost no distinguishing information and is often a candidate to drop (see Variance Threshold feature selection)
• The building block for standard deviation, covariance, and the bias-variance tradeoff
• PCA explicitly seeks the directions of maximum variance in the data — variance is the literal quantity being optimized

Advantages

• Uses every data point's exact distance from the mean, capturing spread precisely
• Has convenient mathematical properties that make it tractable inside optimization (it's differentiable, unlike some rank-based spread measures)

Limitations

• Expressed in squared units, which usually isn't directly interpretable (variance of "40 rupees²" doesn't mean much intuitively) — standard deviation exists to fix this
• Sensitive to outliers, since squaring amplifies the effect of large deviations disproportionately

Common Mistakes

• Using the wrong denominator (\(n\) vs \(n-1\)) without realizing which one a specific library function defaults to — as shown above, NumPy and Pandas disagree by default.
• Interpreting the raw variance value directly instead of converting to standard deviation first for intuitive comparison.
• Comparing variances across features measured in different units (e.g. income in rupees vs age in years) as if they were directly comparable — they aren't, without first standardizing.

Interview Relevance

Q: "Why do we divide by n-1 instead of n when computing sample variance?" Bessel's correction — because the sample mean is computed from the same sample used to compute deviations, those deviations are systematically slightly smaller than true deviations from the (unknown) population mean would be; dividing by \(n-1\) instead of \(n\) corrects this bias so \(s^2\) is an unbiased estimator of \(\sigma^2\).

Q: "np.var() and pandas .var() gave me different answers on the same data — why?" NumPy's np.var() defaults to population variance (dividing by \(n\)), while Pandas' .var() defaults to sample variance (dividing by \(n-1\)) — pass ddof=1 to NumPy to match Pandas' default.

Practice Question

Compute the population variance of \([3, 6, 6, 9]\) by hand: find the mean, the deviations, the squared deviations, and the final variance.

Related ML Notes

• Standard Deviation
• Covariance
• Bias-Variance Tradeoff