Standard Deviation

Standard deviation is the square root of variance — it measures the same thing (how spread out the data is) but back in the data's original, interpretable units, which is why it's reported far more often than raw variance.

Why Take the Square Root At All?

Variance squares every deviation before averaging, which means its units are squared too — if your feature is measured in rupees, variance is in "rupees squared," a unit with no intuitive meaning. Taking the square root converts back to plain rupees, so you can directly say things like "scores typically deviate from the mean by about 8.5 points" — a sentence that makes sense to a human in a way "the variance is 72.25 points²" simply doesn't.

Formula — Population vs Sample

Both are simply \(\sqrt{\text{variance}}\) — everything about population vs sample, and the \(n-1\) correction, carries over directly from Variance.

Graphical Intuition — Comparing Spread

Step-by-Step Numerical Example

Continuing directly from the Variance note's dataset \(2, 4, 4, 4, 5, 5, 7, 9\), where population variance \(\sigma^2 = 4\) and sample variance \(s^2 \approx 4.571\):

The population standard deviation is exactly 2 — meaning, on average, values in this dataset sit about 2 units away from the mean of 5.

Python Implementation

# From scratch — literally just the square root of variance
import math

def std_from_scratch(data, sample=True):
    n = len(data)
    mean = sum(data) / n
    squared_deviations = [(x - mean) ** 2 for x in data]
    denominator = (n - 1) if sample else n
    variance = sum(squared_deviations) / denominator
    return math.sqrt(variance)

data = [2, 4, 4, 4, 5, 5, 7, 9]
print(std_from_scratch(data, sample=False))   # 2.0
print(std_from_scratch(data, sample=True))     # 2.1380899...

# NumPy / Pandas — same ddof caveat as variance
import numpy as np
import pandas as pd

print(np.std(data))           # 2.0  -- population (ddof=0 default)
print(np.std(data, ddof=1))    # 2.1380899...  -- sample
print(pd.Series(data).std())    # 2.1380899...  -- Pandas defaults to sample

The 68-95-99.7 Rule, for Roughly Normal Data

For data that's approximately normally distributed, standard deviation has a very useful rule of thumb:

This is exactly the intuition behind the common "flag as outlier if \(|z| > 3\)" rule from Outlier Treatment — a point more than 3 standard deviations from the mean is, for roughly normal data, extremely rare (about 0.3% of all values), which is why it's treated as suspicious.

Why Standard Deviation Matters for ML

• Standardization divides every value by the feature's standard deviation, directly using this statistic to rescale features
• The central limit theorem and confidence intervals both use standard deviation to quantify uncertainty in an estimate
• Comparing a model's performance across multiple cross-validation folds by both mean AND standard deviation tells you not just "how good," but "how consistent"

Advantages

• Interpretable in the original units of the data, unlike variance
• Directly usable for outlier detection (z-scores) and standardization

Limitations

• Inherits variance's sensitivity to outliers — a few extreme values inflate it substantially
• The 68-95-99.7 rule only holds meaningfully for roughly normal data; it can be misleading on strongly skewed distributions

Common Mistakes

• Reporting standard deviation without checking whether the underlying distribution is anywhere close to normal — the 68-95-99.7 intuition doesn't transfer to heavily skewed data.
• Comparing two models' standard deviation of accuracy across folds without also comparing their means — a model with lower variance but also lower mean accuracy isn't automatically "better."
• Confusing standard deviation (spread of the raw data) with standard error (spread of a sample statistic, like the mean, across repeated samples) — these are related but answer different questions.

Interview Relevance

Q: "Two models both average 85% cross-validation accuracy. Model A has std 0.5%, Model B has std 4%. Which would you trust more?" Model A — its performance is far more consistent across folds, suggesting it will behave more predictably on new data; Model B's high variance suggests its accuracy may be unstable or overly dependent on which specific data ended up in each fold.

Q: "What does a z-score of 2.5 mean in plain language?" The value is 2.5 standard deviations above the mean — using the 68-95-99.7 rule, that puts it in roughly the top 1% of the distribution (for roughly normal data), a fairly unusual, though not extreme, observation.

Practice Question

Using the dataset \([3, 6, 6, 9]\) from the Variance note's practice question, compute the population standard deviation by taking the square root of the variance you found there.

Related ML Notes

• Variance
• Standardization
• Outlier Treatment