The expectation (or expected value) of a random variable is its long-run average — not necessarily a value it can actually take, but the weighted average of every possible outcome, weighted by how likely each one is.
Formula
Each possible value \(x_i\) is multiplied by its probability, then summed — outcomes that are more likely contribute more to the average.
Numerical Example — A Fair Die
3.5 is the expected value — notice it's not even a possible die outcome, which is exactly the point: expectation is a long-run average, not a prediction of any single roll.
import numpy as np
outcomes = [1, 2, 3, 4, 5, 6]
probabilities = [1/6] * 6
expected_value = sum(x * p for x, p in zip(outcomes, probabilities))
print(expected_value) # 3.5
# Confirm with simulation
rolls = np.random.randint(1, 7, size=1_000_000)
print(rolls.mean()) # approximately 3.5Linearity of Expectation
Expectation scales and shifts predictably — if you double every outcome and add 10, the expected value simply doubles and adds 10 too. This property holds even when variables are dependent, which makes it one of the most useful facts in probability.
Why This Matters for ML
- A loss function is typically defined as the expected error across the data distribution — training minimizes an estimate of this expectation, computed from your finite training sample
- The mean of a dataset (used constantly in preprocessing and statistics) IS the sample-based estimate of \(E[X]\)
- Variance is defined directly in terms of expectation: \(\text{Var}(X) = E[(X - E[X])^2]\)
Common Mistakes
- Expecting \(E[X]\) to be a value \(X\) can actually take — as the die example shows, it often isn't.
- Assuming \(E[XY] = E[X]E[Y]\) in general — that only holds when \(X\) and \(Y\) are independent; otherwise you need to account for their covariance.
Interview Relevance
Q: "What's the relationship between the mean of your training data and expectation?" The sample mean is an estimate of the true expected value \(E[X]\) of the underlying distribution the data was drawn from — as your sample size grows, the sample mean converges toward the true expectation.
Practice Question
A biased coin lands heads (X=1) with probability 0.7 and tails (X=0) with probability 0.3. Compute \(E[X]\).
