Linear regression needs a precise, numeric definition of "best fit" before it can search for it — the cost function (Mean Squared Error) is exactly that definition, turning "how good is this line?" into a single number to minimize.
Formula
\(y_i\) is the actual value, \(\hat{y}_i = b_0 + b_1x_i\) is the model's prediction, and \(n\) is the number of data points. \(J(b_0,b_1)\) is written as a function of the coefficients specifically because it's exactly those two numbers — not the data — that training searches over to minimize this cost.
Why Squared Error, Not Just Error?
- Positive and negative errors don't cancel out — squaring makes every term positive, same reasoning as variance's squared deviations
- Large errors are penalized disproportionately more — an error of 10 contributes 100 to the sum, while an error of 2 contributes only 4; this pushes the model to avoid any single badly-wrong prediction
- It's smooth and differentiable everywhere — required for gradient descent to compute a usable gradient at every point
Numerical Example
Using the residuals already computed in Simple Linear Regression: \(0.2, 0.6, -1.0, -0.6, 0.8\).
from sklearn.metrics import mean_squared_error
y_actual = [52, 58, 62, 68, 75]
y_pred = [51.8, 57.4, 63.0, 68.6, 74.2]
print(mean_squared_error(y_actual, y_pred)) # 0.48The Cost Surface — Why It's Shaped Like a Bowl
Because MSE is a sum of squared terms, the cost surface for linear regression is always a smooth, convex bowl — with exactly one global minimum, no other local minima to get stuck in.
This convexity is a genuinely important, practical guarantee: unlike deep neural networks (whose cost surfaces have many local minima), linear regression's cost surface is provably bowl-shaped — gradient descent (or the closed-form Normal Equation) is guaranteed to find the single global best-fit line, not just a locally good one.
MSE vs MAE — Why This Specific Choice
| MSE (used here) | MAE | |
|---|---|---|
| Formula | \(\frac{1}{n}\sum(y_i-\hat{y}_i)^2\) | \(\frac{1}{n}\sum|y_i-\hat{y}_i|\) |
| Outlier sensitivity | High — squaring amplifies large errors | Lower — errors scale linearly |
| Differentiable everywhere? | Yes, smoothly | Not at zero (a sharp corner) |
| Standard choice for training linear regression? | Yes | Less common, though usable |
Common Mistakes
- Confusing the cost function (computed on the training set, used to fit the model) with an evaluation metric reported on a held-out test set — they can use the identical formula but serve different purposes.
- Assuming a lower training MSE always means a better model — an unreasonably low training MSE relative to validation MSE is a sign of overfitting.
Interview Relevance
Q: "Why is linear regression's cost function guaranteed to have a single global minimum?" Because MSE, as a function of the coefficients, is convex (bowl-shaped) — a mathematical property of sums of squared linear terms — so there are no other local minima for gradient descent or the Normal Equation to get stuck in.
Practice Question
Given actual values \([10, 20, 30]\) and predictions \([12, 18, 33]\), compute the MSE by hand.
