This note applies the general gradient descent algorithm specifically to linear regression's MSE cost function — deriving the exact update rules scikit-learn (and every other library) runs internally.
Deriving the Gradients
Starting from \(J(b_0,b_1) = \frac{1}{n}\sum(y_i - (b_0+b_1x_i))^2\), the partial derivatives (using the chain rule) work out to:
Both gradients are, at their core, the average residual — scaled by \(x_i\) for the slope term, since a change in \(b_1\) affects predictions more for larger \(x\) values.
The Update Rule
This is exactly the general gradient descent formula \(\theta := \theta - \alpha\frac{\partial J(\theta)}{\partial \theta}\) from the Gradient Descent note, applied to two specific parameters instead of one.
From-Scratch Implementation
import numpy as np
x = np.array([1, 2, 3, 4, 5])
y = np.array([52, 58, 62, 68, 75])
n = len(x)
b0, b1 = 0.0, 0.0 # start from zero
alpha = 0.01 # learning rate
epochs = 10000
for _ in range(epochs):
y_pred = b0 + b1 * x
error = y - y_pred
grad_b0 = -(2/n) * np.sum(error)
grad_b1 = -(2/n) * np.sum(error * x)
b0 = b0 - alpha * grad_b0
b1 = b1 - alpha * grad_b1
print(round(b0, 2), round(b1, 2)) # approximately 46.2 5.6 -- matches the closed-form answerExpected output: after enough iterations, \(b_0\) and \(b_1\) converge to (approximately) the exact same values — 46.2 and 5.6 — computed by hand in Simple Linear Regression. Gradient descent and the closed-form formula are two different roads to the identical destination.
Gradient Descent vs the Normal Equation — Which One Actually Runs
| Normal Equation | Gradient Descent | |
|---|---|---|
| Type | Exact, closed-form (one calculation) | Iterative, approximate (many small steps) |
| Formula | \(\vec{b} = (X^TX)^{-1}X^T\vec{y}\) | Repeated updates, as above |
| Speed on small/medium data | Very fast | Slower — needs many iterations |
| Speed on very large/high-dimensional data | Slow — matrix inversion is \(O(n^3)\) | Scales much better |
What scikit-learn's LinearRegression uses | Yes, by default (via a numerically stable variant) | No — but SGDRegressor uses gradient descent explicitly |
Practical Use Cases
- Training linear regression on datasets too large for the Normal Equation's matrix inversion to be practical
- Understanding what's actually happening inside every neural network's training loop, which generalizes this exact idea to millions of parameters
Common Mistakes
- Choosing too large a learning rate and watching \(b_0\)/\(b_1\) diverge instead of converge — see the diverging numeric example in Gradient Descent.
- Forgetting to scale features before gradient descent — unscaled features distort the cost surface into an elongated, hard-to-navigate valley, slowing convergence significantly.
Interview Relevance
Q: "Why does scikit-learn's LinearRegression not use gradient descent by default?" For typical dataset sizes, the closed-form Normal Equation computes the exact optimal coefficients directly and is faster than iterating gradient descent to convergence; gradient descent becomes preferable mainly when the feature count is very large, making the Normal Equation's matrix inversion impractically slow.
Practice Question
Using the gradient formulas above, compute \(\frac{\partial J}{\partial b_0}\) and \(\frac{\partial J}{\partial b_1}\) for a single data point \(x=2\), \(y=10\), at current parameters \(b_0=0, b_1=0\).
