Simple linear regression fits a straight line through data with exactly one feature — the cleanest place to see, by hand, exactly how a regression model's coefficients get computed.
Formula for the Coefficients
This should look familiar — the numerator of \(b_1\) is the same sum used in covariance, and the denominator is the same sum used in variance. In fact, \(b_1 = \text{Cov}(X,Y) / \text{Var}(X)\) exactly — simple linear regression's slope is literally a ratio of covariance to variance.
Step-by-Step Numerical Example
Five students: hours studied \(x = [1,2,3,4,5]\), exam marks \(y=[52,58,62,68,75]\).
| Step | Calculation |
|---|---|
| 1. Means | \(\bar{x}=3\), \(\bar{y}=63\) |
| 2. Deviations \((x_i-\bar{x})\) | \(-2,-1,0,1,2\) |
| 3. Deviations \((y_i-\bar{y})\) | \(-11,-5,-1,5,12\) |
| 4. Products, summed | \(22+5+0+5+24 = 56\) |
| 5. Squared X deviations, summed | \(4+1+0+1+4 = 10\) |
| 6. Slope | \(b_1 = 56/10 = 5.6\) |
| 7. Intercept | \(b_0 = 63 - 5.6(3) = 46.2\) |
Final equation: \(\hat{y} = 46.2 + 5.6x\) — matching exactly what scikit-learn returned in the Linear Regression hub note.
Checking the Predictions and Residuals
| x | Actual y | Predicted ŷ = 46.2 + 5.6x | Residual (y − ŷ) |
|---|---|---|---|
| 1 | 52 | 51.8 | 0.2 |
| 2 | 58 | 57.4 | 0.6 |
| 3 | 62 | 63.0 | −1.0 |
| 4 | 68 | 68.6 | −0.6 |
| 5 | 75 | 74.2 | 0.8 |
Notice the residuals sum to (approximately) zero — this isn't a coincidence, it's a mathematical guarantee of the least-squares fitting method used to derive \(b_0\) and \(b_1\).
Python — From Scratch and scikit-learn
# From scratch, following the formula exactly
x = [1, 2, 3, 4, 5]
y = [52, 58, 62, 68, 75]
n = len(x)
x_mean = sum(x) / n
y_mean = sum(y) / n
numerator = sum((x[i] - x_mean) * (y[i] - y_mean) for i in range(n))
denominator = sum((x[i] - x_mean) ** 2 for i in range(n))
b1 = numerator / denominator
b0 = y_mean - b1 * x_mean
print(b0, b1) # 46.2 5.6
# scikit-learn -- same result, far less code
from sklearn.linear_model import LinearRegression
import numpy as np
model = LinearRegression().fit(np.array(x).reshape(-1, 1), y)
print(model.intercept_, model.coef_) # 46.2 [5.6]Practical Use Cases
- Any two-variable relationship — advertising spend vs sales, temperature vs ice cream sales, experience vs salary
- A fast, interpretable first check before reaching for a more complex model
Common Mistakes
- Forgetting that \(b_1\) is only meaningful for the range of \(x\) actually observed in the data — extrapolating far beyond the training range is unreliable.
- Interpreting \(b_0\) (the intercept) literally when \(x=0\) is outside any realistic or observed range (e.g. "0 hours studied" might genuinely make sense, but "0 square feet" for a house price model wouldn't).
Interview Relevance
Q: "What's the relationship between the regression slope and covariance/variance?" The slope \(b_1\) equals \(\text{Cov}(X,Y)/\text{Var}(X)\) exactly — simple linear regression's coefficient is a direct ratio of how X and Y vary together to how much X varies on its own.
Practice Question
For \(x=[2,4,6,8]\), \(y=[3,7,5,10]\), compute \(\bar{x}\), \(\bar{y}\), and use the formula above to find \(b_1\) and \(b_0\) by hand.
