The chain rule differentiates a function built by composing other functions — one inside another. It's the mathematical trick that makes training multi-step models (logistic regression's sigmoid-over-linear-combination, or a neural network's stacked layers) possible.
Formula
If \(y\) is a function of \(u\), and \(u\) is itself a function of \(x\), the chain rule says: differentiate the "outer" function with respect to its immediate input, then multiply by the derivative of the "inner" function.
Numerical Example
Let \(y = (3x+1)^2\). Set \(u = 3x+1\), so \(y = u^2\).
At \(x=1\): \(\frac{dy}{dx} = 6(3(1)+1) = 6(4) = 24\). Verify by expanding directly: \(y = 9x^2+6x+1 \Rightarrow \frac{dy}{dx}=18x+6\), which at \(x=1\) gives \(18+6=24\) — matches.
def y_direct(x):
return (3*x + 1)**2
def dy_dx_chain_rule(x):
u = 3*x + 1
dy_du = 2 * u
du_dx = 3
return dy_du * du_dx
print(dy_dx_chain_rule(1)) # 24
h = 1e-6
approx = (y_direct(1 + h) - y_direct(1)) / h
print(round(approx, 2)) # 24.0 -- confirms the chain rule resultWhy This Is the Engine Behind Backpropagation
Logistic regression's prediction is a composition: a linear combination \(z = w^Tx+b\), passed through the sigmoid function \(\sigma(z)\), compared against the true label by a loss function. Training needs the derivative of the loss with respect to \(w\) — and the only way to get there is by chaining derivatives through each of those composed steps. A neural network is the same idea, just with many more composed layers — this repeated chain-rule application, layer by layer, is exactly what "backpropagation" means.
Common Mistakes
- Forgetting to multiply by the derivative of the inner function — a very common calculus slip that produces a result missing a constant factor.
- Thinking backpropagation is a separate, mysterious algorithm — it's the chain rule, applied systematically layer by layer.
Interview Relevance
Q: "What mathematical rule does backpropagation rely on?" The chain rule — a neural network's loss is a composition of many functions (layers), and backpropagation computes the gradient of the loss with respect to every weight by chaining derivatives backward through each layer.
Practice Question
Let \(y = (2x - 3)^3\). Using \(u = 2x-3\), apply the chain rule to find \(\frac{dy}{dx}\), then evaluate at \(x=2\).
