The derivative of a function tells you its instantaneous rate of change — how fast the output changes as the input changes by a tiny amount. It is the single most important concept in this entire Calculus category: every weight update during training is driven by a derivative.
Formal Definition
This is exactly the limit from the previous note, applied to the slope of \(f\) between \(x\) and a nearby point \(x+h\), as \(h\) shrinks to zero. Geometrically, \(f'(x)\) is the slope of the line tangent to \(f\)'s graph at the point \(x\).
Common Derivative Rules
| Function | Derivative |
|---|---|
| \(f(x) = c\) (constant) | \(f'(x) = 0\) |
| \(f(x) = x^n\) | \(f'(x) = nx^{n-1}\) |
| \(f(x) = e^x\) | \(f'(x) = e^x\) |
| \(f(x) = \ln(x)\) | \(f'(x) = \dfrac{1}{x}\) |
| \(f(x) = \sin(x)\) | \(f'(x) = \cos(x)\) |
Numerical Example
At \(x=3\), the function \(x^2\) is increasing at a rate of 6 units of output per unit of input — meaning a tiny increase in \(x\) near 3 produces roughly 6 times as large an increase in \(f(x)\).
Geometric Meaning — Slope of the Tangent Line
The derivative at a point is the slope of the straight line that just touches the curve there — steeper means faster change.
Code — Numerical vs Symbolic Derivative
def f(x):
return x ** 2
def numerical_derivative(f, x, h=1e-6):
return (f(x + h) - f(x)) / h # approximates f'(x) using the limit definition
print(numerical_derivative(f, 3)) # approximately 6.000001 -- matches f'(x)=2x at x=3import torch
x = torch.tensor(3.0, requires_grad=True) # tell PyTorch to track gradients for x
y = x ** 2
y.backward() # computes dy/dx automatically via autograd
print(x.grad) # tensor(6.) -- exactly matches the analytical derivative 2xWhere This Shows Up in Deep Learning
Every weight \(w\) in a network has a derivative of the loss with respect to it, \(\frac{\partial L}{\partial w}\) — this tells you exactly which direction to nudge \(w\) to reduce the loss. PyTorch's .backward() call computes these derivatives automatically for every parameter in the network using automatic differentiation, which is built entirely on the chain rule (next note) applied systematically.
Common Mistakes
- Confusing the derivative's sign with its magnitude — the sign tells you which direction increases the function; the magnitude tells you how sensitive the output is to that input. Both matter for how gradient descent updates a weight.
- Forgetting a derivative is only defined where the function is smooth — functions with sharp corners (like ReLU at exactly \(x=0\)) require a special convention, covered later in Activation Functions.
Interview Relevance
Q: "In plain terms, what does loss.backward() compute in PyTorch?" It computes the derivative of the loss with respect to every parameter that has requires_grad=True, using automatic differentiation (repeated application of the chain rule through the computational graph). Those derivatives are what the optimizer then uses to update each weight.
Practice Question
Using the power rule, find the derivative of \(f(x) = 4x^3 - 2x + 7\), then evaluate \(f'(2)\).
