Derivatives

A derivative measures how fast a function's output changes as its input changes — geometrically, it's the slope of the tangent line touching the curve at a single point.

Formula

This is the formal definition — the slope between two points on the curve, as those points get infinitely close together. In practice, you use known rules instead of this limit directly:

Geometric Intuition — The Tangent Line

Numerical Example

For \(f(x) = x^2\): by the power rule, \(f'(x) = 2x\). At \(x=3\): \(f'(3) = 6\) — the curve is rising at a rate of 6 units of \(y\) per unit of \(x\), right at that point.

def f(x):
    return x**2

def derivative_f(x):
    return 2 * x   # analytical derivative via the power rule

print(derivative_f(3))    # 6

# Numerical approximation, using the limit definition directly
h = 1e-6
approx = (f(3 + h) - f(3)) / h
print(round(approx, 4))    # 6.0 (approximately, confirming the formula)

Reading the Sign and Size of a Derivative

Why This Matters for ML

A loss function's derivative with respect to a model parameter tells you exactly which direction to adjust that parameter to reduce the loss — this single idea, applied at every parameter simultaneously, is gradient descent.

Common Mistakes

• Confusing the derivative's sign with the function's sign — a negative derivative means the function is decreasing, not that the function's value is negative.
• Assuming a zero derivative always means a minimum — it could also be a maximum or a saddle point; the second derivative (or a broader check) distinguishes these.

Interview Relevance

Q: "What does it mean when gradient descent 'converges'?" The derivative (or gradient, in multiple dimensions) has gotten close to zero — the loss function is nearly flat at the current parameters, so further steps produce negligible improvement.

Practice Question

Using the power rule, find \(f'(x)\) for \(f(x) = x^3\), and evaluate it at \(x=2\).

Related ML Notes

• Partial Derivatives
• Gradient
• Gradient Descent