Partial Derivatives

A neural network's loss doesn't depend on just one number — it depends on every weight simultaneously, often millions of them. A partial derivative measures how the loss changes with respect to just one of those weights, holding all the others fixed.

Notation and Definition

The symbol \(\partial\) (instead of \(d\)) signals "this function has more than one input, and we're only varying \(x\), treating every other input as a constant for this calculation."

Numerical Example

Code

import torch

x = torch.tensor(2.0, requires_grad=True)
y = torch.tensor(1.0, requires_grad=True)

f = x**2 * y + 3*y
f.backward()

print(x.grad)   # tensor(4.) -- df/dx
print(y.grad)   # tensor(7.) -- df/dy

Where This Shows Up in Deep Learning

A network's loss \(L(w_1, w_2, \ldots, w_n)\) is a function of every single weight. Training needs \(\frac{\partial L}{\partial w_i}\) for every weight \(w_i\) — a whole collection of partial derivatives, one per parameter. Collecting all of them into a single vector is exactly what the next note, Gradient, does.

Common Mistakes

• Forgetting to treat other variables as constants when computing one partial derivative — mixing them up (e.g. accidentally differentiating \(y\) too when computing \(\partial f/\partial x\)) is the most common manual-calculation error.
• Assuming partial derivatives require computing them one at a time by hand in real deep learning code — in practice, autograd computes every partial derivative for every parameter automatically and in parallel in a single .backward() call.

Interview Relevance

Q: "Why does a neural network need partial derivatives instead of a single derivative?" Because the loss is a function of many weights simultaneously (potentially millions), not a single variable. A partial derivative isolates how the loss changes with respect to one specific weight, holding all others fixed — exactly the per-parameter update signal training needs.

Practice Question

For \(f(x, y) = 3x^2y^3\), compute \(\frac{\partial f}{\partial x}\) and \(\frac{\partial f}{\partial y}\), then evaluate both at \((x,y) = (1,2)\).

Related DL Notes

• Derivatives
• Gradient
• Gradient Vector