Backpropagation Weight Updates

Backpropagation's job ends the moment every layer's gradients — \(\frac{\partial L}{\partial \mathbf{W}^{(l)}}\) and \(\frac{\partial L}{\partial \mathbf{b}^{(l)}}\) for every layer \(l\) — have been computed. What happens next, turning those gradients into actual changes to the weights, is handled by a separate component entirely: the optimizer.

Where Backpropagation's Responsibility Ends

Why This Separation Matters

Every optimizer covered in the Optimization & LR Scheduling category — plain SGD, momentum, Adam, AdamW — takes the exact same input: a gradient for every parameter, exactly what backpropagation produces. They differ only in how they turn that gradient into an update (see Gradient Descent for the simplest case, or Adam Optimizer for a more sophisticated one). This clean separation is exactly why swapping optimizers in real code (torch.optim.SGD vs torch.optim.Adam) requires no changes at all to the model's forward pass or its .backward() call.

The Simplest Case, Restated

This is plain gradient descent's update rule, applied identically to every layer's weights and biases using the gradients backpropagation just computed. A more sophisticated optimizer like Adam would use these same gradients but combine them with momentum and adaptive per-parameter scaling before applying the update — the gradients themselves are unaffected by which optimizer receives them.

Code

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Sequential(nn.Linear(4, 3), nn.ReLU(), nn.Linear(3, 1))
optimizer = optim.Adam(model.parameters(), lr=0.001)   # could equally be SGD, AdamW, etc.

x = torch.randn(1, 4)
y_true = torch.tensor([[1.0]])

y_pred = model(x)                              # forward pass
loss = nn.MSELoss()(y_pred, y_true)              # loss
loss.backward()                                   # backpropagation -- computes every layer's gradients

optimizer.step()      # the optimizer consumes those gradients and updates every weight
optimizer.zero_grad()   # reset for the next iteration

Common Mistakes

• Conflating backpropagation with the optimization algorithm itself — backpropagation is purely a gradient-computation procedure; "how to use the gradient" is a separate, swappable design choice, covered in full in the Optimization category.
• Forgetting optimizer.zero_grad() before the next backward pass — this was already flagged in Gradient Vector, but it's worth repeating here since it's specifically at this boundary between backprop and the optimizer where the bug tends to surface.

Interview Relevance

Q: "Is backpropagation the same thing as gradient descent?" No — they're two distinct, sequential steps. Backpropagation computes the gradient of the loss with respect to every parameter, using the chain rule. Gradient descent (or any other optimizer, like Adam) is a separate algorithm that takes those already-computed gradients and decides how to actually update each parameter. Backpropagation answers "what is the gradient?"; the optimizer answers "what do I do with it?"

Practice Question

If you swap a training script's optimizer from SGD to Adam, does anything about how the gradients themselves are computed change? Why or why not?

Related DL Notes

• Gradient Calculation
• Gradient Descent
• Adam Optimizer