Gradient Descent (Intro)

This note introduces gradient descent as the algorithm that actually uses backpropagation's gradients to improve the network — closing the loop between "measuring error" and "getting better." The full Optimization category covers its many variants (SGD, Adam, etc.) in depth.

The Core Update Rule

\(\eta\) (eta) is the learning rate — a small positive hyperparameter controlling how large a step to take. \(\nabla L(\mathbf{w})\) is the gradient computed by backpropagation, pointing toward steepest loss increase (see Gradient) — subtracting it moves the weights toward decreasing loss.

Numerical Example

A single weight \(w=2.0\), gradient \(\nabla L = 3.0\), learning rate \(\eta=0.1\):

The weight moved from 2.0 to 1.7 — a small step in the direction that reduces the loss, based on the current gradient. Repeating this process many times, across every weight in the network simultaneously, is what "training" means.

Why the Learning Rate Matters So Much

Code — A Complete Minimal Training Step

import torch
import torch.nn as nn

model = nn.Linear(1, 1)
x = torch.tensor([[2.0]])
y_true = torch.tensor([[10.0]])
learning_rate = 0.01

y_pred = model(x)                       # forward propagation
loss = nn.MSELoss()(y_pred, y_true)      # loss function
loss.backward()                           # backpropagation -- computes gradients

with torch.no_grad():                     # gradient descent update -- no gradient tracking needed here
    for param in model.parameters():
        param -= learning_rate * param.grad
        param.grad.zero_()                 # reset gradient for the next step

In practice, you'll almost always use an optimizer object (torch.optim.SGD, torch.optim.Adam, ...) to handle this update loop instead of writing it manually — covered in the PyTorch category — but this is exactly what those optimizers do underneath.

The Complete Training Flow, Now Fully Assembled

This closes the loop first previewed in Why Calculus for Neural Networks: Input → Weighted Sum → Activation → Prediction (forward propagation) → Loss (loss function) → Gradient (backpropagation) → Weight Update (gradient descent) — repeated over and over, across many batches and epochs, is the entirety of how a neural network learns.

Common Mistakes

• Setting the learning rate too high "to train faster" — this frequently causes the loss to oscillate or diverge entirely instead, as shown in the diagram above.
• Forgetting to zero out gradients between steps (as flagged already in Gradient Vector) — this silently accumulates stale gradients from previous steps into the current update.

Interview Relevance

Q: "What happens if the learning rate is set far too high?" Each weight update overshoots the loss surface's minimum by a large margin. Instead of smoothly converging, the loss can oscillate wildly or increase without bound (diverge) — the optimizer keeps "jumping over" the minimum in a self-reinforcing way rather than settling into it.

Practice Question

A weight is currently 5.0, with gradient 2.0. Using learning rate 0.5, compute the updated weight. What would happen to the update's size if the learning rate were 5.0 instead — is that likely to help or hurt convergence?

Related DL Notes

• Backpropagation (Intro)
• Weight Updates
• Learning Rate