This note ties every piece of this category together into one complete, fully numerical example: a small but genuine multi-layer network, one full forward pass, one full backward pass computing every gradient by hand, and a weight update — verified against PyTorch's autograd at the end.
The Network
A 2-input, 2-hidden-neuron, 1-output network, sigmoid activations throughout, trained with squared error loss:
Step 1 — Forward Pass
Step 2 — Loss
Step 3 — Backward Pass: Output Layer's Error Signal
Step 4 — Backward Pass: Hidden Layer's Error Signal
Step 5 — Gradient Calculation
Step 6 — Weight Update (\(\eta=0.5\), for a visible change)
The output layer's weights both increased slightly — correct, since the network under-predicted (\(\hat y=0.657 < y_{\text{true}}=1\)) and increasing these weights pushes \(\hat y\) higher.
Verifying Every Number with PyTorch Autograd
import torch
x = torch.tensor([0.5, 0.8])
y_true = torch.tensor(1.0)
W1 = torch.tensor([[0.1, 0.3], [0.2, 0.4]], requires_grad=True)
b1 = torch.tensor([0.0, 0.0], requires_grad=True)
W2 = torch.tensor([0.5, 0.6], requires_grad=True)
b2 = torch.tensor(0.0, requires_grad=True)
z1 = W1 @ x + b1
a1 = torch.sigmoid(z1)
z2 = W2 @ a1 + b2
y_pred = torch.sigmoid(z2)
loss = (y_true - y_pred) ** 2
loss.backward()
print("y_pred:", y_pred.item()) # matches 0.6566 above
print("loss:", loss.item()) # matches 0.1179 above
print("dL/dW2:", W2.grad) # matches [-0.0886, -0.0934] above
print("dL/dW1:", W1.grad) # matches the W1 gradient matrix aboveCommon Mistakes
- Losing track of which cached forward-pass values (\(\mathbf{a}^{(1)}\), \(\mathbf{z}^{(1)}\), \(\mathbf{z}^{(2)}\)) feed into which backward-pass formula — working through a full example like this one, step by step, is the most reliable way to build the habit of tracking them correctly.
- Forgetting the sigmoid derivative's convenient form, \(\sigma'(z)=\sigma(z)(1-\sigma(z))\), and instead re-deriving it from scratch each time — reusing the cached activation value directly is both simpler and exactly what an efficient implementation does.
Interview Relevance
Q: "Walk through backpropagation for a small 2-layer network with real numbers." This exact worked example is the kind of answer that demonstrates genuine understanding rather than memorized formulas — being able to compute a forward pass, derive both layers' error signals via the chain rule, compute the resulting weight gradients via the outer product formula, and apply an update, all with concrete numbers, is a strong signal of true comprehension.
Practice Question
Using the same network and the newly computed gradient for \(\mathbf{W}^{(1)}\), compute the updated \(\mathbf{W}^{(1)}\) with \(\eta=0.5\).
