Matrix addition combines two matrices of the same shape by adding corresponding elements. It's the simplest matrix operation — and it's the exact mechanism behind adding a bias term and behind residual (skip) connections in modern architectures.
Formula
Each entry of the result is simply the sum of the corresponding entries — this is called an element-wise operation. \(\mathbf{A}\) and \(\mathbf{B}\) must have identical shapes (unless one is broadcastable — see Broadcasting).
Numerical Example
Code
import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(A + B)
# [[ 6 8]
# [10 12]]import torch
A = torch.tensor([[1., 2.], [3., 4.]])
B = torch.tensor([[5., 6.], [7., 8.]])
print(A + B)Where This Shows Up in Deep Learning
- Bias addition: every linear layer computes \(\mathbf{W}\mathbf{x} + \mathbf{b}\) — the \(+\mathbf{b}\) is matrix/vector addition (broadcast across the batch).
- Residual connections: ResNet's core idea is \(\text{output} = \mathbf{x} + F(\mathbf{x})\) — the input is added directly to the transformed output, letting gradients flow through the addition unimpeded. This single operation is central to training very deep networks (see ResNet).
Common Mistakes
- Trying to add matrices of incompatible shapes and expecting an error to explain itself clearly — always check
.shapefirst when addition fails. - Confusing matrix addition with matrix multiplication — addition is element-wise and requires equal shapes; multiplication has completely different shape rules (see Matrix Multiplication).
Interview Relevance
Q: "What role does matrix addition play in a residual connection?" A residual block computes \(\mathbf{x} + F(\mathbf{x})\) — the original input is added directly to the block's output. Because addition passes gradients through unchanged (derivative of a sum is 1), this addition is what allows gradients to flow to early layers in very deep networks, addressing vanishing gradients.
Practice Question
Given \(\mathbf{A} = \begin{bmatrix}2 & -1\\0 & 3\end{bmatrix}\) and \(\mathbf{B} = \begin{bmatrix}-2 & 1\\4 & 4\end{bmatrix}\), compute \(\mathbf{A} + \mathbf{B}\) by hand, then verify with code.
