Matrix Multiplication

Matrix multiplication is the single most-executed operation in deep learning — every linear layer, every attention score, every convolution (reshaped) reduces to it. Understanding its shape rule is non-negotiable for debugging real models.

Formula and Shape Rule

If \(\mathbf{A}\) has shape \((m, n)\) and \(\mathbf{B}\) has shape \((n, p)\), the inner dimensions must match (\(n = n\)), and the result \(\mathbf{C}\) has shape \((m, p)\) — the outer dimensions.

How Each Entry Is Computed

Full Numerical Example

Matrix Multiplication Is Not Commutative

In general \(\mathbf{A}\mathbf{B} \ne \mathbf{B}\mathbf{A}\) — order matters, unlike scalar multiplication. This is also distinct from element-wise (Hadamard) multiplication, written \(\mathbf{A} \odot \mathbf{B}\), which multiplies corresponding entries and requires equal shapes.

Code

import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

print(A @ B)          # matrix multiplication -> [[19 22] [43 50]]
print(A * B)           # element-wise (Hadamard) -> [[5 12] [21 32]]  -- different!

import torch
A = torch.tensor([[1., 2.], [3., 4.]])
B = torch.tensor([[5., 6.], [7., 8.]])

print(torch.matmul(A, B))   # or A @ B
print(A * B)                 # element-wise, NOT the same operation

Where This Shows Up in Deep Learning

Every linear (fully-connected) layer computes \(\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}\) — a matrix multiplication plus a bias. A batch of \(N\) inputs, each of size \(d_{in}\), passed through a layer producing \(d_{out}\) outputs, is one matrix multiplication: \((N, d_{in}) \times (d_{in}, d_{out}) \rightarrow (N, d_{out})\). This same operation, at massive scale, is also the core of self-attention (\(\mathbf{Q}\mathbf{K}^\top\)) covered later in this hub.

Common Mistakes

• Confusing * (element-wise) with @/matmul (true matrix multiplication) — this is one of the most common silent shape/logic bugs in PyTorch code.
• Forgetting the inner-dimension rule and getting a shape-mismatch error without knowing why — always write out both shapes and check the inner numbers match.

Interview Relevance

Q: "What shape does a batch of 32 samples, each with 10 features, become after a linear layer with 5 output units?" The input is \((32, 10)\), the weight matrix is \((10, 5)\), so the matrix multiplication produces \((32, 5)\) — 32 samples, each now with 5 output features.

Practice Question

Can you multiply a \((4, 3)\) matrix by a \((4, 3)\) matrix using standard matrix multiplication? If not, what shape would the second matrix need to be, and what would the result's shape be?

Related DL Notes

• Dot Product
• Matrix Transpose
• Query, Key, Value