The transpose of a matrix flips it over its diagonal — rows become columns and columns become rows. It's a small operation that shows up constantly, from aligning shapes for multiplication to how gradients are computed during backpropagation.
Formula
Numerical Example
A \((2,3)\) matrix becomes a \((3,2)\) matrix — the shape's dimensions swap.
Key Properties
| Property | Statement |
|---|---|
| Double transpose | \((\mathbf{A}^\top)^\top = \mathbf{A}\) |
| Transpose of a sum | \((\mathbf{A}+\mathbf{B})^\top = \mathbf{A}^\top + \mathbf{B}^\top\) |
| Transpose of a product (order reverses!) | \((\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top \mathbf{A}^\top\) |
| Symmetric matrix | A matrix where \(\mathbf{A}^\top = \mathbf{A}\) (e.g. a covariance matrix) |
Code
import numpy as np
A = np.array([[1, 2, 3], [4, 5, 6]])
print(A.T)
# [[1 4]
# [2 5]
# [3 6]]
print(A.T.shape) # (3, 2)import torch
A = torch.tensor([[1., 2., 3.], [4., 5., 6.]])
print(A.T) # or A.transpose(0, 1)
print(A.T.shape) # torch.Size([3, 2])Where This Shows Up in Deep Learning
- Aligning shapes: if you need a dot product between two row vectors but the shapes don't line up for
matmul, transposing one side is usually the fix — e.g. \(\mathbf{x}^\top\mathbf{w}\) instead of an invalid \(\mathbf{x}\mathbf{w}\). - Backpropagation: the gradient of a linear layer's loss with respect to its input involves the weight matrix's transpose, \(\mathbf{W}^\top\) — this is literally how error signals flow backward through a layer (see Backpropagation Weight Updates).
- Attention: \(\mathbf{Q}\mathbf{K}^\top\) requires transposing the key matrix so its shape aligns for matrix multiplication against the query matrix.
Common Mistakes
- Assuming transpose changes the values of a matrix — it only rearranges their positions; no arithmetic is performed.
- Forgetting that transpose reverses multiplication order: \((\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top\mathbf{A}^\top\), not \(\mathbf{A}^\top\mathbf{B}^\top\).
Interview Relevance
Q: "Why does backpropagation through a linear layer involve the transpose of the weight matrix?" The forward pass computes \(\mathbf{y}=\mathbf{W}\mathbf{x}\); by the chain rule, the gradient flowing back to \(\mathbf{x}\) is \(\mathbf{W}^\top \cdot (\text{gradient w.r.t. } \mathbf{y})\) — the transpose is what correctly routes each output's gradient back to the inputs that produced it.
Practice Question
If \(\mathbf{A}\) has shape \((5, 3)\) and \(\mathbf{B}\) has shape \((5, 3)\), what operation involving a transpose would let you compute a valid \((3,3)\) matrix product?
