A norm measures the "size" or "length" of a vector as a single non-negative number. Different norms define "size" differently — and that choice directly shapes how regularization and gradient clipping behave in deep learning.
The Common Norms
| Norm | Formula | Also Called |
|---|---|---|
| L1 norm | \(\|\mathbf{x}\|_1 = \sum_i |x_i|\) | Manhattan norm |
| L2 norm | \(\|\mathbf{x}\|_2 = \sqrt{\sum_i x_i^2}\) | Euclidean norm |
| L∞ norm | \(\|\mathbf{x}\|_\infty = \max_i |x_i|\) | Max norm |
| General Lp norm | \(\|\mathbf{x}\|_p = \left(\sum_i |x_i|^p\right)^{1/p}\) | — |
Numerical Example
L1 vs L2 — Why the Shape of Their "Unit Circle" Matters
The L1 norm's sharp corners (on the axes) are why L1 regularization tends to push weights to exactly zero; the L2 norm's smooth boundary shrinks weights toward zero without eliminating them.
Code
import numpy as np
x = np.array([3.0, -4.0])
print(np.linalg.norm(x, ord=1)) # 7.0
print(np.linalg.norm(x, ord=2)) # 5.0
print(np.linalg.norm(x, ord=np.inf)) # 4.0import torch
x = torch.tensor([3.0, -4.0])
print(torch.norm(x, p=1)) # tensor(7.)
print(torch.norm(x, p=2)) # tensor(5.)Where This Shows Up in Deep Learning
- L2 regularization / weight decay: adds \(\lambda\|\mathbf{w}\|_2^2\) to the loss to discourage large weights and reduce overfitting (see L2 Regularization).
- L1 regularization: adds \(\lambda\|\mathbf{w}\|_1\) — its sharp-cornered geometry tends to drive some weights to exactly zero, producing sparse models (see L1 Regularization).
- Gradient clipping: rescales a gradient vector so its L2 norm doesn't exceed a threshold, preventing exploding gradients (see Gradient Clipping).
Common Mistakes
- Assuming "norm" always means the Euclidean (L2) length — always check which \(p\) is intended, since L1 and L2 regularization produce meaningfully different model behavior.
- Confusing a vector's norm with "distance" in general — a norm measures a single vector's size; distance between two points is the norm of their difference, \(\|\mathbf{a}-\mathbf{b}\|\).
Interview Relevance
Q: "Why does L1 regularization tend to produce sparse weights while L2 doesn't?" Geometrically, the L1 norm's constraint region has sharp corners exactly on the coordinate axes (where some weights are exactly zero), and the loss's optimal point tends to land on those corners. The L2 norm's constraint region is smooth (a sphere), so its optimum shrinks weights toward zero without usually reaching it exactly.
Practice Question
Compute the L1, L2 and L∞ norms of \([1, -2, 2]\) by hand, then verify with code.
