If the gradient tells you the slope of the loss surface, the Hessian tells you its curvature — is the surface bowl-shaped, saddle-shaped, or flat around this point? That distinction matters for understanding why some points where the gradient is zero are good minima and others aren't.
Definition
The Hessian is the matrix of all second-order partial derivatives of a scalar-valued function — it's the Jacobian of the gradient. For a function of \(n\) variables, it's an \((n,n)\) square matrix.
Numerical Example
Both diagonal entries are positive and off-diagonal terms are zero — this Hessian is positive definite everywhere, confirming \(f\) curves upward in every direction: it's a bowl shape with a true minimum at the origin.
What the Hessian Tells You About a Critical Point
| Hessian Property | Meaning at a Point Where the Gradient Is Zero |
|---|---|
| Positive definite (all eigenvalues > 0) | Local minimum — the function curves upward in every direction |
| Negative definite (all eigenvalues < 0) | Local maximum — the function curves downward in every direction |
| Indefinite (mixed positive/negative eigenvalues) | Saddle point — curves up in some directions, down in others |
This connects directly back to Eigenvalues: the Hessian's eigenvalues determine which of these three cases you're in.
Why Deep Learning Mostly Avoids Computing the Hessian
For a network with \(n\) parameters, the Hessian is an \((n,n)\) matrix — for a modest model with a million parameters, that's \(10^{12}\) entries, computationally infeasible to form or invert. This is exactly why deep learning optimizers (SGD, Adam, etc.) use only first-order information (the gradient), not the Hessian, unlike some classical optimization methods (e.g. Newton's method) that use curvature directly. Adam and similar optimizers approximate some curvature-like behavior cheaply, without ever forming the true Hessian — covered in the Optimization category.
Code — Computing a Small Hessian
import torch
def f(v):
x, y = v[0], v[1]
return x**2 + y**2
x = torch.tensor([1.0, 1.0], requires_grad=True)
H = torch.autograd.functional.hessian(f, x)
print(H)
# tensor([[2., 0.],
# [0., 2.]])Common Mistakes
- Assuming a zero gradient always means a minimum — it only means a critical point; the Hessian is what distinguishes a true minimum from a maximum or (very common in high-dimensional loss surfaces) a saddle point.
- Expecting deep learning training to routinely use the Hessian — in practice it almost never is, purely for computational-cost reasons, not because curvature information wouldn't be useful.
Interview Relevance
Q: "Why is it risky to assume a point where the gradient is zero is a good solution during training?" A zero gradient only signals a critical point — it could be a true minimum, a maximum, or (especially in high-dimensional non-convex loss surfaces typical of deep networks) a saddle point, where the surface curves up in some directions and down in others. The Hessian's eigenvalues are what distinguish these cases, though it's rarely computed directly in practice due to cost.
Practice Question
For \(f(x,y) = x^2 - y^2\), compute the Hessian. Based on its diagonal entries, is the critical point at the origin a minimum, maximum, or saddle point?
