AdaGrad (Adaptive Gradient) introduced a fundamentally different idea: instead of one shared learning rate for every weight, give each parameter its own adaptive learning rate, based on how much that specific parameter's gradient has historically varied.
Formula
\(g_t\) is the gradient for one specific parameter at step \(t\). \(G_t\) accumulates the sum of squared gradients for that parameter across all of training so far. \(\epsilon\) (a tiny constant, e.g. \(10^{-8}\)) prevents division by zero. Crucially, this is computed per parameter — every weight in the network gets its own independently-accumulated \(G_t\) and effective learning rate.
The Intuition — Why Per-Parameter Rates Help
Parameters that receive large, frequent gradients (common features) get their effective learning rate shrunk faster, preventing overshooting. Parameters that receive small, infrequent gradients (rare features) keep a relatively larger effective learning rate, letting them still learn meaningfully despite infrequent updates. This is especially valuable for sparse data — like NLP tasks where most words are rare, and their associated embedding weights only get updated occasionally.
Numerical Example
With \(\eta=0.1\), \(\epsilon\) negligible: after gradients \(g=[2, 2, 2]\) over 3 steps for one parameter, \(G_3 = 4+4+4=12\), effective learning rate becomes \(\frac{0.1}{\sqrt{12}} \approx 0.029\) — noticeably shrunk from the original 0.1, purely from this parameter's own gradient history.
The Fatal Flaw: Learning Rate Only Ever Shrinks
Because \(G_t\) is a running sum that only ever grows (squared values are always non-negative), the effective learning rate \(\frac{\eta}{\sqrt{G_t}+\epsilon}\) monotonically decreases over the entire course of training — and eventually approaches zero, at which point the parameter effectively stops learning entirely, regardless of how large the true gradient currently is. This is AdaGrad's well-known weakness, and it's exactly what RMSProp (next note) was designed to fix.
Code
import torch.optim as optim
optimizer = optim.Adagrad([w], lr=0.1) # PyTorch's built-in AdaGrad implementationCommon Mistakes
- Using AdaGrad for long training runs — its ever-shrinking learning rate makes it poorly suited to training that needs to continue learning over many epochs; it's more commonly seen historically or for specific sparse-feature use cases than as a general-purpose modern default.
Interview Relevance
Q: "What real problem does AdaGrad have that limits its use in modern deep learning?" Its per-parameter learning rate is based on the sum of ALL past squared gradients, which only ever grows — causing the effective learning rate to shrink monotonically and eventually approach zero, effectively halting learning prematurely, even in the middle of a long training run. RMSProp fixes this by using a decaying average instead of an ever-growing sum.
Practice Question
Explain why AdaGrad's accumulator \(G_t\) can only increase over time, never decrease, and why that specifically causes the learning rate to shrink monotonically.
