Exponential Decay

Exponential decay shrinks the learning rate smoothly and continuously every single step, rather than in the sudden jumps of step decay — trading step decay's simplicity for a gentler, gradually-shrinking curve.

Formula

\(\gamma\) (e.g. 0.95, 0.99) is a decay rate applied every single epoch (or step) — unlike step decay's \(\gamma\), which only applies at discrete intervals.

Numerical Example

With \(\eta_0=0.1\), \(\gamma=0.95\): epoch 0: \(0.1\); epoch 10: \(0.1\times0.95^{10}\approx0.0599\); epoch 30: \(0.1\times0.95^{30}\approx0.0215\) — a smooth, continuous shrinkage rather than step decay's flat-then-drop pattern.

Graph

Code

import torch.optim as optim
from torch.optim.lr_scheduler import ExponentialLR

optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = ExponentialLR(optimizer, gamma=0.95)

for epoch in range(30):
    # ... training loop for this epoch ...
    scheduler.step()
    print(scheduler.get_last_lr())

Step Decay vs Exponential Decay

Common Mistakes

• Choosing a \(\gamma\) too close to 1 (e.g. 0.999) expecting meaningful decay — over a typical training run, this decays extremely slowly and may barely differ from a constant learning rate in practice.
• Choosing a \(\gamma\) too far from 1 (e.g. 0.7) — the learning rate can shrink to near-zero within just a handful of epochs, effectively freezing training prematurely.

Interview Relevance

Q: "When might you prefer exponential decay over step decay?" When you want a smoother loss curve without the small, temporary jumps that can appear right at step decay's sudden drop points — exponential decay changes the learning rate a tiny amount every single step instead of in occasional large jumps, at the cost of being slightly less intuitive to reason about directly.

Practice Question

With \(\eta_0=0.05\) and \(\gamma=0.9\), what is the learning rate at epoch 5? (You can approximate \(0.9^5 \approx 0.59\).)

Related DL Notes

• Step Decay
• Cosine Annealing
• Learning Rate Scheduling