Batch Gradient Descent

Batch Gradient Descent computes the gradient using the entire training dataset before making a single weight update. It's the most accurate variant of gradient descent — and the least practical one for real-world dataset sizes.

Formula

Every one of the \(N\) training examples contributes to the gradient before \(\mathbf{w}\) is updated even once.

Advantages and Disadvantages

Why It's Rarely Used in Practice

For a dataset with millions of examples, computing just one gradient update requires processing the entire dataset — with modern datasets, this means a single weight update could take minutes to hours, and you'd need many thousands of updates to actually train a model. This impracticality is exactly what motivates Stochastic and Mini-Batch Gradient Descent, covered next.

Code

import torch

# Batch gradient descent: ALL data used for every single update
def batch_gradient_descent(X, y, w_init, lr=0.01, epochs=100):
    w = w_init.clone().requires_grad_(True)
    for epoch in range(epochs):
        predictions = X @ w
        loss = ((predictions - y) ** 2).mean()   # uses the FULL dataset X, y every time
        loss.backward()
        with torch.no_grad():
            w -= lr * w.grad
        w.grad.zero_()
    return w

X = torch.randn(10000, 5)   # 10,000 examples -- every epoch processes all of them before updating
y = torch.randn(10000)
w = batch_gradient_descent(X, y, torch.zeros(5, requires_grad=True))

Common Mistakes

• Assuming "batch" in "batch gradient descent" means the same thing as a "batch" in mini-batch training — it doesn't; here "batch" specifically means the entire dataset, the opposite of the small batches used in mini-batch gradient descent and PyTorch's DataLoader.
• Using batch gradient descent by default for large modern datasets — this is essentially never done in practice for deep learning; mini-batch is the near-universal standard.

Interview Relevance

Q: "Why isn't batch gradient descent used for training modern deep learning models?" It requires computing gradients over the entire dataset before a single weight update, which is computationally impractical for datasets with millions of examples — both in time (one update could take extremely long) and memory. Mini-batch gradient descent achieves nearly the same gradient quality with vastly more frequent, faster updates.

Practice Question

If a dataset has 1 million examples and batch gradient descent is used, how many weight updates happen per epoch? How does this compare to mini-batch gradient descent with a batch size of 256?

Related DL Notes

• Gradient Descent
• Mini-Batch Gradient Descent
• Stochastic Gradient Descent