Grid search is the simplest systematic hyperparameter search strategy: define a discrete set of candidate values for each hyperparameter, then exhaustively try every possible combination.
The Core Idea
learning_rates = [1e-4, 1e-3, 1e-2]
batch_sizes = [32, 64, 128]
dropout_rates = [0.2, 0.5]
best_config, best_val_acc = None, 0
for lr in learning_rates:
for bs in batch_sizes:
for dropout in dropout_rates:
model = build_model(dropout=dropout)
train(model, lr=lr, batch_size=bs)
val_acc = evaluate(model, val_loader)
if val_acc > best_val_acc:
best_val_acc, best_config = val_acc, (lr, bs, dropout)
print(best_config, best_val_acc)The Combinatorial Explosion Problem
For the example above: \(3\times3\times2=18\) combinations — manageable. But grid search's cost grows multiplicatively with each additional hyperparameter and each additional candidate value per hyperparameter — 5 hyperparameters with 4 values each would require \(4^5=1024\) full training runs, quickly becoming computationally infeasible for anything beyond a handful of hyperparameters with a handful of values each.
Using scikit-learn's GridSearchCV
from sklearn.model_selection import GridSearchCV
from sklearn.neural_network import MLPClassifier
param_grid = {'hidden_layer_sizes': [(64,), (128,)], 'alpha': [1e-4, 1e-3], 'learning_rate_init': [1e-3, 1e-2]}
grid_search = GridSearchCV(MLPClassifier(max_iter=200), param_grid, cv=3)
grid_search.fit(X_train, y_train)
print(grid_search.best_params_)When Grid Search Still Makes Sense
Despite its scaling limitations, grid search remains a reasonable choice when tuning just one or two hyperparameters with a small number of candidate values each — its exhaustiveness guarantees the best combination within the specified grid will be found, with no risk of randomly missing a good combination the way random search (next note) theoretically could.
Common Mistakes
- Applying grid search to more than a handful of hyperparameters simultaneously — the combinatorial explosion quickly makes this computationally infeasible for real deep learning training runs, where each individual combination can itself take hours.
- Choosing grid values without any informed reasoning (e.g. arbitrary, evenly-spaced values on a linear scale for a hyperparameter like learning rate, which behaves more naturally on a logarithmic scale) — poorly chosen grid points can miss the actually useful region of the search space entirely.
Interview Relevance
Q: "Why does grid search become impractical as the number of hyperparameters being tuned increases?" Its total cost is the product of the number of candidate values across every hyperparameter — this grows multiplicatively (combinatorially), not additively, with each new hyperparameter or additional candidate value. Even a modest number of hyperparameters with a modest number of values each can require an infeasibly large number of full training runs, especially when each individual run (as with most deep learning training) is itself computationally expensive.
Practice Question
If you're tuning 4 hyperparameters, each with 5 candidate values, how many total training runs would exhaustive grid search require?
