Grid search exhaustively tries every combination of specified hyperparameter values — thorough and simple to reason about, but its cost grows multiplicatively with every added dimension.
Formula — Total Cost
\(n_i\) is the number of values specified for hyperparameter \(i\), \(d\) is the number of hyperparameters being tuned, and \(k\) is the number of cross-validation folds.
Worked Example
Tuning an SVM's \(C\) (3 values) and \(\gamma\) (2 values) with 5-fold cross-validation:
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
param_grid = {"C": [0.1, 1, 10], "gamma": [0.01, 0.1]}
search = GridSearchCV(SVC(), param_grid, cv=5, scoring="accuracy", verbose=1)
search.fit(X_train, y_train)
# scikit-learn will report "Fitting 5 folds for each of 6 candidates, totalling 30 fits" --
# exactly matching the hand calculationHow Fast This Explodes
| Hyperparameters | Values Each | Combinations | Total Fits (5-fold CV) |
|---|---|---|---|
| 2 | 3, 2 | 6 | 30 |
| 3 | 3, 2, 4 | 24 | 120 |
| 4 | 3, 2, 4, 5 | 120 | 600 |
This multiplicative growth — the "curse of dimensionality" applied to search, not just distance — is exactly why grid search becomes impractical past 3-4 hyperparameters, and why random search or Bayesian optimization take over for larger search spaces.
Inspecting Full Results
import pandas as pd
results = pd.DataFrame(search.cv_results_)
print(results[["params", "mean_test_score", "std_test_score"]].sort_values("mean_test_score", ascending=False))Practical Use Cases
- Small search spaces (1-3 hyperparameters, few values each), where exhaustive coverage is affordable
- Final, focused fine-tuning around a promising region already identified by a coarser search
Advantages
- Guaranteed to find the best combination within the specified grid
- Simple, deterministic, and easy to explain
Limitations
- Computational cost grows multiplicatively with every added hyperparameter
- Wastes effort on unpromising regions of the search space just as much as promising ones
- Only as good as the specific values you chose to include in the grid — a good setting between two grid points will never be found
Common Mistakes
- Specifying an overly fine-grained grid on many hyperparameters simultaneously, causing an impractically long search.
- Not checking whether the best result landed at the edge of the specified grid — if so, the true optimum may lie outside the tested range entirely.
Interview Relevance
Q: "Why does grid search become impractical with many hyperparameters?" Its cost grows as the product of the number of values per hyperparameter — adding even one more hyperparameter with a few values multiplies the total search cost, quickly becoming computationally infeasible, unlike random search's cost, which grows only with the number of iterations you choose to run.
Practice Question
You want to tune 3 hyperparameters with 4, 3, and 5 values respectively, using 10-fold cross-validation. How many total model fits does grid search require?
