Random Search

Random search replaces grid search's exhaustive combination testing with randomly sampled hyperparameter combinations — and, perhaps counterintuitively, often finds better results with the same compute budget.

The Core Idea

import random

def sample_hyperparameters():
    return {
        'lr': 10 ** random.uniform(-5, -1),           # sampled on a LOG scale -- appropriate for learning rate
        'batch_size': random.choice([16, 32, 64, 128]),
        'dropout': random.uniform(0.1, 0.6)
    }

best_config, best_val_acc = None, 0
num_trials = 20   # far fewer than a full grid, yet often finds comparable or better results
for _ in range(num_trials):
    config = sample_hyperparameters()
    model = build_model(dropout=config['dropout'])
    train(model, lr=config['lr'], batch_size=config['batch_size'])
    val_acc = evaluate(model, val_loader)
    if val_acc > best_val_acc:
        best_val_acc, best_config = val_acc, config

Why Random Search Often Outperforms Grid Search at the Same Cost

A well-known empirical finding (Bergstra & Bengio, 2012): in practice, not every hyperparameter matters equally — a few tend to have a much larger impact on final performance than the rest. Grid search wastes a large fraction of its budget exhaustively varying every hyperparameter combination, including many along the (typically several) low-impact dimensions. Random search, by sampling every hyperparameter independently on every trial, naturally explores the important dimensions more thoroughly per unit of compute — for the same total number of trials, it tends to cover the truly consequential hyperparameters' ranges more densely.

Diagram — Why This Matters

Common Mistakes

• Sampling a hyperparameter like learning rate uniformly on a linear scale instead of a logarithmic one — learning rate's meaningful effect spans several orders of magnitude, and linear sampling wastes most trials in a narrow, less-informative range.
• Using too few trials relative to the number of hyperparameters being tuned — random search's advantage relies on having enough trials to meaningfully cover the important dimensions; too few trials can still miss good combinations by chance.

Interview Relevance

Q: "Why can random search often outperform grid search for the same computational budget?" Not every hyperparameter has equal impact on final performance — typically a small number matter much more than the rest. Grid search's exhaustive combinations waste significant budget varying every dimension equally, including low-impact ones. Random search, sampling each hyperparameter independently every trial, naturally provides denser coverage of the high-impact dimensions for the same total trial count, since it doesn't waste structure enforcing uniform coverage across every dimension including unimportant ones.

Practice Question

Why is it important to sample a hyperparameter like learning rate on a logarithmic scale rather than a linear scale during random search?

Related DL Notes

• Grid Search
• Bayesian Optimization
• Optuna