Regularization

Regularization directly fights overfitting by adding a penalty for model complexity to the training objective — the model must now balance fitting the data well against keeping its parameters small and simple.

The General Formula

\(J(w)\) is the original cost function (e.g. MSE or log-loss). \(R(w)\) is a penalty term based on the model's parameters. \(\lambda\) (lambda) controls how strongly that penalty is enforced — \(\lambda=0\) recovers the unregularized model; larger \(\lambda\) forces smaller, simpler parameters at the cost of fitting the training data slightly less precisely.

The Three Common Penalty Choices

Why Penalizing Large Coefficients Prevents Overfitting

An overfit model often relies on a few large-magnitude coefficients to chase individual noisy training points precisely. By adding a cost for coefficient size, regularization forces the model to only use large coefficients when the resulting fit improvement is worth the penalty — genuinely strong, broad patterns easily clear that bar; noise-chasing, narrow adjustments usually don't.

Python Implementation

from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import cross_val_score
import numpy as np

for alpha in [0.01, 0.1, 1, 10, 100]:   # scikit-learn calls lambda "alpha"
    model = Ridge(alpha=alpha)
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="neg_mean_squared_error")
    print(f"alpha={alpha}: mean CV MSE = {-scores.mean():.3f}")
# Look for the alpha that minimizes validation error -- neither 0 nor extremely large

Regularization Applies Beyond Linear Models

The same core idea appears throughout ML under different names: tree pruning penalizes tree complexity, XGBoost's built-in L1/L2 terms regularize boosted trees, and dropout/weight decay serve the same purpose in neural networks. "Add a cost for complexity" is a universal pattern, not just a linear-regression trick.

Practical Use Cases

• Any model showing signs of overfitting, especially with many features relative to samples
• Automatically simplifying a model, either by shrinking (L2) or eliminating (L1) less useful coefficients

Common Mistakes

• Choosing \(\lambda\)/alpha arbitrarily instead of tuning it via cross-validation — too little does nothing, too much causes underfitting.
• Applying regularization to unscaled features — the penalty treats all coefficients on the same numeric footing, so a feature with a naturally larger scale gets unfairly penalized relative to one with a smaller scale, unless features are standardized first.

Interview Relevance

Q: "Why must you scale features before applying L1/L2 regularization?" The penalty term sums coefficient magnitudes directly — an unscaled feature with a naturally small range would need a large coefficient to have real predictive effect, and that large coefficient gets penalized more heavily than it deserves purely due to scale, distorting which features the regularization favors or eliminates.

Practice Question

You tune \(\lambda\) and see training error rise steadily while validation error first falls, then eventually rises too. Explain what's happening at each stage in terms of the bias-variance tradeoff.

Related ML Notes

• L1 Regularization
• L2 Regularization
• Overfitting