The RBF (Radial Basis Function) kernel — also called the Gaussian kernel — is the most commonly used non-linear kernel, flexible enough to wrap decision boundaries around complex, curved cluster shapes.
Formula
\(\lVert x-y \rVert^2\) is the squared Euclidean distance between two points, and \(\gamma\) (gamma) controls how quickly similarity falls off with distance. This function measures similarity, not distance directly: nearby points (\(\lVert x-y \rVert\) small) get a kernel value close to 1; distant points get a value close to 0.
Numerical Example
For \(x=[1,2]\), \(y=[2,3]\): \(\lVert x-y \rVert^2 = (1-2)^2+(2-3)^2 = 1+1 = 2\).
import numpy as np
x = np.array([1, 2])
y = np.array([2, 3])
gamma = 0.5
squared_dist = np.sum((x - y) ** 2)
k = np.exp(-gamma * squared_dist)
print(squared_dist, k) # 2 0.368The Role of Gamma — A Critical Hyperparameter
| Gamma Value | Effect | Risk |
|---|---|---|
| Small gamma | Similarity falls off slowly — even distant points count as "similar," producing a smoother, simpler boundary | Underfitting |
| Large gamma | Similarity falls off fast — only very nearby points count as "similar," producing a highly flexible, wiggly boundary | Overfitting |
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split, cross_val_score
X, y = make_circles(n_samples=300, noise=0.1, factor=0.4, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
for gamma in [0.01, 0.1, 1, 10, 100]:
model = SVC(kernel="rbf", gamma=gamma)
scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"gamma={gamma}: CV accuracy = {scores.mean():.3f}")Like \(k\) in KNN and \(C\) in SVM generally, gamma is virtually always tuned via cross-validation rather than picked by intuition — scikit-learn's default, gamma="scale", computes a reasonable starting value automatically based on the data's variance, but rarely the optimal one for every dataset.
Why RBF Is Considered "Infinite-Dimensional"
The RBF kernel corresponds to an implicit feature mapping \(\phi\) into an infinite-dimensional space (a fact provable via a Taylor series expansion of the exponential function) — a genuinely striking mathematical property that the kernel trick makes entirely computationally practical, despite that infinite dimensionality never being touched directly.
Practical Use Cases
- Any classification problem with a genuinely curved, non-linear class boundary
- A strong general-purpose default when there's no specific reason to prefer a linear or polynomial kernel
Common Mistakes
- Tuning \(C\) and \(\gamma\) independently instead of jointly via grid search — the two interact, and the best value of one often depends on the current value of the other.
- Using an extremely large gamma "to fit the data better," without realizing this typically produces a severely overfit, wiggly boundary.
Interview Relevance
Q: "What does a very large gamma value do to an RBF SVM's decision boundary?" It makes the notion of "similarity" extremely local — only points very close together are considered similar — producing a highly flexible, tightly-wrapped boundary around individual training points, a strong overfitting risk analogous to a very small k in KNN.
Practice Question
For \(x=[0,0]\) and \(y=[3,4]\), compute the RBF kernel value with \(\gamma=0.1\).
