Support Vector Machine (SVM)

A Support Vector Machine (SVM) finds the decision boundary that separates two classes with the widest possible margin — not just any separating line, but specifically the one furthest from the nearest points of either class.

The Core Idea — Why "Widest Margin" Matters

Many lines can separate two linearly separable classes perfectly on the training data. SVM's insight is that not all of them generalize equally well — a boundary that barely squeezes between the two classes is fragile, while one with the widest possible buffer zone on both sides is more robust to new, slightly different data. SVM is specifically the algorithm that searches for that maximum-margin boundary.

Formula — The Decision Boundary

This is the same linear form as logistic regression's \(z\) — a weighted sum of features plus a bias. What's different is how \(w\) and \(b\) get chosen: not by minimizing log-loss, but by explicitly maximizing the margin around this boundary.

The Anatomy of an SVM, Visually

Minimal Working Example

from sklearn.svm import SVC
import numpy as np

X_train = np.array([[2.5,0],[3,1],[1.5,0],[1,1]])
y_train = np.array([1, 1, -1, -1])

model = SVC(kernel="linear")
model.fit(X_train, y_train)

print(model.coef_, model.intercept_)   # approximately [[2, 0]]  [-4]
print(model.support_vectors_)           # the specific points that determine the boundary

See SVM Margin for this exact \(w=[2,0]\), \(b=-4\) example computed and verified by hand.

Practical Use Cases

• Text classification and high-dimensional data, where SVM has historically been a strong performer
• Image classification with a moderate number of features
• Bioinformatics — gene expression classification, where feature counts often exceed sample counts

Advantages

• Effective in high-dimensional spaces, even when features outnumber samples
• The kernel trick lets it capture non-linear boundaries without explicit feature engineering
• Memory-efficient at prediction time — only the support vectors matter, not the full training set

Limitations

• Doesn't scale well to very large datasets — training time grows significantly with sample count
• Requires careful feature scaling, same as any distance/margin-based algorithm
• Choosing the right kernel and its hyperparameters (\(C\), \(\gamma\)) requires real tuning effort
• Doesn't directly output well-calibrated probabilities the way logistic regression does

Common Mistakes

• Running SVM on unscaled features — margin-based distance calculations are just as scale-sensitive as KNN's.
• Defaulting to an RBF kernel without ever trying a linear kernel first, especially on high-dimensional, already near-linearly-separable data (like text) where linear often wins and trains far faster.

Interview Relevance

Q: "Why does SVM maximize the margin instead of just finding any separating boundary?" A wider margin generalizes better to new data — a boundary that barely separates the training classes is more likely to misclassify new points that fall in the previously-unseen space near that thin margin; maximizing the margin is a direct way of building in robustness.

Practice Question

Explain, in your own words, why moving a non-support-vector point (one safely far from the boundary) doesn't change the SVM's decision boundary at all.

Related ML Notes

• SVM Margin
• Support Vectors
• Kernel Trick