SVM Classification

A full walkthrough of how SVM makes a classification decision, including the crucial idea of a soft margin — allowing a few points to violate the margin (or even be misclassified) in exchange for a much more robust overall boundary.

The Optimization Problem — Hard Margin

This says: find the smallest possible \(w\) (which, as shown in SVM Margin, directly maximizes the margin \(2/\lVert w \rVert\)) such that every training point is correctly classified with at least the minimum required margin. This "hard margin" version requires the data to be perfectly linearly separable — unrealistic for most real data.

The Soft Margin — Allowing Some Violations

Each \(\xi_i\) (slack variable) measures how much point \(i\) is allowed to violate the margin. \(C\) controls the tradeoff: a large \(C\) penalizes violations heavily (narrower margin, fits training data more closely — risk of overfitting); a small \(C\) tolerates more violations (wider margin, more tolerant of noise — risk of underfitting).

Python Implementation

from sklearn.svm import SVC
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("svm", SVC(kernel="linear")),
])

param_grid = {"svm__C": [0.01, 0.1, 1, 10, 100]}
grid_search = GridSearchCV(pipeline, param_grid, cv=5, scoring="accuracy")
grid_search.fit(X_train, y_train)

print("Best C:", grid_search.best_params_)
best_model = grid_search.best_estimator_
print(classification_report(y_test, best_model.predict(X_test)))

Getting Probabilities From SVM

# SVC doesn't produce probabilities by default -- it must be explicitly enabled,
# which fits an extra internal calibration model (Platt scaling) and slows training
model = SVC(kernel="linear", probability=True)
model.fit(X_train, y_train)
print(model.predict_proba(X_test[:5]))

Unlike logistic regression, which produces probabilities as a natural mathematical output, SVM's core objective only produces a hard decision (which side of the boundary) — probabilities require this extra calibration step, and are considered somewhat less reliable than logistic regression's native probabilities.

Practical Use Cases

• Any binary classification problem where a small number of noisy or borderline points shouldn't be allowed to distort the whole boundary
• Multi-class problems — scikit-learn handles this automatically via one-vs-one or one-vs-rest strategies

Common Mistakes

• Leaving C at its default value without tuning it — like k in KNN, it's a genuinely important hyperparameter, not a safe-to-ignore default.
• Enabling probability=True without accounting for the extra training time it adds, when only hard class predictions are actually needed.

Interview Relevance

Q: "What does the C hyperparameter control in SVM?" The tradeoff between margin width and margin violations — large C penalizes violations heavily, producing a narrower margin that fits the training data closely (overfitting risk); small C tolerates more violations for a wider, more generalizable margin (underfitting risk if too small).

Practice Question

You train an SVM with a very large C and observe near-perfect training accuracy but poor test accuracy. What would you try adjusting first, and why?

Related ML Notes

• SVM Margin
• Kernel Trick
• Overfitting