A voting classifier combines several different models' predictions using a simple, fixed rule — either a majority vote on hard labels, or an average of predicted probabilities — the most straightforward ensemble technique to build and reason about.
Hard Voting vs Soft Voting
| Hard Voting | Soft Voting | |
|---|---|---|
| Combines | Each model's final class prediction | Each model's predicted class probabilities |
| Formula | \(\hat{y} = \text{mode}(\hat{y}_1, \dots, \hat{y}_m)\) | \(\hat{y} = \arg\max_c \frac{1}{m}\sum_i P_i(c)\) |
| Requires | Just class predictions | Calibrated predict_proba() from every model |
| Typically more accurate? | Simpler, but discards confidence information | Usually yes — retains more information |
Worked Example — Hard Voting
Three models predict a binary outcome for the same input: Logistic Regression → 1, Random Forest → 1, SVM → 0. Majority: two votes for 1, one for 0 → final prediction: 1.
Worked Example — Soft Voting
The same three models' predicted probabilities for class 1: \(0.7, 0.65, 0.3\).
Averaged probability 0.55 is above the 0.5 threshold, so soft voting also predicts class 1 here — but notice it retains how confident the ensemble is (0.55, fairly close to the boundary), information hard voting's simple 2-vs-1 tally completely discards.
Python Implementation
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
models = [
("logreg", LogisticRegression(max_iter=1000)),
("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
("svm", SVC(probability=True)),
]
hard_voting = VotingClassifier(estimators=models, voting="hard")
hard_voting.fit(X_train, y_train)
print("Hard voting:", hard_voting.score(X_test, y_test))
soft_voting = VotingClassifier(estimators=models, voting="soft")
soft_voting.fit(X_train, y_train)
print("Soft voting:", soft_voting.score(X_test, y_test))Weighted Voting
# Give a more trusted/accurate model a bigger say in the final vote
weighted_voting = VotingClassifier(
estimators=models, voting="soft", weights=[1, 2, 1] # Random Forest counts double
)Voting vs Stacking — When Each Fits
Voting uses a fixed, hand-chosen combination rule; stacking learns the combination rule from data via a meta-model. Voting is simpler, faster, and easier to reason about — a reasonable default when base models perform similarly well; stacking is worth the added complexity when base models' relative strengths vary meaningfully across different situations.
Practical Use Cases
- Quickly combining several already-trained, reasonably diverse models with minimal extra complexity
- A simple, robust baseline ensemble before considering the added complexity of stacking
Common Mistakes
- Using hard voting when soft voting is available and every base model supports
predict_proba()— soft voting almost always performs at least as well, since it uses more information. - Combining models that are highly correlated (make very similar mistakes) — voting adds the most value when base models genuinely disagree in different situations, same as any ensemble.
Interview Relevance
Q: "When would you prefer hard voting over soft voting?" When one or more base models can't produce reliable probability estimates (or predict_proba() isn't available/well-calibrated for them) — otherwise soft voting is almost always preferable, since it uses each model's confidence, not just its final label.
Practice Question
Three models predict probabilities for class 1: 0.9, 0.4, 0.4. Compute the soft-voting result, and compare it to what hard voting alone would decide (assuming each model's hard prediction is thresholded at 0.5).
