Naive Bayes classifies by directly applying Bayes' theorem — combining how common each class is with how likely the observed features are for each class — while making one deliberately simplifying assumption that gives the algorithm its name.
The Formula
\(P(y)\) is the prior — how common class \(y\) is overall. \(P(x_i\mid y)\) is the likelihood of feature \(x_i\) given class \(y\). The prediction is whichever class \(y\) maximizes this product: \(\hat{y} = \arg\max_y P(y)\prod_i P(x_i\mid y)\).
Why It's Called "Naive"
This formula only works cleanly if every feature is treated as conditionally independent given the class — i.e. \(P(x_1,x_2\mid y) = P(x_1\mid y)P(x_2\mid y)\), not the true joint probability. In real data this is almost never exactly true (word choices in an email are correlated with each other, not independent) — hence "naive." Remarkably, the algorithm still performs well in practice even when this assumption is clearly violated, because it usually only needs to rank classes correctly, not compute perfectly calibrated probabilities.
Worked Example — Spam Classification
10 training emails: 6 spam, 4 not spam. Features: whether the email contains "free," and whether it contains "meeting."
| P(contains "free") | P(contains "meeting") | |
|---|---|---|
| Spam (6 emails) | 5/6 ≈ 0.833 | 1/6 ≈ 0.167 |
| Not Spam (4 emails) | 1/4 = 0.25 | 3/4 = 0.75 |
New email: contains "free," does NOT contain "meeting." Priors: \(P(\text{spam})=0.6\), \(P(\text{not spam})=0.4\).
Normalizing: \(0.417/(0.417+0.025) \approx \mathbf{0.943}\) — the model predicts spam with about 94.3% confidence.
p_spam_prior, p_notspam_prior = 0.6, 0.4
p_free_spam, p_meeting_spam = 5/6, 1/6
p_free_notspam, p_meeting_notspam = 1/4, 3/4
score_spam = p_spam_prior * p_free_spam * (1 - p_meeting_spam)
score_notspam = p_notspam_prior * p_free_notspam * (1 - p_meeting_notspam)
p_spam = score_spam / (score_spam + score_notspam)
print(round(p_spam, 3)) # 0.943The Three Common Variants
| Variant | Feature Type | Full Note |
|---|---|---|
| Gaussian | Continuous numeric | Gaussian Naive Bayes |
| Multinomial | Counts (e.g. word frequencies) | Multinomial Naive Bayes |
| Bernoulli | Binary (present/absent) | Bernoulli Naive Bayes |
Practical Use Cases
- Spam/ham filtering — one of Naive Bayes' original, most successful applications
- Sentiment classification and other text categorization tasks
- Real-time classification where prediction speed matters — Naive Bayes is extremely fast
Advantages
- Very fast to train and predict, even on large datasets
- Works well with relatively little training data compared to more complex models
- Naturally handles multi-class problems
- Surprisingly competitive despite its simplifying assumption, especially for text
Limitations
- The independence assumption can hurt performance when features are strongly correlated
- Struggles when a feature value never appears with a class in training — see the smoothing fix in Multinomial Naive Bayes
- Probability estimates, while often good enough for ranking, aren't as well-calibrated as logistic regression's
Common Mistakes
- Assuming the independence assumption disqualifies Naive Bayes from ever working well — empirically it often performs strongly regardless.
- Forgetting Laplace (additive) smoothing, causing a single unseen feature value to zero out an entire class's probability.
Interview Relevance
Q: "Why does Naive Bayes work well in practice despite its unrealistic independence assumption?" It usually only needs to rank the correct class highest, not compute perfectly calibrated probabilities — even with correlated features, the assumption's errors often affect all classes similarly enough that the relative ranking (and thus the final prediction) stays correct.
Practice Question
Using the spam example's numbers, recompute the prediction for an email that does NOT contain "free" but DOES contain "meeting."
