Bernoulli Naive Bayes

Bernoulli Naive Bayes handles purely binary features — a word either appears in a document or it doesn't, with no regard for how many times — a distinct modeling choice from Multinomial NB's frequency counts.

Formula

\(x_i \in \{0,1\}\) indicates whether feature \(i\) is present. \(p_{i,y}\) is the probability that feature \(i\) is present at all, among class-\(y\) training documents (regardless of how many times it occurs). Critically, this formula also explicitly penalizes the absence of a feature (via the \((1-p_{i,y})^{1-x_i}\) term) — something Multinomial NB doesn't directly account for.

Worked Example — Reusing the Naive Bayes Hub's Spam Data

This is exactly the example already computed in the Naive Bayes hub note — it's Bernoulli by construction, since "contains 'free'" and "contains 'meeting'" are pure yes/no features, with no frequency information involved:

For a new email containing "free" but NOT "meeting," the calculation explicitly used \((1-0.167)\) and \((1-0.75)\) — the absence terms — which is exactly Bernoulli NB's formula in action, giving \(P(\text{spam}\mid x) \approx 0.943\).

from sklearn.naive_bayes import BernoulliNB
import numpy as np

# has_free, has_meeting -> spam(1)/not spam(0)
X_train = np.array([
    [1,0],[1,0],[1,0],[1,0],[1,1],[0,0],   # spam examples
    [0,1],[0,1],[0,1],[1,0],               # not spam examples
])
y_train = np.array([1,1,1,1,1,1, 0,0,0,0])

model = BernoulliNB()
model.fit(X_train, y_train)

print(model.predict_proba([[1, 0]]))   # contains "free", not "meeting"

Multinomial vs Bernoulli — The Key Difference

Why Explicitly Modeling Absence Matters

Bernoulli NB's penalty for a feature's absence can be a genuine advantage: if "meeting" is common in ham emails, its absence in a new email is itself mild evidence against ham — information Multinomial NB simply doesn't use (a word count of 0 just contributes nothing, rather than actively counting as evidence).

Practical Use Cases

• Short-text classification where word presence, not frequency, is the meaningful signal
• Any dataset naturally represented as binary features (e.g. "has this attribute" checkboxes)

Common Mistakes

• Feeding raw word counts into BernoulliNB without binarizing them first — scikit-learn does binarize automatically by default (binarize=0.0), but it's worth knowing this happens rather than assuming counts are used directly.
• Choosing between Multinomial and Bernoulli NB arbitrarily instead of based on whether frequency genuinely carries signal for the task.

Interview Relevance

Q: "When would you choose Bernoulli Naive Bayes over Multinomial?" When feature presence/absence itself is the meaningful signal rather than frequency (e.g. short texts, or tasks where a word appearing once vs many times shouldn't be treated differently) — and when you specifically want the model to account for the evidential value of a feature's absence, which Multinomial NB doesn't directly capture.

Practice Question

Explain, using the formula, why Bernoulli NB treats an email containing "free" once identically to an email containing "free" ten times.

Related ML Notes

• Multinomial Naive Bayes
• Naive Bayes
• One-Hot Encoding