Raw text — a review, a support ticket, a product description — has to be converted into numbers before any classical ML algorithm can use it. The two standard techniques, Bag-of-Words and TF-IDF, turn a document into a vector of word-based signal.
Bag-of-Words — Counting, Nothing More
from sklearn.feature_extraction.text import CountVectorizer
docs = ["the product is great", "great product, fast delivery", "terrible product, slow delivery"]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(docs)
print(vectorizer.get_feature_names_out())
print(X.toarray())Each document becomes a row, each unique word becomes a column, and each cell counts how many times that word appears in that document — word order is completely discarded, which is exactly what "bag of words" means.
TF-IDF — Weighting Words by How Informative They Are
\(\text{TF}(t,d)\) is how often term \(t\) appears in document \(d\). \(N\) is the total number of documents. \(\text{DF}(t)\) is how many documents contain term \(t\) at all. A word that appears in every document (like "the" or "product" in a product-review dataset) gets a low weight — it's not distinguishing between documents. A word that appears often in one document but rarely across the whole collection gets a high weight — it's genuinely informative about that specific document.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(docs)
print(vectorizer.get_feature_names_out())
print(X.toarray().round(2))
# "product" (appears in all 3 docs) gets a LOWER weight than "terrible" or "great"
# (each appearing in only 1 doc) -- exactly the intended effectSimple Text-Derived Numeric Features
import pandas as pd
df = pd.DataFrame({"review": ["Great product, highly recommend!", "meh, it's okay", "TERRIBLE. NEVER BUYING AGAIN!!!"]})
df["char_count"] = df["review"].str.len()
df["word_count"] = df["review"].str.split().str.len()
df["exclamation_count"] = df["review"].str.count("!")
df["uppercase_ratio"] = df["review"].apply(lambda t: sum(1 for c in t if c.isupper()) / max(len(t), 1))
print(df)These simple statistical features (length, punctuation intensity, capitalization) can be surprisingly predictive on their own — a short, all-caps, exclamation-heavy review is a strong, cheap-to-compute signal even before you touch word-level content.
Practical Use Cases
- Spam detection, sentiment classification, support ticket categorization
- Search and document ranking, where TF-IDF has been a long-standing industry baseline
Limitations — Where Bag-of-Words/TF-IDF Fall Short
- Completely ignores word order and context — "not good" and "good" share the word "good" with no signal that negation flipped its meaning
- Produces very high-dimensional, sparse vectors for large vocabularies
- Modern approaches (word/sentence embeddings, transformer-based models) capture semantic meaning and context far better, though bag-of-words/TF-IDF remain fast, simple, effective baselines
Common Mistakes
- Fitting a
CountVectorizer/TfidfVectorizeron the full dataset (train + test) instead of the training set only — the vocabulary itself can leak test-set information. - Not removing extremely common "stop words" (the, is, a) when they add noise rather than signal for a given task — though for some tasks (sentiment, authorship) even stop words carry useful signal, so this isn't a universal rule.
Interview Relevance
Q: "Why does TF-IDF often outperform simple word counts for text classification?" Because it down-weights words that appear in nearly every document (uninformative, like "the" or "product") and up-weights words that are rare across the collection but concentrated in specific documents — directly capturing which words actually distinguish one document from another.
Practice Question
Given a corpus of 100 customer reviews, the word "delivery" appears in 90 of them, while the word "moldy" appears in only 3. Which word will TF-IDF weight more heavily, and why?
