Beyond the basic one-hot/ordinal encoding choice already covered in preprocessing, engineering categorical features well means handling rare categories, high cardinality, and extracting structure a plain encoder would miss.
The High-Cardinality Problem, Revisited
A "zip code" or "product_id" column with thousands of unique values one-hot encodes into an impractically wide, sparse matrix. Two engineering techniques handle this better than raw encoding:
| Technique | How It Works | Tradeoff |
|---|---|---|
| Frequency encoding | Replace each category with how often it appears in the training data | Simple, but loses category identity — two categories with the same frequency become indistinguishable |
| Target encoding | Replace each category with the mean target value for that category (computed on training data only) | Powerful, but a major leakage risk if not done with proper cross-validation folds |
| Grouping rare categories | Bucket infrequent categories into a single "Other" label | Reduces dimensionality, but can hide a genuinely predictive rare category |
Python Implementation
import pandas as pd
df = pd.DataFrame({"city": ["Delhi","Mumbai","Delhi","Pune","Kochi","Delhi","Mumbai","Ranchi"]})
# Frequency encoding
freq_map = df["city"].value_counts(normalize=True)
df["city_freq"] = df["city"].map(freq_map)
# Grouping rare categories (appearing fewer than 2 times) into "Other"
counts = df["city"].value_counts()
rare = counts[counts < 2].index
df["city_grouped"] = df["city"].replace(rare, "Other")
print(df)Expected output: Kochi and Ranchi (each appearing once) collapse into "Other" in city_grouped, reducing the effective category count from 5 to 4 without discarding those rows entirely.
Target Encoding — Why It Needs Extra Care
Computing "average target value per category" using the entire training set, then using that same encoding on every row (including the rows that contributed to their own category's average), leaks target information into the features. The standard fix is K-fold target encoding: encode each fold's categories using only the target means computed from the other folds — this is exactly the kind of leakage prevention a pipeline discipline is meant to enforce.
Practical Use Cases
- High-cardinality categorical features in tabular business data (product IDs, zip codes, merchant IDs)
- Reducing noisy, rarely-seen categories into a manageable "Other" bucket before one-hot encoding the rest
Common Mistakes
- Target-encoding without cross-validation folds — one of the most common real-world sources of data leakage in tabular ML.
- Grouping rare categories based on an arbitrary frequency cutoff without checking whether any of them are actually strongly predictive despite being rare.
Interview Relevance
Q: "Why is target encoding riskier than one-hot encoding?" Because it directly incorporates target information into a feature — done naively (fit on the whole training set, applied to those same rows), it leaks the label into the feature and inflates training performance in a way that won't generalize; proper K-fold target encoding is required to avoid this.
Practice Question
A "merchant_id" column has 50,000 unique values in a fraud detection dataset. Explain why one-hot encoding is a poor choice here, and propose an alternative.
